A dataset determination method, apparatus, medium, and product

Abstract

The application discloses a kind of dataset determination method, device, medium and product, it is related to data processing technical field.The document length and the different content component are fragmented by using adaptive processing pipeline strategy, the hierarchical segmentation of the depth mining of document structure is realized.Fusion multiple node relations makes the graph semantics more rich, to generate the dynamic maintenance knowledge graph containing node and relation.Fusion user role and diversified node query strategy generation, improve the ability of cross-text information integration, analysis comparison and complex reasoning.Standard answer is decomposed to obtain fact grain, so that standard answer is changed from fuzzy to clear fact coverage in subsequent dataset evaluation process.According to the evaluation dimension of dataset, the closed-loop quality control mode evaluation of multi-agent collaborative verification mechanism is formed, the quality of dataset is improved, and the evaluation effect and reliability of RAG system are also improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, medium, and product for determining a dataset. Background Technology

[0002] Retrieval-Augmented Generation (RAG) systems retrieve relevant contextual information from an external knowledge base before generating answers, using the user's query. This information is then provided to a pre-trained model along with the original query to generate reliable and accurate answers. Before evaluating a RAG system, a testing dataset needs to be constructed. However, this dataset may contain incorrect, ambiguous, or incomplete answers; it may also suffer from single-hop evaluation based solely on a single text block, leading to a lack of diversity in evaluation dimensions; or it may have a shallow understanding of the knowledge base's internal structure. These issues result in poor dataset quality, further impacting the evaluation performance of the RAG system.

[0003] Therefore, improving the quality of constructed datasets to enhance the evaluation performance of RAG evaluation systems is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] The purpose of this invention is to provide a method, apparatus, medium, and product for determining datasets, in order to solve the problem that quality angle differences during the dataset determination process affect the evaluation performance of the RAG evaluation system.

[0005] To address the aforementioned technical problems, this invention provides a method for determining a dataset, comprising:

[0006] Obtain the target document and perform segmentation processing on the target document according to the document length and content components to obtain multiple segment nodes;

[0007] Feature parameters are extracted from multiple block nodes to form nodes of the knowledge graph, and relationships are constructed between the nodes according to the multi-source semantic relationship strategy to obtain the knowledge graph.

[0008] User roles are assigned to the knowledge graph to generate target questions based on the user roles and node query strategies. The target questions are then input into the target model to output standard answers.

[0009] The standard answer is decomposed to obtain fact particles, and a quadruple data of the target question, standard answer, block nodes and fact particles is constructed.

[0010] The quadruple data is evaluated and processed according to the dataset evaluation collaboration mechanism to determine the final dataset.

[0011] On the one hand, the target document is segmented according to its length and content components to obtain multiple chunk nodes, including:

[0012] The target document is parsed to obtain a text file, and the structural information corresponding to the content components is retained;

[0013] The text file is classified according to its document length to obtain a target length file, wherein the target length file is either a first length file or a second length file, and the length of the first length file is greater than the length of the second length file;

[0014] When the target length file is a first length file, the first length file is segmented according to the structure information to obtain the corresponding block nodes;

[0015] When the target length file is a second length file, the second length file is used as a block node.

[0016] On the other hand, the first-length file is segmented based on structural information to obtain corresponding block nodes, including:

[0017] Initiate the hierarchical segmentation mechanism;

[0018] The structural information is added to the first-length file to obtain a third-length file;

[0019] The third-length file is segmented using a hierarchical segmentation mechanism to obtain multiple text blocks, and these text blocks are used as corresponding block nodes.

[0020] On the other hand, feature extraction is performed on multiple block nodes to obtain corresponding feature parameters, including:

[0021] Initiate a parallel feature extraction mechanism; wherein, the parallel feature extraction mechanism includes at least a summary extraction mechanism, a topic extraction mechanism, a keyword extraction mechanism, a named entity recognition mechanism, a question generation mechanism, and a vector embedding extraction mechanism;

[0022] Feature parameters are obtained by extracting features from multiple block nodes using the parallel feature extraction mechanism.

[0023] On the other hand, relationships are constructed between multiple nodes based on a multi-source semantic relationship strategy to obtain a knowledge graph, including:

[0024] The vector relationship between multiple nodes is obtained by processing the vector feature parameters of the block nodes within multiple nodes based on the vector similarity relationship strategy.

[0025] Obtain the linear reading order of the target document, and determine the hierarchical structure relationship strategy based on the linear reading order;

[0026] The hierarchical relationship strategy is used to process multiple nodes to obtain the sequential relationship between them.

[0027] The co-occurrence relationship of entities is determined by processing the named entities of multiple nodes according to the entity co-occurrence relationship strategy;

[0028] Multiple nodes are processed according to the reasoning strategy to determine the corresponding causal relationships;

[0029] A knowledge graph with multiple nodes is constructed based on the vector relationships, the sequence relationships, the entity co-occurrence relationships, and the causal relationships.

[0030] On the other hand, user roles are defined for the knowledge graph, including:

[0031] A global analysis and processing of the knowledge graph yields a corresponding list of topics;

[0032] The target subtopic is determined by selecting from the subtopics in the topic list.

[0033] Generate corresponding user roles based on the target subtopic.

[0034] On the other hand, the target question is generated based on the user role and node query strategy, including:

[0035] In the knowledge graph, the node type corresponding to the target node is determined, wherein the node type is a single-hop node type and a multi-hop node type;

[0036] When the target node is a single-hop node type, the first pre-trained model is invoked, and the target node and user role are input into the first pre-trained model to obtain the first target problem;

[0037] When the target node is a multi-hop node type, the corresponding target relationship is determined in the knowledge graph; the second pre-trained model is called, and the target relationship, the target node and the user role are input into the second pre-trained model to obtain the second target question.

[0038] On the other hand, before inputting the target relationship, the target node, and the user role into the second pre-trained model to obtain the second target problem, the method further includes:

[0039] When the target relationship of multiple target nodes is a preset target relationship and the corresponding content is complementary, or when the target relationship of multiple target nodes is a preset target relationship and the corresponding content is different but shares the same subtopic, the process proceeds to the step of inputting the target relationship, the target node and the user role into the second pre-trained model to obtain the second target problem.

[0040] On the other hand, the standard answer is decomposed to obtain fact particles, including:

[0041] The standard answer is segmented to obtain the corresponding word units;

[0042] The third pre-trained model is invoked, and the lexical units are input into the third pre-trained model for decomposition processing to obtain the corresponding fact particles; wherein, the fact particles at least include the subject, attribute and value.

[0043] On the other hand, the quadruple data is evaluated and processed according to the dataset evaluation collaboration mechanism, including:

[0044] Natural language processing tools are used to perform semantic analysis on the target question in order to determine the question quality.

[0045] The fourth pre-trained model is invoked, and the context information corresponding to the standard answer and the block node is input into the fourth pre-trained model to determine the semantic similarity between the standard answer and the block node;

[0046] The fidelity of the answer is determined based on the semantic similarity.

[0047] The fifth pre-trained model is invoked, and the standard answer and the target question are input into the fifth pre-trained model to determine the semantic coverage of the standard answer and the target question;

[0048] The completeness of the answer is determined based on the semantic coverage.

[0049] The context information of the segmented nodes, the relevance of the target question and the standard answer are determined based on a similarity method.

[0050] The sixth pre-trained model is invoked, and the fact particles are input into the sixth pre-trained model to determine the independence and integrity of the fact particles;

[0051] The dataset evaluation results are determined based on the weight parameters corresponding to the question quality, answer fidelity, answer completeness, relevance of the standard answer, and independent integrity of the fact particles.

[0052] On the other hand, when it is determined that the dataset evaluation results have not reached a preset threshold, the method further includes:

[0053] The evaluation results of the dataset are analyzed and processed to determine the correction strategy;

[0054] When the correction strategy is a problem-oriented strategy, a new target problem is regenerated, and the new target problem, context information, and target problem are input into the target model to obtain a new standard answer.

[0055] Alternatively, when the correction strategy is a block node strategy, new context information is determined based on the embedding vector of the target problem and the node query strategy.

[0056] Alternatively, when the correction strategy is a quadruple matching strategy, the quadruple data is discarded, the next target document is obtained, and the process proceeds to the step of segmenting the target document according to the document length and content components to obtain multiple chunk nodes, until new quadruple data is determined.

[0057] On the other hand, the dataset consists of at least one quadruple data set, and the process of constructing the dataset includes:

[0058] Obtain the target model corresponding to the construction of multiple quadruple data;

[0059] If the target model is the same, the requests for constructing multiple quadruple data will be merged into the same batch for batch processing.

[0060] On the other hand, when the target models are not the same, the method further includes:

[0061] Multiple target models are classified to determine the target models corresponding to different categories;

[0062] Among target models with different classifications, identify the target category that has the largest number of target models under the same category;

[0063] In the target classification, requests to construct the quadruple data corresponding to the target classification are merged into the first batch;

[0064] This process continues until the requests for the four-tuple data corresponding to the category with the smallest number of target models under the same category are merged into the Nth batch.

[0065] Based on network bandwidth, the target model in different batches is called in a balanced manner to generate quadruple data for the corresponding batch.

[0066] On the other hand, the process of determining the target model includes:

[0067] When the target node is a single-hop node, the first performance model with the greatest performance effect in the performance model set is selected as the target model;

[0068] When the target node is a multi-hop node type, the first model with the highest comprehensive score in the model set is selected as the target model.

[0069] On the other hand, after determining the final dataset, the following also includes:

[0070] The system calls the preset model corresponding to the retrieval enhancement generation system and inputs the final dataset into the preset model to output the current answer;

[0071] The fact particles of the current answer and the corresponding quadruple data of the dataset are parsed and processed to determine the target proportion of the evaluation index containing fact particles.

[0072] The model test results of the preset model are determined based on the target proportions of multiple evaluation indicators.

[0073] On the other hand, when a new target document is added, the knowledge graph construction process includes:

[0074] The knowledge graph constructed from the target document is used as the first knowledge graph;

[0075] When a new target document is added, the process proceeds to the step of segmenting the target document according to the document length and content components to obtain multiple chunk nodes, until a new node is determined.

[0076] A second knowledge graph is obtained by constructing relationships between multiple new nodes based on a multi-source semantic relationship strategy.

[0077] The first knowledge graph and the second knowledge graph are fused together to obtain the final knowledge graph.

[0078] On the other hand, the method also includes:

[0079] The timeliness of viewing isolated nodes and relationships connecting multiple nodes in the knowledge graph according to the first preset period;

[0080] If the timeliness meets the preset conditions, then the target nodes and isolated nodes of the target relationship that meet the preset conditions are classified as cold nodes.

[0081] The cold nodes are re-analyzed according to the second preset period to determine a new knowledge graph.

[0082] To address the aforementioned technical problems, the present invention also provides a dataset determination apparatus, comprising:

[0083] Memory, used to store computer programs;

[0084] A processor for implementing the steps of the dataset determination method as described above when executing the computer program.

[0085] To address the aforementioned technical problems, the present invention also provides a non-volatile storage medium storing a computer program, which, when executed by a processor, implements the steps of the dataset determination method described above.

[0086] To address the aforementioned technical problems, the present invention also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the dataset determination method.

[0087] The beneficial effects of this invention are as follows: First, by segmenting the target document according to document length and content components to obtain multiple segmented nodes, compared to the conventional brute-force segmentation method that results in a superficial knowledge base document structure, the segmentation process using an adaptive processing pipeline strategy based on document length and different content components achieves hierarchical segmentation for in-depth mining of the document structure. Second, feature extraction is performed on multiple segmented nodes to obtain corresponding feature parameters, which, together with the corresponding segmented nodes, constitute nodes in the knowledge graph. Nodes are constructed based on segmented nodes and feature parameters to facilitate the subsequent weaving of discrete nodes into a knowledge graph. A multi-source semantic relationship strategy is used to construct the knowledge graph based on the relationships between multiple nodes. The multi-source semantic relationships, combined with various node relationships, enrich the semantics of the graph, generating a dynamically maintained knowledge graph containing nodes and relationships. Third, user roles are assigned to the knowledge graph, and user roles are integrated with diverse node query strategies to avoid generating target questions solely through single-hop queries. This facilitates diversified and comprehensive evaluation dimensions, moving beyond basic "retrieval-reading" capabilities and improving the ability to integrate, analyze, compare, and reason across textual information. Fourth, the target question is input into the target model to obtain the standard answer. The standard answer is then decomposed into fact particles, with each standard answer broken down into a set of atomic facts. This transforms the fuzzy nature of the standard answer into clear factual coverage during subsequent dataset evaluation. Finally, a quadruple dataset of the target question, standard answer, chunk nodes, and fact particles is constructed. Based on the original triplet consisting of question-answer-chunk nodes, the quadruple dataset is evaluated using a dataset evaluation collaboration mechanism to determine the final dataset. Fact particles are then added to improve the accuracy of subsequent evaluations. In terms of evaluation processing, evaluation is performed according to the dataset evaluation dimensions, forming a closed-loop quality control method with a multi-agent collaborative verification mechanism. This effectively eliminates low-quality samples, corrects factual errors and answer illusions, fundamentally ensuring that every sample included in the dataset undergoes rigorous testing. This improves both the dataset quality and the evaluation effectiveness and reliability of the RAG system.

[0088] Secondly, in the segmentation process, different segmentation strategies are selected based on document characteristics. For long documents with strong structure, hierarchical segmentation based on titles or semantics is adopted, while short documents with weak structure are processed as a whole. Compared with the conventional brute-force segmentation, the segmentation strategy in this embodiment is more diverse and improves accuracy. The first-length file is segmented based on structural information to obtain corresponding block nodes, breaking away from the original brute-force segmentation. By incorporating structural information, the segmentation process lays the foundation for subsequent knowledge graph construction and improves the reference basis. The feature extraction using a parallel feature extraction mechanism determines the feature parameters, improving processing efficiency, shortening task time, increasing system throughput, supporting high concurrency requirements, and enhancing system scalability. Relationships are constructed between multiple nodes based on a multi-source semantic relationship strategy to obtain a knowledge graph. This integrates four relationship sources: vector semantics, text structure, entity co-occurrence, and advanced reasoning, making the graph semantics richer.

[0089] In addition, the present invention also provides a dataset determination apparatus, medium, and product, which have the same beneficial effects as the dataset determination method described above. Attached Figure Description

[0090] To more clearly illustrate the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0091] Figure 1 A flowchart of a dataset determination method provided in an embodiment of the present invention;

[0092] Figure 2 This is a schematic diagram of a RAG system evaluation dataset provided in an embodiment of the present invention;

[0093] Figure 3 A structural diagram of a dataset determination device provided in an embodiment of the present invention;

[0094] Figure 4 This is a structural diagram of a dataset determination device provided in an embodiment of the present invention. Detailed Implementation

[0095] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the present invention.

[0096] The core of this invention is to provide a method, apparatus, medium, and product for determining datasets, in order to solve the problem that quality angle differences during the dataset determination process affect the evaluation effect of the RAG evaluation system.

[0097] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0098] There are some shortcomings in the use of pre-trained models. One is "knowledge truncation," where the model's knowledge is limited to the date of its training data cutoff, and it cannot access the latest information. Another is "model illusion," where the model tends to fabricate seemingly reasonable but actually erroneous information when answering questions outside its knowledge scope or when uncertain. Therefore, conducting scientific, efficient, and comprehensive evaluations of RAG system performance becomes crucial, and high-quality benchmark data is a prerequisite for effective evaluation. Common methods for determining datasets include:

[0099] 1. Manual construction method:

[0100] This is a traditional and high-quality method, in which human annotators manually write questions, locate the source of answers, and provide standard answers based on large-scale corpora (such as Wikipedia).

[0101] Advantages: High-quality questions, accurate and reliable answers, and the ability to design challenging and complex questions.

[0102] Disadvantages: Extremely high cost, lengthy processing time, and difficult to scale for mass production. More importantly, this method is difficult to quickly customize for private knowledge bases in specific vertical domains, resulting in a disconnect between general datasets and real-world application scenarios.

[0103] 2. Template- or simple rule-based automation methods:

[0104] These methods attempt to automatically generate fill-in-the-blank questions by extracting entities and relationships from text using predefined sentence templates or rules.

[0105] Advantages: High degree of automation and fast speed.

[0106] Disadvantages: The generated questions are of a limited type, the language is rigid, and there is a lack of diversity and depth. It cannot simulate the complex questioning intentions of real users, resulting in limited evaluation effectiveness.

[0107] 3. Preliminary automated generation method based on large language model (LLM):

[0108] This method leverages the powerful language understanding and generation capabilities of LLM to directly generate question-answer pairs from a given text chunk. For example, some open-source frameworks provide basic functionality for generating question-answer pairs from text.

[0109] Advantages: Compared to template-based methods, the generated questions are more natural and diverse. It can quickly generate a batch of evaluation data for a single document.

[0110] The disadvantages and challenges of the above solutions:

[0111] (1) Uncontrollable quality: The quality of directly generated question-and-answer pairs varies, and may include factual errors, vague questions or incomplete answers, lacking a rigorous verification process.

[0112] (2) Single evaluation dimension: Most of the generated questions are "single-hop" questions based on a single text block, which makes it difficult to evaluate the RAG system's ability to integrate cross-text information, conduct comparative analysis, and perform complex reasoning.

[0113] (3) The evaluation method is unstable: The evaluation usually uses a semantic similarity-based model to compare the generated answer with the standard answer. This type of indicator is sensitive to changes in wording and cannot accurately measure the "factual correctness" of the answer. An answer may be semantically similar to the standard answer, but the key facts may be different.

[0114] (4) High cost: In order to ensure the quality of generation and evaluation, it is usually necessary to continuously call the top-level LLM, which leads to the high cost of building large-scale datasets and limits its application in budget-constrained scenarios.

[0115] (5) Static construction process: Existing methods are mostly one-time, linear processing flows, lacking in-depth mining of the internal structure of the knowledge base, and also lacking a dynamic mechanism to feed back the evaluation results to optimize the data generation process.

[0116] In summary, while some progress has been made in automating the construction of RAG evaluation datasets, core challenges remain, including quality, depth, evaluation paradigms, cost, and dynamism.

[0117] The present invention provides a method for determining a dataset, which can solve the above-mentioned technical problems.

[0118] Figure 1 A flowchart of a dataset determination method provided in an embodiment of the present invention is shown below. Figure 1 As shown, the method includes:

[0119] S11: Obtain the target document and perform fragmentation on the target document according to the document length and content components to obtain multiple fragment nodes;

[0120] S12: Extract features from multiple block nodes to obtain corresponding feature parameters, and use them to form nodes of the knowledge graph with the corresponding block nodes. Then, construct relationships between multiple nodes according to the multi-source semantic relationship strategy to obtain the knowledge graph.

[0121] S13: Assign user roles to the knowledge graph, generate target questions based on user roles and node query strategies, input the target questions into the target model, and output the standard answer;

[0122] S14: Decompose the standard answer to obtain fact particles, and construct a quadruple data of target question, standard answer, block node and fact particles;

[0123] S15: Evaluate the quadruple data according to the dataset evaluation collaboration mechanism to determine the final dataset.

[0124] Specifically, the target document can be the source document, such as Portable Document Format (PDF), Markdown, Hyper Text Markup Language (HTML), or Microsoft Office Open XML (DOCX). The target document is segmented based on its length and content components to obtain multiple chunk nodes. This is achieved through preliminary parsing and cleaning of the document content using a knowledge base. The system does not employ a fixed brute-force segmentation method (such as a fixed number of tokens). Instead, it initiates a document statistical feature analysis submodule. This submodule calculates the document set's length distribution (number of tokens, number of paragraphs), structure level (heading levels, list and table proportions), and other content component features. The segmentation process in step S11 aims to divide the target document based on its length and internal structure (headings, lists, tables, etc.) using different segmentation strategies to obtain multiple chunk nodes.

[0125] In step S12, feature extraction is performed on multiple block nodes to obtain corresponding feature parameters, which, along with the corresponding block nodes, constitute the nodes of the knowledge graph to be constructed subsequently. Feature extraction here can employ one or multiple feature extraction methods. In the case of multiple feature extraction methods, a feature extraction pipeline composed of multiple parallel feature extraction devices (Extractors) is used, such as for summarization, topic, keywords, named entities, question generation, and vector embedding. This construction process can be implemented using mapping relationships.

[0126] Multi-source semantic relations establish various types of connections between multiple nodes, weaving discrete nodes into a network-like knowledge graph. Examples include vector similarity relationships, hierarchical structure relationships, entity co-occurrence relationships, and inference relationships, enriching the semantic depth of the graph.

[0127] In step S13, user roles are set for the knowledge graph. To make the generated target questions more closely resemble real-world application scenarios, the user persona automatic generation module performs a global analysis of the knowledge graph and then calls a model to generate a series of virtual user roles. The generation process is as follows: The LLM receives the instruction: Based on the following core themes [topic list], please generate 5 different user personas for the knowledge base. Each persona should include: [role name], [knowledge background] (e.g., novice, expert, interdisciplinary researcher), [core interests], and [typical questioning style] (e.g., concise, inquiry-based, critical). Example: If the knowledge base is about "artificial intelligence," the generated personas might be: "A social activist concerned about biases against artificial intelligence (AI)," "A technical manager hoping to implement AI principles in products," "A philosophy student researching the long-term social impact of AI," etc. The target questions are generated based on the user roles and node query strategies. Here, the node query strategy is either a single-hop query for a single node or a multi-hop query where multiple nodes have certain specific semantic relationships, corresponding to the generation of their respective target questions.

[0128] Input the target question into the target model to output a standard answer. The standard answer is then decomposed. The purpose of this decomposition is to establish a more stable and interpretable evaluation paradigm, following strict instructions to break it down into a series of independent, indivisible atomic fact units (nuggets), i.e., fact particles. Example instruction: You are a fact extraction engine. Please decompose the following text into a set of independent, basic fact statements. Each fact statement should be a JavaScript Object Notation (JSON) object containing a "subject," "predicate," and "value." If a sentence contains multiple facts, extract them separately, ensuring that each fact is atomic.

[0129] After decomposition, the quadruple data of target question-standard answer-block node-fact particle is constructed. In step S15, the quadruple data is evaluated according to the dataset evaluation collaboration mechanism. The collaboration mechanism here includes question quality, answer fidelity, answer completeness, contextual relevance between block nodes, and fact particle quality. The evaluation agent is used to evaluate these aspects and output a structured diagnostic report (including scores and classifications for each dimension) and an overall quality score for each quadruple data, so that multiple quadruple data can be combined into the final dataset.

[0130] The beneficial effects of this invention are as follows: First, by segmenting the target document according to document length and content components to obtain multiple segmented nodes, compared to the conventional brute-force segmentation method that results in a superficial knowledge base document structure, the segmentation process using an adaptive processing pipeline strategy based on document length and different content components achieves hierarchical segmentation for in-depth mining of the document structure. Second, feature extraction is performed on multiple segmented nodes to obtain corresponding feature parameters, which, together with the corresponding segmented nodes, constitute nodes in the knowledge graph. Nodes are constructed based on segmented nodes and feature parameters to facilitate the subsequent weaving of discrete nodes into a knowledge graph. A knowledge graph is constructed based on relationships between multiple nodes using a multi-source semantic relationship strategy. The multi-source semantic relationships, combined with various node relationships, enrich the semantics of the graph, generating a dynamically maintained knowledge graph containing nodes and relationships. Third, user roles are assigned to the knowledge graph, and user roles are integrated with diverse node query strategies to avoid generating target questions solely through single-hop queries. This facilitates diversified and comprehensive evaluation dimensions, moving beyond basic "retrieval-reading" capabilities and improving the ability to integrate, analyze, compare, and reason across textual information. Fourth, the target question is input into the target model to obtain the standard answer. The standard answer is then decomposed into fact particles, with each standard answer broken down into a set of atomic facts. This transforms the standard answer from vague to clear factual coverage during subsequent dataset evaluation. Finally, a quadruple dataset of the target question, standard answer, chunk nodes, and fact particles is constructed. Based on the original triplet consisting of question-answer-chunk nodes, the quadruple dataset is evaluated using a dataset evaluation collaboration mechanism to determine the final dataset. Fact particles are then added to improve the accuracy of subsequent evaluations. In terms of evaluation processing, evaluation is performed according to the dataset evaluation dimensions, forming a closed-loop quality control method with a multi-agent collaborative verification mechanism. This effectively eliminates low-quality samples, corrects factual errors and answer illusions, fundamentally ensuring that every sample included in the dataset undergoes rigorous testing. This improves both the dataset quality and the evaluation effectiveness and reliability of the RAG system.

[0131] In some embodiments, the target document is segmented according to document length and content components to obtain multiple chunk nodes, including:

[0132] The target document is parsed to obtain a text file, and the structural information corresponding to the content components is preserved.

[0133] The text files are classified according to their length to obtain the target length file. The target length file is either the first length file or the second length file, and the length of the first length file is greater than the length of the second length file.

[0134] When the target length file is a first length file, the first length file is segmented according to the structure information to obtain the corresponding block nodes;

[0135] When the target length file is a second length file, the second length file is used as the block node.

[0136] Specifically, the target document is parsed to obtain a text file. For example, the system receives a folder containing dozens of Markdown documents about the "History of Artificial Intelligence" as input. The document processing and knowledge graph construction module starts, first calling the "unstructured.io" library to parse the Markdown document into plain text content, while retaining the title structure information, that is, retaining the structural information corresponding to the content components.

[0137] Text files are categorized by length to obtain target length files. There will be long documents or medium-to-short documents, which can be assigned using a word count threshold. Documents exceeding the first word count threshold are classified as long documents (first-length files), and those within the first word count threshold are classified as medium-to-short documents (second-length files). When the target length file is a first-length file, it is segmented based on structural information to obtain block nodes. This segmentation process takes into account the generally long document length and clear heading hierarchy. The structural information corresponding to the headings needs to be included in the segmentation to obtain block nodes.

[0138] When the target length file is the second length file, deep segmentation is skipped, and the entire document is directly treated as a chunk node to avoid fragmentation of context information caused by excessive segmentation. Each node is designed as a flexible data structure based on a data model, containing a unique identifier (Universally Unique Identifier, UUID), node type ("DocumentNode", "ChunkNode"), original content, and a dynamically expandable "metadata" dictionary to store various features and source information (such as source file name and page number) extracted later.

[0139] The segmentation process provided in this embodiment selects different segmentation strategies based on document characteristics. For long documents with strong structure, hierarchical segmentation based on headings or semantics is adopted, while short documents with weak structure are processed as a whole. Compared with the conventional brute-force segmentation, the segmentation strategy of this embodiment has multiple styles and improves accuracy.

[0140] In some embodiments, the first-length file is segmented according to structural information to obtain corresponding block nodes, including:

[0141] Initiate the hierarchical segmentation mechanism;

[0142] A third-length file is obtained by adding structural information to the first-length file.

[0143] The third-length file is split into multiple text blocks using a hierarchical segmentation mechanism, and these text blocks are used as corresponding block nodes.

[0144] Specifically, in terms of segmentation, a hierarchical segmentation mechanism is enabled, prioritizing the use of a hierarchical segmenter based on headings or semantics. For example, the "RecursiveCharacterTextSplitter" from "LangChain" can be used as a reference, but with added support for Markdown headings or HTML tags ("..."). <h1>”，"< / h1> <h2>This involves explicit identification of text chunks ("ChunkNodes"), which divides a long document into multiple logically continuous and semantically complete text chunks. The original document itself acts as a high-level "DocumentNode," while its segmented text chunks act as "ChunkNodes," connected to it via "contains" relationships. This approach preserves the document's inherent structure and is crucial for subsequent context-aware retrieval.

[0145] In other words, structural information is added to the first-length file to obtain a third-length file. The third-length file is then segmented using a hierarchical segmentation mechanism to obtain multiple text blocks, which serve as corresponding block nodes.

[0146] The system selects "Recursive Character Text Splitter" and configures it to use Markdown headings as the primary delimiter, setting a reasonable block size (e.g., 1024 tokens) as a secondary delimiter. For example, a chapter on "Neural Networks" is divided into multiple logically coherent "ChunkNodes" such as "Perceptron," "Backpropagation Algorithm," and "Convolutional Neural Networks." Each "ChunkNode" is instantiated through a data model, obtaining a UUID and recording the source file name and title. All "ChunkNodes" originating from the same document establish a "CONTAINS" relationship with the "DocumentNode" representing that document.

[0147] This embodiment provides a method for segmenting a first-length file based on structural information to obtain corresponding block nodes. This breaks away from the original brute-force segmentation and incorporates structural information, thus laying the foundation for subsequent knowledge graph construction and improving the reference basis.

[0148] In some embodiments, feature extraction is performed on multiple block nodes to obtain corresponding feature parameters, including:

[0149] Initiate a parallel feature extraction mechanism; wherein, the parallel feature extraction mechanism includes at least a summary extraction mechanism, a topic extraction mechanism, a keyword extraction mechanism, a named entity recognition mechanism, a question generation mechanism, and a vector embedding extraction mechanism;

[0150] Feature parameters are obtained by extracting features from multiple block nodes through a parallel feature extraction mechanism.

[0151] Specifically, the parallel feature extraction mechanism is driven by a pre-trained model (such as a language model), works by submitting text content with specific instructions (Prompt) to the model, and utilizes asynchronous input / output (IO) tools to achieve large-scale parallel processing.

[0152] The parallel feature extraction mechanisms here include at least a summary extraction mechanism, a topic extraction mechanism, a keyword extraction mechanism, a named entity recognition mechanism, a question generation mechanism, and a vector embedding extraction mechanism, which are elaborated below:

[0153] Summary Extractor: Generates a concise summary for each "ChunkNode" and stores it in metadata['summary']. The summary is not only used for rapid understanding but is also crucial for subsequent computation of semantic relationships across summaries. For example, a Convolutional Neural Network (CNN) is a deep learning model particularly well-suited for processing image data. It simulates the human visual cortex through convolutional layers, pooling layers, and fully connected layers, achieving great success in image recognition, object detection, and other fields. This summary is stored in "metadata['summary']".

[0154] Themes Extractor: Extracts 1-3 core themes or topics to provide material for generating themes-related queries. Entities identified: [Convolutional Neural Networks, CNNs, Deep Learning, Image Recognition, Object Detection].

[0155] Keyphrases Extractor: Extracts key phrases from text for building more accurate sparse vector indexes (such as text retrieval algorithms (Best Matching 25, BM25)) or for entity linking in knowledge graphs. Potential Question: What are the core components of a convolutional neural network?

[0156] Named Entity Extractor (NERExtractor): Identifies and categorizes entities such as person names, place names, organization names, product names, and dates in text. These entities are an important source for constructing fact-based problems.

[0157] Question Extractor: Several potential questions are pre-generated for each text block. These questions can serve as simple evaluation samples or as "seeds" for synthesizing subsequent complex questions.

[0158] The embedding extractor mechanism calls a specialized embedding model to generate high-dimensional vector representations of the original text and summary for each node, storing them in the node's "embedding" and "summary_embedding" attributes. This is the foundation for calculating semantic similarity. High-dimensional vectors are generated for the original text and summary of "ChunkNode" and stored in the "embedding" and "summary_embedding" fields respectively. All these extraction tasks are executed concurrently through "asyncio.gather," significantly reducing processing time.

[0159] Based on the feature extraction mechanism described above, features are extracted from multiple block nodes to obtain the corresponding feature parameters. The feature extraction process here is the same as the conventional method, but it can also be different, and no limitation is made here.

[0160] The feature parameters determined by the feature extraction using the parallel feature extraction mechanism provided in this embodiment improve processing efficiency, shorten task time, increase system throughput, support high concurrency requirements, and enhance system scalability.

[0161] In some embodiments, relationships are constructed between multiple nodes according to a multi-source semantic relationship strategy to obtain a knowledge graph, including:

[0162] The vector relationship between multiple nodes is obtained by processing the vector feature parameters of the block nodes within multiple nodes based on the vector similarity relationship strategy.

[0163] Obtain the linear reading order of the target document and determine the hierarchical structure relationship strategy based on the linear reading order;

[0164] The hierarchical relationship strategy is used to process multiple nodes to obtain the sequential relationship between them;

[0165] The co-occurrence relationship of entities is determined by processing the named entities of multiple nodes according to the entity co-occurrence relationship strategy;

[0166] Multiple nodes are processed according to the reasoning strategy to determine the corresponding causal relationships;

[0167] A knowledge graph with multiple nodes is constructed based on vector relationships, sequence relationships, entity co-occurrence relationships, and causal relationships.

[0168] Specifically, for vector similarity relationships, batch matrix multiplication (calculating cosine similarity) is performed on the embedding vectors of all block nodes. Node pairs with similarity scores higher than a preset threshold (e.g., 0.85, which can be fine-tuned with small sample data) are selected, and a "SIMILAR_TO" relationship is established between them. The relationship itself can carry a similarity score as an attribute. For long documents, this builder may prioritize using the embedding vectors of their summaries for calculation to capture a more macroscopic thematic similarity. For example, if the summaries of the "Convolutional Neural Network" node and the "Recurrent Neural Network" node have a similarity of 0.86 (higher than the threshold of 0.85), a "SIMILAR_TO (score=0.86)" relationship is established between them.

[0169] Based on the linear reading order of the target document, a hierarchical relationship strategy is determined to ensure the order of multiple nodes. The "CONTAINS" relationship from "DocumentNode" to its "ChunkNode," as well as the "prior" / "inferior" relationships between "ChunkNodes," are automatically established, preserving the linear reading order of the document.

[0170] Based on the entity co-occurrence relationship strategy, named entities of multiple nodes are processed to determine entity co-occurrence relationships, and key named entities that appear together in different nodes are analyzed. If two nodes share a sufficient number of key entities (e.g., the number of shared entities > 2), an "ENTITY_CO_OCCURRENCE" relationship is established between them. This is very effective for discovering content that is topic-related but has large differences in textual expression. For example, if the "F Test" node and the "E Meeting" node both frequently contain entities such as "John McCarthy" and "Marvin Minsky", then an "ENTITY_CO_OCCURRENCE" relationship is established between them.

[0171] The system processes multiple nodes using a reasoning strategy to determine corresponding causal relationships. For key nodes, the model can be invoked to infer higher-level relationships. For example, given the content of two nodes, the system queries the LLM to determine if a "cause-effect," "contrast," or "example-of" relationship exists between them, greatly enriching the semantic depth of the graph. For instance, for the key nodes "backpropagation algorithm" and "the rise of deep learning," the system can invoke the model for higher-level relationship reasoning, asking: "Node A1 describes the backpropagation algorithm, and node B1 describes the rise of deep learning. What is the most likely relationship between A1 and B1?" The model might answer "cause-effect," and the system can then establish a "cause-effect" relationship.

[0172] Based on the different relationships mentioned above, a knowledge graph is constructed between multiple nodes to provide important reference information.

[0173] This embodiment provides a knowledge graph that constructs relationships between multiple nodes based on a multi-source semantic relationship strategy. This integrates four relationship sources: vector semantics, text structure, entity co-occurrence, and advanced reasoning, making the graph semantics richer.

[0174] In some embodiments, user roles are assigned to the knowledge graph, including:

[0175] A global analysis and processing of the knowledge graph yields a corresponding list of topics;

[0176] The target subtopic is determined by selecting from the subtopics in the topic list.

[0177] Generate corresponding user roles based on the target subtopic.

[0178] Specifically, the knowledge graph is analyzed globally to obtain a list of corresponding topics, such as "symbolic AI" and "connectionist AI." Based on the various sub-major graphs in the topic list, target sub-topics, or core topics, are selected and processed. Corresponding user roles are then generated based on these core topics, such as instructing a model to generate a persona. Outputs might include: "A high school student curious about the history of AI," "A computer science undergraduate seeking the evolution of technology," and "A humanities and social science scholar interested in the philosophical debates surrounding AI technology."

[0179] To make the generated evaluation questions more closely resemble real-world application scenarios, the user persona generation module first performs a global analysis of the knowledge graph (such as statistical topic and entity frequency), and then calls a highly creative top-level LLM to generate a series of virtual user roles.

[0180] Generation process: The LLM will receive instructions, such as generating 5 different user personas for the knowledge base based on the following core themes [topic list]. Each persona should include: [role name], [knowledge background] (e.g., novice, expert, interdisciplinary researcher), [core interests], and [typical questioning style] (e.g., concise, inquiry-based, critical).

[0181] Example: If the knowledge base is about "issues related to artificial intelligence", the generated personas might be: "a social activist who is concerned about AI bias", "a technical manager who wants to implement guidelines in products", "a philosophy student who studies the long-term social impact of AI", etc.

[0182] The user roles provided in this embodiment often have diverse goals and skill backgrounds, and the needs of different roles vary significantly. Setting roles can achieve customized matching of functions and permissions, reduce invalid operations, reduce cognitive load, and significantly improve usage efficiency and experience.

[0183] In some embodiments, generating a target question based on user roles and node query strategies includes:

[0184] In the knowledge graph, determine the node type corresponding to the target node, where the node type is either a single-hop node type or a multi-hop node type;

[0185] When the target node is a single-hop node type, the first pre-trained model is invoked, and the target node and user role are input into the first pre-trained model to obtain the first target problem;

[0186] When the target node is a multi-hop node type, the corresponding target relationship is determined in the knowledge graph; the second pre-trained model is called, and the target relationship, target node and user role are input into the second pre-trained model to obtain the second target question.

[0187] In these knowledge graphs, determine the single-hop and multi-hop node types corresponding to the target node. A single-hop query randomly selects a high-information-density block node, combines this target node with a random persona, calls the first pre-trained model, and inputs the user role into the model to obtain the first target question. Submit an instruction to the LLM: As the [persona], please formulate a question related to your interests based on the following text. This question should conform to your questioning habits (e.g., concise, in-depth, critical, application-oriented, etc.) and focus on the core topic of the text. Objective: To evaluate the RAG system's basic "retrieval-generation" capability for single, isolated knowledge points.

[0188] Multi-hop queries are used to find combinations of nodes with specific relationship paths within the knowledge graph. For example, finding two nodes, A2 and B2, connected by the "SIMILAR_TO" relationship but with complementary content. An instruction is submitted to the LLM: Assume a [persona] and, combining information from two contexts [context A2] and [context B2], formulate a question that requires reference to both contexts for a complete answer. The answer should clearly integrate information from both sides. Objective: To evaluate the system's ability to perform complex reasoning and information integration. The generated standard answer will explicitly label the information sources ("source_nodes": ["uuid_A2", "uuid_B2"]), providing precise source tracing evidence for subsequent evaluation.

[0189] Taking multi-hop query as an example, the target relationship is determined, and the target relationship, target node and user role are input into the second pre-trained model to obtain the second target problem.

[0190] In this embodiment, the strategies for single-hop and multi-hop node types are input into the corresponding pre-trained models to generate target questions. Compared to conventional question generation, which often involves simple single-point fact retrieval and cannot effectively evaluate the system's complex reasoning capabilities, this embodiment, combined with dynamically generated user personas, can systematically generate questions covering both single-hop and multi-hop types. It can deeply consider the RAG system's performance in advanced cognitive tasks such as information integration and logical reasoning, making the evaluation uniquely comprehensive and challenging.

[0191] In some embodiments, before inputting the target relationship, target node, and user role into the second pre-trained model to obtain the second target problem, the method further includes:

[0192] When the target relationship of multiple target nodes is a preset target relationship and the corresponding content is complementary, or when the target relationship of multiple target nodes is a preset target relationship and the corresponding content is different but shares the same subtopic, the process proceeds to the step of inputting the target relationship, target nodes and user roles into the second pre-trained model to obtain the second target problem.

[0193] Specifically, to further enhance the challenge and comprehensiveness of the evaluation, the system also supports generating complex queries, such as comparative queries, to find two nodes in the graph that have a "CONTRAST" relationship or share the same topic but have different content (e.g., descriptions of two different technical solutions). The LLM command generates questions comparing their advantages, disadvantages, similarities, and differences. For example: Please compare and summarize the core differences between the "attention mechanism" described in node A3 and the "recurrent neural network" described in node B3 when processing sequence data, and their respective applicable scenarios. This applies to scenarios where multiple target nodes have a pre-defined target relationship and their corresponding content is complementary.

[0194] Negative / Counterfactual Query: The process involves randomly selecting a node and generating a question whose answer does not exist in the current knowledge base. This is crucial for evaluating whether the RAG system truly "knows what it knows and doesn't know what it doesn't know," rather than fabricating answers (illusions). For example: Please describe a detailed case study in this knowledge base regarding the application of quantum computers in marketing. (Assuming this content does not exist in the database). The standard answer should be: Based on existing data, the knowledge base does not contain specific cases about the application of quantum computers in marketing; that is, scenarios where multiple target nodes have a pre-defined target relationship, corresponding to different content, and share the same sub-topic.

[0195] It should be noted that both of the above scenarios are implemented based on multi-hop node type queries, enabling more complex queries.

[0196] The multi-hop query strategy for comparative and negative queries provided in this embodiment enhances the depth and breadth of the evaluation, enabling a comprehensive assessment of the overall capabilities of the RAG system.

[0197] In some embodiments, the standard answer is decomposed to obtain fact particles, including:

[0198] The standard answer is segmented to obtain the corresponding word units;

[0199] The third pre-trained model is invoked, and the lexical units are input into the third pre-trained model for decomposition processing to obtain the corresponding fact particles; wherein, the fact particles contain at least the subject, attribute and value.

[0200] Specifically, the standard answer is decomposed by first segmenting it to obtain the corresponding word units, and then calling the third pre-trained model to decompose the word units into a series of independent fact particles. These fact particles at least include the subject, attribute, and value.

[0201] This module invokes an LLM that has been fine-tuned with instructions and is adept at structured output. It receives the text of the standard answer and follows strict instructions to break it down into a series of independent, indivisible atomic units of fact (nuggets).

[0202] Example Instruction: You are a fact extraction engine. Please break down the following text into a set of independent, basic fact statements. Each fact statement should be a JSON object containing a subject, predicate, and object. If a sentence contains multiple facts, extract them separately. Ensure that each fact is atomic.

[0203] Example:

[0204] Standard answer: A4 Corporation is a multinational technology company headquartered in Location B4, founded by C and D on April 1, 1976.

[0205] Deconstructed factual particles (JSON format): [

[0207] {"subject": "A4 Company", "predicate": "type", "object": "multinational technology company"}

[0208] {"subject": "Company A4", "predicate": "Headquarters Location", "object": "B4"}

[0209] {"subject": "A4 Company", "predicate": "Founder", "object": "C"}

[0210] {"subject": "A4 Company", "predicate": "Founder", "object": "C"}

[0211] {"subject": "A4 Company", "predicate": "Founder", "object": "D"}

[0212] {"subject": "A4 Company", "predicate": "Date of Establishment", "object": "April 1, 1976"}

[0213] ].

[0214] This embodiment provides a further processing method for standard answers to obtain fact granules. Traditional text similarity evaluation methods, such as Recall-Oriented Understudy for Gisting Evaluation (ROUGE), are easily affected by wording changes and cannot accurately measure "factual correctness." This invention innovatively introduces a "fact granulation deconstruction" technique into the automated process, breaking down each standard answer into a set of atomic facts. When evaluating the dataset, the assessment of answer quality shifts from vague semantic similarity to clear factual coverage, making the dataset more stable and robust to interference.

[0215] In some embodiments, the quadruple data is evaluated according to a dataset evaluation collaboration mechanism, including:

[0216] Natural language processing tools are used to perform semantic analysis on the target question in order to determine the question quality;

[0217] The fourth pre-trained model is invoked, and the context information corresponding to the standard answer and the block node is input into the fourth pre-trained model to determine the semantic similarity between the standard answer and the block node.

[0218] Determine the fidelity of the answer based on semantic similarity;

[0219] The fifth pre-trained model is invoked, and the standard answer and the target question are input into the fifth pre-trained model to determine the semantic coverage of the standard answer and the target question;

[0220] Determine the completeness of the answer based on semantic coverage;

[0221] The relevance of the block nodes to the target question and the standard answer is determined based on similarity.

[0222] The sixth pre-trained model is invoked, and the fact particles are input into the sixth pre-trained model to determine the independence and integrity of the fact particles;

[0223] The dataset evaluation results are determined based on the weight parameters corresponding to the question quality, answer fidelity, answer completeness, relevance of the standard answer, and independent integrity of the fact particles.

[0224] Specifically, corresponding evaluation processing strategies are set based on different evaluation dimensions. In this process, the generation of quadruple data is formed by a generation agent, and a judgment agent is used in this stage. The core of this agent is a lightweight, fine-tuned scoring model. By fine-tuning on a small number (hundreds to a thousand) of high-quality manually labeled samples, a small to medium-sized model can accurately score and classify the generated content across multiple dimensions.

[0225] This mainly includes question quality, answer fidelity, answer completeness, contextual information of chunk nodes, relevance between the target question and the standard answer (contextual relevance), and independent integrity of fact particles (fact particle quality).

[0226] Question Quality: Are the questions clear and unambiguous? Are they too broad or too specific? Is the difficulty level appropriate? (Score 1-5)

[0227] Answer Faithfulness: Can every sentence in the answer be supported by the given context? Is there any fabricated (illusionary) content? (Classification: High, Medium, Low);

[0228] Answer Completeness: Does the answer fully address all aspects of the question? Are any key pieces of information from the context omitted? (Classification: High, Medium, Low)

[0229] Context Relevance: Is the provided context necessary to answer the question? Is there a large amount of redundant or completely irrelevant context? (Rating 1-5)

[0230] Fact Nugget Quality: Are the fact granules generated from the deconstruction of the standard answer semantically complete, with clear subject-verb-object structure and consistent granularity? Are there any deconstruction errors, missing information, or redundant repetitions? (Score 1-5)

[0231] We set the weight parameters for each of the different dimensions mentioned above to obtain a comprehensive dataset evaluation result.

[0232] Regarding the use of different pre-trained models, different pre-trained models can be set according to the actual situation. They can be selected based on the characteristics of the evaluation dimensions, or they can be the same pre-trained model. There are no restrictions here.

[0233] The dataset evaluation collaboration mechanism provided in this embodiment evaluates the quadruple data and provides evaluation reports for each dimension to facilitate subsequent improvement strategies.

[0234] In some embodiments, when it is determined that the dataset evaluation result has not reached a preset threshold, the method further includes:

[0235] Analyze and process the dataset evaluation results to determine correction strategies;

[0236] When the strategy is corrected to a problem-oriented strategy, a new target problem is generated, and the new target problem, context information, and target problem are input into the target model to obtain a new standard answer.

[0237] Alternatively, when the strategy is modified to a block node strategy, new context information is determined based on the embedding vector of the target question and the node query strategy.

[0238] Alternatively, when the strategy is changed to a four-tuple matching strategy, the four-tuple data is discarded, the next target document is obtained, and the process proceeds to the step of splitting the target document into multiple chunk nodes based on the document length and content components, until new four-tuple data is determined.

[0239] Specifically, the diagnostic report received by the agent is modified. For unqualified samples (such as those with an overall score below the preset threshold of 4.0), it will adopt different strategies based on the specific "cause":

[0240] When the correction strategy is a question-based strategy, if the question is unclear (e.g., there are unclear pronoun references), it will attempt to rewrite the question. If the answer contains slight illusions or missing information, it will instruct the LLM to regenerate a more faithful and complete answer based on the context and the original question.

[0241] When the correction strategy is a chunk node strategy, if the diagnosis is "context irrelevant", the correction agent can perform a query to find a more relevant "ChunkNode" in the knowledge graph as the new context using the question's embedding vector, and then regenerate the answer.

[0242] When the correction strategy is a quadruple matching strategy, if the problem does not match the context at all, or if the quality is still low after multiple rounds of correction, it will directly discard the sample, and can choose to record this failure case for analysis of potential problems in the knowledge graph or generation strategy.

[0243] All corrected samples will be resubmitted to the evaluation agent for a new round of review, forming an iterative cycle of "generation-evaluation-correction". Only samples that pass the quality threshold will be accepted into the final dataset.

[0244] This embodiment presents a quality control system that mimics peer review to address the quality challenges of automatically generated content. The system consists of three intelligent agents with different roles (generating, judging, and correcting), forming a dynamic, iterative quality improvement loop. It is not a simple "post-generation filtering" but rather a self-improving mechanism that includes "automatic correction" and "iterative review."

[0245] In some embodiments, the dataset consists of at least one quadruple data set, and the process of constructing the dataset includes:

[0246] Obtain the target model corresponding to the construction of multiple quadruple data;

[0247] If the target model is the same, the requests for constructing multiple quadruple data will be merged into the same batch for batch processing.

[0248] Specifically, in the above embodiments, the dataset construction process corresponding to steps S11 to S15 only involves one record. The dataset consists of multiple records, so there are also a large number of pre-trained model calls during the entire dataset construction process. Mature open-source tools and strategies will be used to manage these tasks efficiently and robustly.

[0249] Smart Batching: This method merges multiple requests sent to the same model's Application Programming Interface (API) into a single batch, significantly reducing network overhead and the number of API calls. In this example, if multiple quadruple data sets correspond to the same target model, then these requests need to be merged into a single batch for batch processing.

[0250] When the target models provided in this embodiment belong to the same type, multiple requests are merged and processed in the same batch during the dataset construction, reducing network overhead and the number of API interface calls.

[0251] In some embodiments, when the target models are not the same, the method further includes:

[0252] Multiple target models are classified to determine the target models corresponding to different categories;

[0253] Among target models with different classifications, identify the target category that has the largest number of target models under the same category;

[0254] In target classification, requests to construct the four-tuple data corresponding to the target classification are merged into the first batch;

[0255] This process continues until the requests for the four-tuple data corresponding to the category with the smallest number of target models under the same category are merged into the Nth batch.

[0256] Based on network bandwidth, the target model in different batches is called in a balanced manner to generate quadruple data for the corresponding batch.

[0257] Specifically, when the target models are not the same, multiple target models need to be classified to determine the target category with the largest number of target models under the same category. This category is then merged into the first batch, and so on. The requests for the four-tuple data corresponding to the category with the smallest number of target models under the same category are merged into the Nth batch. The various batches are then called in a balanced manner according to the network bandwidth to generate the four-tuple data under each batch.

[0258] It includes a built-in rate controller for external APIs (such as OpenAI, Anthropic, and Google AI Platform), which can be implemented using libraries like tenacity. When a rate throttling error (HTTP 429) is detected, it automatically reduces the request frequency and uses an exponential backoff algorithm for retries, ensuring the stability and success rate of long-running tasks.

[0259] The scheduling process of target models in different batches provided in this embodiment adopts asynchronous parallel scheduling, which simultaneously initiates and manages hundreds or even thousands of independent model requests, maximizing the utilization of computing resources and network bandwidth, and significantly shortening the end-to-end construction time.

[0260] In some embodiments, the process of determining the target model includes:

[0261] When the target node is a single-hop node, the first performance model with the greatest performance effect in the performance model set is selected as the target model.

[0262] When the target node is a multi-hop node type, the first model with the highest comprehensive score in the model set is selected as the target model.

[0263] The selection of which model to use depends on the type of the target node. If it's a single-hop node, the best-performing model from the set of performance models should be chosen as the target model. This is because it requires strong reading comprehension and content generation skills, but less demanding requirements for top-tier creativity.

[0264] In multi-hop node types, the first model with the highest overall score in the model set is selected as the target model. The rationale is that it requires strong reasoning, creativity, and the ability to follow complex instructions to ensure the novelty and depth of the generated content.

[0265] Additionally, when the task description involves fact granular deconstruction, named entity recognition, and keyword extraction, the recommended model level is the instruction fine-tuning model. The reason is that these models excel in adhering to structured output formats such as JSON, which require high performance.

[0266] The task description evaluates the agent's quality across multiple dimensions, and the recommended model level is a lightweight, fine-tuned model. The rationale is that the task is clearly defined and the domain is fixed; a fine-tuned lightweight model can perform comparablely to a large model on specific tasks, but with a cost that may be only a fraction of the latter.

[0267] When the task description is to generate vector representations of text, the recommended model level is a dedicated embedding model. The rationale is that it is specifically optimized for embedding tasks, offering excellent performance at a very low cost.

[0268] By employing this differentiated, task-driven "intelligent routing" strategy, this invention can reduce the overall computational cost by more than an order of magnitude without sacrificing (or even improving) the quality of the final dataset in some tasks, thus addressing the core pain point of high evaluation costs (especially the cost of the scoring model).

[0269] The target models provided in this embodiment are different for different node types. Based on factors such as model characteristics and cost, a model with a higher cost performance is selected to improve the quality of the dataset.

[0270] In some embodiments, after determining the final dataset, the method further includes:

[0271] Call the preset model corresponding to the retrieval enhancement generation system, and input the final dataset into the preset model to output the current answer;

[0272] The fact particles of the current answer and the corresponding quadruple data in the dataset are parsed and processed to determine the target proportion of the evaluation metric that contains fact particles.

[0273] The model test results of the preset model are determined based on the target proportions of multiple evaluation indicators.

[0274] Specifically, considering that the above embodiments only construct the final dataset, after constructing the final dataset, the evaluation process of the RAG system needs to be performed. This requires calling the preset model of the RAG system, inputting its final dataset, and obtaining the current answer. The current answer and fact particles are then parsed to determine the target proportion of fact particles corresponding to the evaluation metrics. For example, the evaluation metrics are fact recall, fact precision, and F1 score, thereby achieving a quantitative, stable, and interpretable evaluation of the "factual correctness" and "completeness" of the answer.

[0275] Fact Recall: The number of ground-truth fact particles included in the system's answer, reflecting "information coverage capability". It is calculated as: number of correctly mentioned particles in the answer / total number of particles.

[0276] Fact Precision: Of the particles mentioned in the answer, how many are true ground truth, and not errors or extraneous information? That is: Number of correct particles / Total number of particles generated by the system.

[0277] Fact F1 Score: The harmonic mean of the two above is the final overall quality measure.

[0278] The target proportions of the above evaluation indicators are used to determine the model test results of the preset model. If the model test results are qualified, the performance of the preset model under the RAG system is determined to be qualified.

[0279] This embodiment provides an evaluation metric for the RAG system that utilizes fact granularity in addition to constructing the dataset. During evaluation, by calculating the number of "fact granules" covered by the answers generated by the RAG system, precise fact recall, fact precision, and F1 score can be obtained. This evaluation paradigm transforms the assessment of answer quality from the vague "semantic similarity" to the clear "fact coverage," making the evaluation results more stable, interpretable, robust to interference, and fairer.

[0280] In some embodiments, the knowledge graph construction process, when a new target document is added, includes:

[0281] The knowledge graph constructed from the target document is used as the first knowledge graph;

[0282] When a new target document is added, the process proceeds to the step of splitting the target document into multiple chunk nodes based on the document length and content components, until the new node is determined.

[0283] A second knowledge graph is obtained by constructing relationships between multiple new nodes based on a multi-source semantic relationship strategy.

[0284] The first and second knowledge graphs are merged to obtain the final knowledge graph.

[0285] Specifically, during the knowledge graph construction process, if a new document is added, it is not necessary to completely rebuild the graph. Steps S11 to S12 are only performed on the new document to generate new nodes and relationships to construct a second knowledge graph. The two graphs are then merged to obtain the final knowledge graph. This merging process can be a direct splicing method or other splicing methods; no specific limitation is made here.

[0286] The new target document provided in this embodiment forms a second knowledge graph after being added, avoiding the need to completely rebuild the knowledge graph. Instead, the second knowledge graph is added on the basis of the original first knowledge graph, saving construction time and improving efficiency.

[0287] In some embodiments, the method further includes:

[0288] Based on the first preset period, examine the timeliness of isolated nodes among multiple nodes in the knowledge graph and the relationships connecting multiple nodes.

[0289] If the timeliness meets the preset conditions, then the target nodes and isolated nodes of the target relationship that meet the preset conditions will be classified as cold nodes.

[0290] The cold nodes are re-analyzed according to the second preset cycle to determine the new knowledge graph.

[0291] Specifically, the periodic health check of each node in the knowledge graph requires examining the timeliness of isolated nodes and node relationships within the knowledge graph according to the first preset cycle. If the timeliness meets preset conditions, the target relationship corresponding to the timeliness that meets the preset conditions is treated as a weak connection and classified as a cold node in the same way as isolated nodes, so that the cold nodes can be re-analyzed and processed to determine a new knowledge graph.

[0292] Additionally, if a generated question-answer pair is deemed "low-quality," particularly due to "contextual irrelevance" or "answer illusion," the system will send this feedback signal back to the knowledge graph. For example, if nodes a and b are used to generate a failed multi-hop question, the system can reduce the weight of their "SIMILAR_TO" relationship or mark these two nodes as requiring further human review.

[0293] This embodiment provides a method for periodically checking the node status in the knowledge graph, periodically running scripts to check isolated nodes and weakly connected parts in the graph, or identifying "cold spots" and "hot spots" in the knowledge base based on user query logs (if available), and may trigger re-analysis or enhancement of cold spot areas.

[0294] Figure 2 This is a schematic diagram of a RAG system evaluation dataset provided in an embodiment of the present invention, as shown below. Figure 2 As shown, this example uses a collection of Markdown documents about the "history of artificial intelligence development," but this is not a limitation of the present invention. The method and system can be applied to any document corpus in any field and any format.

[0295] Hardware platform: One or more cloud servers equipped with NVIDIA A100 or equivalent graphics processing units (GPUs).

[0296] Software stack:

[0297] Core libraries: data modeling, asynchronous processing, document parsing, framework-like components (text segmentation), numerical computation, and retry logic.

[0298] Data storage: A vector database is used to store node embeddings, and a graph database or in-memory graph structure is used to store the nodes and relationships of the knowledge graph.

[0299] LLM service: Accesses multiple model services via API for fine-tuned evaluation proxies.

[0300] In the query synthesis module, the knowledge graph is first analyzed to identify core themes such as "symbolic AI" and "connectionist AI." Then, the model is invoked and instructed to generate personas. Outputs might include: "A high school student curious about the history of AI," "A computer science student exploring the evolution of technology," and "A humanities and social science scholar interested in the philosophical debates surrounding AI technology."

[0301] Path selection: The query synthesis engine randomly samples a path in the knowledge graph: "E Meeting" node ENTITY_CO_OCCURRENCE --> "F Test" node.

[0302] Character and Instructions: Randomly select the character "Computer Science Major Undergraduate".

[0303] LLM Invocation: Submitting Instructions to the Model: Imagine yourself as a computer science student tracing the development of AI technology. Based on the following two texts about the 'E Conference' and the 'F Test,' formulate a profound question that requires integrating information from both perspectives, and provide a standard answer. The question should demonstrate an exploration of historical connections.

[0304] Result generated:

[0305] Question: How did H's 'F test' lay the conceptual foundation for establishing artificial intelligence as an independent discipline at the 'E conference' ten years later? What are the connections and differences between the two in their approaches to defining 'intelligence'?

[0306] The standard answer states: "The F-test, for the first time, provided a standard for judging whether a machine is intelligent at the operational level, equating intelligence with the ability to converse indistinguishably from humans. This 'behavioralist' definition sets a grand but conceivable goal for AI research. While participants at the E-Conference had different views on the implementation path, they all shared the core belief that 'machines can simulate intelligence,' which is a continuation of Turing's ideas. The connection lies in their attempts to define and realize machine intelligence; the difference is that the F-test provides an external, black-box evaluation standard, while the E-Conference's topics focus more on specific internal implementation mechanisms, such as symbolic reasoning and neural networks."

[0307] Deconstructing the facts of the standard answer into granular details:

[0308] The above standard answer was immediately sent to the fact particle deconstruction module.

[0309] LLM call: Invokes the model and provides strict JSON output instructions.

[0310] Destructuring result (JSON format): [

[0312] {"subject": "F test", "predicate": "definition method", "object": "determines whether the machine is intelligent from an operational perspective"}

[0313] {"subject": "F-test", "predicate": "core idea", "object": "intelligence equivalent to the ability to converse indistinguishable from humans"}

[0314] {"subject": "F-test", "predicate": "set for AI research", "object": "a grand but imaginable goal"}

[0315] {"subject": "E-meeting", "predicate": "shared belief", "object": "machines can simulate intelligence"}

[0316] {"subject": "Concepts of the E Conference", "predicate": "is a continuation of...", "object": "Turing ideas"}

[0317] {"subject": "F test and E meeting", "predicate": "association", "object": "both attempt to define and implement machine intelligence"}

[0318] {"subject": "F test", "predicate": "provide", "object": "external, black-box evaluation criteria"}

[0319] {"subject": "E-meeting", "predicate": "predicate", "object": "internal implementation mechanism"}

[0320] ].

[0321] This set of factual particles, along with the question, standard answer, and source node UUID, are saved together to form a complete evaluation sample.

[0322] When the evaluation agent intervenes in the third stage, the model evaluation agent, fine-tuned on 1000 manually labeled samples, receives the sample. It evaluates each dimension separately:

[0323] It rated the "quality of the question" as 5 points (clear and in-depth).

[0324] Compare the answer with the context (the original text of "F Test" and "E Meeting") and determine the "answer fidelity" as "high";

[0325] Check whether the answer addresses all parts of the question ("relevant" vs. "different"), and determine the "answer completeness" as "high";

[0326] If both context nodes are confirmed to be necessary to answer the question, the "context relevance" score is 5.

[0327] The final overall quality score is 4.8.

[0328] The third stage of the correction agent's decision-making: The correction agent receives a score report of 4.8, which is higher than the preset threshold of 4.0. Therefore, the correction agent determines that the sample is "qualified" and approves it directly into the final dataset without any correction.

[0329] Example of handling unqualified samples: Suppose another sample's evaluation report is "Answer fidelity: Low (illusion)". The correction agent will identify the problematic sentence, and then, based on the original context and question, instruct it to regenerate a more faithful answer. The corrected sample is then sent back to the evaluation agent for a second round of review. Only samples that pass the review will be ultimately adopted.

[0330] The manifestation of intelligent cost routing: Throughout the entire process, the routing strategy of the cost optimization execution module continues to play a role.

[0331] When the RAG system evaluates a creative persona like "high school student," it routes to model e.

[0332] When generating answers to simple single-hop questions, the route is routed to model f.

[0333] When executing a fact granular destructuring process that strictly adheres to the JSON format, routing is performed to model g.

[0334] When performing quality scoring tasks that require processing hundreds of times per second, routing is performed to a low-cost, locally deployed fine-tuning model h.

[0335] When performing vector embedding, routing is performed to a dedicated model i.

[0336] In the fourth phase, batch execution and efficiency are key: the system packages hundreds or thousands of such generation-validation tasks. Calls to external LLM APIs are initiated in parallel. For example, while waiting for model e to generate a complex problem, the system may have already completed dozens of vector embeddings and a dozen quality scoring tasks performed by lightweight models. The built-in tenacity library ensures that tasks automatically and intelligently retry when an API is temporarily unavailable or returns a rate-limiting error, guaranteeing the stable completion of large-scale build tasks.

[0337] Through close collaboration across the four stages described above, this embodiment successfully transformed a batch of unordered Markdown documents into a RAG evaluation dataset containing hundreds of high-quality, multi-type, and structured fact granules. The entire process was highly automated, and costs were effectively controlled.

[0338] Furthermore, considering ecosystem compatibility, all core modules of the system (such as the segmenter, extractor, and validator) define standard input / output interfaces and use a neutral data model (such as the Pydantic model) for data exchange. This allows the invention to be free from deep binding to any specific framework (such as LangChain or LlamaIndex). Users can easily: integrate privately deployed embedded models (such as through Ollama), replace them with the company's self-developed document segmentation algorithm, and integrate LLM services from different vendors.

[0339] Although this embodiment primarily focuses on text documents, its core architecture has excellent potential for multimodal expansion.

[0340] For example, node expansion: the metadata of a knowledge graph ChunkNode can be easily expanded to store identifiers or Uniform Resource Identifiers (URIs) that point to non-text data such as images, audio, and tables.

[0341] Feature extraction extension: It can integrate multimodal models to extract vector embeddings and text descriptions of images and store them in nodes.

[0342] Relationship building extension: The relationship builder can be extended to include new cross-modal algorithms capable of calculating text-image similarity and text-table association.

[0343] Query synthesis extension: It can generate multimodal questions that require a combination of text and image information to answer (e.g., "Based on the image below and related text description, what is the design concept of this product?").

[0344] This provides a clear technical path for future evaluations of more complex RAG systems that incorporate multimodal information.

[0345] exist Figure 2 The process comprises four stages. The first stage is the adaptive knowledge graph construction based on source documents. This stage transforms the input set of unstructured documents into a knowledge graph (KG) rich in semantic information and structured relationships through an intelligent processing pipeline. This graph is the foundation for all subsequent operations.

[0346] The second stage involves query synthesis that integrates user personas with factual particles. Building upon the constructed knowledge graph, this stage simulates different user perspectives to generate challenging query-answer pairs covering various types, including single-point facts, multi-point connections, comparative analysis, and even counterfactual reasoning. It also innovatively deconstructs standard answers into atomic "factual particles."

[0347] The third stage is content verification and refinement based on multi-agent collaboration. This stage introduces a closed-loop system consisting of intelligent agents with three roles: "generator," "judge," and "corrector." This system cross-validates and self-corrects the synthesized query-answer pairs to ensure the quality, accuracy, and reliability of the generated content. Furthermore, after correction, the results can be fed back into the knowledge graph for further optimization.

[0348] The fourth stage is high-efficiency batch execution and deployment. This stage utilizes a series of optimization strategies, including intelligent task routing, asynchronous parallel processing, and cost-aware model selection, to maximize processing efficiency and minimize computing resource costs while ensuring quality.

[0349] Meanwhile, an automated construction system is provided for the four stages. The document processing and knowledge graph construction module is responsible for executing the tasks of the first stage, including sub-modules such as intelligent document parsing, adaptive sharding, multi-dimensional feature extraction, multi-source relationship construction, and dynamic maintenance of the knowledge graph.

[0350] The query synthesis module is responsible for executing the tasks of the second stage, including sub-modules such as global topic analysis, dynamic persona generation, multi-strategy query synthesis (single-hop, multi-hop, comparison, negation, etc.), and factual granular deconstruction of standard answers.

[0351] Multi-agent verification module: Responsible for executing the third stage tasks, including generating agents, evaluating agents, and correcting agents, forming a closed-loop "generate-review-modify" quality control process.

[0352] Cost optimization execution module: responsible for executing the tasks in the fourth stage, including sub-modules such as intelligent batch processing, asynchronous scheduling, API rate control, and intelligent cost routing based on task complexity model services.

[0353] The various embodiments corresponding to the dataset determination method have been described in detail above. Based on this, the present invention also discloses a dataset determination device corresponding to the above method. Figure 3 This is a structural diagram of a dataset determination device provided in an embodiment of the present invention. Figure 3 As shown, the dataset identifies the following devices:

[0354] The acquisition module 11 is used to acquire the target document and perform segmentation processing on the target document according to the document length and content components to obtain multiple segment nodes;

[0355] The relation construction module 12 is used to extract features from multiple block nodes to obtain corresponding feature parameters, so as to form nodes of the knowledge graph with the corresponding block nodes, and to construct relations between multiple nodes according to the multi-source semantic relation strategy to obtain the knowledge graph.

[0356] Output module 13 is used to set user roles for the knowledge graph, generate target questions based on user roles and node query strategies, input the target questions into the target model, and output standard answers;

[0357] The decomposition processing module 14 is used to decompose the standard answer to obtain fact particles, and construct a quadruple data of target question, standard answer, block node and fact particles;

[0358] Evaluation processing module 15 is used to evaluate the quadruple data according to the dataset evaluation collaboration mechanism to determine the final dataset.

[0359] Since the embodiments of the device part correspond to the embodiments described above, please refer to the embodiments of the method part for the description of the device part, and will not be repeated here.

[0360] For an introduction to the dataset determination device provided by the present invention, please refer to the above method embodiments. The present invention will not be described in detail here, but it has the same beneficial effects as the above dataset determination method.

[0361] Figure 4 A structural diagram of a dataset determination device provided in an embodiment of the present invention is shown below. Figure 4 As shown, the device includes:

[0362] Memory 21 is used to store computer programs;

[0363] Processor 22 is used to implement the steps of a dataset determination method when executing a computer program.

[0364] The dataset determination device provided in this embodiment may include, but is not limited to, smartphones, tablets, laptops, or desktop computers.

[0365] The processor 22 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 22 may be implemented using at least one hardware form selected from Digital Signal Processors (DSPs), Field-Programmable Gate Arrays (FPGAs), and programmable logic arrays. The processor 22 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 22 may integrate a GPU, which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 22 may also include an AI processor, which handles computational operations related to machine learning.

[0366] The memory 21 may include one or more non-volatile storage media, which may be non-transitory. The memory 21 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 21 is used to store at least the following computer program 211, which, after being loaded and executed by the processor 22, is capable of implementing the relevant steps of the dataset determination method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 21 may also include an operating system 212 and data 213, etc., and the storage method may be temporary storage or permanent storage. The operating system 212 may include Windows, Unix, Linux, etc. The data 213 may include, but is not limited to, the data involved in the dataset determination method, etc.

[0367] In some embodiments, the dataset determination device may further include a display screen 23, an input / output interface 24, a communication interface 25, a power supply 26, and a communication bus 27.

[0368] Those skilled in the field can understand, Figure 4 The structure shown does not constitute a limitation on the dataset determination device and may include more or fewer components than illustrated.

[0369] The processor 22 implements the dataset determination method provided in any of the above embodiments by calling instructions stored in the memory 21.

[0370] For a description of the dataset determination device provided by the present invention, please refer to the above method embodiments. The present invention will not be described in detail here, but it has the same beneficial effects as the above dataset determination method.

[0371] Furthermore, the present invention also provides a non-volatile storage medium on which a computer program is stored, and when the computer program is executed by the processor 22, it implements the steps of the dataset determination method described above.

[0372] It is understood that if the methods in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in a non-volatile storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0373] For an introduction to the non-volatile storage medium provided by the present invention, please refer to the above method embodiments. The present invention will not be described in detail here, but it has the same beneficial effects as the above-described data set determination method.

[0374] Furthermore, the present invention also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of a dataset determination method.

[0375] For an introduction to the computer program product provided by the present invention, please refer to the above method embodiments. The present invention will not be described in detail here, but it has the same beneficial effects as the above-described dataset determination method.

[0376] The foregoing has provided a detailed description of a dataset determination method, apparatus, medium, and product provided by the present invention. The various embodiments in the specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make several improvements and modifications to the present invention without departing from the principles of the invention, and these improvements and modifications also fall within the protection scope of the present invention.

[0377] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.< / h2>

Claims

1. A data set determination method characterized by, The method comprises the following steps: acquiring a target document, and performing slicing processing on the target document according to a document length and a content component to obtain a plurality of block nodes; performing feature extraction on the plurality of block nodes to obtain corresponding feature parameters, to form nodes of a knowledge graph with the corresponding block nodes, and performing relationship construction on the plurality of nodes according to a multi-source semantic relationship strategy, to obtain the knowledge graph; setting a user role for the knowledge graph, to generate a target question according to the user role and a node query strategy, and inputting the target question into a target model to output a standard answer; performing decomposition processing on the standard answer to obtain a fact grain, and constructing four-tuple data of the target question, the standard answer, the block node and the fact grain; performing evaluation processing on the four-tuple data according to a data set evaluation cooperative mechanism, to determine a final data set; correspondingly, setting a user role for the knowledge graph comprises: performing global analysis processing on the knowledge graph to obtain a corresponding topic list; performing selection processing on a sub-topic in the topic list to determine a target sub-topic; generating a corresponding user role according to the target sub-topic; correspondingly, generating a target question according to the user role and a node query strategy comprises: determining a node type corresponding to a target node in the knowledge graph, wherein the node type is a single-hop node type and a multi-hop node type; when the target node is a single-hop node type, inputting the target node and the user role into a first pre-training model to obtain a first target question; when the target node is a multi-hop node type, determining a corresponding target relationship in the knowledge graph; inputting the target relationship, the target node and the user role into a second pre-training model to obtain a second target question; correspondingly, when a data set evaluation result obtained by performing evaluation processing on the four-tuple data according to a data set evaluation cooperative mechanism does not reach a preset threshold, the method further comprises: performing analysis processing on the data set evaluation result to determine a correction strategy; when the correction strategy is a question strategy, generating a new target question, inputting the new target question, context information and the target question into a target model to obtain a new standard answer; or, when the correction strategy is a block node strategy, determining new context information according to an embedding vector of the target question and the node query strategy; or, when the correction strategy is a four-tuple matching strategy, discarding the four-tuple data, acquiring a next target document, and entering the step of performing slicing processing on the target document according to a document length and a content component to obtain a plurality of block nodes, until a new four-tuple data is determined.

2. The data set determination method of claim 1, wherein, performing slicing processing on the target document according to a document length and a content component to obtain a plurality of block nodes comprises: performing analysis processing on the target document to obtain a text file, and retaining structure information corresponding to the content component; According to the document length, the text file is subjected to length classification processing to obtain a target length file, wherein the target length file is a first length file or a second length file, and the first length file has a page length greater than that of the second length file; When the target length file is the first length file, the first length file is subjected to segmentation processing according to structure information to obtain a corresponding block node; When the target length file is the second length file, the second length file is taken as a block node.

3. The data set determination method according to claim 2, characterized in that, The segmentation processing of the first length file according to structure information to obtain a corresponding block node comprises: starting a hierarchical segmentation mechanism; adding the structure information to the first length file to obtain a third length file; segmenting the third length file by the hierarchical segmentation mechanism to obtain a plurality of text blocks, and taking the text blocks as corresponding block nodes.

4. The data set determination method of claim 1, wherein, The feature extraction of a plurality of block nodes to obtain corresponding feature parameters comprises: starting a parallel feature extraction mechanism; wherein the parallel feature extraction mechanism at least comprises an abstract extraction mechanism, a theme extraction mechanism, a keyword extraction mechanism, a named entity recognition mechanism, a question generation mechanism and a vector embedding extraction mechanism; the feature extraction of a plurality of block nodes by the parallel feature extraction mechanism to obtain feature parameters.

5. The data set determination method of claim 4, wherein, According to a multi-source semantic relationship strategy, the relationship between a plurality of nodes is constructed to obtain a knowledge graph, comprising: processing the vector feature parameters of the block nodes in the plurality of nodes according to a vector similarity relationship strategy to obtain the vector relationship between the plurality of nodes; obtaining the linear reading order of the target document, and determining a hierarchical structure relationship strategy according to the linear reading order; processing the plurality of nodes according to the hierarchical structure relationship strategy to obtain the precedence relationship between the plurality of nodes; processing the named entities of the plurality of nodes according to an entity co-occurrence relationship strategy to determine the entity co-occurrence relationship; processing the plurality of nodes according to an inference relationship strategy to determine the corresponding causal relationship; constructing the knowledge graph of the plurality of nodes according to the vector relationship, the precedence relationship, the entity co-occurrence relationship and the causal relationship.

6. The data set determination method of claim 1, wherein, Before inputting the target relationship, the target node and the user role into the second pre-training model to obtain a second target question, further comprising: when the target relationship of the plurality of target nodes is a preset target relationship and the corresponding content is complementary, or when the target relationship of the plurality of target nodes is a preset target relationship and the corresponding content is different and shares the same sub-theme, entering the step of inputting the target relationship, the target node and the user role into the second pre-training model to obtain a second target question.

7. The data set determination method of claim 1, wherein, The decomposition processing of the standard answer to obtain a fact grain comprises: segmenting the standard answer to obtain corresponding word units; calling a third pre-training model, inputting the word units into the third pre-training model for decomposition processing to obtain corresponding fact grains; wherein the fact grains at least contain a subject, an attribute and a value.

8. The data set determination method according to claim 7, characterized in that, The evaluation processing of the four-tuple data according to a data set evaluation collaboration mechanism comprises: performing semantic analysis on the target question by using a natural language processing tool to determine question quality; calling a fourth pre-trained model, inputting the standard answer and context information corresponding to the block node into the fourth pre-trained model to determine semantic similarity between the standard answer and the block node; determining answer fidelity according to the semantic similarity; calling a fifth pre-trained model, inputting the standard answer and the target question into the fifth pre-trained model to determine semantic coverage of the standard answer and the target question; determining answer completeness according to the semantic coverage; determining relevance of context information of the block node, the target question and the standard answer according to the similarity mode; calling a sixth pre-trained model, inputting the fact granule into the sixth pre-trained model to determine independent integrity of the fact granule; determining a data set evaluation result based on respective weight parameters corresponding to question quality, answer fidelity, answer completeness, relevance of the standard answer and independent integrity of the fact granule.

9. The data set determination method of claim 1, wherein, The data set is composed of at least one four-tuple data, and the composition process of the data set comprises: obtaining a target model corresponding to a plurality of four-tuple data; if the target models are the same, merging requests for constructing a plurality of four-tuple data into the same batch for batch processing.

10. The data set determination method of claim 9, wherein, When the target models are not the same, the method further comprises: classifying a plurality of target models to determine target models corresponding to different classifications; determining a target classification with the largest number of target models in the same classification among different classifications of target models; in the target classification, merging requests for constructing four-tuple data corresponding to the target classification into a first batch; by analogy, until the request for the four-tuple data corresponding to the classification with the smallest number of target models in the same classification is merged into the Nth batch; According to the network bandwidth, the target models in different batches are balanced to generate four-tuple data in the corresponding batch.

11. The data set determination method of claim 1, wherein, The determination process of the target model comprises: when the target node is a single-hop node type, selecting a first performance model with the largest performance effect in a performance model set as the target model; when the target node is a multi-hop node type, selecting a first model with the largest comprehensive score in a model set as the target model.

12. The data set determination method of claim 1, wherein, After determining the final data set, further comprising: calling a pre-set model corresponding to a retrieval enhancement generation system, inputting the final data set into the pre-set model to output a current answer; analyzing the current answer and the fact granule of the four-tuple data corresponding to the data set to determine a target proportion of the evaluation index containing the fact granule; determining a model test result of the pre-set model according to the target proportions of a plurality of evaluation indexes.

13. The data set determination method according to any one of claims 2 to 5, characterized in that, When a new target document is added, the construction process of the knowledge graph comprises: constructing the knowledge graph of the target document as a first knowledge graph; When a new target document is added, go to the step of performing sharding processing on the target document according to the document length and the content component to obtain a plurality of block nodes until a new node is determined; According to the multi-source semantic relation strategy, a plurality of new nodes are subjected to relation construction to obtain a second knowledge graph; The first knowledge graph and the second knowledge graph are fused to obtain a final knowledge graph.

14. The data set determination method according to any one of claims 2 to 5, characterized in that, The method further comprises: According to a first preset period, checking isolated nodes in the plurality of nodes of the knowledge graph and time effectiveness of relations connected by the plurality of nodes; If the time effectiveness satisfies a preset condition, target nodes and isolated nodes of target relations of the time effectiveness satisfying the preset condition are divided into cold nodes; According to a second preset period, the cold nodes are reanalyzed to determine a new knowledge graph.

15. A data set determination apparatus characterized by comprising: Comprise: A memory for storing a computer program; A processor for executing the computer program to implement the steps of the data set determination method according to any one of claims 1 to 14.

16. A non-volatile storage medium, comprising: The non-volatile storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the data set determination method according to any one of claims 1 to 14.

17. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instruction is executed by the processor to implement the steps of the data set determination method according to any one of claims 1 to 14.

Images