Data set determination method and device, medium and product

Abstract

The invention discloses a data set determination method and device, a medium and a product, and relates to the technical field of data processing. Hierarchical segmentation of deep mining of a document structure is realized through fragmentation processing of document lengths and different content components by adopting a self-adaptive processing pipeline strategy. Various node relationships are fused, so that graph semantics are richer, and a dynamically maintained knowledge graph containing nodes and relationships is generated. User roles and diversified node query strategy generation are fused, and the capabilities of cross-text information integration, analysis and comparison and complex reasoning are improved. And decomposing the standard answer to obtain fact particles, so that the standard answer is converted from fuzzy to clear fact coverage in the subsequent data set evaluation process. According to the method, evaluation processing is carried out according to the data set evaluation dimension, closed-loop quality control mode evaluation of a multi-agent collaborative verification mechanism is formed, and the evaluation effect and reliability of the RAG system are improved while the data set quality is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, device, medium and product for determining a data set. Background Art

[0002] The Retrieval-Augmented Generation (RAG) system uses user queries to retrieve relevant contextual information from an external knowledge base before generating answers. This information is then fed into a pre-trained model along with the original query to generate reliable and accurate answers. Before evaluating a RAG system, a dataset must be constructed. However, this process can lead to poor dataset quality due to incorrect, ambiguous, or incomplete answers, single-hop evaluations where the evaluation dimension consists of only a single block of text, or shallow exploration of the knowledge base's inherent structure. This can further impact the effectiveness of the RAG evaluation system.

[0003] Therefore, how to improve the quality of constructed data sets to enhance the evaluation effect of the RAG evaluation system is an urgent problem that those skilled in the art need to solve. Summary of the Invention

[0004] The purpose of the present invention is to provide a data set determination method, device, medium and product to solve the problem that the quality angle difference in the data set determination process affects the evaluation effect of the RAG evaluation system.

[0005] To solve the above technical problems, the present invention provides a method for determining a data set, comprising: Obtaining a target document, and segmenting the target document into multiple segment nodes according to the document length and content components; Extract features from multiple block nodes to obtain corresponding feature parameters, which together with the corresponding block nodes constitute nodes of the knowledge graph, and construct relationships between multiple nodes according to the multi-source semantic relationship strategy to obtain a knowledge graph; Setting a user role for the knowledge graph to generate a target question based on the user role and the node query strategy, inputting the target question into the target model, and outputting a standard answer; Decomposing the standard answer to obtain fact particles, and constructing four-tuple data of the target question, standard answer, block node and fact particle; The quadruple data is evaluated and processed according to the data set evaluation coordination mechanism to determine the final data set.

[0006] On the one hand, the target document is segmented according to the document length and content components to obtain multiple segment nodes, including: Parsing the target document to obtain a text file, and retaining structural information corresponding to the content components; Performing length classification processing on the text file according to the document length to obtain a target length file, wherein the target length file is a first length file or a second length file, and the length of the first length file is greater than the length of the second length file; When the target length file is a first length file, segmenting the first length file according to the structure information to obtain corresponding block nodes; When the target length file is a second length file, the second length file is used as a block node.

[0007] On the other hand, the first length file is segmented according to the structure information to obtain corresponding block nodes, including: Start the hierarchical segmentation mechanism; Adding the structural information to the first length file to obtain a third length file; The third length file is segmented by a hierarchical segmentation mechanism to obtain a plurality of text blocks, and the text blocks are used as corresponding block nodes.

[0008] On the other hand, feature extraction is performed on multiple block nodes to obtain corresponding feature parameters, including: Initiating a parallel feature extraction mechanism; wherein the parallel feature extraction mechanism includes at least a summary extraction mechanism, a topic extraction mechanism, a keyword extraction mechanism, a named entity recognition mechanism, a question generation mechanism, and a vector embedding extraction mechanism; The parallel feature extraction mechanism is used to extract features from multiple block nodes to obtain feature parameters.

[0009] On the other hand, the relationship between multiple nodes is constructed according to the multi-source semantic relationship strategy to obtain a knowledge graph, including: Processing the vector feature parameters of the block nodes within the multiple nodes according to the vector similarity relationship strategy to obtain the vector relationship between the multiple nodes; Obtaining a linear reading order of a target document, and determining a hierarchical structure relationship strategy based on the linear reading order; Processing multiple nodes according to the hierarchical structure relationship strategy to obtain a sequence relationship between the multiple nodes; Processing named entities of multiple nodes according to entity co-occurrence relationship strategy to determine entity co-occurrence relationship; Process multiple nodes according to the inference relationship strategy to determine the corresponding causal relationship; A knowledge graph of multiple nodes is constructed based on the vector relationship, the sequential relationship, the entity co-occurrence relationship and the causal relationship.

[0010] On the other hand, setting user roles for the knowledge graph includes: Performing a global analysis on the knowledge graph to obtain a corresponding topic list; Determine the target sub-topic by selecting and processing the sub-topics in the topic list; Generate corresponding user roles based on the target sub-topic.

[0011] On the other hand, a target question is generated according to the user role and the node query strategy, including: Determining a node type corresponding to a target node in the knowledge graph, wherein the node type is a single-hop node type and a multi-hop node type; When the target node is a single-hop node type, calling a first pre-trained model, inputting the target node and the user role into the first pre-trained model to obtain a first target problem; When the target node is a multi-hop node type, the corresponding target relationship is determined in the knowledge graph; the second pre-trained model is called, and the target relationship, the target node and the user role are input into the second pre-trained model to obtain a second target problem.

[0012] On the other hand, before inputting the target relationship, the target node, and the user role into the second pre-training model to obtain the second target problem, the method further includes: When the target relationship of multiple target nodes is a preset target relationship and the corresponding contents are complementary, or when the target relationship of multiple target nodes is a preset target relationship and the corresponding contents are different and share the same subtopic, the step of inputting the target relationship, the target node and the user role into the second pre-trained model to obtain the second target problem is entered.

[0013] On the other hand, the standard answer is decomposed to obtain fact particles, including: Segmenting the standard answer to obtain corresponding word units; The third pre-trained model is called, and the word-unit is input into the third pre-trained model for decomposition processing to obtain corresponding fact particles; wherein the fact particles at least include a subject, an attribute, and a value.

[0014] On the other hand, the quadruple data is evaluated and processed according to the data set evaluation coordination mechanism, including: Using natural language processing tools to perform semantic analysis on the target question to determine the quality of the question; Calling a fourth pre-trained model, inputting context information corresponding to the standard answer and the block node into the fourth pre-trained model to determine the semantic similarity between the standard answer and the block node; determining answer fidelity based on the semantic similarity; Calling a fifth pre-trained model, inputting the standard answer and the target question into the fifth pre-trained model to determine the semantic coverage of the standard answer and the target question; determining answer completeness based on the semantic coverage; Determining the relevance of the context information of the block node, the target question, and the standard answer based on a similarity method; calling a sixth pre-trained model, and inputting the fact granules into the sixth pre-trained model to determine the independent integrity of the fact granules; The dataset evaluation results are determined based on the corresponding weight parameters of question quality, answer fidelity, answer completeness, relevance of standard answers, and independent integrity of fact particles.

[0015] On the other hand, when it is determined that the evaluation result of the data set does not reach the preset threshold, the method further includes: Analyzing and processing the evaluation results of the data set to determine a correction strategy; When the correction strategy is a question strategy, a new target question is regenerated, and the new target question, context information, and the target question are input into a target model to obtain a new standard answer; Alternatively, when the correction strategy is a block node strategy, new context information is determined according to the embedding vector of the target question and the node query strategy; Alternatively, when the correction strategy is a quadruple matching strategy, the quadruple data is discarded, and the next target document is obtained, and the step of segmenting the target document according to the document length and content components to obtain multiple block nodes is entered until the new quadruple data is determined.

[0016] On the other hand, the data set is composed of at least one quadruple data, and the process of constructing the data set includes: Obtain and construct target models corresponding to multiple quadruple data; If the target model is the same, the requests corresponding to constructing multiple quadruple data are merged into the same batch for batch processing.

[0017] On the other hand, when the target models are not the same, the method further includes: Classify multiple target models to determine target models corresponding to different classifications; Determine the target category with the largest number of target models under the same category among the target models of different categories; In the target classification, requests for constructing quad-tuple data corresponding to the target classification are merged into a first batch; This process is repeated until the requests for the four-tuple data corresponding to the category with the smallest number of target models under the same category are merged into the Nth batch; The target models under different batches are called evenly according to the network bandwidth to generate quadruple data under the corresponding batches.

[0018] On the other hand, the process of determining the target model includes: When the target node is a single-hop node, the first performance model with the greatest performance effect is selected from the performance model set as the target model; When the target node is a multi-hop node type, the first model with the largest comprehensive score in the model set is selected as the target model.

[0019] On the other hand, after the final dataset is determined, it also includes: Calling a preset model corresponding to the retrieval enhancement generation system, inputting the final data set into the preset model to output a current answer; Parsing the fact particles of the four-tuple data corresponding to the current answer and the data set to determine the target ratio of the fact particles corresponding to the evaluation index; The model test result of the preset model is determined according to the target ratio of multiple evaluation indicators.

[0020] On the other hand, when a new target document is added, the knowledge graph construction process includes: Using the knowledge graph constructed from the target document as the first knowledge graph; When a new target document is added, the process proceeds to the step of slicing the target document according to the document length and content components to obtain a plurality of segment nodes until a new node is determined; According to the multi-source semantic relationship strategy, the relationship between multiple new nodes is constructed to obtain the second knowledge graph; The first knowledge graph and the second knowledge graph are fused to obtain a final knowledge graph.

[0021] In another aspect, the method further comprises: Checking the timeliness of isolated nodes among the plurality of nodes of the knowledge graph and the relationships connected by the plurality of nodes according to a first preset period; If the timeliness meets the preset conditions, the target nodes and isolated nodes of the target relationship that meet the timeliness corresponding to the preset conditions are classified as cold nodes; The cold nodes are re-analyzed according to a second preset period to determine a new knowledge graph.

[0022] To solve the above technical problems, the present invention further provides a data set determination device, comprising: memory for storing computer programs; A processor is configured to implement the steps of the data set determination method when executing the computer program.

[0023] To solve the above technical problem, the present invention further provides a non-volatile storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the data set determination method as described above are implemented.

[0024] To solve the above technical problem, the present invention further provides a computer program product, comprising a computer program / instruction, which implements the steps of the data set determination method when executed by a processor.

[0025] The beneficial effects of the present invention are as follows: first, the target document is segmented according to the document length and content components to obtain multiple block nodes. Compared with the conventional brute force segmentation method, which results in the knowledge base document structure remaining on the surface, the segmentation process is performed by using the document length and different content components and an adaptive processing pipeline strategy, thereby realizing hierarchical segmentation for deep mining of the document structure. Second, feature extraction is performed on multiple block nodes to obtain corresponding feature parameters, which are used to form nodes of the knowledge graph with the corresponding block nodes. Nodes are constructed based on the block nodes and feature parameters to facilitate the subsequent weaving of discrete nodes into a knowledge graph. According to the multi-source semantic relationship strategy, the relationship between multiple nodes is constructed to construct a knowledge graph. The multi-source semantic relationship and the fusion of multiple node relationships make the graph semantics richer, so as to generate a knowledge graph containing nodes and relationships that is dynamically maintained. Third, user roles are set for the knowledge graph, and the generation of user roles and diversified node query strategies is integrated to avoid generating target questions in a single-hop query mode, so as to facilitate the diversification and comprehensiveness of subsequent evaluation dimensions, no longer limited to the basic "retrieval-reading" ability, and improve the ability of cross-text information integration, analysis and comparison, and complex reasoning. Fourth, input the target question into the called target model to obtain the standard answer. Decompose the standard answer to obtain fact particles, and decompose each standard answer into a set of atomic facts, so that the standard answer is transformed from vague to clear fact coverage in the subsequent data set evaluation process. Finally, construct the four-tuple data of the target question, standard answer, block node and fact particles. On the basis of the original triple consisting of question-answer-block node, the four-tuple data is evaluated and processed according to the data set evaluation collaborative mechanism to determine the final data set, add fact particles, and improve the accuracy of subsequent evaluation. In terms of evaluation processing, the evaluation is carried out according to the data set evaluation dimension to form a closed-loop quality control evaluation method of the multi-agent collaborative verification mechanism, which effectively eliminates low-quality samples, corrects factual errors and answer illusions, and fundamentally ensures that each sample included in the data set has been strictly tested. While improving the quality of the data set, it also improves the evaluation effect and reliability of the RAG system.

[0026] Secondly, in the fragmentation process, different segmentation strategies are selected according to the document features. For long documents with strong structure, hierarchical segmentation based on titles or semantics is adopted, and for short documents with weak structure, overall processing is performed. Compared with the conventional unified brute force segmentation, the segmentation strategy of this embodiment has multiple styles and improves accuracy. The first length file is segmented according to the structural information to obtain the corresponding block nodes, breaking the original brute force segmentation and adding structural information, so that the segmentation process lays a mechanism for the subsequent knowledge graph construction and improves the reference basis. The feature parameters determined by the feature extraction performed by the parallel feature extraction mechanism improve processing efficiency, shorten the task time, increase system throughput, support high concurrency requirements, and enhance system scalability. According to the multi-source semantic relationship strategy, the relationship between multiple nodes is constructed to obtain a knowledge graph, which realizes the integration of four relationship sources: vector semantics, text structure, entity co-occurrence and high-level reasoning, making the graph semantics richer.

[0027] In addition, the present invention also provides a data set determination device, a medium, and a product, which have the same beneficial effects as the above-mentioned data set determination method. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present invention, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0029] Figure 1 A flow chart of a method for determining a data set provided by an embodiment of the present invention; Figure 2 A schematic diagram of a RAG system evaluation data set provided by an embodiment of the present invention; Figure 3 A structural diagram of a data set determination device provided in an embodiment of the present invention; Figure 4 This is a structural diagram of a data set determination device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0031] The core of the present invention is to provide a data set determination method, device, medium and product to solve the problem that the quality angle difference in the data set determination process affects the evaluation effect of the RAG evaluation system.

[0032] In order to enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0033] Pre-trained models have some flaws during use. One is "knowledge truncation," where the model's knowledge is limited to the end of its training data and it cannot access the latest information. Another is "model hallucination," where the model tends to fabricate seemingly reasonable but actually incorrect information when answering questions outside of its knowledge or uncertainty. Scientifically, efficiently, and comprehensively evaluating the performance of RAG systems is crucial, and high-quality benchmark data is a prerequisite for effective evaluation. Conventional dataset determination methods include: 1. Manual construction method: This is a traditional and high-quality method, in which human annotators manually write questions based on large-scale corpora (such as Wikipedia), locate the source of answers, and provide standard answers.

[0034] Advantages: High-quality questions, accurate and reliable answers, and the ability to design challenging and complex questions.

[0035] Disadvantages: Extremely expensive, time-consuming, and difficult to scale. More importantly, this approach is difficult to quickly customize for private knowledge bases in specific verticals, resulting in a disconnect between general datasets and real-world application scenarios.

[0036] 2. Automated methods based on templates or simple rules: This type of method attempts to automatically generate "fill-in-the-blank" questions by extracting entities and relationships from text through predefined sentence templates or rules.

[0037] Advantages: high degree of automation and high speed.

[0038] Disadvantages: The generated question types are single, the language expression is rigid, lacks diversity and depth, cannot simulate the complex questioning intentions of real users, and the evaluation effect is limited.

[0039] 3. Preliminary automated generation method based on Large Language Model (LLM): This approach leverages the powerful language understanding and generation capabilities of LLM to generate question-answer pairs directly from a given chunk of text. For example, some open source frameworks provide basic "generating question-answer pairs from text" functionality.

[0040] Advantages: Compared with the template method, the generated questions are more natural and diverse. It can quickly generate a batch of evaluation data for a document.

[0041] Disadvantages and challenges of the above solutions: (1) Uncontrollable quality: The quality of directly generated question and answer pairs varies greatly, and may contain factual errors, ambiguous questions, or incomplete answers, lacking a rigorous verification process.

[0042] (2) Single evaluation dimension: Most of the generated questions are “single-hop” questions based on a single text block, which makes it difficult to evaluate the RAG system’s ability to integrate cross-text information, conduct comparative analysis, and conduct complex reasoning.

[0043] (3) Unstable evaluation methods: Evaluations often use semantic similarity-based models to compare generated answers with standard answers. This type of metric is sensitive to wording changes and cannot accurately measure the “factual correctness” of an answer. An answer may be semantically similar to the standard answer, but differ in key facts.

[0044] (4) High cost: In order to ensure the quality of generation and evaluation, it is usually necessary to continuously call the top-level LLM, which leads to high costs for building large-scale datasets and limits its application in budget-constrained scenarios.

[0045] (5) Static construction process: Existing methods are mostly one-time, linear processing flows, lacking in-depth exploration of the internal structure of the knowledge base, and also lacking a dynamic mechanism for using evaluation results as feedback to optimize the data generation process.

[0046] In summary, although some progress has been made in the automated construction of RAG evaluation datasets, core challenges such as quality, depth, evaluation paradigm, cost, and dynamism are still commonly faced.

[0047] The present invention provides a data set determination method that can solve the above technical problems.

[0048] Figure 1 A flow chart of a method for determining a data set provided by an embodiment of the present invention is shown as follows: Figure 1 As shown, the method includes: S11: Obtain a target document and segment the target document according to the document length and content components to obtain multiple segment nodes; S12: Extract features from multiple block nodes to obtain corresponding feature parameters, and use them together with the corresponding block nodes to form nodes of the knowledge graph. Then, build relationships between the multiple nodes according to the multi-source semantic relationship strategy to obtain a knowledge graph. S13: Set user roles for the knowledge graph to generate target questions based on the user roles and node query strategies, input the target questions into the target model, and output standard answers; S14: Decompose the standard answer to obtain fact particles, and construct the four-tuple data of target question, standard answer, block node and fact particle; S15: Evaluate and process the quadruple data according to the data set evaluation coordination mechanism to determine the final data set.

[0049] Specifically, the target document can be a source document such as Portable Document Format (PDF), Lightweight Markup Language (Markdown), Hypertext Markup Language (HTML), or Microsoft Office Open XML (DOCX). The target document is segmented based on document length and content components to produce multiple chunk nodes, performing preliminary parsing and cleaning of the document content using a knowledge base. The system does not employ a fixed brute-force chunking approach (e.g., a fixed number of tokens). Instead, it initiates a document statistical feature analysis submodule. This submodule calculates content component features such as the length distribution (number of tokens, number of paragraphs) and the degree of structure (proportion of heading hierarchy, lists, and tables) of the document collection. The purpose of the chunking process in step S11 is to assign different chunking strategies to the target document based on its length and internal structure, such as headings, lists, and tables, to produce multiple chunk nodes (ChunkNodes).

[0050] In step S12, feature extraction is performed on multiple block nodes to obtain corresponding feature parameters, which together with the corresponding block nodes constitute the nodes of the knowledge graph to be constructed subsequently. Feature extraction here can adopt a single feature method or multiple feature methods. In the case of multiple feature methods, a feature extraction pipeline composed of multiple parallel feature extraction devices (Extractors) is used, such as abstracts, topics, keywords, named entities, question generation, vector embedding, etc. The construction process here can be implemented using a mapping relationship.

[0051] Multi-source semantic relationships establish various types of connections between multiple nodes, weaving discrete nodes into a network-like knowledge graph. These relationships include vector similarity, hierarchical relationships, entity co-occurrence relationships, and inference relationships, enriching the semantic depth of the graph.

[0052] In step S13, user personas are assigned to the knowledge graph. To ensure that the generated target questions are more relevant to real-world application scenarios, the user persona automatic generation module performs a global analysis of the knowledge graph and then uses a model to generate a series of virtual user personas. Generation process: The LLM receives an instruction: Based on the following core topic [Topic List], please generate five different user personas for the knowledge base. Each persona should include: [Role Name], [Knowledge Background] (e.g., Novice, Expert, Interdisciplinary Researcher), [Core Interests], and [Typical Questioning Style] (e.g., Concise, Inquiring, Critical). For example, if the knowledge base is about "artificial intelligence," the generated personas might include: "A social activist concerned about bias in artificial intelligence (AI)," "A technical manager who hopes to implement AI standards in products," or "A philosophy student studying the long-term societal impact of AI." Target questions are generated based on the user personas and node query strategies. The node query strategies here include single-hop queries for a single node and multi-hop queries for nodes with specific semantic relationships, each generating its own target question.

[0053] Input the target question into the target model to output the standard answer. The standard answer is decomposed. The purpose of this decomposition is to establish a more stable and interpretable evaluation paradigm. Following strict instructions, it is decomposed into a series of independent, indivisible atomic fact units (Nuggets), i.e., fact granules. Instruction example: You are a fact extraction engine. Please decompose the following text into a set of independent, basic fact statements. Each fact statement should be a data exchange format (JavaScript Object Notation, JSON) object containing a "subject", "predicate", and "value (object)". If a sentence contains multiple facts, please extract them separately to ensure that each fact is atomic.

[0054] After the decomposition is completed, the four-tuple data of target question-standard answer-block node-fact particle is constructed. In step S15, the four-tuple data is evaluated and processed according to the data set evaluation collaborative mechanism. The collaborative mechanism here is question quality, answer fidelity, answer completeness, contextual relevance between block nodes, fact particle quality, etc. The judgment agent is used in these aspects to output a structured diagnostic report (including scores and classifications of each dimension) and an overall quality score for each four-tuple data, so as to form the final data set with multiple four-tuple data.

[0055] The beneficial effects of the embodiments of the present invention are as follows: first, the target document is segmented according to document length and content components to obtain multiple block nodes. Compared with the conventional brute force segmentation method, which results in the knowledge base document structure remaining superficial, the segmentation process using document length and different content components and an adaptive processing pipeline strategy achieves hierarchical segmentation for deep mining of the document structure. Second, feature extraction is performed on multiple block nodes to obtain corresponding feature parameters, which are used to form nodes of the knowledge graph with the corresponding block nodes. Nodes are constructed based on the block nodes and feature parameters to facilitate the subsequent weaving of discrete nodes into a knowledge graph. A knowledge graph is constructed based on the relationship between multiple nodes according to a multi-source semantic relationship strategy. Multi-source semantic relationships and the integration of multiple node relationships enrich the graph semantics to generate a dynamically maintained knowledge graph containing nodes and relationships. Third, user roles are set for the knowledge graph, and the generation of user roles and diversified node query strategies is integrated to avoid generating target questions only in a single-hop query mode, facilitating the subsequent diversified and comprehensive evaluation dimensions, no longer limited to the basic "retrieval-reading" capabilities, and improving the ability to integrate cross-text information, analyze and compare, and conduct complex reasoning. Fourth, the target question is input into the called target model to obtain the standard answer. The standard answer is decomposed to obtain fact particles, and each standard answer is decomposed into a set of atomic facts, so that the standard answer is transformed from vague to clear fact coverage in the subsequent data set evaluation process. Finally, the four-tuple data of the target question, standard answer, block node and fact particle is constructed. On the basis of the original triple consisting of question-answer-block node, the four-tuple data is evaluated and processed according to the data set evaluation collaborative mechanism to determine the final data set, add fact particles, and improve the accuracy of subsequent evaluation. In terms of evaluation processing, the evaluation is carried out according to the data set evaluation dimension to form a closed-loop quality control evaluation method of the multi-agent collaborative verification mechanism, which effectively eliminates low-quality samples, corrects factual errors and answer illusions, and fundamentally ensures that each sample included in the data set has been strictly tested. While improving the quality of the data set, it also improves the evaluation effect and reliability of the RAG system.

[0056] In some embodiments, the target document is segmented according to the document length and content components to obtain multiple segment nodes, including: Parse the target document to obtain a text file, and retain the structural information corresponding to the content components; Performing length classification processing on the text file according to the document length to obtain a target length file, wherein the target length file is a first length file or a second length file, and the length of the first length file is greater than the length of the second length file; When the target length file is a first length file, the first length file is segmented according to the structure information to obtain corresponding block nodes; When the target length file is the second length file, the second length file is used as a block node.

[0057] Specifically, the target document is parsed to produce a text file. For example, the system receives as input a folder containing dozens of Markdown documents on the "History of Artificial Intelligence Development." The document processing and knowledge graph construction module is activated, first calling the "unstructured.io" library to parse the Markdown document into plain text while preserving the title structure information, that is, the structural information corresponding to the content components.

[0058] Text files are categorized by length based on their length to create target-length files. These files may be long or medium-short. These files can be assigned using a word count threshold: documents exceeding the first word count threshold are classified as long (first-length files), while documents below the first word count threshold are classified as medium-short (second-length files). When the target-length file is the first-length file, the first-length file is segmented based on the structural information to create block nodes. This segmentation process takes into account the fact that documents are generally long and have a clear title hierarchy. The structural information corresponding to the title needs to be incorporated into the segmentation to create the block nodes.

[0059] When the target length file is the second length file, deep segmentation is skipped and the entire document is directly processed as a block node to avoid the fragmentation of context information caused by excessive segmentation. Each node is designed as a flexible data structure based on the data model, which contains a universally unique identifier (UUID), node type ("DocumentNode", "ChunkNode"), original content, and a dynamically expandable "metadata" dictionary for storing various features and source information (such as source file name and page number) extracted subsequently.

[0060] The segmentation process provided in this embodiment selects different segmentation strategies according to document features. For long documents with strong structure, hierarchical segmentation based on titles or semantics is adopted, and for short documents with weak structure, overall processing is performed. Compared with the conventional unified brute force segmentation, the segmentation strategy of this embodiment has multiple styles and improves accuracy.

[0061] In some embodiments, segmenting the first length file according to the structure information to obtain corresponding block nodes includes: Start the hierarchical segmentation mechanism; Adding structural information to the first length file to obtain a third length file; The third length file is segmented by a hierarchical segmentation mechanism to obtain a plurality of text blocks, and the text blocks are used as corresponding block nodes.

[0062] Specifically, in terms of segmentation processing, a hierarchical segmentation mechanism is enabled, preferably a hierarchical segmenter based on titles or semantics. For example, the "RecursiveCharacterTextSplitter" in "LangChain" can be used as a reference, but with the addition of support for Markdown titles or HTML tags (" <h1>”，"< / h1> <h2>"). This splits a long document into multiple logically coherent and semantically complete chunks of text (Chunks). The original document itself acts as a high-level "DocumentNode", and the segmented chunks act as "ChunkNodes" and are connected to it via the "CONTAINS" relationship. This approach preserves the inherent structure of the document and is crucial for subsequent context-aware retrieval.

[0063] That is, the structural information is added to the first-length file to obtain a third-length file, and the third-length file is segmented through a hierarchical segmentation mechanism to obtain multiple text blocks as corresponding block nodes.

[0064] The system selects the "Recursive Character Text Splitter" and configures it to use Markdown headings as primary delimiters, with a reasonable chunk size (e.g., 1024 tokens) as a secondary delimiter. For example, a chapter on "Neural Networks" is split into multiple logically coherent "ChunkNodes," such as "Perceptron," "Backpropagation Algorithm," and "Convolutional Neural Network." Each "ChunkNode" is instantiated using the data model, assigned a UUID, and records the source file name and title. All "ChunkNodes" originating from the same document establish a "CONTAINS" relationship with the "DocumentNode" representing that document.

[0065] This embodiment provides a method for segmenting the first-length file according to the structural information to obtain corresponding block nodes, breaking the original brute force segmentation and adding structural information, so that the segmentation process lays a mechanism for subsequent knowledge graph construction and improves the reference basis.

[0066] In some embodiments, feature extraction is performed on multiple block nodes to obtain corresponding feature parameters, including: Initiate a parallel feature extraction mechanism; wherein the parallel feature extraction mechanism includes at least a summary extraction mechanism, a topic extraction mechanism, a keyword extraction mechanism, a named entity recognition mechanism, a question generation mechanism, and a vector embedding extraction mechanism; The feature parameters are obtained by extracting features from multiple block nodes through a parallel feature extraction mechanism.

[0067] Specifically, the parallel feature extraction mechanism is driven by a pre-trained model (such as a language model). It works by submitting text content with specific instructions (Prompt) to the model and using asynchronous input and output (IO) tools to achieve large-scale parallel processing.

[0068] The parallel feature extraction mechanism here includes at least summary extraction mechanism, topic extraction mechanism, keyword extraction mechanism, named entity recognition mechanism, question generation mechanism and vector embedding extraction mechanism, which are respectively expanded as follows: Summary Extractor: Generates a concise summary for each ChunkNode and stores it in metadata['summary']. The summary is not only used for quick comprehension but also crucial for subsequently calculating semantic relationships across summaries. For example, the Convolutional Neural Network (CNN) is a deep learning model particularly well-suited for processing image data. It mimics the human visual cortex through convolutional, pooling, and fully connected layers, achieving significant success in image recognition and object detection. This summary is stored in metadata['summary']].

[0069] Themes Extractor: Extracts 1-3 core themes or topics to provide material for subsequent topic-related queries. Entity Identification: [Convolutional Neural Networks, CNNs, Deep Learning, Image Recognition, Object Detection].

[0070] Keyphrases Extractor: Extracts key phrases from text for use in building more accurate sparse vector indexes (such as text retrieval algorithms (Best Matching 25, BM25)) or entity linking for knowledge graphs. Generate potential questions: What are the core components of convolutional neural networks?

[0071] Named Entity Recognition (NERExtractor): Identifies and classifies entities such as names of people, places, organizations, products, and dates in text. These entities are important sources for constructing fact-based questions.

[0072] Question Extractor: Generates several potential questions for each text block. These questions can serve as simple evaluation samples or as "seeds" for synthesizing subsequent complex questions.

[0073] Embedding Extractor: This mechanism uses a specialized embedding model to generate high-dimensional vector representations for the original text and summary of each node. These representations are stored in the node's "embedding" and "summary_embedding" attributes, which serve as the basis for calculating semantic similarity. High-dimensional vectors are generated for the original text and summary of a ChunkNode and stored in the "embedding" and "summary_embedding" fields, respectively. All of these extraction tasks are executed concurrently using asyncio.gather, significantly reducing processing time.

[0074] According to the above feature extraction mechanism, feature extraction is performed on multiple block nodes to obtain corresponding feature parameters. The feature extraction process here is the same as the conventional method, or it can be different, which is not limited here.

[0075] The feature parameters determined by feature extraction using a parallel feature extraction mechanism provided in this embodiment improve processing efficiency, shorten task time, increase system throughput, support high concurrency requirements, and enhance system scalability.

[0076] In some embodiments, building relationships among multiple nodes according to a multi-source semantic relationship strategy to obtain a knowledge graph includes: Processing the vector feature parameters of the block nodes within the multiple nodes according to the vector similarity relationship strategy to obtain the vector relationship between the multiple nodes; Obtaining the linear reading order of the target document and determining the hierarchical structure relationship strategy according to the linear reading order; Process multiple nodes according to the hierarchical relationship strategy to obtain the order relationship between the multiple nodes; Processing named entities of multiple nodes according to entity co-occurrence relationship strategy to determine entity co-occurrence relationship; Process multiple nodes according to the inference relationship strategy to determine the corresponding causal relationship; Construct a knowledge graph with multiple nodes based on vector relationships, sequential relationships, entity co-occurrence relationships, and causal relationships.

[0077] Specifically, the vector similarity relationship performs batch matrix multiplication (calculating cosine similarity) on the embedding vectors of all block nodes, screening out node pairs with similarity above a preset threshold (for example, 0.85, which can be fine-tuned using small sample data) and establishing a "SIMILAR_TO" relationship between them. The relationship itself can carry a similarity score as an attribute. For long documents, this builder prioritizes using the embedding vector of their summary to capture more macroscopic similarity in the main idea. For example, if the vector similarity between the summary of the "Convolutional Neural Network" node and the summary of the "Recurrent Neural Network" node is 0.86 (higher than the threshold of 0.85), a "SIMILAR_TO (score=0.86)" relationship is established between the two.

[0078] Based on the linear reading order of the target document, a hierarchical relationship strategy is determined to achieve a precedence order between multiple nodes. A "CONTAINS" relationship is automatically established from a "DocumentNode" to its "ChunkNode", as well as a "precedes / after" relationship between "ChunkNodes", preserving the linear reading order of the document.

[0079] Based on the entity co-occurrence strategy, named entities across multiple nodes are processed to determine entity co-occurrence relationships, analyzing key named entities that co-occur across nodes. If two nodes share a sufficient number of key entities (for example, > 2 shared entities), an "ENTITY_CO_OCCURRENCE" relationship is established between them. This is very effective for discovering topically related content with significant differences in textual representation. For example, if entities such as "John McCarthy" and "Marvin Minsky" are found to frequently appear in both the "F Test" node and the "E Conference" node, an "ENTITY_CO_OCCURRENCE" relationship is established between them.

[0080] Based on the inference relationship strategy, multiple nodes are processed to determine corresponding causal relationships. For key nodes, the model can be used to infer higher-level relationships. For example, given the content of two nodes, the LLM can be asked whether there is a "cause-effect," "contrast," or "example-of" relationship between them. This greatly enriches the semantic depth of the graph. For example, for the two key nodes "backpropagation algorithm" and "the rise of deep learning," the system can use the model to perform high-level relationship reasoning, asking: "Node A1 describes the backpropagation algorithm, and node B1 describes the rise of deep learning. What is the most likely relationship between A1 and B1?" The model may answer "cause-effect," and the system can then establish a "cause-effect" relationship based on this.

[0081] Based on the above different relationships, a knowledge graph between multiple nodes is constructed to provide important reference basis.

[0082] This embodiment provides a method for constructing relationships between multiple nodes based on a multi-source semantic relationship strategy to obtain a knowledge graph, which integrates four sources of relationships: vector semantics, text structure, entity co-occurrence, and advanced reasoning, making the graph semantics richer.

[0083] In some embodiments, setting a user role for a knowledge graph includes: Perform global analysis on the knowledge graph to obtain the corresponding topic list; Determine the target sub-topic by selecting and processing the sub-topics in the topic list; Generate corresponding user roles based on the target sub-topic.

[0084] Specifically, a global analysis of the knowledge graph is performed to generate a list of corresponding topics, such as "Symbolic AI" and "Connectionist AI." The target subtopics, or core topics, are then selected and processed based on the sub-graphs within the topic list. Based on these core topics, corresponding user personas are generated, such as by instructing a model to generate a persona. Possible outputs include: "a high school student curious about the history of AI," "a computer science major seeking insights into the evolution of technology," and "a humanities and social science scholar interested in the philosophical debates surrounding AI technology."

[0085] To make the generated evaluation questions more relevant to real-world application scenarios, the user persona automatic generation module first performs a global analysis of the knowledge graph (such as counting topics and entity frequencies), and then calls a highly creative top-level LLM to generate a series of virtual user roles.

[0086] Generation process: The LLM receives instructions such as "Generating five different user personas for the knowledge base based on the following core topic [Topic List]. Each persona should include: [Role Name], [Knowledge Background] (e.g., Novice, Expert, Interdisciplinary Researcher), [Core Interests], and [Typical Questioning Style] (e.g., Concise, Inquiring, Critical).

[0087] For example, if the knowledge base is about "issues related to artificial intelligence," the generated personas might be: "a social activist concerned about AI bias," "a technical manager who wants to implement guidelines in products," or "a philosophy student studying the long-term societal impact of AI."

[0088] The user roles provided in this embodiment often have diverse goals and skill backgrounds, and the needs of different roles vary significantly. Setting roles can achieve customized matching of functions and permissions, reduce invalid operations, reduce cognitive load, and significantly improve usage efficiency and experience.

[0089] In some embodiments, generating a target question based on the user role and the node query strategy includes: Determine the node type corresponding to the target node in the knowledge graph, where the node type is a single-hop node type and a multi-hop node type; When the target node is a single-hop node type, calling the first pre-trained model, inputting the target node and the user role into the first pre-trained model to obtain a first target problem; When the target node is a multi-hop node type, the corresponding target relationship is determined in the knowledge graph; the second pre-trained model is called, and the target relationship, target node and user role are input into the second pre-trained model to obtain the second target problem.

[0090] Within these knowledge graphs, determine the single-hop and multi-hop node types corresponding to the target node. A single-hop query involves randomly selecting a high-information-density block node, combining this target node with a random persona, and then invoking the first pre-trained model. This model, along with the user persona, is then fed into the model to generate the first target question. Submit instructions to the LLM: "As the [persona], please ask a question relevant to your interests based on the following text." This question should align with your questioning style (e.g., concise, in-depth, critical, and practically application-oriented) and focus on the core topic of the text. Objective: To evaluate the RAG system's basic "retrieval-generation" capabilities for single, isolated knowledge points.

[0091] Multi-hop query searches for node combinations with specific relationship paths in the knowledge graph. For example, look for two nodes A2 and B2 that are connected by the "SIMILAR_TO" relationship but have complementary content. Submit instructions to the LLM: Play the role of [personality], combine the information of the following two contexts [context A2] and [context B2], and ask a question that requires reference to both to fully answer. The answer needs to clearly integrate information from both sides. Goal: To evaluate the system's ability to perform complex reasoning and information integration. The generated standard answer will clearly mark the source of the information ("source_nodes": ["uuid_A2", "uuid_B2"]), providing accurate traceability for subsequent evaluation.

[0092] Taking multi-hop query as an example, the target relationship is determined, and the target relationship, target node and user role are input into the second pre-training model to obtain the second target question.

[0093] This embodiment provides strategies for both single-hop and multi-hop node types. By inputting parameters specific to each strategy into the corresponding pre-trained model, target questions are generated. Compared to conventional question generation, which often involves simple single-point fact retrieval, this approach cannot effectively evaluate the system's complex reasoning capabilities. This embodiment, combined with dynamically generated user personas, can systematically generate questions covering both single-hop and multi-hop types, enabling in-depth consideration of the RAG system's performance on advanced cognitive tasks such as information integration and logical reasoning. This provides a comprehensive and challenging evaluation.

[0094] In some embodiments, before inputting the target relationship, the target node, and the user role into the second pre-trained model to obtain the second target question, the method further includes: When the target relationship of multiple target nodes is a preset target relationship and the corresponding contents are complementary, or when the target relationship of multiple target nodes is a preset target relationship and the corresponding contents are different and share the same subtopic, the step of inputting the target relationship, target node and user role into the second pre-trained model to obtain the second target problem is entered.

[0095] Specifically, to further enhance the challenge and comprehensiveness of the evaluation, the system also supports generating complex queries, such as comparative queries, which search for two nodes in the graph that have a "CONTRAST" relationship or share the same topic but differ in content (e.g., descriptions of two different technical solutions). LLM is instructed to generate questions comparing their strengths and weaknesses, similarities and differences. For example, compare and summarize the core differences and applicable scenarios between the "attention mechanism" described in node A3 and the "recurrent neural network" described in node B3 when processing sequence data. This refers to scenarios where the target relationships of multiple target nodes are pre-defined and the corresponding content is complementary.

[0096] Negative / Counterfactual Query: Process: Randomly select a node and generate a question whose answer does not exist in the entire current knowledge base. This is crucial for evaluating whether the RAG system "knows what it knows and doesn't know what it doesn't know," rather than fabricating answers (illusions). For example, please describe a detailed case study on "the application of quantum computers in marketing" in this knowledge base (assuming this content does not exist in the knowledge base). The standard answer should be: Based on existing information, the knowledge base does not contain a specific case study on the application of quantum computers in marketing. This refers to a scenario where the target relationships of multiple target nodes are pre-defined, correspond to different content, and share the same sub-topic.

[0097] It should be noted that the above two scenarios are both implemented based on multi-hop node type queries to achieve more complex queries.

[0098] The multi-hop query strategy of comparative and negative queries provided in this embodiment improves the depth and breadth of evaluation and can comprehensively measure the comprehensive capabilities of the RAG system.

[0099] In some embodiments, decomposing the standard answer to obtain fact particles includes: Segment the standard answer to obtain the corresponding word units; The third pre-trained model is called, and the word unit is input into the third pre-trained model for decomposition processing to obtain corresponding fact particles; wherein the fact particles at least include a subject, an attribute, and a value.

[0100] Specifically, the standard answer is decomposed and segmented to obtain the corresponding word units, and the third pre-trained model is called to decompose the word units into a series of independent fact particles. The fact particles here at least contain a subject, an attribute, and a value.

[0101] This module calls an LLM that is fine-tuned by instructions and is good at structured output. It receives the standard answer text and follows strict instructions to break it down into a series of independent and indivisible atomic fact units (Nuggets).

[0102] Example: You are a fact extraction engine. Please break the following text into a set of independent, basic fact statements. Each fact statement should be a JSON object containing a subject, a predicate, and a value. If a sentence contains multiple facts, extract them separately. Ensure that each fact is atomic.

[0103] Example: Model answer: A4 is a multinational technology company headquartered at B4. It was founded by C and D on April 1, 1976.

[0104] Deconstructed fact granules (JSON format): [ {"subject":"A4 Company","predicate":"type","object":"Multinational Technology Company"}, {"subject":"A4 Company","predicate":"Headquarters Location","object":"B4"}, {"subject":"A4 Company","predicate":"Founder","object":"C"}, {"subject":"A4 Company","predicate":"Founder","object":"C"}, {"subject":"A4 Company","predicate":"Founder","object":"D"}, {"subject":"A4 Company","predicate":"Date of Establishment","object":"April 1, 1976"} ].

[0105] This embodiment further processes the standard answers to generate fact granules. Traditional text similarity assessment methods, such as the Recall-Oriented Understudy for Gisting Evaluation (ROUGE), are susceptible to wording variations and cannot accurately measure "factual correctness." This innovative "fact granulation deconstruction of standard answers" technology is introduced into the automated process, breaking each standard answer down into a set of atomic facts. When evaluating a dataset, the assessment of answer quality shifts from fuzzy semantic similarity to clear factual coverage, making the dataset more stable and resistant to interference.

[0106] In some embodiments, the quadruple data is evaluated and processed according to the data set evaluation coordination mechanism, including: Use natural language processing tools to perform semantic analysis on target questions to determine the quality of the questions; Calling the fourth pre-trained model, inputting the context information corresponding to the standard answer and the block node into the fourth pre-trained model to determine the semantic similarity between the standard answer and the block node; Determine answer fidelity based on semantic similarity; Calling the fifth pre-trained model, inputting the standard answer and the target question into the fifth pre-trained model to determine the semantic coverage of the standard answer and the target question; Determine the completeness of the answer based on semantic coverage; Determine the relevance of the context information of the block nodes, the target question, and the standard answer based on the similarity method; Calling the sixth pre-trained model and inputting the fact particles into the sixth pre-trained model to determine the independence and integrity of the fact particles; The dataset evaluation results are determined based on the corresponding weight parameters of question quality, answer fidelity, answer completeness, relevance of standard answers, and independent integrity of fact particles.

[0107] Specifically, we set up corresponding evaluation processing strategies based on different evaluation dimensions. During this process, the generation of quadruple data is performed by a generation agent. This stage utilizes a judgment agent, whose core is a lightweight, fine-tuned scoring model. By fine-tuning on a small number (hundreds to a thousand) of high-quality, manually annotated samples, we enable a small to medium-sized model to accurately score and classify multiple dimensions of generated content.

[0108] It mainly includes question quality, answer fidelity, answer completeness, contextual information of block nodes, relevance between target questions and standard answers (context relevance), and independent integrity of fact particles (fact particle quality).

[0109] Question Quality: Is the question clear and unambiguous? Is it too broad or too specific? Is it of appropriate difficulty? (Scale 1-5) Answer Faithfulness: Is every statement in the answer justified in the given context? Is there any fabricated (illusionary) content? (Classification: high, medium, low) Answer Completeness: Does the answer fully address all aspects of the question? Does it omit any key contextual information? (Classification: High, Medium, Low) Context Relevance: Is the context provided necessary to answer the question? Is there a lot of redundant or completely irrelevant context? (Score 1-5) Fact Nugget Quality: Are the fact granules generated by deconstructing the standard answer semantically complete, with clear subject-verb-object structure and consistent granularity? Are there any deconstruction errors, missing information, or redundant repetitions? (Score 1-5).

[0110] Based on the above different dimensions, the respective weight parameters are set to obtain comprehensive data set evaluation results.

[0111] Regarding the above-mentioned calling of different pre-training models, different pre-training models can be set according to actual conditions, and can be selected based on the characteristics of the evaluation dimension, or the same pre-training model can be used. There is no limitation here.

[0112] The data set evaluation collaborative mechanism provided in this embodiment evaluates and processes the quadruple data, and provides evaluation reports in various dimensions to facilitate subsequent implementation of improvement strategies.

[0113] In some embodiments, when it is determined that the dataset evaluation result does not reach a preset threshold, the method further includes: Analyze and process the evaluation results of the data set to determine the correction strategy; When the correction strategy is changed to the question strategy, a new target question is regenerated, and the new target question, context information, and target question are input into the target model to obtain a new standard answer; Alternatively, when the correction strategy is a block node strategy, new context information is determined based on the embedding vector of the target question and the node query strategy; Alternatively, when the correction strategy is a quadruple matching strategy, the quadruple data is discarded, and the next target document is obtained, and the step of slicing the target document according to the document length and content components to obtain multiple block nodes is entered until the new quadruple data is determined.

[0114] Specifically, the correction agent receives the diagnostic report from the evaluation agent. For samples that fail (for example, if the overall score is below the preset threshold of 4.0), it will adopt different strategies based on the specific "cause": When the correction strategy is a question strategy, if the question is unclear (for example, if there is an ambiguous pronoun), it will try to rewrite the question. If the answer is slightly illusory or information is missing, it will instruct the LLM to regenerate a more faithful and complete answer based on the context and the original question.

[0115] When the correction strategy is the chunk node strategy, if the diagnosis is "irrelevant context", the correction agent can execute a query, use the embedding vector of the question to find a more relevant "ChunkNode" in the knowledge graph as the new context, and then regenerate the answer.

[0116] When the correction strategy is a four-tuple matching strategy, if the question does not match the context at all, or the quality is still low after multiple rounds of correction, it will directly discard the sample and optionally record this failure case for analyzing potential problems in the knowledge graph or generation strategy.

[0117] All corrected samples are submitted to the judging agent for another round of review, forming an iterative cycle of "generation-judgment-correction." Only samples that pass the quality threshold are accepted into the final dataset.

[0118] This embodiment provides a quality control system designed to address the quality challenges of automatically generated content, mimicking peer review. This system consists of three intelligent agents with different roles (generation, evaluation, and revision), forming a dynamic, iterative closed loop for quality improvement. Rather than a simple "post-generation filtering" approach, it incorporates a self-improvement mechanism that includes "automatic revision" and "iterative review."

[0119] In some embodiments, the data set is composed of at least one quadruple data, and the process of constructing the data set includes: Obtain and construct target models corresponding to multiple quadruple data; If the target model is the same, the requests corresponding to constructing multiple quadruple data will be merged into the same batch for batch processing.

[0120] Specifically, in the above embodiment, the corresponding dataset construction process in steps S11 to S15 is only for one record. The dataset is composed of multiple records, so there are also massive pre-training model calls in the entire dataset construction process. Mature open source tools and strategies will be used to manage these tasks efficiently and robustly.

[0121] Smart Batching: Multiple requests sent to the same model application programming interface (API) are consolidated into a single batch, significantly reducing network overhead and the number of API calls. In other words, if multiple quad-tuples of data correspond to the same target model, these requests are consolidated into a single batch for processing.

[0122] When the target models provided in this embodiment are of the same type, multiple requests are merged in constructing the data set and processed in the same batch, thereby reducing network overhead and the number of API calls.

[0123] In some embodiments, when the target models are not the same, the method further includes: Classify multiple target models to determine target models corresponding to different classifications; Determine the target category with the largest number of target models under the same category among the target models of different categories; In the target classification, the requests for constructing the four-tuple data corresponding to the target classification are merged into the first batch; This process is repeated until the requests for the four-tuple data corresponding to the category with the smallest number of target models under the same category are merged into the Nth batch; The target models under different batches are called evenly according to the network bandwidth to generate quadruple data under the corresponding batches.

[0124] Specifically, when the target models are not the same, multiple target models need to be classified and processed to determine the target category with the largest number of target models under the same category, and merge them into the first batch. Similarly, the requests for quadruple data corresponding to the category with the smallest number of target models under the same category are merged into the Nth batch, and the multiple batches are called evenly according to the network bandwidth to generate quadruple data under each batch.

[0125] A built-in rate controller for external APIs (such as OpenAI, Anthropic, and Google AI Platform) is available, which can be implemented using libraries such as tenacity. When a rate limit error (HTTP 429) is detected, the request frequency is automatically reduced and retries are performed using an exponential backoff algorithm to ensure the stability and success rate of long-running tasks.

[0126] The scheduling process of target models under different batches provided in this embodiment adopts asynchronous parallel scheduling, simultaneously initiating and managing hundreds or even thousands of independent model requests, maximizing the utilization of computing resources and network bandwidth, and significantly shortening the end-to-end build time.

[0127] In some embodiments, the process of determining the target model includes: When the target node is a single-hop node, the first performance model with the greatest performance effect in the performance model set is selected as the target model; When the target node is a multi-hop node type, the first model with the largest comprehensive score in the model set is selected as the target model.

[0128] The choice of model among different models also depends on the target node type. For single-hop nodes, the best performing model in the performance model set should be selected as the target model. This is because it requires strong reading comprehension and content generation skills, but less demanding on top-level creativity.

[0129] In the case of multi-hop nodes, the first model with the highest comprehensive score in the model set is selected as the target model. The reason is that strong reasoning, creativity, and the ability to follow complex instructions are required to ensure the novelty and depth of the generated content.

[0130] In addition, when the task description is fact granular deconstruction, named entity recognition, or keyword extraction, the recommended model level is: Instruction Fine-tuning Model. Reason: These models excel in this area, as they require high-level capabilities for conforming to structured output formats such as JSON.

[0131] In the task description, the evaluation agent performs multi-dimensional quality scoring. The recommended model level is a lightweight fine-tuned model. Reasoning: For a well-defined task and a fixed domain, a fine-tuned lightweight model can perform comparable to a large model on a specific task, but at a fraction of the cost.

[0132] When the task is to generate vector representations of text, the recommended model level is: Dedicated Embedding Model. Reason: It is optimized for embedding tasks, has good results, and is extremely low cost.

[0133] Through this differentiated, task-driven "intelligent routing" strategy, the present invention can reduce the overall computing cost by more than an order of magnitude without sacrificing (or even improving in some tasks) the quality of the final dataset, thus solving the core pain point of high evaluation costs (especially the cost of the scoring model).

[0134] This embodiment provides different target models corresponding to different node types. Based on factors such as model characteristics and cost, a model with high cost performance is selected to improve the quality of the data set.

[0135] In some embodiments, after determining the final data set, the method further includes: Call the preset model corresponding to the retrieval enhancement generation system, input the final data set into the preset model to output the current answer; Parse the fact particles of the four-tuple data corresponding to the current answer and the dataset to determine the target proportion of fact particles corresponding to the evaluation index; The model test results of the preset model are determined according to the target ratios of multiple evaluation indicators.

[0136] Specifically, considering that the above embodiment only constructs the final dataset, after constructing the final dataset, the RAG system's evaluation process requires invoking the system's preset model, inputting its final dataset, and obtaining the current answer. The current answer and fact particles are then parsed to determine the target proportion of fact particles corresponding to the evaluation metrics. For example, evaluation metrics such as Fact Recall, Fact Precision, and F1 score are used to achieve a quantitative, stable, and interpretable assessment of the answer's "factual correctness" and "completeness."

[0137] Fact Recall: This measure measures the number of ground-truth fact particles contained in the system's answer, reflecting its "information coverage capability." It is calculated as: number of particles correctly mentioned in the answer / total number of particles.

[0138] Fact Precision: How many of the particles mentioned in the answer are ground-truth, rather than errors or redundant information. That is: number of correct particles / total number of particles generated by the system.

[0139] Fact F1 Score: The harmonic mean of the above two is the final comprehensive quality measure.

[0140] The target ratio of the above evaluation indicators is used to determine the model test result of the preset model. If the model test result is qualified, it is determined that the performance of the preset model under the RAG system is qualified.

[0141] This embodiment provides evaluation metrics for RAG systems using fact granules, in addition to constructing a dataset. By calculating the number of "fact granules" covered by the answers generated by the RAG system, precise fact recall, fact precision, and F1 scores can be derived. This evaluation paradigm shifts the assessment of answer quality from the vague concept of "semantic similarity" to the clear concept of "fact coverage," making the evaluation results more stable, interpretable, resilient, and fair.

[0142] In some embodiments, when a new target document is added, the knowledge graph construction process includes: The knowledge graph constructed from the target document is used as the first knowledge graph; When a new target document is added, the target document is segmented according to the document length and content components to obtain multiple segment nodes until a new node is determined; According to the multi-source semantic relationship strategy, the relationship between multiple new nodes is constructed to obtain the second knowledge graph; The first knowledge graph and the second knowledge graph are fused to obtain the final knowledge graph.

[0143] Specifically, during the knowledge graph construction process, if a new document is added, there is no need to completely rebuild the graph. Instead, steps S11 to S12 are simply executed on the new document to generate new nodes and relationships to construct a second knowledge graph. The two are then fused to obtain the final knowledge graph. This fusion process can be done by direct splicing or other splicing methods, which are not limited here.

[0144] The new target document provided in this embodiment is added to form a second knowledge graph, avoiding completely rebuilding the knowledge graph. Instead, the second knowledge graph is added based on the original first knowledge graph, saving construction time and improving efficiency.

[0145] In some embodiments, the method further comprises: Checking the timeliness of isolated nodes among a plurality of nodes of the knowledge graph and relationships connected by the plurality of nodes according to a first preset period; If the timeliness meets the preset conditions, the target nodes and isolated nodes of the target relationship that meet the timeliness corresponding to the preset conditions are classified as cold nodes; The cold nodes are re-analyzed according to a second preset period to determine a new knowledge graph.

[0146] Specifically, regular health checks of each node in the knowledge graph involve checking the timeliness of isolated nodes and node relationships across multiple nodes in the knowledge graph based on a first preset period. If the timeliness meets the preset conditions, the target relationships that meet the timeliness requirements are treated as weak links and classified as cold nodes in the same manner as isolated nodes. These cold nodes are then re-analyzed and processed to determine a new knowledge graph.

[0147] In addition, if a generated question-answer pair is judged to be of "low quality", especially rejected due to "irrelevant context" or "answer hallucination", the system will pass this feedback signal back to the knowledge graph. For example, if nodes a and b are used to generate a failed multi-hop question, the system can reduce the weight of the "SIMILAR_TO" relationship between them, or mark the two nodes as requiring further manual review.

[0148] This embodiment provides for periodic checking of node status in the knowledge graph, periodic running of scripts, checking of isolated nodes and weakly connected parts in the graph, or identification of "cold spots" and "hot spots" in the knowledge base based on user query logs (if available), and may trigger re-analysis or enhancement of cold spot areas.

[0149] Figure 2 A schematic diagram of a RAG system evaluation data set provided by an embodiment of the present invention is shown in FIG. Figure 2 As shown, a Markdown document collection about "History of Development of Artificial Intelligence" is processed as an example, but this is not a limitation of the present invention. The method and system can be applied to document corpora in any fields and in any formats.

[0150] Hardware platform: One or more cloud servers equipped with NVIDIA A100 or equivalent graphics processing units (GPUs).

[0151] Software stack: Core libraries: data modeling, asynchronous processing, document parsing, framework-like components (text segmentation), numerical calculations, and retry logic.

[0152] Data storage: Use a vector database to store node embeddings, and use a graph database or in-memory graph structure to store the nodes and relationships of the knowledge graph.

[0153] LLM service: Access multiple model services through APIs for fine-tuned evaluation agents.

[0154] In the query synthesis module, the knowledge graph is first subject-wise analyzed to identify core themes such as "Symbolic AI" and "Connectionist AI." The model is then invoked to generate personas. Possible outputs include: "a high school student curious about the history of AI," "a computer science major seeking insights into technological evolution," and "a humanities and social science scholar interested in the philosophical debates surrounding AI."

[0155] Path selection: The query synthesis engine randomly samples a path in the knowledge graph: "E Meeting" node ENTITY_CO_OCCURRENCE --> "F Test" node.

[0156] Character and instructions: Randomly select the "Computer Science Major College Student" character.

[0157] LLM Call: Submitting instructions to the model: Please play the role of a computer science student analyzing the development of AI technology. Based on the following two passages about the "E Conference" and the "F Test," please propose a profound question that requires comprehensive information from both sides and provide a standard answer. The question should reflect an exploration of historical relevance.

[0158] Generates the following result: Question: How did H's 'F Test' lay the conceptual foundation for the 'E Conference' ten years later, which established artificial intelligence as an independent discipline? What are the connections and differences between the two approaches to defining 'intelligence'?

[0159] Standard answer: "The F test, for the first time, provided an operational standard for determining whether a machine is intelligent, equating intelligence with the ability to engage in conversation that is indistinguishable from that of humans. This 'behaviorist' definition sets a grand but conceivable goal for AI research. Although the participants of the E conference had different views on the implementation path, they all shared the core belief that 'machines can simulate intelligence', which is a continuation of Turing's thinking. The connection is that both attempt to define and implement machine intelligence; the difference is that the F test provides an external, black-box evaluation standard, while the topics of the E conference focus more on specific internal implementation mechanisms, such as symbolic reasoning, neural networks, etc."

[0160] The factual granular deconstruction of the standard answer: The above standard answers are immediately fed into the fact granule deconstruction module.

[0161] LLM call: Calls the model and gives strict JSON output instructions.

[0162] Deconstruction result (JSON format): [ {"subject":"F test","predicate":"definition method","object":"Judging whether a machine is intelligent from the operational level"}, {"subject":"F-test","predicate":"Core idea","object":"Intelligence is equivalent to the ability to have conversations that are indistinguishable from humans"}, {"subject":"F-test","predicate":"sets","object":"a grand but conceivable goal for AI research"}, {"subject":"E Conference","predicate":"Shared Belief","object":"Machines Can Simulate Intelligence"}, {"subject":"The belief of E Conference","predicate":"is a continuation of...","object":"Turing's Thought"}, {"subject":"F test and E conference","predicate":"association","object":"both attempt to define and implement machine intelligence"}, {"subject":"F-test","predicate":"provide","object":"external, black-box criteria"}, {"subject":"E Meeting","predicate":"More attention","object":"Internal implementation mechanism"} ].

[0163] This set of fact particles is saved together with the question, standard answer, and source node UUID to form a complete evaluation sample.

[0164] When the critic agent in the third stage intervenes, the model critic agent, which has been fine-tuned on 1,000 manually labeled examples, receives the sample. It evaluates each dimension separately: It judged "question quality" as 5 points (clear and in-depth); Compare the answer with the context (the original text of "F Test" and "E Meeting") and judge the "Answer Fidelity" as "High"; Check whether the answer answers all parts of the question ("related" and "different") and judge "Answer Completeness" as "High"; Confirm that both context nodes are necessary to answer the question, and judge "Contextual Relevance" as 5 points.

[0165] Finally, an overall quality score of 4.8 was given.

[0166] The third stage of the revision agent's decision: The revision agent receives a score of 4.8, which is higher than the pre-set threshold of 4.0. Therefore, the revision agent determines that the sample is "passing" and approves it for inclusion in the final dataset without revision.

[0167] Example of handling unqualified examples: Suppose another example receives a "Answer Fidelity: Low (Hallucination)" rating. The correction agent will identify the problematic sentence and, based on the original context and question, instruct it to regenerate a more faithful answer. The corrected example will then be sent back to the judging agent for a second round of review. Only examples that pass this review will be ultimately adopted.

[0168] Manifestation of intelligent cost routing: Throughout the entire process, the routing strategy of the cost optimization execution module continues to play a role: When evaluating the RAG system, when generating creative characters such as "high school students", the route is to model e.

[0169] When generating simple one-hop question answers, route to model f.

[0170] When executing fact granule deconstruction that strictly follows the JSON format, route to model g.

[0171] When performing quality scoring tasks that require processing hundreds of times per second, it is routed to the extremely low-cost, locally deployed fine-tuned model h.

[0172] When doing vector embedding, it is routed to a dedicated model i.

[0173] In the fourth phase, batch execution and efficiency: the system packages hundreds or even thousands of these build-and-verify tasks. Calls to the external LLM API are initiated in parallel. For example, while waiting for model e to generate a complex problem, the system may have already completed dozens of vector embeddings and a dozen quality scoring tasks performed by lightweight models. The built-in tenacity library ensures that if an API is temporarily unavailable or returns a rate limit error, the task is automatically and intelligently retried, ensuring the stable completion of large-scale build tasks.

[0174] Through the close collaboration of the above four stages, this embodiment finally successfully converted a batch of disordered Markdown documents into a RAG evaluation dataset containing hundreds of high-quality, multi-type, and structured fact particles. The entire process was highly automated and the cost was effectively controlled.

[0175] Furthermore, considering ecosystem compatibility, all core modules of the system (such as the segmenter, extractor, and validator) define standard input and output interfaces and use neutral data models (such as the Pydantic model) for data exchange. This frees the system from deep ties to any specific framework (such as LangChain or LlamaIndex). Users can easily access privately deployed embedded models (such as those provided by Ollama), replace them with proprietary document segmentation algorithms, and integrate LLM services from different vendors.

[0176] Although this embodiment mainly processes text documents, its core architecture has good multimodal expansion potential.

[0177] For example, node expansion: the metadata of the knowledge graph ChunkNode can be easily expanded to store identifiers or Uniform Resource Identifiers (URIs) pointing to non-text data such as images, audio, and tables.

[0178] Feature extraction extension: Multimodal models can be integrated to extract vector embeddings and text descriptions of images and store them in nodes.

[0179] Relationship building extension: The relationship builder can be extended to include new cross-modal algorithms that can calculate image-text similarity and document-table relevance.

[0180] Query synthesis expansion: It can generate multimodal questions that require the integration of text and image information to answer (for example, "Based on the image below and the related text description, what is the design concept of this product?").

[0181] This reserves a clear technical path for future support of more complex RAG system evaluations containing multimodal information.

[0182] exist Figure 2 The first phase involves constructing an adaptive knowledge graph based on the source documents. This phase transforms the input unstructured document collection into a knowledge graph (KG) rich in semantic information and structured relationships through an intelligent processing pipeline. This graph serves as the foundation for all subsequent operations.

[0183] The second stage involves query synthesis, integrating user personas with fact granules. This stage, based on the constructed knowledge graph, simulates different user perspectives to generate challenging query-answer pairs covering a variety of scenarios, including single-point facts, multi-point associations, comparative analysis, and even counterfactual reasoning. It also pioneers the deconstruction of standard answers into atomic "fact granules."

[0184] The third stage involves verification and refinement of generated content through multi-agent collaboration. This stage introduces a closed-loop system comprised of intelligent agents with three roles: "generation," "judgment," and "correction." This system cross-checks and self-corrects the synthesized query-answer pairs, ensuring the quality, accuracy, and reliability of the generated content. Furthermore, after corrections, the content can be fed back into the knowledge graph for optimization.

[0185] The fourth phase is cost-effective batch execution and deployment. This phase uses a series of optimization strategies, including intelligent task routing, asynchronous parallel processing, and cost-aware model selection, to maximize processing efficiency and minimize computing resource costs while ensuring quality.

[0186] At the same time, an automated construction system is provided for the four stages, including the document processing and knowledge graph construction module: responsible for executing the tasks of the first stage, including sub-modules such as document intelligent parsing, adaptive segmentation, multi-dimensional feature extraction, multi-source relationship construction, and dynamic maintenance of the knowledge graph.

[0187] Query synthesis module: Responsible for executing the tasks of the second phase, including sub-modules such as global topic analysis, dynamic persona generation, multi-strategy query synthesis (single-hop, multi-hop, comparison, negation, etc.), and fact granular deconstruction of standard answers.

[0188] Multi-agent verification module: Responsible for executing the tasks of the third phase, including generation agents, evaluation agents, and correction agents, forming a closed-loop "generation-review-modification" quality control process.

[0189] Cost Optimization Execution Module: This module is responsible for executing the tasks in the fourth phase and includes sub-modules such as intelligent batch processing, asynchronous scheduling, API rate control, and intelligent cost routing for model services based on task complexity.

[0190] The above describes in detail various embodiments corresponding to the data set determination method. On this basis, the present invention also discloses a data set determination device corresponding to the above method. Figure 3 This is a structural diagram of a data set determination device provided by an embodiment of the present invention. Figure 3 As shown, the data set determination device includes: The acquisition module 11 is used to acquire a target document and segment the target document into multiple segment nodes according to the document length and content components; A relationship construction module 12 is used to extract features from multiple block nodes to obtain corresponding feature parameters, so as to form nodes of the knowledge graph together with the corresponding block nodes, and to construct relationships between the multiple nodes according to a multi-source semantic relationship strategy to obtain a knowledge graph; Output module 13, used to set user roles for the knowledge graph, generate target questions based on user roles and node query strategies, input the target questions into the target model, and output standard answers; The decomposition processing module 14 is used to decompose the standard answer to obtain fact particles and construct the four-tuple data of the target question, the standard answer, the block node and the fact particle; The evaluation processing module 15 is used to evaluate and process the quadruple data according to the data set evaluation coordination mechanism to determine the final data set.

[0191] Since the embodiments of the device part correspond to the above embodiments, the embodiments of the device part please refer to the description of the embodiments of the method part, and will not be repeated here.

[0192] For an introduction to a data set determination device provided by the present invention, please refer to the above method embodiment, and the present invention will not be described in detail here. It has the same beneficial effects as the above data set determination method.

[0193] Figure 4 A structural diagram of a data set determination device provided by an embodiment of the present invention, such as Figure 4 As shown, the device includes: Memory 21, for storing computer programs; The processor 22 is configured to implement the steps of the data set determination method when executing a computer program.

[0194] The data set determination device provided in this embodiment may include but is not limited to a smart phone, a tablet computer, a laptop computer, or a desktop computer.

[0195] The processor 22 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 22 may be implemented in at least one of the following hardware forms: a digital signal processor (DSP), a field-programmable gate array (FPGA), or a programmable logic array. The processor 22 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 22 may be integrated with a GPU, which is responsible for rendering and drawing the content required to be displayed on the display screen. In some embodiments, the processor 22 may also include an AI processor for processing computing operations related to machine learning.

[0196] The memory 21 may include one or more non-volatile storage media, which may be non-transitory. The memory 21 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 21 is at least used to store the following computer program 211, wherein, after the computer program is loaded and executed by the processor 22, it can implement the relevant steps of the data set determination method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 21 may also include an operating system 212 and data 213, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 212 may include Windows, Unix, Linux, etc. The data 213 may include but is not limited to data involved in the data set determination method, etc.

[0197] In some embodiments, the data set determination device may further include a display screen 23 , an input / output interface 24 , a communication interface 25 , a power supply 26 , and a communication bus 27 .

[0198] Those skilled in the art will understand that Figure 4 The structure shown in the figure does not constitute a limitation on the data set determination device, and may include more or fewer components than shown in the figure.

[0199] The processor 22 implements the data set determination method provided in any of the above embodiments by calling instructions stored in the memory 21 .

[0200] For an introduction to a data set determination device provided by the present invention, please refer to the above method embodiment, and the present invention will not be repeated here. It has the same beneficial effects as the above data set determination method.

[0201] Furthermore, the present invention also provides a non-volatile storage medium, on which a computer program is stored. When the computer program is executed by the processor 22, the steps of the above-mentioned data set determination method are implemented.

[0202] It is understandable that if the methods in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0203] For an introduction to a non-volatile storage medium provided by the present invention, please refer to the above method embodiment, which will not be described in detail herein. The method has the same beneficial effects as the above data set determination method.

[0204] Furthermore, the present invention also provides a computer program product, comprising a computer program / instruction, which implements the steps of the data set determination method when executed by a processor.

[0205] For an introduction to a computer program product provided by the present invention, please refer to the above method embodiment, which will not be described in detail herein. The computer program product has the same beneficial effects as the above data set determination method.

[0206] The above is a detailed introduction to a data set determination method, device, medium and product provided by the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, the present invention can also be improved and modified in several ways, and these improvements and modifications also fall within the scope of protection of the present invention.

[0207] It should also be noted that, in this specification, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.< / h2>

Claims

1. A method for determining a data set, characterized in that: include: Obtaining a target document, and segmenting the target document into multiple segment nodes according to the document length and content components; Extract features from multiple block nodes to obtain corresponding feature parameters, which together with the corresponding block nodes constitute nodes of the knowledge graph, and construct relationships between multiple nodes according to the multi-source semantic relationship strategy to obtain a knowledge graph; Setting a user role for the knowledge graph to generate a target question based on the user role and the node query strategy, inputting the target question into the target model, and outputting a standard answer; Decomposing the standard answer to obtain fact particles, and constructing four-tuple data of the target question, standard answer, block node and fact particle; The quadruple data is evaluated and processed according to the data set evaluation coordination mechanism to determine the final data set.

2. The data set determination method according to claim 1, characterized in that: The target document is segmented according to the document length and content components to obtain multiple segment nodes, including: Parsing the target document to obtain a text file, and retaining structural information corresponding to the content components; Performing length classification processing on the text file according to the document length to obtain a target length file, wherein the target length file is a first length file or a second length file, and the length of the first length file is greater than the length of the second length file; When the target length file is a first length file, segmenting the first length file according to the structure information to obtain corresponding block nodes; When the target length file is a second length file, the second length file is used as a block node.

3. The data set determination method according to claim 2, characterized in that: Segmenting the first-length file according to the structure information to obtain corresponding block nodes includes: Start the hierarchical segmentation mechanism; Adding the structural information to the first length file to obtain a third length file; The third length file is segmented by a hierarchical segmentation mechanism to obtain a plurality of text blocks, and the text blocks are used as corresponding block nodes.

4. The method for determining a data set according to claim 1, wherein: Feature extraction is performed on multiple block nodes to obtain corresponding feature parameters, including: Initiating a parallel feature extraction mechanism; wherein the parallel feature extraction mechanism includes at least a summary extraction mechanism, a topic extraction mechanism, a keyword extraction mechanism, a named entity recognition mechanism, a question generation mechanism, and a vector embedding extraction mechanism; The parallel feature extraction mechanism is used to extract features from multiple block nodes to obtain feature parameters.

5. The method for determining a data set according to claim 4, wherein: Build relationships between multiple nodes based on the multi-source semantic relationship strategy to obtain a knowledge graph, including: Processing the vector feature parameters of the block nodes within the multiple nodes according to the vector similarity relationship strategy to obtain the vector relationship between the multiple nodes; Obtaining a linear reading order of a target document, and determining a hierarchical structure relationship strategy based on the linear reading order; Processing multiple nodes according to the hierarchical structure relationship strategy to obtain a sequence relationship between the multiple nodes; Processing named entities of multiple nodes according to entity co-occurrence relationship strategy to determine entity co-occurrence relationship; Process multiple nodes according to the inference relationship strategy to determine the corresponding causal relationship; A knowledge graph of multiple nodes is constructed based on the vector relationship, the sequential relationship, the entity co-occurrence relationship and the causal relationship.

6. The method for determining a data set according to claim 1, wherein: Setting user roles for the knowledge graph includes: Performing a global analysis on the knowledge graph to obtain a corresponding topic list; Determine the target sub-topic by selecting and processing the sub-topics in the topic list; Generate corresponding user roles based on the target sub-topic.

7. The method for determining a data set according to claim 6, wherein: Generate target questions based on the user role and node query strategy, including: Determining a node type corresponding to a target node in the knowledge graph, wherein the node type is a single-hop node type and a multi-hop node type; When the target node is a single-hop node type, calling a first pre-trained model, inputting the target node and the user role into the first pre-trained model to obtain a first target problem; When the target node is a multi-hop node type, the corresponding target relationship is determined in the knowledge graph; the second pre-trained model is called, and the target relationship, the target node and the user role are input into the second pre-trained model to obtain a second target problem.

8. The method for determining a data set according to claim 7, wherein: Before inputting the target relationship, the target node, and the user role into the second pre-training model to obtain the second target problem, the method further includes: When the target relationship of multiple target nodes is a preset target relationship and the corresponding contents are complementary, or when the target relationship of multiple target nodes is a preset target relationship and the corresponding contents are different and share the same subtopic, the step of inputting the target relationship, the target node and the user role into the second pre-trained model to obtain the second target problem is entered.

9. The method for determining a data set according to claim 1, wherein: The standard answer is decomposed to obtain fact particles, including: Segmenting the standard answer to obtain corresponding word units; The third pre-trained model is called, and the word-unit is input into the third pre-trained model for decomposition processing to obtain corresponding fact particles; wherein the fact particles at least include a subject, an attribute, and a value.

10. The method for determining a data set according to claim 9, wherein: The quadruple data is evaluated and processed according to the data set evaluation coordination mechanism, including: Using natural language processing tools to perform semantic analysis on the target question to determine the quality of the question; Calling a fourth pre-trained model, inputting context information corresponding to the standard answer and the block node into the fourth pre-trained model to determine the semantic similarity between the standard answer and the block node; determining answer fidelity based on the semantic similarity; Calling a fifth pre-trained model, inputting the standard answer and the target question into the fifth pre-trained model to determine the semantic coverage of the standard answer and the target question; determining answer completeness based on the semantic coverage; Determining the relevance of the context information of the block node, the target question, and the standard answer based on a similarity method; calling a sixth pre-trained model, and inputting the fact granules into the sixth pre-trained model to determine the independent integrity of the fact granules; The dataset evaluation results are determined based on the corresponding weight parameters of question quality, answer fidelity, answer completeness, relevance of standard answers, and independent integrity of fact particles.

11. The method for determining a data set according to claim 10, wherein: When it is determined that the dataset evaluation result does not reach the preset threshold, the method further includes: Analyzing and processing the evaluation results of the data set to determine a correction strategy; When the correction strategy is a question strategy, a new target question is regenerated, and the new target question, context information, and the target question are input into a target model to obtain a new standard answer; Alternatively, when the correction strategy is a block node strategy, new context information is determined according to the embedding vector of the target question and the node query strategy; Alternatively, when the correction strategy is a quadruple matching strategy, the quadruple data is discarded, and the next target document is obtained, and the step of segmenting the target document according to the document length and content components to obtain multiple block nodes is entered until the new quadruple data is determined.

12. The method for determining a data set according to claim 1, wherein: The data set is composed of at least one quadruple data, and the process of forming the data set includes: Obtain and construct target models corresponding to multiple quadruple data; If the target model is the same, the requests corresponding to constructing multiple quadruple data are merged into the same batch for batch processing.

13. The method for determining a data set according to claim 12, wherein: When the target models are not the same, the method further includes: Classify multiple target models to determine target models corresponding to different classifications; Determine the target category with the largest number of target models under the same category among the target models of different categories; In the target classification, requests for constructing quad-tuple data corresponding to the target classification are merged into a first batch; This process is repeated until the requests for the four-tuple data corresponding to the category with the smallest number of target models under the same category are merged into the Nth batch; The target models under different batches are called evenly according to the network bandwidth to generate quadruple data under the corresponding batches.

14. The method for determining a data set according to claim 7, wherein: The process of determining the target model includes: When the target node is a single-hop node, the first performance model with the greatest performance effect is selected from the performance model set as the target model; When the target node is a multi-hop node type, the first model with the largest comprehensive score in the model set is selected as the target model.

15. The method for determining a data set according to claim 1, wherein: After finalizing the dataset, it also includes: Calling a preset model corresponding to the retrieval enhancement generation system, inputting the final data set into the preset model to output a current answer; Parsing the fact particles of the four-tuple data corresponding to the current answer and the data set to determine the target ratio of the fact particles corresponding to the evaluation index; The model test result of the preset model is determined according to the target ratio of multiple evaluation indicators.

16. The method for determining a data set according to any one of claims 2 to 5, wherein: When a new target document is added, the knowledge graph construction process includes: Using the knowledge graph constructed from the target document as the first knowledge graph; When a new target document is added, the process proceeds to the step of slicing the target document according to the document length and content components to obtain a plurality of segment nodes until a new node is determined; According to the multi-source semantic relationship strategy, the relationship between multiple new nodes is constructed to obtain the second knowledge graph; The first knowledge graph and the second knowledge graph are fused to obtain a final knowledge graph.

17. The method for determining a data set according to any one of claims 2 to 5, wherein: The method further comprises: Checking the timeliness of isolated nodes among the plurality of nodes of the knowledge graph and the relationships connected by the plurality of nodes according to a first preset period; If the timeliness meets the preset conditions, the target nodes and isolated nodes of the target relationship that meet the timeliness corresponding to the preset conditions are classified as cold nodes; The cold nodes are re-analyzed according to a second preset period to determine a new knowledge graph.

18. A data set determination device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the data set determination method according to any one of claims 1 to 17 when executing the computer program.

19. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data set determination method according to any one of claims 1 to 17 are implemented.

20. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of the data set determination method according to any one of claims 1 to 17 are implemented.

Images