Unified adaptation and standardized evaluation method and system for heterogeneous RAG framework

Abstract

The application discloses a unified adaptation and standardized evaluation method and system for a heterogeneous RAG framework. The method constructs a three-layer decoupling architecture of the arrangement layer, the adaptation layer and the evaluation layer. First, the ability characteristics of the target framework are automatically loaded through the framework capability perception matrix; then, an independent API adapter is assigned to encapsulate the heterogeneous framework into a unified operation interface, and an intelligent uploading strategy is executed and the heterogeneous response format is parsed into a unified four-tuple structure; then, six mutually orthogonal evaluation dimensions are established, and automatic evaluation is carried out based on a large language model evaluator; finally, combined with the framework capability, a differentiated evaluation standard is applied, different failure modes are mapped to specific dimension combinations to realize accurate performance attribution, and a comparative analysis report is output. The application effectively solves the problems that the heterogeneous framework cannot be uniformly evaluated and a single index is difficult to locate the bottleneck, and significantly improves the scientificity, reproducibility and fairness of cross-framework comparison of the evaluation.

Description

Technical Field

[0001] This invention relates to the testing and evaluation of large language model application systems, and more particularly to a unified adaptation and standardized evaluation method and system for heterogeneous RAG frameworks. Background Technology

[0002] With the rapid development of large language model technology, Retrieval-Augmented Generation (RAG) has become an advanced system architecture for overcoming the illusion of large models and introducing external domain knowledge. Its applications have expanded widely to various scenarios such as enterprise knowledge bases, intelligent customer service, and professional question answering. By combining information retrieval and text generation capabilities, RAG systems can provide accurate answers in complex and even highly specialized tasks. However, with the emergence of various open-source and commercial RAG frameworks (such as AnythingLLM, RAGFlow, MaxKB, and Metaso), the differentiation in system implementations has significantly increased. Different frameworks exhibit significant differences in API interface design, document upload methods, response formats, and source information return mechanisms. Therefore, in the implementation and application of RAG systems, it is particularly important to conduct unified evaluations of these heterogeneous frameworks to ensure that the system can maintain high efficiency in various tasks.

[0003] Existing RAG evaluation methods have been applied in model capability validation in recent years, typically assessing response quality through a single metric or expert scoring. However, these methods still have significant shortcomings. First, heterogeneous frameworks cannot be evaluated uniformly; existing methods usually only test a single framework, failing to achieve fair comparisons across frameworks. Specifically, incompatibility of API interfaces, heterogeneous response formats, differences in upload strategies, and inconsistent source information detection among different frameworks lead to a lack of fairness in cross-framework evaluation. Second, existing evaluation methods have fundamental flaws; a single metric cannot pinpoint performance bottlenecks. For example, existing factual single-metric methods cannot distinguish whether performance issues originate from the retrieval or generation end. Furthermore, expert evaluation is costly and not reproducible; manual evaluation requires domain experts to score each question individually, which is time-consuming, and scoring standards vary from person to person. Finally, existing methods lack framework awareness, treating all RAG frameworks as black boxes and failing to adjust evaluation criteria based on framework capability differences. The existence of these problems indicates that solving the problem of unified evaluation of heterogeneous RAG frameworks and establishing a scientifically reproducible automated evaluation method are the core challenges that current RAG evaluation technology urgently needs to address. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention proposes a unified adaptation and six-dimensional standardized evaluation method for heterogeneous RAG frameworks. It constructs a three-layer decoupled architecture consisting of an orchestration layer, an adaptation layer, and an evaluation layer. Multiple heterogeneous RAG frameworks are encapsulated into a unified operation interface through a pluggable framework adapter pattern, thus solving the problem of inconsistent evaluation of heterogeneous frameworks. Next, six mutually orthogonal evaluation dimensions are established: completeness, factuality, context utilization, technical depth, relevance, and support quality. Combined with a framework capability awareness matrix, differentiated evaluation standards are applied to different types of frameworks, achieving fair and automated evaluation of heterogeneous frameworks. Furthermore, the reproducibility of the evaluation is ensured through thought chain reasoning and probability-weighted scoring of labeled units using a large language model evaluator based on G-Eval. This method excels in handling incompatibility between multiple framework interfaces and heterogeneous response formats. It effectively addresses the shortcomings of existing evaluation methods, such as the inability of a single indicator to locate performance bottlenecks and the high cost of expert scoring. It possesses strong and accurate attribution capabilities, enabling different failure modes to be mapped to specific dimensional combinations. Furthermore, by replacing manual evaluation with an automated evaluation process, it can significantly improve the scientific rigor, reproducibility, and fairness of cross-framework comparisons in RAG system evaluation.

[0005] A unified adaptation and standardized evaluation method for heterogeneous RAG frameworks includes the following steps:

[0006] Step S1, Automatic Framework Capability Awareness: The system maintains the capability feature description of each supported RAG framework through the framework capability awareness matrix, and automatically loads the capability features of the target framework during evaluation initialization.

[0007] Step S2, Unified Operation Interface Adaptation: Assign an independent API adapter to the target framework, and encapsulate the heterogeneous RAG framework into a unified operation interface that includes document upload, query and answer, and source information extraction through a pluggable framework adapter pattern.

[0008] Step S3: Execution of intelligent upload strategy: Based on the framework capability awareness matrix and user configuration, automatically select and execute the corresponding document upload strategy;

[0009] Step S4, Response Format Standardization and Parsing: Call the query and answer interface to obtain the raw response data, and use the framework-specific parsing logic to convert the heterogeneous response format into a unified four-tuple structure for standardized parsing;

[0010] Step S5, Automated Evaluation and Precise Attribution: Establish six mutually orthogonal evaluation dimensions for automated evaluation based on a large language model evaluator; combine framework capability awareness with differentiated evaluation criteria, and map different failure modes to specific dimension combinations to achieve performance attribution;

[0011] Step S6: Generate a comparative analysis report: Summarize the evaluation results into a standardized comparative report output, revealing the differentiated capability profiles between the frameworks.

[0012] Furthermore, in step S1, the capability feature description of the supported RAG framework includes the following dimensions:

[0013] Knowledge base upload support tags: Whether knowledge base documents can be uploaded via API;

[0014] Default to skip upload tag: Whether to skip the upload step by default;

[0015] Manually pre-configure tags: Whether the knowledge base needs to be manually pre-configured;

[0016] Framework feature type tags: distinguish between full RAG type, manual configuration type, or search engine type;

[0017] Upload strategy tags: distinguish between API upload support and manual upload required;

[0018] When encountering an unregistered framework, the system automatically uses the default security setting that does not support knowledge base uploads and skips the upload step.

[0019] Furthermore, in step S2, the definitions of each unified interface are as follows:

[0020] Document upload interface: Input file path, return upload result including success or failure flags and detailed information;

[0021] Question and Answer Interface: Input the question text and context information, and it returns the framework's answer text and raw response data;

[0022] Source information extraction interface: Extracts source information from the raw response data and returns standardized source information detection results, including whether source information was found, the number of source information items, detailed content, and detection method identifier.

[0023] Furthermore, in step S3, the specific process of the intelligent upload strategy includes the following steps:

[0024] Step S31: Check if the user has explicitly specified skipping upload. If specified, the user setting is used directly. If not specified, the frame default value in the frame capability awareness matrix is ​​read. When skipping upload, it is assumed that the knowledge has been uploaded in advance.

[0025] Step S32: When it is determined that an upload is required, a temporary document file is created for each gold context source file according to the context content, and the document upload interface of the corresponding framework adapter is called.

[0026] Step S33: Confirm the upload result according to the success determination logic specific to the target framework, calculate the upload success rate, and wait for a preset time after the upload is completed to ensure that the vector index construction is completed;

[0027] Step S34: Record all detailed error information of failed uploads into a failure list and output it to the evaluation report; continue to perform subsequent queries and evaluations when some uploads fail and mark the incomplete upload status in the results.

[0028] Furthermore, in step S4, the heterogeneous response formats are converted into a unified four-tuple structure, which includes: whether source information was detected, the amount of source information, frame-specific details, and source detection method identifier; the parsing strategies for each frame are as follows:

[0029] Complete RAG-type framework: Directly extract source information arrays and source counts from raw response data;

[0030] Manually configured framework: First, check the explicit source information array. If it is empty, it will be downgraded to heuristic detection to extract the frequency of domain-specific terms and technical content features in the answers. When the knowledge usage index is greater than zero or the technical content feature is true, it is inferred that the framework uses knowledge base content.

[0031] Search engine-type framework: Extracts an array of source information while retaining session identifiers and result identifiers for tracing.

[0032] Furthermore, in step S5, the automated evaluation based on the large language model evaluator includes the following steps:

[0033] Step S51: Generate a clear assessment objective description for each dimension, describing the specific RAG system capabilities assessed for that dimension and the corresponding 10-level scoring criteria;

[0034] Step S52: Design structured reasoning steps for each dimension to guide the evaluation model to think along the process. The process includes: identifying key statements in the answers, comparing each statement with the source material one by one, marking statements that are consistent, inconsistent or unverifiable, and making a comprehensive judgment to give a score.

[0035] Step S53: Call the evaluation model through the G-Eval module, extract the label unit level probability when the model generates scores, normalize and weight the output probabilities of candidate scores from 1 to 10 to obtain continuous score values.

[0036] Step S54: Independently evaluate the RAG response for each test case across six dimensions, outputting a six-dimensional score vector. The six mutually orthogonal evaluation dimensions specifically include:

[0037] Completeness: Assess the extent to which the answer covers all the necessary technical elements in the query;

[0038] Factual: Assess the consistency between the response and the source material; detect hallucinations;

[0039] Context utilization: Evaluates the actual proportion of the retrieved contextual content used in the response;

[0040] Technical depth: Assessing the sophistication of the analysis and reasoning;

[0041] Relevance: Assess the degree to which the answer focuses on the query, and detect off-topic content and information redundancy;

[0042] Support quality: Assess the specificity and accuracy of the citations.

[0043] Furthermore, in step S5, the differentiated evaluation criteria applied to the capability perception of each framework are as follows:

[0044] Manually configurable framework: The system evaluates the knowledge usage in the answers using a knowledge usage metric heuristic method: it detects the frequency of domain-specific terms appearing in the answers and evaluates whether the answers contain specific data points that only appear in the knowledge base documents to infer the utilization of the knowledge base;

[0045] Search engine-type frameworks allow for greater evaluation tolerance in terms of context utilization.

[0046] Complete RAG-type framework: applying the standard six-dimensional evaluation criteria without special adjustments.

[0047] A unified and standardized evaluation system for heterogeneous RAG frameworks, achieving automated evaluation through a three-layer decoupled architecture, specifically including:

[0048] The orchestration layer is responsible for evaluation process management and strategy selection. It loads completed evaluation records through configuration management, supports breakpoint resume mechanism, and only performs incremental evaluation on incomplete test cases.

[0049] The adaptation layer handles heterogeneous framework interfaces uniformly through an independent API adapter class, including a mechanism for identifying and handling specific errors that already exist in files within search engine frameworks.

[0050] The evaluation layer establishes six mutually orthogonal evaluation dimensions to independently perform evaluations, and automatically generates capability profiles for each framework's strengths and weaknesses based on the evaluation results.

[0051] Furthermore, after the system completes the evaluation, it summarizes the evaluation results of all frameworks into a standardized comparison report, including:

[0052] Test case level records: Complete evaluation data for each test case, including test identifier, test type, question text, standard answer, actual answer in the RAG framework, six-dimensional evaluation score, source information detection results, upload success rate, and response time;

[0053] Dimensional aggregation: mean, median, standard deviation, extreme values, quartiles, and correlation matrix between dimensions for each dimension;

[0054] Performance bottleneck diagnosis: Automatically identify performance bottleneck types through dimensional combination patterns and output corresponding optimization suggestions;

[0055] Cross-frame / cross-model comparison table: Generates a comparison matrix between frames or models based on six-dimensional indicators, and ranks them according to the comprehensive score of the six dimensions and the score of each individual dimension;

[0056] Framework capability profile: Automatically generate a capability profile for each framework based on the evaluation results, identifying its strengths and weaknesses;

[0057] The results are saved in both structured data format and tabular format, and support breakpoint resume—loading completed evaluation records and performing evaluations only on incomplete test cases to avoid duplication of work.

[0058] Furthermore, the system supports two complementary evaluation modes, ensuring the comparability of results through strict variable control:

[0059] Cross-model comparison mode: The RAG framework is fixed, and the basic large language model is replaced one by one; the variables that remain unchanged in the evaluation include: the same test case set, the same knowledge base content, the same chunking strategy, the same upload process, and the same query method; the only variable that changes is the underlying large language model.

[0060] Cross-framework comparison mode: Fix the basic large language model and replace the RAG framework one by one; it is necessary to ensure that each framework obtains the same knowledge base content, and manual configuration of the framework requires uploading the same documents in advance and using the same test case set;

[0061] Both modes use the exact same six-dimensional metrics and evaluation process to ensure consistency in scoring standards; configuration management uses a JSON configuration file to unify the management framework connection parameters, including API address, authentication key, and workspace identifier, and supports switching evaluation targets without modifying the code.

[0062] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0063] 1. Pluggable heterogeneous framework unified adaptation: By using the adapter pattern, various RAG frameworks with completely different API interfaces are encapsulated into a unified operation interface. When adding a new framework, only the adapter class needs to be implemented, without modifying the evaluation logic.

[0064] 2. Framework Capability Awareness Matrix: For the first time, a systematic description system of RAG framework capability features is established, covering multiple dimensions such as knowledge base support, upload strategy, and framework type, providing prior guidance for the automatic selection of evaluation strategies and assessment standards.

[0065] 3. Six-dimensional orthogonal evaluation enables accurate attribution: The combination of six independent dimensions can accurately distinguish three typical failure modes: retrieval failure, generation failure, and attention dilution, providing a clear direction for RAG system optimization.

[0066] 4. Automated evaluation based on G-Eval replaces human experts: The automated evaluation process based on thought chain reasoning and probability weighting of labeled units achieves complete reproducibility while maintaining evaluation quality, and significantly reduces evaluation costs.

[0067] 5. Differentiated evaluation of framework capabilities: Different evaluation criteria are applied to different types of frameworks, such as manually configured, search engine, and full RAG. In particular, the heuristic method of knowledge usage indicators solves the problem of fair evaluation of frameworks whose knowledge base content cannot be directly verified.

[0068] 6. Dual-mode evaluation design: The cross-model comparison and cross-framework comparison modes strictly control variables, respectively isolating differences in model capabilities and differences in framework design, to ensure the scientific nature of the evaluation conclusions.

[0069] 7. Structured exception handling and breakpoint resume: Framework-specific error handling and incremental evaluation mechanisms based on existing results improve the robustness of the system in actual deployment. Attached Figure Description

[0070] Figure 1 This is a schematic diagram of the unified adaptation and standardized evaluation method for heterogeneous RAG frameworks proposed in this invention.

[0071] Figure 2 This is a cross-frame six-dimensional evaluation comparison radar chart provided according to an embodiment of the present invention.

[0072] Figure 3 This is a comparison chart of cross-model six-dimensional evaluation results provided according to an embodiment of the present invention.

[0073] Figure 4 This is a six-dimensional comprehensive score ranking chart based on the configuration of various frameworks and models provided in the embodiments of the present invention. Detailed Implementation

[0074] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings. The technical solutions in the embodiments of the present invention will be clearly and completely described. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0075] A unified adaptation and standardized evaluation method for heterogeneous RAG frameworks includes the following steps: Figure 1 As shown:

[0076] Step S1, Automatic Framework Capability Awareness: The system maintains the capability feature description of each supported RAG framework through the framework capability awareness matrix, and automatically loads the capability features of the target framework during evaluation initialization.

[0077] Step S2, Unified Operation Interface Adaptation: Assign an independent API adapter to the target framework, and encapsulate the heterogeneous RAG framework into a unified operation interface that includes document upload, query and answer, and source information extraction through a pluggable framework adapter pattern.

[0078] Step S3: Execution of intelligent upload strategy: Based on the framework capability awareness matrix and user configuration, automatically select and execute the corresponding document upload strategy;

[0079] Step S4, Response Format Standardization and Parsing: Call the query and answer interface to obtain the raw response data, and use the framework-specific parsing logic to convert the heterogeneous response format into a unified four-tuple structure for standardized parsing;

[0080] Step S5, Automated Evaluation and Precise Attribution: Establish six mutually orthogonal evaluation dimensions for automated evaluation based on a large language model evaluator; combine framework capability awareness with differentiated evaluation criteria, and map different failure modes to specific dimension combinations to achieve performance attribution;

[0081] Step S6: Generate a comparative analysis report: Summarize the evaluation results into a standardized comparative report output, revealing the differentiated capability profiles between the frameworks.

[0082] Furthermore, in step S1, the capability feature description of the supported RAG framework includes the following dimensions:

[0083] Knowledge base upload support tags: Whether knowledge base documents can be uploaded via API;

[0084] Default to skip upload tag: Whether to skip the upload step by default;

[0085] Manually pre-configure tags: Whether the knowledge base needs to be manually pre-configured;

[0086] Framework feature type tags: distinguish between full RAG type, manual configuration type, or search engine type;

[0087] Upload strategy tags: distinguish between API upload support and manual upload required;

[0088] When encountering an unregistered framework, the system automatically uses the default security setting that does not support knowledge base uploads and skips the upload step.

[0089] Furthermore, in step S2, the definitions of each unified interface are as follows:

[0090] Document upload interface: Input file path, return upload result including success or failure flags and detailed information;

[0091] Question and Answer Interface: Input the question text and context information, and it returns the framework's answer text and raw response data;

[0092] Source information extraction interface: Extracts source information from the raw response data and returns standardized source information detection results, including whether source information was found, the number of source information items, detailed content, and detection method identifier.

[0093] Furthermore, in step S3, the specific process of the intelligent upload strategy includes the following steps:

[0094] Step S31: Check if the user has explicitly specified skipping upload. If specified, the user setting is used directly. If not specified, the frame default value in the frame capability awareness matrix is ​​read. When skipping upload, it is assumed that the knowledge has been uploaded in advance.

[0095] Step S32: When it is determined that an upload is required, a temporary document file is created for each gold context source file according to the context content, and the document upload interface of the corresponding framework adapter is called.

[0096] Step S33: Confirm the upload result according to the success determination logic specific to the target framework, calculate the upload success rate, and wait for a preset time after the upload is completed to ensure that the vector index construction is completed;

[0097] Step S34: Record all detailed error information of failed uploads into a failure list and output it to the evaluation report; continue to perform subsequent queries and evaluations when some uploads fail and mark the incomplete upload status in the results.

[0098] Furthermore, in step S4, the heterogeneous response formats are converted into a unified four-tuple structure, which includes: whether source information was detected, the amount of source information, frame-specific details, and source detection method identifier; the parsing strategies for each frame are as follows:

[0099] Complete RAG-type framework: Directly extract source information arrays and source counts from raw response data;

[0100] Manually configured framework: First, check the explicit source information array. If it is empty, it will be downgraded to heuristic detection to extract the frequency of domain-specific terms and technical content features in the answers. When the knowledge usage index is greater than zero or the technical content feature is true, it is inferred that the framework uses knowledge base content.

[0101] Search engine-type framework: Extracts an array of source information while retaining session identifiers and result identifiers for tracing.

[0102] Furthermore, in step S5, the automated evaluation based on the large language model evaluator includes the following steps:

[0103] Step S51: Generate a clear assessment objective description for each dimension, describing the specific RAG system capabilities assessed for that dimension and the corresponding 10-level scoring criteria;

[0104] Step S52: Design structured reasoning steps for each dimension to guide the evaluation model to think along the process. The process includes: identifying key statements in the answers, comparing each statement with the source material one by one, marking statements that are consistent, inconsistent or unverifiable, and making a comprehensive judgment to give a score.

[0105] Step S53: Call the evaluation model through the G-Eval module, extract the label unit level probability when the model generates scores, normalize and weight the output probabilities of candidate scores from 1 to 10 to obtain continuous score values.

[0106] Step S54: Independently evaluate the RAG response for each test case across six dimensions, outputting a six-dimensional score vector. The six mutually orthogonal evaluation dimensions specifically include:

[0107] Completeness: Assess the extent to which the answer covers all the necessary technical elements in the query;

[0108] Factual: Assess the consistency between the response and the source material; detect hallucinations;

[0109] Context utilization: Evaluates the actual proportion of the retrieved contextual content used in the response;

[0110] Technical depth: Assessing the sophistication of the analysis and reasoning;

[0111] Relevance: Assess the degree to which the answer focuses on the query, and detect off-topic content and information redundancy;

[0112] Support quality: Assess the specificity and accuracy of the citations.

[0113] Furthermore, in step S5, the differentiated evaluation criteria applied to the capability perception of each framework are as follows:

[0114] Manually configurable framework: The system evaluates the knowledge usage in the answers using a knowledge usage metric heuristic method: it detects the frequency of domain-specific terms appearing in the answers and evaluates whether the answers contain specific data points that only appear in the knowledge base documents to infer the utilization of the knowledge base;

[0115] Search engine-type frameworks allow for greater evaluation tolerance in terms of context utilization.

[0116] Complete RAG-type framework: applying the standard six-dimensional evaluation criteria without special adjustments.

[0117] A unified and standardized evaluation system for heterogeneous RAG frameworks, achieving automated evaluation through a three-layer decoupled architecture, specifically including:

[0118] The orchestration layer is responsible for evaluation process management and strategy selection. It loads completed evaluation records through configuration management, supports breakpoint resume mechanism, and only performs incremental evaluation on incomplete test cases.

[0119] The adaptation layer handles heterogeneous framework interfaces uniformly through an independent API adapter class, including a mechanism for identifying and handling specific errors that already exist in files within search engine frameworks.

[0120] The evaluation layer establishes six mutually orthogonal evaluation dimensions to independently perform evaluations, and automatically generates capability profiles for each framework's strengths and weaknesses based on the evaluation results.

[0121] Furthermore, after the system completes the evaluation, it summarizes the evaluation results of all frameworks into a standardized comparison report, including:

[0122] Test case level records: Complete evaluation data for each test case, including test identifier, test type, question text, standard answer, actual answer in the RAG framework, six-dimensional evaluation score, source information detection results, upload success rate, and response time;

[0123] Dimensional aggregation: mean, median, standard deviation, extreme values, quartiles, and correlation matrix between dimensions for each dimension;

[0124] Performance bottleneck diagnosis: Automatically identify performance bottleneck types through dimensional combination patterns and output corresponding optimization suggestions;

[0125] Cross-frame / cross-model comparison table: Generates a comparison matrix between frames or models based on six-dimensional indicators, and ranks them according to the comprehensive score of the six dimensions and the score of each individual dimension;

[0126] Framework capability profile: Automatically generate a capability profile for each framework based on the evaluation results, identifying its strengths and weaknesses;

[0127] The results are saved in both structured data format and tabular format, and support breakpoint resume—loading completed evaluation records and performing evaluations only on incomplete test cases to avoid duplication of work.

[0128] Furthermore, the system supports two complementary evaluation modes, ensuring the comparability of results through strict variable control:

[0129] Cross-model comparison mode: The RAG framework is fixed, and the basic large language model is replaced one by one; the variables that remain unchanged in the evaluation include: the same test case set, the same knowledge base content, the same chunking strategy, the same upload process, and the same query method; the only variable that changes is the underlying large language model.

[0130] Cross-framework comparison mode: Fix the basic large language model and replace the RAG framework one by one; it is necessary to ensure that each framework obtains the same knowledge base content, and manual configuration of the framework requires uploading the same documents in advance and using the same test case set;

[0131] Both modes use the exact same six-dimensional metrics and evaluation process to ensure consistency in scoring standards; configuration management uses a JSON configuration file to unify the management framework connection parameters, including API address, authentication key, and workspace identifier, and supports switching evaluation targets without modifying the code.

[0132] To further illustrate the technical solution of the present invention, the following embodiments are used for further explanation:

[0133] Example 1: Cross-Frame Six-Dimensional Assessment Comparison

[0134] The method of this invention was used to perform a unified evaluation of four heterogeneous RAG frameworks (AnythingLLM, RAGFlow, MaxKB, and Metaso). The six-dimensional evaluation results of each framework under its optimal model configuration were taken as follows: Figure 2 As shown in Table 1.

[0135] Table 1. Average scores of the six-dimensional evaluation under the optimal configuration of each framework.

[0136]

[0137] As can be seen from the radar chart and data table, the six-dimensional evaluation system of this invention can clearly reveal the differentiated capability profiles of each framework:

[0138] AnythingLLM performed best in the context utilization dimension (0.889), indicating that its retrieval-generation pipeline has the highest information delivery efficiency; it also ranked highest in factuality (0.850) and completeness (0.850), reflecting the comprehensive advantages of a complete RAG-type framework.

[0139] RAGFlow performed best in the relevance dimension (0.855) and had the most balanced scores across the six dimensions (with the smallest standard deviation), indicating that the framework has no obvious weaknesses in any of its capabilities.

[0140] MaxKB and Metaso are relatively weak in two dimensions: technical depth (0.694 / 0.691) and support quality (0.714 / 0.707), revealing the structural limitations of manually configured and search engine-based frameworks in the accurate utilization of knowledge.

[0141] Example 2: Cross-model six-dimensional evaluation comparison

[0142] The AnythingLLM framework was used as the standard evaluation platform. Three different underlying large language models (DeepSeek-V3.2-Exp, Qwen2.5-72B-Instruct, and Gemini-2.5-Pro) were used, with the same 200 test cases and the same knowledge base for evaluation. The results are as follows: Figure 3 As shown in Table 2.

[0143] Table 2. Average scores of the six-dimensional evaluation for different models under the fixed framework.

[0144]

[0145] The above cross-model comparison results demonstrate the diagnostic value of the six-dimensional orthogonal assessment:

[0146] Significant differences in model capabilities: Gemini-2.5-Pro ​​leads by a large margin in all six dimensions, with an overall average score (0.837) that is 31.2% higher than DeepSeek-V3.2-Exp (0.638) and 39.3% higher than Qwen2.5-72B (0.601).

[0147] The context utilization dimension had the lowest discriminative power: the three models had the smallest score difference in this dimension (0.765-0.889, range 0.124), indicating that the AnythingLLM framework's retrieval pipeline provided relatively consistent context support for all models.

[0148] The quality of support dimension had the highest discriminative power: the score difference was the largest (0.541-0.814, range 0.273), indicating that the different models had the most significant differences in their ability to cite evidence and support arguments.

[0149] Example 3: Comprehensive Ranking Combining Multiple Frameworks and Models

[0150] All tested frameworks were ranked according to their combined model configurations based on the average score across six dimensions, as shown below. Figure 4 As shown in Table 3.

[0151] Table 3: Overall Ranking of All Configurations

[0152]

[0153] The comprehensive ranking data revealed the following key findings:

[0154] Model selection is more important than framework selection: within the same AnythingLLM framework, changing the model caused the overall score to change from 0.601 to 0.837, a range of 0.236; while different frameworks, under optimal configuration, had an overall score range of 0.741-0.837, with a range of only 0.096. This indicates that in RAG system deployment, the choice of the base model has a greater impact on the final performance than the framework itself.

[0155] Complete RAG-type frameworks occupy the top and bottom of the rankings: This seemingly contradictory phenomenon precisely verifies the diagnostic capability of the six-dimensional evaluation system—the performance of complete RAG-type frameworks is highly dependent on the quality of the underlying model, and choosing an inappropriate model will cause the framework's advantages to be completely unrealized.

[0156] The necessity of framework capability awareness: Manually configured (MaxKB) and search engine (Metaso) frameworks performed stably in the middle ranking range, indicating that the performance of these frameworks is relatively less affected by model selection, but there is still a performance ceiling.

[0157] The evaluation scale and system operation statistics are shown in Table 4.

[0158] Table 4. Statistics on the Operation of the Unified Evaluation System

[0159]

[0160] As can be seen from the system operation statistics in Table 4, the unified evaluation system of this invention demonstrates extremely high evaluation efficiency and scientific rigor in actual operation, specifically revealing the following key findings:

[0161] Heterogeneous compatibility and extensive evaluation scale: The system successfully covers four mainstream heterogeneous RAG frameworks (AnythingLLM, RAGFlow, MaxKB, and Metaso), and fully includes three different framework types: complete RAG, manually configured, and search engine. Combined with more than three large language models (such as DeepSeek-V3.2-Exp and Qwen2.5-72B) and 200 carefully selected test cases covering complex scenarios such as robustness, multi-hop inference, and generation quality, the system completed more than 10,000 six-dimensional evaluations. This demonstrates that the "adaptation layer" design of this invention possesses strong versatility and is capable of handling large-scale, multi-variable, and complex cross-evaluation tasks.

[0162] High efficiency with zero human intervention throughout the entire process: In tens of thousands of evaluation samples, the system required zero human intervention throughout the entire process, achieving complete automation. This fully verifies that the system can effectively replace traditional human expert evaluation, completely solving the fundamental defects mentioned in the background technology, such as high cost and excessive time consumption of manual scoring.

[0163] Absolute reproducibility of evaluation results: Statistical results clearly indicate that, under the premise of fixed random seeds and evaluation models, the evaluation results of this system are completely reproducible. Combined with six orthogonal dimensions of scientific measurement, this feature overcomes the subjectivity problem of varying scoring standards in traditional evaluations, providing an objective, stable, and scientific benchmark for the selection and performance diagnosis of RAG systems.

[0164] The above descriptions are merely preferred embodiments of this application, and the present invention is not limited to the above embodiments. It is understood that other improvements and variations directly derived or conceived by those skilled in the art without departing from the spirit and concept of the present invention should be considered to be included within the protection scope of the present invention.

Claims

1. A unified adaptation and standardized evaluation method for heterogeneous RAG frameworks, characterized in that, Includes the following steps: Step S1, Automatic Framework Capability Awareness: The system maintains the capability feature description of each supported RAG framework through the framework capability awareness matrix, and automatically loads the capability features of the target framework during evaluation initialization. Step S2, Unified Operation Interface Adaptation: Assign an independent API adapter to the target framework, and encapsulate the heterogeneous RAG framework into a unified operation interface that includes document upload, query and answer, and source information extraction through a pluggable framework adapter pattern. Step S3: Execution of intelligent upload strategy: Based on the framework capability awareness matrix and user configuration, automatically select and execute the corresponding document upload strategy; Step S4, Response Format Standardization and Parsing: Call the query and answer interface to obtain the raw response data, and use the framework-specific parsing logic to convert the heterogeneous response format into a unified four-tuple structure for standardized parsing; Step S5, Automated Evaluation and Precise Attribution: Establish six mutually orthogonal evaluation dimensions for automated evaluation based on a large language model evaluator; combine framework capability awareness with differentiated evaluation criteria, and map different failure modes to specific dimension combinations to achieve performance attribution; Step S6: Generate a comparative analysis report: Summarize the evaluation results into a standardized comparative report output, revealing the differentiated capability profiles between the frameworks.

2. The unified adaptation and standardized evaluation method for heterogeneous RAG frameworks according to claim 1, characterized in that, In step S1, the capability characteristics description of the supported RAG framework includes the following dimensions: Knowledge base upload support tags: Whether knowledge base documents can be uploaded via API; Default to skip upload tag: Whether to skip the upload step by default; Manually pre-configure tags: Whether the knowledge base needs to be manually pre-configured; Framework feature type tags: distinguish between full RAG type, manual configuration type, or search engine type; Upload strategy tags: distinguish between API upload support and manual upload required; When encountering an unregistered framework, the system automatically uses the default security setting that does not support knowledge base uploads and skips the upload step.

3. The unified adaptation and standardized evaluation method for heterogeneous RAG frameworks according to claim 1, characterized in that, In step S2, the definitions of each unified interface are as follows: Document upload interface: Input file path, return upload result including success or failure flags and detailed information; Question and Answer Interface: Input the question text and context information, and it returns the framework's answer text and raw response data; Source information extraction interface: Extracts source information from the raw response data and returns standardized source information detection results, including whether source information was found, the number of source information items, detailed content, and detection method identifier.

4. The unified adaptation and standardized evaluation method for heterogeneous RAG frameworks according to claim 1, characterized in that, In step S3, the specific process of the intelligent upload strategy includes the following steps: Step S31: Check if the user has explicitly specified skipping upload. If specified, the user setting is used directly. If not specified, the frame default value in the frame capability awareness matrix is ​​read. When skipping upload, it is assumed that the knowledge has been uploaded in advance. Step S32: When it is determined that an upload is required, a temporary document file is created for each gold context source file according to the context content, and the document upload interface of the corresponding framework adapter is called. Step S33: Confirm the upload result according to the success determination logic specific to the target framework, calculate the upload success rate, and wait for a preset time after the upload is completed to ensure that the vector index construction is completed; Step S34: Record all detailed error information of failed uploads into a failure list and output it to the evaluation report; continue to perform subsequent queries and evaluations when some uploads fail and mark the incomplete upload status in the results.

5. The unified adaptation and standardized evaluation method for heterogeneous RAG frameworks according to claim 1, characterized in that, In step S4, the heterogeneous response formats are converted into a unified four-tuple structure, which includes: whether source information was detected, the amount of source information, frame-specific details, and source detection method identifier; the parsing strategies for each frame are as follows: Complete RAG-type framework: Directly extract source information arrays and source counts from raw response data; Manually configured framework: First, check the explicit source information array. If it is empty, it will be downgraded to heuristic detection to extract the frequency of domain-specific terms and technical content features in the answers. When the knowledge usage index is greater than zero or the technical content feature is true, it is inferred that the framework uses knowledge base content. Search engine-type framework: Extracts an array of source information while retaining session identifiers and result identifiers for tracing.

6. The unified adaptation and standardized evaluation method for heterogeneous RAG frameworks according to claim 1, characterized in that, In step S5, the automated evaluation based on the large language model evaluator includes the following steps: Step S51: Generate a clear assessment objective description for each dimension, describing the specific RAG system capabilities assessed for that dimension and the corresponding 10-level scoring criteria; Step S52: Design structured reasoning steps for each dimension to guide the evaluation model to think along the process. The process includes: identifying key statements in the answers, comparing each statement with the source material one by one, marking statements that are consistent, inconsistent or unverifiable, and making a comprehensive judgment to give a score. Step S53: Call the evaluation model through the G-Eval module, extract the label unit level probability when the model generates scores, normalize and weight the output probabilities of candidate scores from 1 to 10 to obtain continuous score values. Step S54: Independently evaluate the RAG response for each test case across six dimensions, outputting a six-dimensional score vector. The six mutually orthogonal evaluation dimensions specifically include: Completeness: Assess the extent to which the answer covers all the necessary technical elements in the query; Factual: Assess the consistency between the response and the source material; detect hallucinations; Context utilization: Evaluates the actual proportion of the retrieved contextual content used in the response; Technical depth: Assessing the sophistication of the analysis and reasoning; Relevance: Assess the degree to which the answer focuses on the query, and detect off-topic content and information redundancy; Support quality: Assess the specificity and accuracy of the citations.

7. The unified adaptation and standardized evaluation method for heterogeneous RAG frameworks according to claim 1, characterized in that, In step S5, the following differentiated evaluation criteria are applied to the capability perception of each framework: Manually configurable framework: The system evaluates the knowledge usage in the answers using a knowledge usage metric heuristic method: it detects the frequency of domain-specific terms appearing in the answers and evaluates whether the answers contain specific data points that only appear in the knowledge base documents to infer the utilization of the knowledge base; Search engine-type frameworks allow for greater evaluation tolerance in terms of context utilization. Complete RAG-type framework: applying the standard six-dimensional evaluation criteria without special adjustments.

8. A unified adaptation and standardized evaluation system for heterogeneous RAG frameworks, characterized in that, The unified adaptation and standardization evaluation system for heterogeneous RAG frameworks is applicable to the unified adaptation and standardization evaluation method for heterogeneous RAG frameworks as described in any one of claims 1 to 7. The unified adaptation and standardization evaluation system for heterogeneous RAG frameworks achieves automated evaluation through a three-layer decoupled architecture, specifically including: The orchestration layer is responsible for evaluation process management and strategy selection. It loads completed evaluation records through configuration management, supports breakpoint resume mechanism, and only performs incremental evaluation on incomplete test cases. The adaptation layer handles heterogeneous framework interfaces uniformly through an independent API adapter class, including a mechanism for identifying and handling specific errors that already exist in files within search engine frameworks. The evaluation layer establishes six mutually orthogonal evaluation dimensions to independently perform evaluations, and automatically generates capability profiles for each framework's strengths and weaknesses based on the evaluation results.

9. The unified adaptation and standardized evaluation system for heterogeneous RAG frameworks according to claim 8, characterized in that, After the system completes the evaluation, it summarizes the evaluation results of all frameworks into a standardized comparison report, including: Test case level records: Complete evaluation data for each test case, including test identifier, test type, question text, standard answer, actual answer in the RAG framework, six-dimensional evaluation score, source information detection results, upload success rate, and response time; Dimensional aggregation: mean, median, standard deviation, extreme values, quartiles, and correlation matrix between dimensions for each dimension; Performance bottleneck diagnosis: Automatically identify performance bottleneck types through dimensional combination patterns and output corresponding optimization suggestions; Cross-frame / cross-model comparison table: Generates a comparison matrix between frames or models based on six-dimensional indicators, and ranks them according to the comprehensive score of the six dimensions and the score of each individual dimension; Framework capability profile: Automatically generate a capability profile for each framework based on the evaluation results, identifying its strengths and weaknesses; The results are saved in both structured data format and tabular format, and support breakpoint resume—loading completed evaluation records and performing evaluations only on incomplete test cases to avoid duplication of work.

10. The unified adaptation and standardized evaluation system for heterogeneous RAG frameworks according to claim 8, characterized in that, The system supports two complementary evaluation modes, and ensures the comparability of results through strict variable control: Cross-model comparison mode: The RAG framework is fixed, and the basic large language model is replaced one by one; the variables that remain unchanged in the evaluation include: the same test case set, the same knowledge base content, the same chunking strategy, the same upload process, and the same query method; the only variable that changes is the underlying large language model. Cross-framework comparison mode: Fix the basic large language model and replace the RAG framework one by one; it is necessary to ensure that each framework obtains the same knowledge base content, and manual configuration of the framework requires uploading the same documents in advance and using the same test case set; Both modes use the exact same six-dimensional metrics and evaluation process to ensure consistency in scoring standards; configuration management uses a JSON configuration file to unify the management framework connection parameters, including API address, authentication key, and workspace identifier, and supports switching evaluation targets without modifying the code.

Images