RAG system intelligent evaluation method and device based on large model, and computer equipment

Abstract

The invention discloses an intelligent evaluation method and device for an RAG system based on a large model, and computer equipment. The method comprises the following steps: acquiring a knowledge base source file required by a to-be-evaluated RAG system; analyzing and converting the knowledge base source file, creating a structured document block set, and executing preprocessing operation to obtain an intermediate knowledge base; automatically generating an evaluation data set including the question, the answer and the document block of the corresponding evidence according to the intermediate knowledge base based on LLM; performing performance evaluation on the to-be-evaluated RAG system by using an NLP evaluation index and an LLM-based multi-dimensional scoring mechanism based on the evaluation data set to obtain an evaluation result; and generating a report according to the evaluation result, and outputting the report. By implementing the method provided by the invention, the performance evaluation and iterative optimization of the RAG system can be comprehensively and accurately improved at low cost.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of financial risk control, and more particularly to an RAG system intelligent evaluation method and device based on a large model and a computer device. BACKGROUND

[0002] With the continuous progress of LLM (Large Language Model), RAG (Retrieval-Augmented Generation) has emerged as a key technology in the field of NLP (Natural Language Processing). RAG systems effectively solve the information loss problem caused by the traditional context memory limitation by combining external document retrieval with the powerful language generation function of LLM, and show excellent performance in complex application scenarios such as open domain question answering, knowledge-intensive tasks and multi-turn dialogue.

[0003] However, the current means of evaluating RAG systems have obvious limitations. Mainly reflected in the evaluation data generation efficiency is low, the data type is single, it is difficult to keep up with the pace of rapid iteration of technology, and it cannot fully cover the complex scenarios in the real world such as negative samples, multi-hop reasoning, etc. These are key elements to measure the performance of RAG systems. In addition, traditional evaluation indicators tend to focus on the surface quality of the generated text, and fail to accurately capture the subtle differences at the semantic level, which makes it necessary to rely on manual judgment, but this way is subjective and time-consuming, and cannot provide efficient, objective and comprehensive quantitative evaluation results.

[0004] In particular, when applying RAG technology in the professional field of financial risk control, the above challenges are more pronounced. Financial risk control involves a large number of highly specialized terms and information with strong context dependence, and the understanding of terms is extremely high. At the same time, due to the high sensitivity of financial data, strict data security and privacy protection regulations limit the authenticity and diversity of evaluation data. Therefore, evaluating RAG systems in the context of financial risk control needs to pay more attention to accuracy, robustness and reliability.

[0005] Although the existing RAG system evaluation method covers multiple links such as document parsing, data generation, automatic evaluation and report generation, there are still deficiencies in the specific implementation details and technical depth. For example, by receiving the original document data provided by the user, using NLP technology to divide it into multiple semantically complete document blocks, and then automatically generating questions and answers in different scenarios, and comprehensively evaluating through multiple evaluation indexes. However, the existing scheme mainly relies on expert manual writing or using LLM to automatically generate question and answer pairs in the data generation stage, which not only has high cost and low efficiency, but also the types of generated questions are not comprehensive enough to fully test the ability of RAG system to face complex and diverse problems in the real world.

[0006] Therefore, it is necessary to design a new method to realize low-cost, comprehensive and accurate performance evaluation and iterative optimization of RAG system. SUMMARY

[0007] The purpose of the present application is to overcome the defects of the prior art and provide a RAG system intelligent evaluation method and device based on a large model and a computer device.

[0008] To achieve the above purpose, the following technical solutions are adopted: a RAG system intelligent evaluation method based on a large model, comprising: Obtaining a knowledge base source file required by a RAG system to be evaluated; Parsing and converting the knowledge base source file and creating a structured document block set to perform a preprocessing operation to obtain an intermediate knowledge base; Based on LLM, automatically generating an evaluation dataset including document blocks of questions, answers and corresponding evidence based on the intermediate knowledge base; Based on the evaluation dataset, using NLP evaluation indexes and a multi-dimensional scoring mechanism based on LLM to evaluate the performance of the RAG system to be evaluated to obtain an evaluation result; Generating a report according to the evaluation result and outputting the report.

[0009] The further technical solutions are: the parsing and converting of the knowledge base source file and the creation of a structured document block set to perform a preprocessing operation to obtain an intermediate knowledge base, comprising: Processing structured or semi-structured documents and PDF documents with text layers in the knowledge base source file, and for scanned PDF or image format documents, using OCR technology, layout analysis and table recognition technology for processing, and performing preprocessing operations to obtain processed documents; Segmenting the processed documents into independent document blocks to obtain an intermediate knowledge base.

[0010] The further technical solutions are: each document block is associated with metadata information.

[0011] Further technical solutions are as follows: the structured or semi-structured documents in the knowledge base source file and the PDF documents with a text layer are processed, and for the scanned PDF or image format documents, the OCR technology, layout analysis and table recognition technology are used for processing, and preprocessing operations are performed to obtain the processed documents, including: The structured or semi-structured documents in the knowledge base source file and the PDF documents with a text layer are processed, and for the scanned PDF or image format documents, the OCR technology, layout analysis and table recognition technology are used for processing, and invalid content is removed, and sensitive information is desensitized to obtain the processed documents.

[0012] Further technical solutions are as follows: the evaluation data set including the document blocks of questions, answers and corresponding evidences is automatically generated based on the intermediate knowledge base according to the LLM, including: Setting prompt words; According to the prompt words, different types of questions and answers are generated to obtain data entries; According to the data entries and the intermediate knowledge base, an evaluation data set is generated.

[0013] Further technical solutions are as follows: the performance evaluation of the RAG system to be evaluated is performed based on the evaluation data set using the NLP evaluation index and the multi-dimensional scoring mechanism based on the LLM to obtain the evaluation result, including: Comparing the generated answers with the expected answers based on the evaluation data set, the text similarity is preliminarily quantified to obtain a first result; Combining the scoring system of the LLM, the RAG system output is analyzed and scored from multiple dimensions to obtain a scoring result; Calculate the recall rate and the precision rate; The content quality of the evaluation data set is evaluated by using the fact coincidence degree analysis and the grammar checking method to obtain a second result; The evaluation data set is input into the LLM, and the overall performance of the RAG system to be evaluated is comprehensively evaluated to obtain a third result; The first result, the scoring result, the recall rate and the precision rate, the second result and the third result are summarized to obtain the evaluation result.

[0014] Further technical solutions are as follows: the evaluation data set is compared with the expected answer based on the evaluation data set, and the text similarity is preliminarily quantified to obtain a first result, including: Using BLEU and ROUGE indexes, the generated answers are compared with the expected answers based on the evaluation data set, and the text similarity is preliminarily quantified to obtain a first result.

[0015] Further technical solutions are as follows: The method further comprises: generating a report according to the evaluation result, and outputting the report. According to the evaluation result, automatic data filling, chart rendering and intelligent analysis are performed through a preset report template to obtain a report, and the report is output.

[0016] The application also provides a large model-based RAG system intelligent evaluation device, which comprises: An acquisition unit is configured to acquire a knowledge base source file required by a RAG system to be evaluated; An analysis unit is configured to analyze and convert the knowledge base source file and create a structured document block set, and perform a preprocessing operation to obtain an intermediate knowledge base; A data set generation unit is configured to automatically generate an evaluation data set including document blocks of questions, answers and corresponding evidences based on the intermediate knowledge base according to an LLM; An evaluation unit is configured to perform performance evaluation of the RAG system to be evaluated based on the evaluation data set by using an NLP evaluation index and a multi-dimensional scoring mechanism based on an LLM to obtain an evaluation result; A report generation unit is configured to generate a report according to the evaluation result, and output the report.

[0017] The application also provides a computer device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the above method when executing the computer program.

[0018] Compared with the prior art, the application has the following beneficial effects: the application creates a structured intermediate knowledge base by automatically acquiring and processing a knowledge base source file, and automatically generates an evaluation data set containing questions, answers and evidences by using a large language model; then, the RAG system is comprehensively evaluated by using an NLP evaluation index and a multi-dimensional scoring mechanism based on an LLM, and finally, a detailed report is generated. This method realizes low-cost data processing and evaluation data set generation, improves the comprehensiveness and accuracy of evaluation, and provides a solid foundation and clear direction for iterative optimization of the system. This not only reduces labor costs and time consumption, but also ensures the depth and breadth of the evaluation result, thereby effectively improving the performance and reliability of the RAG system.

[0019] The application will be further described below in combination with the drawings and specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0021] Figure 1 The application scenario schematic diagram of the RAG system intelligent evaluation method based on a large model provided by the embodiments of the present application is shown in the figure. Figure 2 The flowchart of the RAG system intelligent evaluation method based on a large model provided by the embodiments of the present application is shown in the figure. Figure 3 The schematic block diagram of the RAG system intelligent evaluation device based on a large model provided by the embodiments of the present application is shown in the figure. Figure 4 The schematic block diagram of the computer device provided by the embodiments of the present application is shown in the figure. DETAILED DESCRIPTION

[0022] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.

[0023] It should be understood that when used in the present specification and the appended claims, the terms "comprise" and "include" indicate the presence of the described features, integers, steps, operations, elements, and / or components, but do not exclude one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0024] It should also be understood that the terms used in the present specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the present specification and the appended claims, the singular forms "a", "an" and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0025] It should be further understood that the term "and / or" used in the present specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0026] Please refer to Figure 1 and Figure 2 , Figure 1An application scenario schematic diagram of the intelligent evaluation method of the RAG system based on a large model provided by the embodiment of the present application. Figure 2 An illustrative flowchart of the intelligent evaluation method of the RAG system based on a large model provided by the embodiment of the present application. The intelligent evaluation method of the RAG system based on a large model is applied in a server, which interacts with a terminal for data, creates a structured intermediate knowledge base by acquiring and processing multi-source knowledge base files, automatically generates an evaluation data set including questions, answers and their evidence, and uses NLP evaluation indicators and a multi-dimensional scoring mechanism of LLM to evaluate the performance of the RAG system. This method uses OCR technology and layout analysis to process knowledge sources of different formats, ensuring the comprehensiveness and accuracy of the data; at the same time, by automatically generating evaluation data sets and reports, it reduces manual intervention and reduces costs. Combined with text similarity measurement, fact overlap analysis and grammar checking, etc. Various evaluation dimensions, precise quantification and comprehensive evaluation of RAG system output are realized, thereby supporting the iterative optimization of the system to realize efficient, low-cost and comprehensive improvement of the performance evaluation quality of the RAG system.

[0027] Figure 2 A flowchart of the intelligent evaluation method of the RAG system based on a large model provided by the embodiment of the present application. As shown in Figure 2 , the method comprises the following steps S110 to S150.

[0028] S110, acquire the knowledge base source files required for the RAG system to be evaluated.

[0029] In this embodiment, the knowledge base source file refers to the basic data set required for evaluating the performance of the RAG (Retrieval-Augmented Generation) system. These source files include but are not limited to PDF documents, DOCX documents, XLSX tables, and image files in various formats. They may contain structured, semi-structured or unstructured information, covering professional knowledge, business logic, rules and cases in specific fields such as financial risk control.

[0030] Step S110 is the starting point of the entire evaluation process, and its main task is to collect and organize real-world data that can reflect the working environment and application scenarios of the RAG system to be evaluated from the user. Specifically, this process involves the following aspects: Diversity: In order to ensure the comprehensiveness and representativeness of the evaluation results, the selected knowledge base source files should cover as many types and sources of data as possible. For example, in the field of financial risk control, this may include policy and regulation documents, market analysis reports, customer credit scoring model descriptions, historical transaction records, etc. In this way, various situations that the RAG system may encounter in future actual applications can be simulated.

[0031] Realism: The source files should originate from actual business scenarios, rather than fictional or idealized situations. This helps to accurately assess the performance of the RAG system in handling real-world problems, including its ability to understand complex contexts, search accuracy, and the quality of answer generation.

[0032] Security and Compliance: Given that many application scenarios, especially in the financial sector, involve sensitive information, it is crucial to strictly adhere to relevant privacy protection laws and industry standards when collecting and using these source files. This means that advanced technical means such as OCR, layout analysis, table recognition, and sensitive information desensitization must be used to ensure the safe flow of data and prevent the improper disclosure of any personal information.

[0033] Preprocessing Preparation: Although detailed parsing and conversion will be done in subsequent steps, preliminary screening and sorting are also needed in step S110 to remove obviously irrelevant or low-quality content, laying a good foundation for further processing.

[0034] In summary, step S110 is not just a simple data collection process, but also involves preliminary judgment of data quality and applicability, which is crucial for ensuring the smooth progress of subsequent steps and the effectiveness of the final evaluation results. By carefully selecting and preparing these knowledge base source files, a solid data support can be provided for the intelligent evaluation system based on LLM, thereby achieving accurate quantitative evaluation and continuous optimization of the RAG system.

[0035] S120, parsing and converting the knowledge base source files and creating a structured document block set, performing preprocessing operations to obtain an intermediate knowledge base.

[0036] In this embodiment, the intermediate knowledge base refers to the set of structured document blocks after processing. These document blocks not only retain the core information and logical structure of the original document, but also add rich metadata information such as document ID, location information, data sensitivity level, etc. The intermediate knowledge base provides a high-quality data source for the subsequent data generation module, supports automatic generation of evaluation data sets, and provides accurate search basis for the intelligent evaluation module.

[0037] In an embodiment, the above step S120 can include steps S121-S122.

[0038] S121, processing structured or semi-structured documents and PDF documents with text layers in the knowledge base source files, and for scanned PDF or image format documents, using OCR technology, layout analysis and table recognition technology, and performing preprocessing operations to obtain processed documents.

[0039] Specifically, structured or semi-structured documents in the knowledge base source file and PDF documents with a text layer are processed, and for scanned PDF or image format documents, OCR technology, layout analysis and table recognition technology are used for processing, and invalid content is removed, and sensitive information is desensitized to obtain processed documents.

[0040] In this embodiment, the processed document refers to clean text formed after parsing, conversion and preprocessing, which maintains the information integrity and accuracy of the original document while eliminating potential security risks.

[0041] Specifically, this step includes the following aspects: Document parsing: First, for structured (such as XLSX) or semi-structured (such as DOCX) documents, a special parser is called to read their internal structure or text stream to extract text content and its logical structure (such as paragraphs, titles, lists, etc.). For PDF documents containing optional text layers, similar parsing methods are also adopted.

[0042] Image document processing: For scanned PDF or image format documents, OCR technology is needed to recognize the text content, and layout analysis and table recognition technology are combined to accurately extract and reconstruct complex table structures. This step ensures that even paper reports or contract images can be effectively converted into usable text information.

[0043] Preprocessing operations: Remove invalid content: For example, non-key information such as headers and footers, page numbers, and advertisements.

[0044] Text cleaning: including removing extra spaces, unifying punctuation symbol formats, etc., to improve text quality.

[0045] Sensitive information desensitization: According to the strict requirements of the financial risk control field, use rules, pattern matching or deep learning models to identify and process personal identity information, account information, transaction details and other sensitive data to ensure information security.

[0046] S122, the processed document is segmented into independent document blocks to obtain an intermediate knowledge base.

[0047] In this embodiment, each of the document blocks is associated with metadata information.

[0048] Specifically, in this step, the system divides the document into several independent document blocks according to the natural paragraph boundaries and chapter hierarchical structure (such as titles, subtitles, etc.) of the document. Each document block is a relatively complete semantic unit, facilitating subsequent accurate retrieval and use.

[0049] While generating the document blocks, the system automatically associates them with corresponding metadata information, such as the unique identifier of the original document (document ID), the location information of the document block in the original document (page number and paragraph number), and labels specific to the financial risk control field (such as data sensitivity level, data source, data owner, etc.). These metadata help to achieve fine-grained access control, audit tracking, and provide important support for intelligent evaluation.

[0050] Through the above two steps, an intermediate knowledge base composed of multiple independent but interrelated document blocks is finally formed. This knowledge base not only lays the foundation for building high-quality evaluation data sets, but also provides a solid data guarantee for the continuous optimization of the RAG system.

[0051] Specifically, the goal of the above step S120 is to efficiently and accurately transform the user-submitted original documents into a structured set of document blocks. In the financial risk control field, which has extremely high requirements for data privacy, security, and compliance, the safe flow and effective management of information are emphasized throughout the data processing process, laying a solid and reliable data foundation for subsequent data generation and intelligent evaluation.

[0052] First, multiple formats of original documents, such as PDF, DOCX, XLSX, and image files, are processed. For structured or semi-structured documents (such as DOCX, XLSX), and PDF documents that internally contain optional text layers, a dedicated parser is called to directly read the internal structure or text stream of the document, efficiently extracting the text content while preserving logical structure information such as paragraphs, titles, and lists. For paper reports, contract images, and other scanned PDF or image format documents commonly found in the financial risk control field, this module integrates OCR (Optical Character Recognition), layout analysis, and table recognition technologies to intelligently identify document titles, paragraphs, and accurately extract and reconstruct complex table structures.

[0053] At this stage, necessary preprocessing operations are also performed, such as removing headers, footers, page numbers, advertisements, and other invalid or noise content, and performing text cleaning, such as removing extra spaces, and normalizing punctuation. For financial risk control data, the preprocessing stage will introduce sensitive information recognition and desensitization mechanisms (such as rule-based, pattern matching, or deep learning models), including but not limited to anonymization, pseudonymization, or partial masking of personal identity information, account information, transaction details, and other key sensitive data, to ensure that sensitive information protection is completed before the data flows into the knowledge base. This series of operations aims to preserve the core information of the document and improve the accuracy and compliance of subsequent processing.

[0054] Subsequently, the document enters the chunking phase. Based on the natural paragraph boundaries of the document, the chapter hierarchy (such as titles, subtitles, etc.), the document is divided into several independent document blocks (such as paragraphs, all content under subheadings, etc.), ensuring that each document block is a relatively complete semantic unit, facilitating subsequent accurate retrieval.

[0055] At the same time of document block generation, the system automatically associates each document block with its corresponding metadata information, including but not limited to the unique identifier of the original document (document ID), the location information of the document block in the original document (page number and paragraph number), and the data labels for the financial risk control field (such as data sensitivity level, data source, data owner, etc.). Among them, the location information of the document block forms a unique chunking number, which provides key support for subsequent fine access control, audit tracking, and intelligent evaluation. The processed document blocks are represented in structured JSON.

[0056] Finally, all the document blocks and their metadata processed above are stored in a structured manner, together building a queryable document knowledge base. At the storage level, strict adherence to the data storage security standards of the financial industry, combined with multi-level access control strategies, ensures the security and compliance of the document knowledge base data throughout its lifecycle. This not only guarantees the effectiveness and security of the data, but also provides a reliable foundation for subsequent data generation and intelligent evaluation. In this way, the entire process ensures that the conversion process from the original document to the structured document block set is both efficient and secure, meeting the high standards of data privacy and security in the financial risk control field.

[0057] S130, automatically generating an evaluation dataset including document blocks of questions, answers and corresponding evidence based on the intermediate knowledge base according to the LLM.

[0058] In this embodiment, the evaluation dataset refers to a collection composed of a series of structured questions, answers and evidence document blocks supporting the answers, aiming to comprehensively evaluate the performance of the RAG system.

[0059] In an embodiment, the above step S130 can include steps S131-S133.

[0060] S131, setting a prompt word.

[0061] In this embodiment, first, to guide the LLM to generate specific types of questions and answers, it is necessary to carefully design the prompt word (Prompt Engineering). These prompts aim to clearly indicate the content direction of the LLM's attention, the expected answer length (short answer or long answer), and the source of the answer (no source, single document block or multiple document blocks). Through customized prompts, it can be ensured that the generated data entries are both diverse and targeted.

[0062] Setting appropriate prompt words is a key step to ensure the quality of generated questions and answers. It directly affects the relevance, accuracy, and practicality of the final generated data.

[0063] S132, generating different types of questions and answers according to the prompt words to obtain data entries.

[0064] In this embodiment, data entries refer to specific units contained in the evaluation data set, consisting of a specific question, its corresponding answer, and supporting evidence document blocks.

[0065] Based on the prompt words set in step S131, the powerful context understanding and text generation functions of LLM are used to generate questions and their corresponding answers covering different difficulties and complexities. These questions may include negative sample questions, single-block short / long answer questions, multi-block short / long answer questions, etc., as described in the previous section. Each type of question aims to test the performance of the RAG system in different scenarios.

[0066] This step ensures that the generated data entries can comprehensively cover various evaluation needs, from simple information retrieval to complex logical reasoning, thereby providing a comprehensive basis for evaluating the performance of the RAG system.

[0067] S133, generating an evaluation data set according to the data entries and the intermediate knowledge base.

[0068] In this embodiment, in this step, the data entries generated in the previous step are combined with relevant document blocks in the intermediate knowledge base to further construct a complete evaluation data set. Each data entry not only contains questions and answers, but also includes supporting evidence document blocks, ensuring that all data has clear sources and basis. This data set follows a unified JSON format, facilitating subsequent processing and analysis.

[0069] Building a structured evaluation data set is the core of the entire process, which provides a solid data foundation for the intelligent evaluation module, enabling the system to perform efficient, objective, and comprehensive quantitative evaluation.

[0070] Through the execution of the above steps S131 to S133, this embodiment can automatically generate evaluation data sets suitable for the financial risk control field, not only improving the efficiency of evaluation data generation, but also greatly enriching the diversity and depth of data, which helps to more accurately evaluate the performance of the RAG system and promote its continuous optimization. This process is of great significance in solving the limitations of traditional methods in generating evaluation data.

[0071] Step S120 of the embodiment focuses on automatically generating synthetic data containing questions, answers, and evidence document blocks using a large language model (LLM) to build a diverse, high-quality, and comprehensive evaluation dataset covering complex scenarios in the field of financial risk control. This module fully leverages the powerful capabilities of LLM in context understanding, logical reasoning, and text generation, and uses the structured document knowledge base processed by the file parsing module as the basic data source. Through customized prompt instructions (Prompt Engineering), various properties of synthetic data can be flexibly controlled, including answer sources and answer lengths.

[0072] Specifically, the data generation module can generate multiple types of questions: Negative sample questions: These questions do not have direct or indirect answers in the provided document blocks, used to test the RAG system's "refusal" ability and anti-illusion ability when facing irrelevant information, unsolvable problems, or needing to refuse to answer.

[0073] Single-block short answer questions: Require the RAG system to perform precise information retrieval from a single document block and generate a short, factual answer, directly testing its basic retrieval and information extraction capabilities.

[0074] Single-block long answer questions: Require the RAG system to generate a coherent long text answer based on understanding a single paragraph, further testing its understanding ability and text generation ability of a single evidence source.

[0075] Multi-block short answer questions: Require the RAG system to integrate information from multiple different document blocks to form a final short answer, focusing on its cross-document information integration and filtering capabilities.

[0076] Multi-block long answer questions: The most challenging type of question, requiring the RAG system not only to retrieve and integrate information from multiple document blocks, but also to perform complex logical reasoning, comparative analysis, or comprehensive judgment based on this, ultimately forming a long answer with clear structure and sufficient argumentation, fully evaluating its comprehensive performance and reasoning limits when handling complex, multi-source information queries.

[0077] During actual generation, different generation strategies are used for optimization according to different types of questions. For single-block short / long answer questions, a simplified and explicit constraint prompt is used to guide the LLM to focus on a single specified document block, efficiently generating basic evaluation data in batches. For multi-block short / long answer questions, higher intelligence and complexity are demonstrated, with the system first guiding the LLM to perform deep semantic analysis and relevance identification on the input candidate document block set, actively identifying and filtering multiple blocks with specific logical relationships, and then generating comprehensive, deep, and complex questions based on these association relationships.

[0078] To ensure the standardization and usability of the generated data, all data follows a unified JSON format, clearly defining fields such as question type, question, answer, and evidence document blocks, facilitating automated parsing, retrieval, and source analysis by the subsequent evaluation framework. To ensure the high quality, accuracy, and internal consistency of the LLM-generated data, the module also employs multiple quality assurance mechanisms, including using a verification prompt to guide the LLM in self-checking and a small number of manual spot checks to calibrate the generation process.

[0079] Each system group must contain at least two distinct document blocks. The same document block may have different or the same associations with multiple other document blocks, but the same document block should not appear repeatedly within the same group. The output must strictly adhere to the JSON data format described above.

[0080] S140. Based on the evaluation dataset, NLP evaluation metrics and an LLM-based multidimensional scoring mechanism are used to evaluate the performance of the RAG system to be evaluated, so as to obtain the evaluation results.

[0081] In this embodiment, the evaluation result refers to the final performance evaluation score or rating that summarizes multiple aspects such as text similarity, multi-dimensional scoring, recall and precision, factual overlap analysis, and comprehensive evaluation.

[0082] In one embodiment, step S140 described above may include steps S141 to S146.

[0083] S141. Based on the evaluation dataset, compare the generated answer with the expected answer to preliminarily quantify the text similarity and obtain the first result.

[0084] In this embodiment, the generated answer is first compared with the expected answer to initially quantify text similarity. This stage mainly focuses on the degree of consistency between the generated answer and the expected answer at the lexical or semantic level. The first result typically includes one or more numerical indicators (such as BLEU, ROUGE, etc.), which can reflect the similarity between the generated answer and the expected answer.

[0085] Specifically, the BLEU and ROUGE metrics are used to compare the generated and expected answers based on the evaluation dataset to initially quantify text similarity and obtain the first result.

[0086] S142. Combining the LLM scoring system, analyze and score the RAG system output from multiple dimensions to obtain the scoring results.

[0087] In this embodiment, the scoring result refers to the quantitative evaluation of the RAG system output from multiple dimensions such as accuracy, completeness, fidelity, illusion rate, and fluency, based on the LLM scoring system.

[0088] Next, the LLM scoring system is used to analyze and score the RAG system output from multiple dimensions. This step not only considers the accuracy of the answers, but also includes the completeness, fidelity, hallucination rate, and fluency of the answers. In this way, a score reflecting the performance of the RAG system can be obtained, providing an important basis for subsequent comprehensive evaluation.

[0089] S143, calculate the recall rate and precision rate.

[0090] In this embodiment, the recall rate refers to the proportion of the number of relevant documents successfully retrieved to the total number of actual relevant documents in the retriever evaluation, measuring the ability of the retriever to find all relevant information.

[0091] The precision rate refers to the proportion of the number of relevant documents retrieved to the total number of documents retrieved in the retriever evaluation, reflecting the relevance of the results returned by the retriever.

[0092] Recall rate and precision rate are key indicators of retriever performance. Recall rate refers to the ability of the retriever to successfully find all relevant documents, while precision rate reflects how many of the results returned by the retriever are truly relevant. These two indicators are crucial for evaluating the effectiveness of the RAG system in information retrieval.

[0093] S144, evaluate the content quality of the evaluation dataset using fact overlap analysis and grammar checking methods to obtain a second result.

[0094] In this embodiment, the second result refers to the quality evaluation result of the fact accuracy and language standardization of the generated answers through fact overlap analysis and grammar checking.

[0095] In this step, the content quality of the evaluation dataset is evaluated using fact overlap analysis and grammar checking. The second result aims to verify whether the fact statements in the generated answers can be effectively supported by the retrieved information, and to ensure that the answers conform to basic language norms and logical coherence.

[0096] S145, input the evaluation dataset into the LLM, and comprehensively evaluate the overall performance of the RAG system to be evaluated to obtain a third result.

[0097] In this embodiment, the third result refers to the comprehensive score or rating obtained by inputting the evaluation dataset into the LLM and comprehensively considering the overall performance of the RAG system to be evaluated in information retrieval, answer generation, etc.

[0098] Subsequently, the evaluation dataset is input into the LLM to comprehensively evaluate the overall performance of the entire RAG system to be evaluated. The third result should be a comprehensive score or rating that reflects the overall performance of the RAG system in handling complex queries, including but not limited to its efficiency in information retrieval, answer generation, etc.

[0099] S146, aggregating the first result, the scoring result, the recall and precision, the second result, and the third result to obtain an evaluation result.

[0100] Finally, the first result, the scoring result, the recall and precision, the second result, and the third result obtained in the previous steps are aggregated to form the final evaluation result. This comprehensive evaluation result not only objectively reflects the performance level of the RAG system to be evaluated, but also provides valuable feedback for subsequent system optimization and improvement.

[0101] Through the above steps, the embodiment provides a comprehensive and detailed evaluation method that can effectively improve the performance evaluation accuracy of the RAG system and help the continuous optimization and development of the system.

[0102] In the embodiment, the step S140 described above comprehensively evaluates the retriever, the generator, and the overall performance of the RAG system by integrating industry-standard natural language processing (NLP) evaluation indicators and a multi-dimensional intelligent scoring mechanism based on a large language model (LLM). This module effectively addresses the evaluation challenges of the RAG system in terms of factual accuracy, hallucination rate, and complex reasoning ability. The unique professional terminology system and strong context dependency in the financial risk control field pose challenges to general language models. Therefore, the model is fine-tuned using a large amount of internally accumulated risk control business data to improve the depth and breadth of the model's understanding of complex financial expressions and optimize the accuracy and robustness of semantic similarity calculation, thereby alleviating the impact of domain terminology barriers on the performance of the evaluation system.

[0103] Specifically, the following content is included: Retriever evaluation: This part focuses on whether the RAG system can efficiently and accurately retrieve the most relevant document blocks from the knowledge base to the user's question. The semantic similarity between the retrieval results and the question is calculated to evaluate the context relevance; Recall@K and Precision@K are used to measure the ability of the retriever to identify all relevant documents and the proportion of truly relevant documents among them, respectively; NDCG and MRR are used to evaluate the quality of the retrieval result ranking.

[0104] Generator Evaluation: Focuses on the ability of RAG systems to generate high-quality answers based on retrieved information. Fact accuracy of generated answers is verified through fact overlap analysis and keyword matching; relevance of answers is measured using semantic similarity calculations; and readability of answers is evaluated using pre-set grammar check rules and traditional metrics such as BLEU and ROUGE.

[0105] Joint Evaluation: To simulate human evaluators' deep semantic evaluation, this module introduces a multi-dimensional intelligent scoring mechanism based on LLM. LLM, as the evaluation subject, comprehensively judges the question, retrieved information, and generated answer, and outputs multiple quantitative indicators: answer accuracy, evaluating the degree of response and semantic relevance of the generated answer to the user's question; answer completeness, measuring the degree of utilization and semantic relevance of the generated answer to the retrieved information; answer fidelity, evaluating the degree of fidelity of the factual content in the generated answer to the retrieved information, and determining whether the factual statements in the answer can be supported by the retrieved information; illusion rate, quantifying false information in the answer that cannot be inferred from the retrieved information or known facts; answer fluency, evaluating the grammatical, syntactic, logical coherence, and overall readability of the generated answer. These indicators collectively provide a comprehensive quantitative evaluation of RAG system performance, with a particular focus on LLM's ability to judge semantic depth and factual accuracy. In addition, this module also introduces perplexity (PPL). Perplexity is calculated based on the model's prediction probability for each word in the text sequence. The lower the perplexity, the stronger the model's prediction ability for the text sequence, i.e., the higher the probability assigned to the sequence, considering it to be more "determined" and "natural".

[0106] ; where H(W) represents the cross-entropy, measuring the difference between the model's output distribution and the true distribution. The lower the cross-entropy, the closer the model's prediction to the true distribution. N represents the number of words in the sequence. wi represents the i-th word in the sequence. P(wi) represents the probability of the model predicting the i-th word given the first i-1 words in the sequence.

[0107] Here it is innovatively used to evaluate the impact of retrieval augmentation on the "confidence" of the generation model. Specifically, by comparing the perplexity of the same base generation model generating answers with retrieval augmentation and without retrieval augmentation, it quantitatively analyzes whether retrieval information effectively reduces the model's generation entropy (i.e., the model's generation uncertainty). A lower perplexity indicates that the model, with the aid of external retrieval information, has a more determined choice for the generated content, showing a higher "confidence", thus effectively proving the positive role and value of retrieval results on the generated answer. The greater the decline, the better the effect of retrieval augmentation, and the more "confident" the model is in generating answers.

[0108] Meanwhile, the average response speed and P95 / P99 latency of the RAG system are calculated to measure the response speed of the system. Finally, manual evaluation can be used as a supplement and verification method for the automated evaluation system, and samples with specific difficult problems or doubts about the results of automated evaluation are sampled for review. By collecting human verification results, a human preference dataset can be constructed for model fine-tuning to better match human judgment standards, providing a more reliable reference benchmark for automated evaluation.

[0109] Comprehensive diagnosis: The various quantitative indicators at different levels are combined to generate a comprehensive score reflecting the overall performance of the system. This not only provides a comprehensive and operational evaluation, but also allows for adjusting the weights of each indicator according to the needs of specific application scenarios to achieve specific optimization goals. At the same time, manual evaluation is used as an important supplement to the automated evaluation system, and through manual review of difficult problems or samples with doubts about the results of automated evaluation, the model is further calibrated to better meet human judgment standards.

[0110] S150, generating a report according to the evaluation results and outputting the report.

[0111] In this embodiment, according to the evaluation results, automatic data filling, chart rendering and intelligent analysis are performed through a pre-set report template to obtain a report, and the report is output.

[0112] The evaluation results described above are an important part of the highly readable, structured and clear visualized report. The core goal is to provide users with intuitive and insightful decision support.

[0113] First, two types of key data are received from the intelligent evaluation module: calculated evaluation indicator scores and multi-dimensional evaluation scores based on large language models (LLM). The received data is cleaned, formatted and missing value processed to ensure data quality and accuracy. Then, through a pre-set report template, data is automatically filled, charts are rendered and intelligent analysis is performed. The entire process does not require human intervention, ensuring the consistency, accuracy and objectivity of each report, thereby providing users with reliable and trustworthy decision-making basis The source data of this embodiment comes from user-uploaded files, which have the characteristics of format diversification and field specificity. The target data generated includes: Structured knowledge document library: such data can be directly used as the core knowledge source of the RAG system.

[0114] High-quality evaluation dataset: can be used for model training, fine-tuning, etc.

[0115] The embodiment can automatically generate multi-type and high-quality evaluation data, covering complex scenarios such as negative samples, single / multi-blocks, and short / long answers that traditional methods cannot reach, thereby obtaining a more comprehensive and more practical application scenario evaluation dataset at a low cost. In addition, by combining traditional evaluation indicators with LLM intelligent scoring mechanisms, the embodiment can automatically evaluate the key sub-modules and overall performance of the RAG system, comprehensively and objectively evaluate the RAG system through multi-dimensional quantitative evaluation indicators, accurately locate performance bottlenecks, and assist the iterative progress of the RAG system.

[0116] The core of the embodiment is to convert raw unstructured documents into high-quality datasets that can be used for automated evaluation through a series of modular processing flows, and finally generate intuitive and insightful reports to support the optimization of the RAG (Retrieval-Augmented Generation) system. The specific process is as follows: First, the user uploads the original unstructured document to the associated metadata document block. In this process, not only is the document structured, but the data security is also enhanced through sensitive information identification and desensitization mechanisms. This mechanism can effectively protect the privacy and sensitive information of users, ensuring that there is no leakage or abuse in subsequent data processing. The processed document forms a structured document knowledge base, which serves as the basis for subsequent module input.

[0117] Based on the structured document knowledge base formed above, the powerful semantic understanding and generation capabilities of large language models (LLM) are used to automatically generate multi-type and high-quality question and answer pairs. This method solves the problem of low efficiency and lack of data diversity in traditional evaluation data generation. LLM can understand the content of the document and generate test cases covering various complex scenarios, including but not limited to negative samples, single / multi-blocks, short / long answers, etc., thereby greatly enriching the content of the evaluation dataset and improving the comprehensiveness and accuracy of the evaluation.

[0118] Combining traditional evaluation indicators and LLM-based intelligent scoring mechanisms, the performance of the RAG system on a given evaluation dataset is comprehensively evaluated. Through this comprehensive evaluation method, not only can the performance of the RAG system be objectively quantified, but also its strengths and weaknesses in different dimensions can be analyzed in depth, accurately locating possible bottleneck problems. This approach provides an efficient, objective, and comprehensive quantitative evaluation standard, providing a clear direction for subsequent system improvement.

[0119] Finally, all evaluation results from the intelligent evaluation module are received and integrated into visual reports that are easy to understand. These reports not only show the specific scores of the evaluation, but also provide deeper analysis and insights through charts and other forms, providing users with a window to intuitively understand the performance of the RAG system. In addition, the reports provide data support for the continuous optimization of the system, helping users to develop reasonable improvement strategies based on actual evaluation results.

[0120] In summary, the embodiment provides a complete solution from document parsing, data generation, intelligent evaluation to report generation through innovative modular design, aiming to improve the evaluation efficiency and quality of the RAG system, while ensuring data security and promoting the continuous optimization and development of the system. Each step in this process is closely connected and together forms an efficient automated evaluation system, solving many challenges in the existing technology.

[0121] The above-mentioned intelligent evaluation method for RAG system based on large model creates a structured intermediate knowledge base by automatically acquiring and processing knowledge base source files, and automatically generates an evaluation data set containing questions, answers and their evidence using a large language model. Then, the RAG system is comprehensively evaluated using natural language processing evaluation indicators and a multi-dimensional scoring mechanism based on LLM, and finally a detailed report is generated. This method realizes low-cost data processing and evaluation data set generation, while improving the comprehensiveness and accuracy of the evaluation, providing a solid foundation and clear direction for the iterative optimization of the system. This not only reduces labor costs and time consumption, but also ensures the depth and breadth of the evaluation results, effectively improving the performance and reliability of the RAG system.

[0122] Figure 3 is a schematic block diagram of an intelligent evaluation device 300 for a RAG system based on a large model provided by an embodiment of the present application. As shown in Figure 3 corresponding to the above-mentioned intelligent evaluation method for RAG system based on large model, the present application further provides an intelligent evaluation device 300 for RAG system based on large model. The device for executing the above-mentioned intelligent evaluation method for RAG system based on large model can be configured in a server. Specifically, please refer to Figure 3 , the intelligent evaluation device 300 for RAG system based on large model includes an acquisition unit 301, an analysis unit 302, a data set generation unit 303, an evaluation unit 304 and a report generation unit 305.

[0123] The acquisition unit 301 is configured to acquire a knowledge base source file required by a RAG system to be evaluated; the parsing unit 302 is configured to parse and convert the knowledge base source file and create a structured document block set, perform a preprocessing operation, and obtain an intermediate knowledge base; the data set generation unit 303 is configured to automatically generate an evaluation data set including a document block of a question, an answer and corresponding evidence based on the intermediate knowledge base according to an LLM; the evaluation unit 304 is configured to perform performance evaluation of the RAG system to be evaluated based on the evaluation data set by using an NLP evaluation index and a multi-dimensional scoring mechanism based on an LLM, to obtain an evaluation result; and the report generation unit 305 is configured to generate a report according to the evaluation result and output the report.

[0124] In an embodiment, the parsing unit 302 comprises: A processing subunit is configured to process structured or semi-structured documents and PDF documents with a text layer in the knowledge base source file, and for scanned PDF or image format documents, adopt an OCR technology, a layout analysis and a table recognition technology for processing, and perform a preprocessing operation to obtain processed documents; and a segmentation subunit is configured to segment the processed documents into independent document blocks to obtain an intermediate knowledge base.

[0125] In an embodiment, the processing subunit is configured to process structured or semi-structured documents and PDF documents with a text layer in the knowledge base source file, and for scanned PDF or image format documents, adopt an OCR technology, a layout analysis and a table recognition technology for processing, and remove invalid content and perform desensitization processing on sensitive information to obtain processed documents.

[0126] In an embodiment, the data set generation unit 303 comprises: A setting subunit is configured to set a prompt word; a first generation subunit is configured to generate different types of questions and answers according to the prompt word to obtain data entries; and a second generation subunit is configured to generate an evaluation data set according to the data entries and the intermediate knowledge base.

[0127] In an embodiment, the evaluation unit 304 comprises: The first evaluation subunit is configured to preliminarily quantify the text similarity by comparing the generated answer with the expected answer based on the evaluation data set, to obtain a first result; the second evaluation subunit is configured to analyze and score the RAG system output from multiple dimensions in combination with the LLM scoring system, to obtain a scoring result; the third evaluation subunit is configured to calculate the recall rate and the precision rate; the fourth evaluation subunit is configured to evaluate the content quality of the evaluation data set by using the fact coincidence degree analysis and the grammar checking method, to obtain a second result; the fifth evaluation subunit is configured to input the evaluation data set into the LLM, and comprehensively evaluate the overall performance of the RAG system to be evaluated, to obtain a third result; and the aggregation subunit is configured to aggregate the first result, the scoring result, the recall rate and the precision rate, the second result and the third result, to obtain an evaluation result.

[0128] In an embodiment, the first evaluation subunit is configured to preliminarily quantify the text similarity by comparing the generated answer with the expected answer based on the evaluation data set using the BLEU and ROUGE indicators, to obtain a first result.

[0129] In an embodiment, the report generation unit 305 is configured to automatically fill data, render charts and intelligently analyze by using a preset report template according to the evaluation result, to obtain a report, and output the report.

[0130] It should be noted that the specific implementation process of the above-mentioned RAG system intelligent evaluation device 300 based on a large model and each unit can be clearly understood by those skilled in the art, and can refer to the corresponding description in the foregoing method embodiments. For the convenience and brevity of description, it will not be repeated here.

[0131] The above-mentioned RAG system intelligent evaluation device 300 based on a large model can be realized in the form of a computer program, which can run on a computer device as shown in the computer device. Figure 4

[0132] Please refer to Figure 4 , Figure 4 is a schematic block diagram of a computer device provided by an embodiment of the present application. The computer device 500 can be a server, wherein the server can be a stand-alone server or a server cluster composed of multiple servers.

[0133] Referring to Figure 4 , the computer device 500 includes a processor 502, a memory and a network interface 505 connected through a system bus 501, wherein the memory can include a non-volatile storage medium 503 and an internal memory 504.

[0134] ​The non-volatile storage medium 503 can store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions that, when executed, cause the processor 502 to perform a large model-based RAG system intelligent evaluation method.

[0135] The processor 502 is configured to provide computing and control capabilities to support the operation of the entire computer device 500.

[0136] The memory 504 provides an environment for the computer program 5032 in the non-volatile storage medium 503 to run, which, when executed by the processor 502, causes the processor 502 to perform a large model-based RAG system intelligent evaluation method.

[0137] The network interface 505 is configured to communicate with other devices over a network. Those skilled in the art can understand that the network interface 505 can be configured to communicate with other devices over a network according to the structure shown in the figure, and the specific implementation is not limited herein. Figure 4 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device 500 to which the scheme of the present application is applied. The specific computer device 500 can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0138] The processor 502 is configured to run the computer program 5032 stored in the memory to implement the following steps: Obtain the knowledge base source file required for the RAG system to be evaluated; parse and convert the knowledge base source file and create a structured document block set, perform preprocessing operations to obtain an intermediate knowledge base; automatically generate an evaluation dataset including questions, answers and corresponding evidence based on the intermediate knowledge base based on LLM; perform performance evaluation of the RAG system to be evaluated based on the evaluation dataset using NLP evaluation indicators and a multi-dimensional scoring mechanism based on LLM to obtain an evaluation result; generate a report according to the evaluation result and output the report.

[0139] In an embodiment, when implementing the step of parsing and converting the knowledge base source file and creating a structured document block set, and performing preprocessing operations to obtain an intermediate knowledge base, the processor 502 specifically implements the following steps: The structured or semi-structured documents in the knowledge base source file and the PDF documents with text layers are processed, and for scanned PDF or image format documents, OCR technology, layout analysis and table recognition technology are used for processing, and preprocessing operations are performed to obtain processed documents; the processed documents are segmented into independent document blocks to obtain an intermediate knowledge base.

[0140] Each of the document blocks is associated with metadata information.

[0141] In an embodiment, the processor 502, in implementing the step of processing structured or semi-structured documents and PDF documents with text layers in the knowledge base source files, and for scanned PDF or image format documents, using OCR technology, layout analysis and table recognition technology, and performing preprocessing operations to obtain the processed documents, implements the following steps: Processing structured or semi-structured documents and PDF documents with text layers in the knowledge base source files, and for scanned PDF or image format documents, using OCR technology, layout analysis and table recognition technology, and removing invalid content, and performing desensitization processing on sensitive information, to obtain the processed documents.

[0142] In an embodiment, the processor 502, in implementing the step of automatically generating an evaluation dataset including document blocks of questions, answers and corresponding evidence based on the intermediate knowledge base according to the LLM, implements the following steps: Setting prompt words; generating different types of questions and answers according to the prompt words to obtain data entries; and generating an evaluation dataset according to the data entries and the intermediate knowledge base.

[0143] In an embodiment, the processor 502, in implementing the step of performing performance evaluation of the RAG system to be evaluated based on the evaluation dataset using NLP evaluation indicators and a multi-dimensional scoring mechanism based on the LLM to obtain an evaluation result, implements the following steps: Preliminarily quantifying the text similarity based on the evaluation dataset by comparing the generated answers with the expected answers to obtain a first result; scoring the RAG system output from multiple dimensions based on the scoring system of the LLM to obtain a scoring result; calculating the recall rate and the precision rate; evaluating the content quality of the evaluation dataset using fact coincidence degree analysis and grammar checking methods to obtain a second result; inputting the evaluation dataset into the LLM to comprehensively evaluate the overall performance of the RAG system to be evaluated to obtain a third result; and aggregating the first result, the scoring result, the recall rate and the precision rate, the second result and the third result to obtain the evaluation result.

[0144] In an embodiment, the processor 502, in implementing the step of preliminarily quantifying the text similarity based on the evaluation dataset by comparing the generated answers with the expected answers to obtain a first result, implements the following steps: Using BLEU and ROUGE indicators to preliminarily quantify the text similarity based on the evaluation dataset by comparing the generated answers with the expected answers to obtain a first result.

[0145] In an embodiment, the processor 502, when implementing the step of generating a report according to the evaluation result and outputting the report, specifically implements the following steps: According to the evaluation result, the report is obtained through automatic data filling, chart rendering and intelligent analysis based on a preset report template, and the report is output.

[0146] It should be understood that, in the embodiments of the present application, the processor 502 can be a central processing unit (CPU), and the processor 502 can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0147] It can be understood by those skilled in the art that all or part of the processes in the method of implementing the above embodiments can be completed by a computer program instructing related hardware. The computer program includes program instructions, and the computer program can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the above-mentioned embodiments of the method.

[0148] Therefore, the present application also provides a storage medium. The storage medium can be a computer-readable storage medium. The storage medium stores a computer program, wherein the computer program is executed by a processor to make the processor execute the following steps: obtaining a knowledge base source file required for evaluating the RAG system; performing parsing conversion on the knowledge base source file and creating a structured document block set, performing a preprocessing operation to obtain an intermediate knowledge base; automatically generating an evaluation dataset including a document block of a question, an answer and corresponding evidence based on the intermediate knowledge base according to the LLM; performing performance evaluation of the RAG system to be evaluated based on the evaluation dataset using NLP evaluation indexes and a multi-dimensional scoring mechanism based on the LLM to obtain an evaluation result; generating a report according to the evaluation result and outputting the report.

[0149] In an embodiment, the processor, when executing the computer program to implement the step of performing parsing conversion on the knowledge base source file and creating a structured document block set, performing a preprocessing operation to obtain an intermediate knowledge base, specifically implements the following steps: processing structured or semi-structured documents and PDF documents with text layers in the knowledge base source file, and for scanned PDF or image format documents, using OCR technology, layout analysis and table recognition technology for processing, and performing preprocessing operations to obtain processed documents; segmenting the processed documents into independent document blocks to obtain an intermediate knowledge base.

[0150] Each of the document blocks is associated with metadata information.

[0151] In an embodiment, when the processor implements the step of processing structured or semi-structured documents and PDF documents with text layers in the knowledge base source file, and for scanned PDF or image format documents, using OCR technology, layout analysis and table recognition technology for processing, and performing preprocessing operations to obtain processed documents, the processor implements the following steps: processing structured or semi-structured documents and PDF documents with text layers in the knowledge base source file, and for scanned PDF or image format documents, using OCR technology, layout analysis and table recognition technology for processing, and removing invalid content, and performing desensitization processing on sensitive information to obtain processed documents.

[0152] In an embodiment, when the processor implements the step of automatically generating an evaluation dataset including document blocks of questions, answers and corresponding evidence based on the intermediate knowledge base according to the LLM, the processor implements the following steps: setting prompt words; generating questions and answers of different types according to the prompt words to obtain data entries; and generating an evaluation dataset according to the data entries and the intermediate knowledge base.

[0153] In an embodiment, when the processor implements the step of performing performance evaluation of a RAG system to be evaluated based on the evaluation dataset using NLP evaluation indicators and a multi-dimensional scoring mechanism based on the LLM to obtain an evaluation result, the processor implements the following steps: preliminarily quantifying text similarity based on the generated answers and expected answers of the evaluation dataset to obtain a first result; combining the scoring system of the LLM to analyze and score the output of the RAG system from multiple dimensions to obtain a scoring result; calculating recall rate and precision rate; evaluating the content quality of the evaluation dataset using fact coincidence degree analysis and grammar checking methods to obtain a second result; inputting the evaluation dataset into the LLM to comprehensively evaluate the overall performance of the RAG system to be evaluated to obtain a third result; and aggregating the first result, the scoring result, the recall rate and the precision rate, the second result and the third result to obtain the evaluation result.

[0154] In an embodiment, when the processor executes the computer program to implement the step of generating a report according to the evaluation result and outputting the report, the following steps are implemented: The BLEU and ROUGE indexes are used to compare the generated answer with the expected answer based on the evaluation data set, to preliminarily quantify the text similarity to obtain a first result.

[0155] In an embodiment, when the processor executes the computer program to implement the step of generating a report according to the evaluation result and outputting the report, the following steps are implemented: According to the evaluation result, the report is obtained through automatic data filling, chart rendering and intelligent analysis based on a preset report template, and the report is output.

[0156] The storage medium can be a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk, and various computer readable storage media that can store program codes.

[0157] Those skilled in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized by electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in general terms in the above description. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0158] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and actual implementation can have another division manner. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0159] The steps in the method of the embodiments of the present application can be adjusted, combined and deleted in sequence according to actual needs. The units in the device of the embodiments of the present application can be combined, divided and deleted according to actual needs. In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit.

[0160] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a storage medium. Based on such an understanding, the technical solutions of the present application essentially or say the part that contributes to the prior art, or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application.

[0161] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for intelligent evaluation of RAG systems based on large models, characterized in that, include: Obtain the knowledge base source files required for the RAG system to be evaluated; The source files of the knowledge base are parsed and transformed to create a structured set of document blocks, and preprocessing operations are performed to obtain an intermediate knowledge base; Based on LLM, an evaluation dataset is automatically generated from the intermediate knowledge base, including document blocks containing questions, answers, and corresponding evidence. Based on the aforementioned evaluation dataset, NLP evaluation metrics and an LLM-based multidimensional scoring mechanism are used to evaluate the performance of the RAG system to be evaluated, so as to obtain the evaluation results. A report is generated based on the evaluation results, and the report is output.

2. The intelligent evaluation method for RAG systems based on large models according to claim 1, characterized in that, The process of parsing and transforming the source files of the knowledge base and creating a structured set of document blocks, followed by preprocessing to obtain an intermediate knowledge base, includes: The document processes structured or semi-structured documents and PDF documents with text layers within the knowledge base source files. For scanned PDF or image format documents, it employs OCR technology, layout analysis, and table recognition technology for processing, and performs preprocessing operations to obtain the processed documents. The processed document is divided into independent document blocks to obtain an intermediate knowledge base.

3. The intelligent evaluation method for RAG systems based on large models according to claim 2, characterized in that, Each document block is associated with metadata information.

4. The intelligent evaluation method for RAG systems based on large models according to claim 2, characterized in that, The process involves processing structured or semi-structured documents and PDF documents with text layers within the knowledge base source files. For scanned PDFs or image-formatted documents, OCR technology, layout analysis, and table recognition technology are used for processing, along with preprocessing operations to obtain the processed documents, including: The system processes structured or semi-structured documents and PDF documents with text layers within the knowledge base source files. For scanned PDF or image documents, it employs OCR technology, layout analysis, and table recognition technology to remove invalid content and de-identify sensitive information to obtain the processed documents.

5. The intelligent evaluation method for RAG systems based on large models according to claim 1, characterized in that, The evaluation dataset, automatically generated based on the LLM and the intermediate knowledge base, includes document blocks containing questions, answers, and corresponding evidence, and comprises: Set prompt words; Based on the provided prompts, different types of questions and answers are generated to obtain data entries; An assessment dataset is generated based on the data entries and the intermediate knowledge base.

6. The intelligent evaluation method for RAG systems based on large models according to claim 1, characterized in that, The performance evaluation of the RAG system under evaluation is conducted using NLP evaluation metrics and an LLM-based multidimensional scoring mechanism based on the evaluation dataset to obtain evaluation results, including: Based on the comparison between the generated answers and the expected answers using the aforementioned evaluation dataset, the text similarity is initially quantified to obtain the first result; By combining the LLM scoring system, the RAG system output is analyzed and scored from multiple dimensions to obtain the scoring results; Calculate recall and precision; The content quality of the evaluation dataset was evaluated using fact overlap analysis and grammar checking to obtain a second result; The evaluation dataset is input into LLM to comprehensively evaluate the overall performance of the RAG system to be evaluated, so as to obtain a third result; The first result, the scoring result, the recall and precision, the second result, and the third result are combined to obtain the evaluation result.

7. The intelligent evaluation method for RAG systems based on large models according to claim 6, characterized in that, The process of comparing the generated answers with the expected answers based on the evaluation dataset to obtain a first result includes: The BLEU and ROUGE metrics were used to compare the generated and expected answers against the evaluation dataset to preliminarily quantify text similarity and obtain the first result.

8. The intelligent evaluation method for RAG systems based on large models according to claim 1, characterized in that, The process of generating and outputting a report based on the evaluation results includes: Based on the evaluation results, the system automatically populates data, renders charts, and performs intelligent analysis using a preset report template to generate and output a report.

9. An intelligent evaluation device for RAG systems based on large models, characterized in that, include: The acquisition unit is used to acquire the knowledge base source files required by the RAG system to be evaluated. The parsing unit is used to parse and transform the source files of the knowledge base and create a structured set of document blocks, and perform preprocessing operations to obtain an intermediate knowledge base; The dataset generation unit is used to automatically generate an evaluation dataset based on the intermediate knowledge base using the LLM, including document blocks containing questions, answers, and corresponding evidence. The evaluation unit is used to evaluate the performance of the RAG system to be evaluated based on the evaluation dataset using NLP evaluation metrics and an LLM-based multidimensional scoring mechanism, so as to obtain the evaluation results. The report generation unit is used to generate a report based on the evaluation results and output the report.

10. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method as described in any one of claims 1 to 8.

Images