Automatic evaluation method and system for RAG intelligent agent system

Abstract

The invention discloses an automatic evaluation method and system for an RAG intelligent agent system, and relates to the technical field of artificial intelligence, and the method comprises the steps: receiving original document data uploaded by a user, carrying out semantic segmentation on an original document based on a natural language processing technology, generating a plurality of document blocks with complete semantics, and according to different conditions, combining the content of the document blocks, carrying out the automatic evaluation of the RAG intelligent agent system. Questions and corresponding answers in different scenes are automatically generated, wherein the questions comprise a question generated based on a single document block, a cross-block question generated by combining a plurality of document blocks and a hidden question generated based on hidden semantic reasoning; according to the automatic evaluation method and system for the RAG intelligent agent system, through an automatic evaluation process and a problem generation method, the dependence on manual intervention is remarkably reduced, the complexity of data annotation and result verification is reduced, and the evaluation time is greatly shortened, so that the requirement of rapid iterative optimization is met.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an automated evaluation method and system for a RAG intelligent agent system. Background Art

[0002] In the current technical field, RAG (Retrieval-Augmented Generation) agent system, as an emerging artificial intelligence technology, has been widely used in scenarios such as text generation and question answering. However, in the specific application process, especially in the financial field.

[0003] The existing RAG system evaluation methods have the following main problems: First, the evaluation efficiency is low. In the existing technology, the performance evaluation of the RAG system usually requires a lot of manual intervention, such as data labeling and result verification, which makes the evaluation process time-consuming and labor-intensive, and it is difficult to meet the needs of rapid iterative optimization; second, there is a lack of systematic evaluation. Existing evaluation methods mostly focus on performance indicators of a single dimension, such as accuracy or generation quality, and fail to form a comprehensive systematic evaluation framework. This limitation makes it impossible for existing technologies to fully identify and optimize the potential deficiencies of the system; in addition, existing technologies are difficult to adapt to the complexity of financial professional scenarios. Application scenarios in the financial field usually require processing high-precision and high-reliability data content, while also meeting the compliance requirements of specific industries. The existing RAG system evaluation methods lack special designs for financial scenarios, which limits their application effects in financial business. Summary of the invention

[0004] The purpose of the present invention is to provide a RAG intelligent agent system automatic evaluation method and system to solve the problems of the prior art.

[0005] To achieve the above object, the present invention provides the following technical solution: an automated evaluation method for a RAG agent system, the method comprising:

[0006] Receive original document data uploaded by the user;

[0007] Based on natural language processing technology, the original document is semantically segmented to generate multiple semantically complete document blocks;

[0008] Based on different conditions and combined with the content of the document block, questions and corresponding answers in different scenarios are automatically generated, including questions generated based on a single document block, cross-block questions generated by combining multiple document blocks, and implicit questions generated based on implicit semantic reasoning;

[0009] The RAG system is automatically evaluated using the questions and answers generated above. The evaluation indicators include the relevance score of the retrieval results and the relevance of the questions, the accuracy score of the retrieval results and the standard answers, the precision score of the document rearrangement ability, and the similarity score of the generated answers and the standard answers.

[0010] Generate a comprehensive evaluation report, providing quantitative indicators and system optimization suggestions.

[0011] Preferably, the semantic segmentation uses a BERT model to encode the document and combines it with a K-means clustering algorithm to generate multiple semantically complete document blocks.

[0012] Preferably, the question generation includes generating questions according to subject keywords, document types and application scenarios input by the user.

[0013] Preferably, the cross-block question generated by combining multiple document blocks includes deducing complex questions based on comprehensive information of different document blocks, and obtaining answers through semantic matching between multiple documents.

[0014] Preferably, the implicit question is generated by analyzing the implicit semantic relationship between document blocks and combining conditional reasoning to generate questions and answers.

[0015] Preferably, the evaluation indicators include the recall rate and precision of the document retrieval component, the language fluency score of the answer generation component, and the semantic similarity score between the user question and the system answer.

[0016] Preferably, the comprehensive evaluation report includes visual charts and quantitative data, showing the performance of the system in evaluations of different dimensions and providing optimization suggestions.

[0017] Preferably, the evaluation method supports dynamic adjustment of evaluation indicators according to financial industry needs, including loan approval, risk analysis and report generation application scenarios.

[0018] Preferably, the generated evaluation data is stored and archived so as to track and analyze the historical performance of the system.

[0019] The RAG agent system automatic evaluation system is used to implement the steps of the RAG agent system automatic evaluation method, and the system includes:

[0020] A data receiving module, used to receive original document data uploaded by users;

[0021] A semantic segmentation module, connected to the data receiving module, is used to perform semantic segmentation on the original document based on natural language processing technology to generate multiple semantically complete document blocks;

[0022] A question generation module is connected to the semantic segmentation module and is used to automatically generate questions and corresponding answers in different scenarios based on different conditions and document block contents. The questions include: questions generated based on a single document block, cross-block questions generated by combining multiple document blocks, and implicit questions generated based on implicit semantic reasoning;

[0023] An automated evaluation module, connected to the question generation module, for automatically evaluating the RAG system using the questions and answers generated above, wherein the evaluation includes: a relevance score of the relevance between the search results and the questions, an accuracy score of the search results and the standard answers, a precision score of the document rearrangement capability, and a similarity score between the generated answers and the standard answers;

[0024] The evaluation report generation module is connected to the automated evaluation module to generate a comprehensive evaluation report, providing quantitative indicators and system optimization suggestions.

[0025] It can be seen from the above technical solution that the present invention has the following beneficial effects:

[0026] The automated evaluation method and system of the RAG intelligent system significantly reduces the reliance on manual intervention, reduces the complexity of data annotation and result verification, and greatly shortens the evaluation time through automated evaluation processes and question generation methods, thereby meeting the needs of rapid iterative optimization. A comprehensive evaluation framework covering multiple dimensions is constructed, including performance indicators such as retrieval relevance, answer accuracy, document re-arrangement capability, and similarity of generated answers. It can comprehensively identify and optimize potential deficiencies of the system and improve the overall performance of the RAG system. An evaluation method is designed specifically for the high-precision and high-reliability requirements of the financial field, supporting complex professional terminology processing, contextual semantic analysis, and industry compliance requirements, thereby improving the applicability and accuracy of the RAG system in financial business. The modular design of the framework and the flexible evaluation indicator definition capabilities allow customized evaluations based on financial business scenarios (such as loan approval, risk analysis, report generation, etc.), effectively meeting the needs of specific application scenarios, and enhancing the intelligent question-answering and data processing capabilities of the RAG system through automated evaluation and optimization, thereby promoting the development of financial business in the direction of intelligence and automation and improving overall business efficiency and competitiveness. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a flow chart of the method of the present invention;

[0028] Figure 2 This is a schematic diagram of the connection of the system modules of the present invention. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] like Figure 1 As shown, the present invention provides a technical solution: an automated evaluation method for a RAG intelligent system, comprising receiving original document data uploaded by a user; semantically segmenting the original document based on natural language processing technology to generate a plurality of semantically complete document blocks; automatically generating questions and corresponding answers in different scenarios based on different conditions and in combination with the content of the document blocks, including questions generated based on a single document block, cross-block questions generated in combination with a plurality of document blocks, and implicit questions generated based on implicit semantic reasoning; utilizing the questions and answers generated above to automatically evaluate the RAG system, wherein the evaluation indicators include a relevance score between the relevance of the retrieval results to the questions, an accuracy score between the retrieval results and the standard answers, a precision score between the document rearrangement capability, and a similarity score between the generated answers and the standard answers; generating a comprehensive evaluation report to provide quantitative indicators and system optimization suggestions.

[0031] In the above method, the original document data is received through the user upload interface. Natural language processing technology (such as word segmentation, syntactic analysis and semantic modeling, etc.) is used to semantically segment the document data to ensure that each document block has semantic integrity, providing a basis for the subsequent evaluation generation process. In the question and answer generation stage, the method combines different conditions, such as single-block content features, logical relationships between multiple blocks of content, and implicit semantic reasoning, to generate questions and answers suitable for different scenarios. Single-block questions are generated based on independent document blocks; cross-block questions are generated using the relevance of different document blocks; implicit questions use implicit semantic reasoning technology (such as semantic embedding or pre-trained language model) to mine deep semantic relationships to ensure the comprehensiveness of question generation. For the evaluation part, the method uses the generated questions and standard answers to comprehensively evaluate the RAG system. The evaluation indicators cover the following aspects: Relevance score: evaluates the relevance of the system's retrieval results to the questions. Accuracy score: measures the consistency between the retrieval results and the standard answers. Document re-ranking accuracy score: analyzes the system's ability to sort the retrieval results. Similarity scoring: Evaluate the textual similarity between the generated answer and the standard answer, for example, based on natural language generation evaluation indicators such as BLEU and ROUGE. Finally, a comprehensive evaluation report is generated based on the above indicators to provide quantitative results and optimization suggestions for system performance. Through natural language processing and automated processes, the need for manual participation is significantly reduced, and the evaluation efficiency is greatly improved. Based on different types of question generation methods, the method can cover a variety of evaluation scenarios to ensure the comprehensiveness of the evaluation results. The RAG system is quantitatively evaluated through multi-dimensional indicators to provide a scientific basis for system optimization. The comprehensive evaluation report provides guiding suggestions for the improvement of the RAG system and improves the actual application effect.

[0032] Semantic segmentation uses the BERT model to encode the document, and combines the K-means clustering algorithm to generate multiple semantically complete document blocks. In the above implementation, the process of semantic segmentation consists of the following steps: the input document is processed using the BERT model, and its bidirectional context modeling capability is used to generate a high-dimensional semantic vector representation of the document content. The BERT model can effectively capture the semantic features of the document content, especially in long text processing, with excellent performance. The K-means clustering algorithm is applied to the generated high-dimensional semantic vector representation, and the document is divided into several clusters based on the similarity of the vectors. Each cluster corresponds to a semantically complete document block, ensuring that the sentences within the block are semantically consistent, and the content between blocks is independent of each other. Through the clustering results, the document content is split into multiple semantically complete document blocks, providing a basis for subsequent question generation and evaluation. The key to this implementation is that BERT provides high-quality semantic representation, and the K-means clustering algorithm uses the similarity of semantic vectors to effectively segment the document content, so that the document block can accurately express the semantic theme of the content. The above method further optimizes the semantic segmentation process in claim 1 by combining the BERT model and the K-means clustering algorithm, and its advantages include: the deep semantic representation ability of the BERT model effectively improves the accuracy of document block segmentation and reduces the loss of semantic information. Combined with K-means clustering, the method can automatically adapt to documents of different types and structures without excessive manual adjustments. Higher-quality document block segmentation provides more stable input data for question generation and evaluation, further improving the reliability and accuracy of the evaluation. Both the BERT model and the K-means clustering algorithm have high computational efficiency and are suitable for processing large-scale document data.

[0033] Question generation includes generating questions based on the subject keywords, document types, and application scenarios input by the user. In the above implementation, the process of question generation is based on the specific conditions input by the user, and mainly includes the following steps: the user inputs the subject keywords, and the system performs semantic analysis on the keywords through natural language processing technology to understand its core meaning and related concepts. During the parsing process, the system will expand or associate the keywords, such as generating synonyms, hyponyms, and hyponyms related to the topic based on the pre-trained language model. The system adjusts the question generation rules according to the document type (such as technical documents, news reports, legal terms, etc.). For example, for technical documents, the main questions generated are concentrated on operating procedures or technical features; while for news documents, they may be biased towards event backgrounds or development trends. Combined with application scenario information (such as educational scenarios, medical scenarios, or business scenarios), adjust the tone, scope, and focus of question generation. For example, in educational scenarios, questions may be mainly guiding questions, while in business scenarios, they focus more on decision support questions. The specific questions generated based on the above conditions are divided into the following categories: questions closely generated around the keywords entered by the user, such as "What is the core technology about [keyword]? Generate detailed questions related to keywords based on document content, such as "What are the implementation steps of [keyword]? Expand the generated questions based on scenario characteristics, such as "How to apply [keyword] in [scenario]? Through the above steps, the system ensures that the generated questions can cover the core content of user needs, while adapting to document characteristics and usage scenarios. According to the keywords, document types and scenarios entered by the user, the system can generate customized questions that better meet user needs, improve the pertinence and practicality of question generation, and the generated questions cover multi-level needs from the core of the subject to the scenario application, which enhances the integrity and depth of the evaluation process. Through automated condition adaptation and question generation process, manual intervention is greatly reduced, and the efficiency of question generation is improved. The scenario adaptation function makes the generated questions more practical, which helps users achieve more precise evaluation goals in specific application fields.

[0034] Cross-block questions generated in combination with multiple document blocks include deducing complex questions based on the comprehensive information of different document blocks, and obtaining answers through semantic matching between multiple documents. In this embodiment, the system analyzes the semantic relationship between multiple document blocks and uses cross-document semantic reasoning technology (such as context understanding and knowledge integration based on language models) to derive answers to complex questions. The generated questions can cover the intersection of multi-document information, such as causal relationships in time series or logical associations between concepts. The answers are obtained through semantic matching and reasoning between multiple documents to ensure the accuracy and consistency of the comprehensive information. This method effectively improves the ability to generate questions and extract answers in cross-document scenarios, making the evaluation coverage more comprehensive; at the same time, the generation and answering of cross-block questions utilize the semantic connection between document blocks, improving the system's ability to integrate complex information and logical reasoning, thereby enhancing the depth and professionalism of the evaluation.

[0035] The generation of implicit questions is achieved by analyzing the implicit semantic relationship between document blocks and combining conditional reasoning to generate questions and answers. In this embodiment, the system first uses semantic analysis technology to deeply mine the implicit semantic relationship between document blocks, for example, discovering potential associations through logical connections between sentences, semantic similarity or contextual dependencies. Subsequently, combined with conditional reasoning methods (such as rule-based reasoning or causal reasoning based on neural networks), implicit questions are constructed and corresponding answers are derived. The generated implicit questions often reveal deep information that is not directly apparent in the document content but can be logically deduced. This method significantly expands the scope of question generation, can mine potential information that is not explicitly expressed in the document, and enhances the intelligence level of the evaluation; the implicit questions and answers generated by conditional reasoning further improve the depth and coverage of the evaluation, providing strong support for the comprehensive evaluation of the RAG system.

[0036] The evaluation indicators include the recall rate and precision of the document retrieval component, the language fluency score of the answer generation component, and the semantic similarity score between the user question and the system answer. In this embodiment, the system comprehensively evaluates the RAG system through the following evaluation indicators: evaluate the coverage and accuracy of the document retrieval component when retrieving the target document, and the calculation method is based on the standard information retrieval evaluation formula. Perform linguistic analysis on the output of the answer generation component, such as generating scores through grammar checking tools or pre-trained language models to evaluate its grammatical correctness and naturalness of expression. Through semantic matching algorithms (such as cosine similarity or sentence conversion models based on embedding), the semantic consistency between user questions and system answers is evaluated to measure the relevance and accuracy of system answers. This method carefully evaluates the RAG system from multiple perspectives of document retrieval, answer generation, and user interaction, which helps to discover the advantages and disadvantages of the system in different modules; the quantitative scoring method provides a clear performance measurement standard, provides clear guidance for the optimization direction of the system, and further enhances the scientificity and practicality of the evaluation.

[0037] The comprehensive evaluation report includes visual charts and quantitative data, which show the performance of the system in different dimensional evaluations and provide optimization suggestions. In this embodiment, the evaluation report is based on multi-dimensional data analysis, and uses visualization tools (such as bar charts, line charts, radar charts, etc.) to intuitively display the system's performance in various evaluation indicators (such as recall, precision, fluency, similarity, etc.). At the same time, combined with quantitative data, a specific score and its weight are provided for each indicator to ensure that users can fully understand the performance of the system. The report also uses the analysis results to generate targeted optimization suggestions, such as recommending adjustment of model parameters or training data when the retrieval accuracy is insufficient, and recommending improvement of the language generation algorithm when the answer generation fluency is low. This method presents complex evaluation data through visualization, allowing users to quickly and intuitively understand the performance of the system in different dimensions; at the same time, combined with quantitative analysis and optimization suggestions, it significantly improves the guidance and practicality of the report, making it easier for users to implement targeted system improvements.

[0038] The evaluation method supports dynamic adjustment of evaluation indicators according to the needs of the financial industry, including loan approval, risk analysis and report generation application scenarios. In this embodiment, the evaluation method dynamically adjusts the evaluation indicators according to the specific needs of the financial industry: evaluates the ability of the RAG system in processing customer credit information, document retrieval and automatically generating approval suggestions. New evaluation indicators may include information integrity check, risk warning accuracy, etc. For risk factor mining associated with multiple documents, the evaluation indicators can be adjusted to the accuracy of cross-document information extraction, association analysis capabilities and logical consistency of reports. Evaluate the language accuracy, format compliance and correct use of domain terms of the RAG system when generating professional reports. The evaluation method dynamically adjusts the indicator weights through predefined rules or user input, and highlights the evaluation results and optimization suggestions directly related to financial applications in the report. This method enhances the industry adaptability of the RAG system evaluation and can meet the diverse application needs of the financial industry; the function of dynamically adjusting the evaluation indicators ensures the high correlation between the evaluation results and the actual business scenarios, enhances the value and guidance of the evaluation, and helps to accelerate the implementation of the system in financial scenarios.

[0039] It also includes storing and archiving the generated evaluation data in order to track and analyze the historical performance of the system. In this embodiment, the system stores the data generated by each evaluation, including the specific scores of the evaluation indicators, detailed records of the generated questions and answers, and the comprehensive evaluation report, in the database. When the data is stored, it is archived in chronological order and marked with information such as application scenarios and configuration versions to facilitate the retrieval and comparison of historical data. Based on the archived data, the system supports multi-dimensional historical performance analysis, such as performance trend evaluation, comparison of improvement effects between different versions, etc., to provide data support for system optimization. This method achieves long-term tracking of the historical performance of the system through the storage and archiving of evaluation data, which facilitates the discovery of performance fluctuations and improvement trends; the archived data can also be used as a reference for subsequent optimization, which improves the continuity and scientific nature of the evaluation work and helps the iteration and performance improvement of the system.

[0040] A RAG agent system automatic evaluation system is also provided, which is used to implement the steps of the RAG agent system automatic evaluation method, and the system includes:

[0041] A data receiving module, used to receive original document data uploaded by users;

[0042] A semantic segmentation module, connected to the data receiving module, is used to perform semantic segmentation on the original document based on natural language processing technology to generate multiple semantically complete document blocks;

[0043] A question generation module is connected to the semantic segmentation module and is used to automatically generate questions and corresponding answers in different scenarios based on different conditions and document block contents. The questions include: questions generated based on a single document block, cross-block questions generated by combining multiple document blocks, and implicit questions generated based on implicit semantic reasoning;

[0044] An automated evaluation module, connected to the question generation module, for automatically evaluating the RAG system using the questions and answers generated above, wherein the evaluation includes: a relevance score of the relevance between the search results and the questions, an accuracy score of the search results and the standard answers, a precision score of the document rearrangement capability, and a similarity score between the generated answers and the standard answers;

[0045] The evaluation report generation module is connected to the automated evaluation module to generate a comprehensive evaluation report, providing quantitative indicators and system optimization suggestions.

[0046] Each module works together to ensure the flexibility and scalability of the system in a modular design. The process from data reception to evaluation report generation is based on automation as the core goal, and efficient data processing and system performance evaluation are achieved through natural language processing technology and evaluation algorithms. The system functions are clearly separated, and each module can be independently developed and optimized, which is easy to expand and maintain. It covers the entire process from data reception, problem generation to evaluation report generation, greatly reducing manual intervention. Multi-dimensional indicators are combined with dynamic adjustment functions to adapt to application needs in different fields. Through quantitative evaluation reports and visual displays, it provides a scientific basis for system optimization and iteration.

[0047] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. The RAG agent system automated evaluation method is characterized by: The method comprises: Receive original document data uploaded by the user; Based on natural language processing technology, the original document is semantically segmented to generate multiple semantically complete document blocks; Based on different conditions and combined with the content of the document block, questions and corresponding answers in different scenarios are automatically generated, including questions generated based on a single document block, cross-block questions generated by combining multiple document blocks, and implicit questions generated based on implicit semantic reasoning; The RAG system is automatically evaluated using the questions and answers generated above. The evaluation indicators include the relevance score of the retrieval results and the relevance of the questions, the accuracy score of the retrieval results and the standard answers, the precision score of the document rearrangement ability, and the similarity score of the generated answers and the standard answers. Generate a comprehensive evaluation report, providing quantitative indicators and system optimization suggestions.

2. The RAG agent system automatic evaluation method according to claim 1, characterized in that: The semantic segmentation uses the BERT model to encode the document and combines it with the K-means clustering algorithm to generate multiple semantically complete document blocks.

3. The RAG agent system automatic evaluation method according to claim 1, characterized in that: The question generation includes generating questions according to subject keywords, document types and application scenarios input by users.

4. The RAG agent system automatic evaluation method according to claim 1, characterized in that: The cross-block questions generated by combining multiple document blocks include deducing complex questions based on comprehensive information of different document blocks, and obtaining answers through semantic matching between multiple documents.

5. The RAG agent system automatic evaluation method according to claim 1, characterized in that: The implicit question is generated by analyzing the implicit semantic relationship between document blocks and combining conditional reasoning to generate questions and answers.

6. The RAG agent system automatic evaluation method according to claim 1, characterized in that: The evaluation indicators include the recall rate and precision of the document retrieval component, the language fluency score of the answer generation component, and the semantic similarity score between the user question and the system answer.

7. The RAG agent system automatic evaluation method according to claim 1, characterized in that: The comprehensive evaluation report includes visual charts and quantitative data, showing the performance of the system in evaluations of different dimensions and providing optimization suggestions.

8. The RAG agent system automatic evaluation method according to claim 1, characterized in that: The evaluation method supports dynamic adjustment of evaluation indicators according to the needs of the financial industry, including loan approval, risk analysis and report generation application scenarios.

9. The RAG agent system automatic evaluation method according to claim 1, characterized in that: It also includes the storage and archiving of the generated measurement data to track and analyze the historical performance of the system.

10. A RAG agent system automated evaluation system, used to implement the steps of the RAG agent system automated evaluation method according to any one of claims 1 to 9, characterized in that: The system comprises: A data receiving module, used to receive original document data uploaded by users; A semantic segmentation module, connected to the data receiving module, is used to perform semantic segmentation on the original document based on natural language processing technology to generate multiple semantically complete document blocks; A question generation module is connected to the semantic segmentation module and is used to automatically generate questions and corresponding answers in different scenarios based on different conditions and document block contents. The questions include: questions generated based on a single document block, cross-block questions generated by combining multiple document blocks, and implicit questions generated based on implicit semantic reasoning; An automated evaluation module, connected to the question generation module, for automatically evaluating the RAG system using the questions and answers generated above, wherein the evaluation includes: a relevance score of the relevance between the search results and the questions, an accuracy score of the search results and the standard answers, a precision score of the document rearrangement capability, and a similarity score between the generated answers and the standard answers; The evaluation report generation module is connected to the automated evaluation module to generate a comprehensive evaluation report, providing quantitative indicators and system optimization suggestions.

Images