Rag evaluation device for real-time performance diagnosis of rag systems based on scheduler, and rag system evaluation method using the same
The RAG evaluation device provides real-time performance diagnosis and dynamic feedback to enhance RAG system stability and efficiency by analyzing accuracy, similarity, and reliability, addressing challenges in existing RAG system evaluation methods.
Patent Information
- Application Number
- KR1020260067102
- Authority / Receiving Office
- KR · KR
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2026-04-14
- Publication Date
- 2026-11-06
- Estimated Expiration
- 2046-04-14
AI Technical Summary
Existing RAG systems face challenges in real-time performance evaluation, including long evaluation cycles, failure to reflect operational changes, unsatisfactory scoring methods, and difficulty in managing quality degradation due to individual metric reliance and unsuitable metric suitability, which hinders stable operation and quality management.
A RAG evaluation device and method that automatically diagnoses performance using a scheduler, analyzing accuracy, similarity, reliability, and hallucination, and dynamically feeds back system settings to improve performance by reflecting the latest documents and reducing administrator intervention.
Enables continuous, real-time performance management with objective and precise diagnosis, identifying abnormalities early, and improving operational reliability and efficiency by dynamically adjusting system settings based on performance indicators.
Smart Images

Figure 112026045326439-PAT00005_ABST
Abstract
Description
Technology Field
[0001] The present disclosure relates to a performance indicator solution for a RAG system, and more specifically, to a RAG evaluation device that diagnoses the performance of a RAG system in real time based on a scheduler, and a method for evaluating a RAG system using the same. Background Technology
[0002] Retrieval-Augmented Generation (RAG) systems, which combine information retrieval and generation capabilities, are being utilized in various fields based on Large Language Models (LLM). RAG systems can operate by searching for relevant documents corresponding to user queries, and then having generative AI generate responses based on the retrieved documents. Evaluation methods for RAG systems can involve using traditional NLP evaluation metrics such as BLEU, ROUGE, and Recall, or performing qualitative evaluations by utilizing LLM as a scoring model. Additionally, to quantitatively evaluate the performance of RAG systems, an automated evaluation framework that includes both the retrieval and generation stages can be applied.
[0003] Meanwhile, there are limiting factors acting as various issues in the performance evaluation of RAG systems, and limitations resulting from these problems exist. For instance, issues may arise such as long evaluation cycles or the failure to reflect evaluation results in the operational system in real time. Additionally, there may be limitations where evaluation metrics rely on a single indicator or fail to reflect the interrelationships between search quality and generation quality by evaluating them individually. Furthermore, scoring methods utilizing LLM may result in unsatisfactory scoring results or a failure to detect performance degradation in the evaluation model. Moreover, if quality degradation within the evaluation pipeline itself or issues with metric suitability are left unaddressed for an extended period, quality management of the entire RAG system may become difficult.
[0004] Therefore, to stably maintain the operational quality of the RAG system, it is necessary to automatically determine the RAG evaluation timing based on a scheduler, thereby enabling multidimensional metric-based evaluation encompassing both the search and creation phases, real-time performance diagnosis, and the immediate reflection of evaluation results as feedback. In particular, to ensure the operational stability of the RAG system, technology capable of cumulatively managing performance metrics—including accuracy scores, similarity scores, reliability scores, and hallucination warning indicators—over time, and evaluating the causes of performance degradation, may be required. In other words, a RAG performance metric-based feedback system is required that can detect performance degradation occurring in the RAG system operating environment at an early stage and continuously improve quality without administrator intervention. Prior art literature
[0005] Korean Registered Patent No. 10-2824126 B1 "Search system and method using layered metadata based on Search Augmented Generation (RGA)" The problem to be solved
[0006] One objective of the present disclosure is to provide a RAG evaluation device and a method for evaluating a RAG system using the same, which can objectively and comprehensively diagnose the performance status of a RAG system even in an operating environment where documents are added or updated in real time by automatically calling the search API and generation API of a RAG system based on a scheduler and analyzing performance evaluation factors including accuracy, similarity, reliability, and whether hallucination occurs by comparing search result and generation response data with a correct answer standard or a search context.
[0007] Another objective of the present disclosure is to provide a RAG evaluation device and a method for evaluating a RAG system using the same, which can identify problems such as delays in reflecting the latest documents, failure to include the latest documents in search results, and failure to reflect the latest documents in generated response data at an early stage by quantitatively scoring performance indicators including an accuracy score, a similarity score, a reliability score, and a hallucination warning indicator based on the analysis results, and evaluating the performance status, abnormal signs, and causes of performance degradation of the RAG system based on whether the performance indicators meet a reference threshold, the trend of change over time, and the pattern of change compared to past history.
[0008] Another objective of the present disclosure is to provide a RAG evaluation device and a method for evaluating a RAG system using the same, which can continuously maintain and improve the reflectivity of the latest information, response stability, and operational reliability of the RAG system in a real-time operating environment while reducing administrator intervention by dynamically feeding back to the RAG system a system setting including at least one of a search setting, a creation setting, an index setting, and an alarm setting along with an evaluation result of the RAG system, and re-evaluating whether performance has been improved by re-executing the same user query or a user query under the same conditions.
[0009] However, the problems that this disclosure aims to solve are not limited to those mentioned above, and other unmentioned problems will be clearly understood by a person skilled in the art from the description below. means of solving the problem
[0010] A RAG evaluation device for diagnosing the performance of a RAG system in real time based on a scheduler according to the present disclosure for achieving the above-described technical problem may include a memory for storing at least one instruction and at least one processor for executing said at least one instruction. The above-mentioned at least one processor collects search result and generated response data by automatically calling the search API and generation API of the RAG system for a user query using a scheduler, and analyzes performance evaluation factors including accuracy, similarity, reliability, and whether hallucination occurs by comparing the search result and generated response data with a correct answer standard or a search context, and quantitatively scores a performance indicator including at least one of an accuracy score, a similarity score, a reliability score, and a hallucination warning indicator based on the analysis results, evaluates the performance status, abnormal signs, and causes of performance degradation of the RAG system based on whether the reference threshold of the performance indicator is met, the trend of change over time, and the pattern of change compared to past history, and can dynamically feed back to the RAG system a system setting including at least one of a search setting, a generation setting, an index setting, and an alarm setting along with the evaluation results for the RAG system to improve the performance of the RAG system.
[0011] In one embodiment, the at least one processor detects at least one of a predefined time condition, an event condition, and a state change condition using the scheduler, determines a RAG evaluation time using the scheduler, automatically executes the search API and the generation API of the RAG system at the determined RAG evaluation time, and can collect the search result and the generation response data based on the call results of the search API and the generation API.
[0012] In one embodiment, the at least one processor detects whether a new document or an updated document has occurred, extracts at least two items among a title, date, and changes from the new document or the updated document, and combines the extracted at least two items to generate a user query for verifying the reflection of the latest document.
[0013] In one embodiment, the at least one processor can calculate a priority value P for a plurality of user query candidates using a recent document update frequency U, document importance C, and a past performance degradation history score E, and determine the execution order of the user queries and the RAG evaluation time so that the user queries with a priority value P greater than or equal to a reference value are executed preferentially.
[0014] In one embodiment, the at least one processor extracts time information including at least one of a document loading time, indexing completion time, search reflection time, and generation response time for a new document or an updated document, and can store the time information in conjunction with the search result and the generation response data.
[0015] In one embodiment, the at least one processor selects a valid correct answer criterion or search context at the time of execution of the user query, and by comparing the search result and the generated response data with the correct answer criterion or search context in at least one of a core fact unit, an attribute item unit, and a sentence unit, the accuracy, the similarity, the reliability, and the occurrence of hallucination, including whether the latest changes are reflected.
[0016] In one embodiment, the at least one processor can quantitatively score the performance indicator by normalizing the accuracy score, the similarity score, the reliability score, and the hallucination warning indicator, and assigning weights by reflecting at least one of whether the latest information is reflected, time accuracy, and evidence accuracy.
[0017] In one embodiment, the at least one processor calculates a search reflection delay value Dr based on the difference between the search reflection time and the document loading time, and a response reflection delay value Da based on the difference between the creation response time and the document loading time, and if the search reflection delay value Dr exceeds a preset reference search delay value τr, the indexing delay is determined to be the cause of performance degradation, and if the search reflection delay value Dr is less than or equal to the reference search delay value τr and the response reflection delay value Da exceeds the reference response delay value τa, the creation reflection delay is determined to be the cause of performance degradation.
[0018] In one embodiment, the at least one processor classifies the performance state of the RAG system into a normal state, a borderline state, or a degraded state based on whether the reference threshold of the performance indicator is met, the trend of change over time, and the pattern of change compared to past history; if the search result does not include the latest document, it evaluates the cause of performance degradation related to the search setting or the index setting; if the search result includes the latest document but the generated response data does not reflect the latest document, it evaluates the cause of performance degradation related to the generation setting; and, according to the evaluation result, changes at least one of the search setting, the generation setting, the index setting, and the alarm setting, and can re-evaluate whether performance has been improved by re-executing the same user query or the user query under the same conditions.
[0019] In one embodiment, the at least one processor may store the search results and the generated response data for each user query, and store the evaluation results regarding the cause of performance degradation and the system settings that are dynamically fed back to the RAG system by mutually labeling them.
[0020] In one embodiment, the at least one processor analyzes the evaluation results for the RAG system according to predefined meta-evaluation criteria, calculates a self-evaluation score regarding at least one of scoring accuracy, scoring consistency, evaluation latency, evaluation cost, false alarm rate, false alarm rate, and distribution drift sensitivity, and diagnoses the evaluation performance of the evaluation method used based on the self-evaluation score.
[0021] In one embodiment, the at least one processor can update the evaluation method by adjusting at least one of an evaluation criterion weight, an evaluation threshold, a scoring prompt template, an LLM selection for evaluation, a batch size, a sampling strategy, a cache expiration time, a worker parallelism, a reporting period, and a feedback policy based on the diagnostic result of the evaluation performance.
[0022] A method for evaluating a RAG system according to the present disclosure for achieving the aforementioned technical objectives may include: a step of collecting search result and generated response data by automatically calling the search API and generation API of the RAG system in response to a user query using a scheduler; a step of analyzing performance evaluation elements including accuracy, similarity, reliability, and whether hallucination occurs by comparing the search result and generated response data with a correct answer standard or a search context; a step of quantitatively scoring a performance indicator including at least one of an accuracy score, a similarity score, a reliability score, and a hallucination warning indicator based on the analysis result; a step of evaluating the performance status, abnormal signs, and causes of performance degradation of the RAG system based on whether the performance indicator meets a reference threshold, the trend of change over time, and the pattern of change compared to past history; and a step of dynamically feeding back to the RAG system a system setting including at least one of a search setting, a generation setting, an index setting, and an alarm setting along with the evaluation result for the RAG system to improve the performance of the RAG system.
[0023] In addition to this, a computer program stored on a computer-readable recording medium for implementing the present disclosure may be further provided.
[0024] In addition to this, a computer-readable recording medium for recording a computer program for implementing the present disclosure may be further provided. Effects of the invention
[0025] According to the aforementioned means for solving the problem of the present disclosure, the RAG evaluation device and the RAG system evaluation method using the same of the present disclosure can automatically call the search API and generation API of the RAG system based on a scheduler and collect search result and generation response data together with time information. Accordingly, the RAG evaluation device and the RAG system evaluation method using the same of the present disclosure can continuously check whether the latest documents are reflected even in an operating environment where documents are added or updated in real time, and can stably manage the performance status of the RAG system.
[0026] In addition, the RAG evaluation device and the RAG system evaluation method using the same disclosed in the present disclosure can multi-dimensionally analyze performance evaluation elements, including accuracy, similarity, reliability, and whether hallucination occurs, by comparing the search results and the generated response data with a correct answer standard or a search context, and can quantify the analysis results into performance indicators including an accuracy score, a similarity score, a reliability score, and a hallucination warning indicator. Accordingly, the RAG evaluation device and the RAG system evaluation method using the same disclosed in the present disclosure can perform objective and precise performance diagnosis that comprehensively reflects the mutual influence relationship between the search stage and the generation stage, and can identify abnormal signs early and distinguish between one-time errors and structural performance degradation by considering together whether the reference threshold of the performance indicator is met, the trend of change over time, and the pattern of change compared to past history.
[0027] Furthermore, the RAG evaluation device and the RAG system evaluation method using the same disclosed in the present disclosure evaluate the causes of performance degradation by distinguishing between search reflection delays, creation reflection delays, non-reflection of the latest document, and non-reflection of the latest document in creation response data, and can dynamically feed back to the RAG system a system setting including at least one of search settings, creation settings, index settings, and alarm settings along with the evaluation results. Accordingly, the RAG evaluation device and the RAG system evaluation method using the same disclosed in the present disclosure can re-evaluate whether performance has been improved by re-executing the same user query or a user query under the same conditions, and can improve operational reliability, maintenance efficiency, and overall system performance management efficiency while reducing administrator intervention.
[0028] The effects of the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by a person skilled in the art from the description below. Brief explanation of the drawing
[0029] FIG. 1 is a conceptual diagram showing the RAG system and RAG evaluation device of the present invention. FIG. 2 is a drawing showing an example of a RAG system of the present invention. FIG. 3 is a diagram showing the block configuration of the RAG evaluation device of the present invention. Figure 4 is a diagram showing the system architecture and data processing flow of the RAG evaluation device of the present invention. FIG. 5 is a flowchart showing the operation of the RAG evaluation device of the present invention. FIG. 6 is a block diagram showing the operation of the RAG evaluation device of the present invention collecting search results and generated response data using a scheduler. FIG. 7 is a flowchart illustrating the operation of the RAG evaluation device of the present invention collecting search result and generated response data by automatically calling the search API and generation API of the RAG system for a user query using a scheduler. FIG. 8 is a diagram showing an example of a multidimensional evaluation index according to automatic scoring of the RAG evaluation device of the present invention. FIG. 9 is a diagram showing an example of the RAG evaluation device of the present invention outputting performance evaluation indicators. Specific details for implementing the invention
[0030] Hereinafter, embodiments are described in detail with reference to the attached drawings. However, various modifications may be made to the embodiments, and thus the scope of the patent application is not limited or restricted by these embodiments. It should be understood that all modifications, equivalents, and substitutions to the embodiments are included within the scope of the rights.
[0031] Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be modified and implemented in various forms. Accordingly, the embodiments are not limited to the specific disclosed forms, and the scope of this specification includes modifications, equivalents, or substitutions that fall within the technical concept.
[0032] Terms such as "first" or "second" may be used to describe various components, but these terms should be interpreted solely for the purpose of distinguishing one component from another. For example, the first component may be named the second component, and similarly, the second component may be named the first component.
[0033] Furthermore, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless explicitly and specifically defined otherwise. In certain cases, terms have been selected at the applicant's discretion, and in such cases, their meanings will be described in detail in the relevant explanatory sections. Accordingly, terms used in this disclosure should be defined not merely by their names, but based on their meanings and the content throughout this disclosure.
[0034] When it is stated that a component is "connected" to another component, it should be understood that it may be directly connected to or coupled with that other component, or that there may be other components in between.
[0035] The terms used in the embodiments are for illustrative purposes only and should not be interpreted as intended to be limiting. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, terms such as "comprising" or "having" are intended to indicate the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.
[0036] Throughout this specification, when a part is described as "comprising" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components. Furthermore, the singular form used in this specification includes the plural form unless specifically stated otherwise in the text. Additionally, the expression "at least one of a, b, and c" described throughout this specification may encompass 'a alone,' 'b alone,' 'c alone,' 'a and b,' 'a and c,' 'b and c,' or 'a, b, and c all.'
[0037] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which the embodiments pertain. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this application.
[0038] Additionally, terms such as “part,” “module,” etc., as described in this specification refer to a unit that processes at least one function or operation, which may be implemented in hardware or software, or a combination of hardware and software. Furthermore, embodiments of the present disclosure may be represented by functional block configurations and various processing steps. These functional blocks may be implemented by various numbers of hardware and / or software configurations that execute specific functions. For example, embodiments of the present disclosure may employ integrated circuit configurations such as memory, processing, logic, look-up tables, etc., which can execute various functions under the control of one or more microprocessors or other control devices.
[0039] Each block of the process flow diagrams attached to this specification and combinations of the flow diagrams may be executed by computer program instructions. Since these computer program instructions may be loaded into the processor of a general-purpose computer, a computer for special purposes, or other programmable data processing equipment, the instructions executed through the processor of the computer or other programmable data processing equipment create means for performing the functions described in the flow diagram block(s).
[0040] These computer program instructions may be stored in computer-available or computer-readable memory that can be directed toward a computer or other programmable data processing equipment to implement a function in a specific way, and the instructions stored in said computer-available or computer-readable memory may also produce a manufactured item containing instruction means that performs the function described in the flowchart block(s).
[0041] Since computer program instructions can be loaded onto a computer or other programmable data processing equipment, instructions that perform a series of operation steps on the computer or other programmable data processing equipment to create a process executed by the computer can also provide steps for executing the functions described in the flowchart block(s).
[0042] Additionally, each block may represent a module, segment, or part of code containing one or more executable instructions for executing a specified logical function(s). Furthermore, in some alternative execution examples, the functions mentioned in the blocks may occur out of order. For instance, two blocks described in succession may actually be executed substantially simultaneously, or the blocks may be executed in reverse order according to their corresponding functions.
[0043] In addition, when describing with reference to the attached drawings, identical components are assigned the same reference numeral regardless of drawing symbols, and redundant descriptions thereof are omitted. In describing the embodiments, if it is determined that a detailed description of related prior art could unnecessarily obscure the essence of the embodiments, such detailed description is omitted.
[0044] FIG. 1 is a conceptual diagram showing the RAG system (110) and RAG evaluation device (120) of the present invention.
[0045] Referring to FIG. 1, the RAG system (110) provides search result and generated response data for a user query to the RAG evaluation device (120), and the RAG evaluation device (120) can dynamically feed back evaluation results related to the RAG system (110) to the RAG system. For example, the RAG evaluation device (120) can quantify the search quality, generation quality, and reliability of the RAG system (110) as multidimensional indicators to perform an accurate performance diagnosis that reflects the mutual influence relationship between the search stage and the generation stage. For example, the RAG evaluation device (120) can automatically execute a user query based on a scheduler and diagnose whether documents updated in real time are reflected in the search results and generated responses in a periodic or event-linked manner.
[0046] In one embodiment, the RAG system (110) may include at least one of a search module, a generation module, a context input module, a vector repository, a document corpus, a log collector, an API gateway, and a security authentication module. The RAG system (110) may provide search results and generation response data for a user query to a RAG evaluation device (120).
[0047] For example, the RAG system (110) may generate at least one of a search result set, a list of document identifiers, a similarity score, a search query normalization result, a context chunk, a source citation span, a generated response text, token usage, response latency, a model version in use, a prompt in use, a search parameter value, a reranking score, a source URI, a source confidence label, and a user session identifier. For example, the RAG system (110) may generate at least one of a document loading time for a new document or an updated document, an indexing completion time, a search reflection time, and a generated response time.
[0048] For example, the RAG system (110) may construct a payload containing all or part of the generated data and transmit the generated data to a RAG evaluation device (120) using at least one of REST, gRPC, a message queue, or a streaming interface.
[0049] For example, the RAG system (110) may receive control signals, prompt update suggestions, search weight adjustment values, re-ranking thresholds, context injection policy change values, response filter rules, re-indexing triggers, cache invalidation instructions, blacklists or forbidden word lists, evaluation tracking identifiers, etc. transmitted from the RAG evaluation device (120), and reflect them in internal parameters. For example, the RAG system (110) may receive a re-evaluation request or a request for verification of the latest document reflection registered by a scheduler, and may provide search results and generated response data corresponding to the re-evaluation request or the request for verification of the latest document reflection.
[0050] In one embodiment, the RAG evaluation device (120) may include at least one of an LLM automatic scoring engine, a performance indicator calculation module, a plug-in indicator loader, a cause analysis module, a dynamic feedback controller, a data collection module, a template-based record module, a security module, and a management API. The RAG evaluation device (120) may dynamically feed back evaluation results related to the RAG system (110) to the RAG system. In addition to the above components, the RAG evaluation device (120) may include a scheduler module, and the scheduler module may determine the time of evaluation execution, the user query to be evaluated, the execution priority, and the re-execution condition.
[0051] For example, the RAG evaluation device (120) may receive at least one of search result data, generated response data, context chunk, similarity score, model and prompt version information, call log, error code, timestamp, and session identifier from the RAG system (110). For example, the RAG evaluation device (120) may receive at least one of the latest document identifier, document loading time, indexing completion time, search reflection time, and generated response time.
[0052] For example, the RAG evaluation device (120) can generate at least one of a conformity score, an accuracy score, a reliability score, a step-by-step quality score, a weighted sum quality index, a confidence interval, an explanation of the basis for evaluation, a cause analysis report, an alarm event, a policy proposal value, a control signal, an audit log, an electronic signature value, and a record timestamp. For example, the RAG evaluation device (120) can generate at least one of a user query for verifying the reflection of the latest document, a search reflection delay value, a response reflection delay value, a performance status, an anomaly, and a candidate cause of performance degradation.
[0053] For example, the RAG evaluation device (120) can transmit the policy proposal value or the control signal to the RAG system (110), record and transmit the evaluation result and audit log to a database or external audit system, and stream summary indicators and alarm events to a dashboard module. For example, the RAG evaluation device (120) can register a re-evaluation schedule with a scheduler module to re-execute the same user query or a user query under the same conditions after generating the policy proposal value or the control signal.
[0054] That is, the RAG evaluation device (120) can provide objective and comprehensive evaluation results for the RAG system (110). In addition, the RAG evaluation device (120) can optimize the performance of the RAG system (110) in real time without administrator intervention, and can maintain stable quality for a long period. Furthermore, by performing an evaluation linked to real-time document updates based on a scheduler, the RAG evaluation device (120) can identify early instances of non-reflection of the latest documents, indexing delays, or creation reflection delays that are difficult to detect with static evaluation methods.
[0055] FIG. 2 is a drawing showing an example of the RAG system (200) of the present invention.
[0056] Referring to FIG. 2, the RAG system (200) of the present invention can provide a response optimized for a user query by integrally performing search and generation processes. The RAG system (200) may include at least one of a generation module (210), a search module (220), a vector repository (230), and a document corpus (240). For example, the RAG system (200) may operate in a manner such that an external evaluation device can track the reflection status of search results and generated responses corresponding to the document update time, so as to be linked with the real-time evaluation structure of the present invention.
[0057] Specifically, the RAG system (200) receives a user query as input, searches for relevant documents in the document corpus (240) through the search module (220), and the searched documents can be selected by comparing their similarity with vectors embedded in the vector repository (230). The selected documents and the query are transmitted as input to the generation module (210), and the generation module (210) can generate a final response that combines the user query and context by utilizing a large-scale language model. The RAG system (200) can transmit the generated response, along with the source documents used in the search process, reliability indicators, and log data of the response generation process, to an external device. The RAG system (200) can receive the transmitted data back in the form of feedback after it has been analyzed by an evaluation device, and the received feedback can be reflected in autonomous performance improvements such as adjusting search weights, updating prompt templates, and updating the document corpus. Accordingly, the RAG system (200) of the present invention can continuously provide stable and optimized generative responses by integrally performing the entire process from search, generation, data recording, and feedback reflection.
[0058] In one embodiment, the generation module (210) can generate a final response by receiving a user's query and a searched context as input based on a large-scale language model. The generation module (210) may include various model architectures, prompt templates, and parameter settings, and can control the length, format, and level of detail of the final response. Additionally, the generation module (210) can record a quality log for the generated final response. The generation module (210) can perform forbidden word removal, sensitive information masking, or format normalization on the response by applying a post-processing filter to the generated response.
[0059] In one embodiment, the search module (220) can convert a user query into an embedding vector or keyword representation and use the converted representation to search for relevant document chunks in the vector repository (230). The search module (220) may include a similarity calculator, a k-nearest neighbor finder, and a reranking model. The search module (220) may calculate a confidence score, source metadata, and citation potential indicators for each search document chunk and attach the calculated indicators to the search result list.
[0060] In one embodiment, the vector storage (230) may be a database configuration for storing embedding vectors generated from original documents included in the document corpus (240) and responding to search queries. The vector storage (230) may include an ANN algorithm, a distributed storage structure, and a memory cache layer to efficiently index and search large-scale high-dimensional embedding vectors. The vector storage (230) may dynamically update or delete document embedding records by managing the document identifier, document source, and update timestamp for the document embedding records together.
[0061] In one embodiment, the document corpus (240) may be a data set that stores original documents referenced by the RAG system. The document corpus (240) may include at least one of internal corporate documents, public technical documents, and domain specialized materials. The document corpus (240) may include various data formats such as text, tables, and image captions. The document corpus (240) may be maintained in a state suitable for search and embedding generation by applying refinement rules, tokenization, and attaching metadata. The document corpus (240) may expand the scope, schema, or storage capacity of the document corpus (240) to meet the needs of the service domain by providing version control, access control, or subject-based partitioning functions.
[0062] FIG. 3 is a diagram showing the block configuration of the RAG evaluation device (300) of the present invention.
[0063] Referring to FIG. 3, the RAG evaluation device (120) of FIG. 1 may be implemented as an electronic device (300). For example, the RAG evaluation device (300) may include at least one of a server PC in an in-house or edge environment that can be connected to a RAG system, a public or private cloud server, and a database. For example, the RAG evaluation device (300) may continuously collect document update information, search results, generation responses, and time information while always connected to the RAG system to be evaluated in order to perform scheduler-based evaluation in a document environment that is updated in real time. In the following description of FIG. 3, the electronic device (300) may be the same device as the RAG evaluation device (300) of the present invention.
[0064] In one embodiment, the electronic device (300) may include an input / output interface (310), a memory (320), a processor (330), and a communication interface (340). However, the present disclosure is not limited thereto. The electronic device (300) may be configured with some of the components shown in FIG. 3 omitted, or may be configured to include other components in addition to those shown in FIG. 3. For example, the electronic device (300) may further include at least one of a scheduler module, a user query generation module, a time information tracking module, a performance status evaluation module, or a setting feedback module.
[0065] In one embodiment, the input / output interface (310), memory (320), processor (330), and communication interface (340) may each be physically / electrically connected to each other.
[0066] In one embodiment, the electronic device (300) may be connected to various types of external devices through an input / output interface (310). In one embodiment, the input / output interface (310) may include at least one of a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, a port for connecting a device equipped with an identification module (SIM), an audio I / O (Input / Output) port, or a video I / O (Input / Output) port. In one embodiment, the input / output interface (310) may include a USB (Universal Serial Bus), HDMI (High Definition Multimedia Interface), or DVI (Digital Visual Interface), etc.
[0067] In one embodiment, the memory (320) can store data used in the electronic device (300). In one embodiment, the memory (320) can store instructions, programs, or modules for the operation of the processor (330).
[0068] In one embodiment, the memory (320) may include at least one type of storage medium among a flash memory type, a hard disk type, an SSD type (Solid State Disk type), an SSD type (Silicon Disk Drive type), a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), RAM (random access memory; RAM), SRAM (static random access memory), ROM (read-only memory; ROM), EEPROM (electrically erasable programmable read-only memory), PROM (programmable read-only memory), magnetic memory, and an optical disk.
[0069] In one embodiment, the memory (320) may include user information and components displayed in the user interface.
[0070] In one embodiment, the processor (330) may include a general-purpose processor such as a CPU (Central Processing Unit), AP (Application Processor), DSP (Digital Signal Processor), or a neural network processing processor such as an NPU (Neural Processing Unit). In one embodiment, the processor (330) may be divided by one or more processors to perform operations. In one embodiment, the processor (330) may control the operation of the electronic device (300). The processor (330) may control the operation of the electronic device (300) according to instructions stored in memory (320).
[0071] In one embodiment, the processor (330) may be implemented as a memory that stores data for an algorithm or a program that reproduces the algorithm for controlling the operation of components within the electronic device (300) of the present disclosure, and as a processor that performs the aforementioned operation using the data stored in the memory. In this case, the memory and the processor may each be implemented as separate chips. Alternatively, the memory and the processor may be implemented as a single chip.
[0072] In addition, the processor may control one or a combination of the components described above in order to implement various embodiments according to the present disclosure through the electronic device (300).
[0073] In one embodiment, the communication interface (340) may include one or more components that enable communication between an external server or an external electronic device and the electronic device (300). In one embodiment, the communication interface (340) may include at least one of a wired communication module or a wireless communication module.
[0074] The wired communication module may include various wired communication modules such as a Local Area Network (LAN) module, a Wide Area Network (WAN) module, or a Value Added Network (VAN) module, as well as various cable communication modules such as USB (Universal Serial Bus), HDMI (High Definition Multimedia Interface), DVI (Digital Visual Interface), RS-232 (recommended standard 232), power line communication, or POTS (plain old telephone service).
[0075] The wireless communication module may include a wireless communication module that supports a wireless communication method including at least one of WiBro (Wireless broadband), GSM (global System for Mobile Communication), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), UMTS (universal mobile telecommunications system), TDMA (Time Division Multiple Access), LTE (Long Term Evolution), 4G, 5G, 6G, Bluetooth™, or Wi-Fi (Wireless-Fidelity).
[0076] In one embodiment, the processor (330) can obtain information from an external electronic device or server or provide information through a communication interface (340).
[0077] A processor (330) according to one embodiment can perform the operation of the RAG evaluation device (300) described below through the drawings. Specifically, the processor (330) can quantify the search quality, generation quality, and reliability of the RAG system as multidimensional indicators to perform an accurate performance diagnosis that reflects the mutual influence relationship between the search stage and the generation stage. In addition, the processor (330) can periodically diagnose the performance of the evaluation pipeline itself and update the parameters of the evaluation method based on scoring accuracy, indicator suitability, resource usage efficiency, etc. Furthermore, the processor (330) can automatically execute user queries based on a scheduler and automatically generate user queries for verifying the reflection of the latest documents from new documents or updated documents. In addition, the processor (330) can track document loading time, indexing completion time, search reflection time, and generation response time, and calculate search reflection delay values and response reflection delay values to distinguish between indexing delay or generation reflection delay. Additionally, the processor (330) can evaluate the performance status, abnormal signs, and causes of performance degradation based on whether the standard threshold of the performance indicator is met, the trend of change over time, and the pattern of change compared to past history. Additionally, the processor (330) can dynamically provide feedback on at least one of search settings, creation settings, index settings, and alarm settings according to the evaluation results, and can re-execute the same user query or a user query under the same conditions to reconfirm whether performance has been improved.
[0078] FIG. 4 is a diagram showing the system architecture and data processing flow of the RAG evaluation device (300) of the present invention.
[0079] The RAG evaluation device (300) can perform a series of processes, such as receiving an evaluation request, normalizing collected data, performing automatic scoring, aggregating indicators, recording results, and generating feedback, as shown in the data flow of the system architecture of FIG. 4. Each block and arrow shown in FIG. 4 may indicate the role of the module constituting the system architecture and the data movement path between modules.
[0080] For example, the RAG evaluation device (300) can collect results and return feedback from the RAG system that forms the evaluation target by linking with the user interface module (410), evaluation system core (420), scheduler module (425), asynchronous processing module (430), LLM automatic scoring module (440), LLM APIs (450), performance indicator module (460), multilayer caching module (470), and database (480).
[0081] Specifically, the user interface module (410) can receive an evaluation request and forward it to the evaluation system core (420). The evaluation system core (420) can perform a cache lookup in the multi-layer caching module (470) and, in the case of a cache miss, forward an execution plan to the scheduler module (425). The scheduler module (425) can determine the timing of evaluation execution based on at least one of a time condition, a document update condition, a re-evaluation condition, or a condition suspected of performance degradation, and can select or automatically generate user queries for which the search API and generation API must be called to the RAG system to be evaluated. The scheduler module (425) can forward the execution order and timing of the selected user queries to the asynchronous processing module (430). The asynchronous processing module (430) can normalize the collected search results and generation responses and forward them to the LLM automatic scoring module (440). The LLM automatic scoring module (440) can call LLM APIs (450) to generate scoring results and commentary and return them to the performance indicator module (460). The performance indicator module (460) can calculate quality indices and confidence intervals and request a record to the database (480). The evaluation system core (420) can transmit a control signal to the RAG system if the aggregate result is below a threshold. Based on the received control signal, the RAG system can adjust at least one of the search weight, re-ranking parameter, context injection policy, prompt parameter, and response filter policy. The scheduler module (425) can re-register a re-evaluation schedule for the same user query or user query under the same conditions after the setting change according to the control signal. The user interface module (410) can display comprehensive indicators, alerts, and cause analysis reports on the dashboard.
[0082] In one embodiment, the user interface module (410) may be a user screen component for setting evaluation scenarios, registering evaluation targets, uploading datasets, and visualizing dashboards. The user interface module (410) may convert evaluation scenarios entered by an operator into a message format and transmit them to the evaluation system core (420), and receive completed evaluation results and alert events to display on the screen. The user interface module (410) may also transmit metadata regarding projects, model versions, prompt versions, thresholds, and notification channels. The user interface module (410) may additionally receive input for evaluation scenarios for verifying the reflection of the latest documents, real-time re-evaluation scenarios, and settings for priority inspection target document groups.
[0083] In one embodiment, the evaluation system core (420) may be a central control module that directs the entire evaluation processing flow. When the evaluation system core (420) receives an evaluation request from the user interface module (410), it may convert the request into a standard evaluation specification and generate an execution plan including API call parameters, batch size, retry policy, experiment setting parameters, etc. The evaluation system core (420) may include a user query identifier, an evaluation scenario identifier, a latest document identifier, a reference threshold, and re-evaluation conditions in the generated execution plan and transmit it to the scheduler module (425). The evaluation system core (420) transmits the execution time, execution priority, and re-execution status calculated by the scheduler module (425) to the asynchronous processing module (430), checks for cache hit status in the multi-layer caching module (470), and, if necessary, approves API calls to the evaluation target RAG system. After the execution is completed, the evaluation system core (420) may delegate result aggregation to the performance indicator module (460) and be responsible for generating feedback signals.
[0084] In one embodiment, the scheduler module (425) may be a key component for controlling evaluation execution in a document environment that is updated in real time. The scheduler module (425) may detect whether a new document or an updated document has occurred, and may automatically generate a user query for verifying the reflection of the latest document by extracting at least some of the title item, date item, and change item from the new document or the updated document. The scheduler module (425) may calculate the execution priority for multiple user query candidates by reflecting the frequency of recent document updates, document importance, and past performance degradation history. The scheduler module (425) may track the document loading time, indexing completion time, search reflection time, and creation response time together, and calculate the search reflection delay value and the response reflection delay value using the time information. If the search reflection delay value or the response reflection delay value deviates from a reference value, the scheduler module (425) may advance the subsequent evaluation schedule or prioritize the registration of a re-evaluation. The scheduler module (425) can shorten the execution cycle or register additional evaluation scenarios for administrator verification when the same type of abnormal signs are repeated.
[0085] In one embodiment, the asynchronous processing module (430) may be an execution layer that reliably handles a large volume of evaluation requests, including a work queue and background workers. The asynchronous processing module (430) performs queuing, backoff, retries, timeouts, stage checkpoints, and progress tracking, and can execute calls between the evaluation target RAG system and the LLM automatic scoring module (440) according to user queries, execution timestamps, execution priorities, and re-execution conditions transmitted from the scheduler module (425). The asynchronous processing module (430) can prevent duplicate calls by referencing a cache key and can group and process requests of the same scenario according to a batch processing policy. The asynchronous processing module (430) can manage call schedule identifiers and progress information together for each execution and provide them to the scheduler module (425) and the evaluation system core (420).
[0086] In one embodiment, the LLM automatic scoring module (440) may be a core evaluation layer that performs automatic scoring by comparing collected search results and generated responses with predefined correct answers or search contexts. The LLM automatic scoring module (440) may apply prompt templates, scoring rules, deduction rules, and grounds citation verification rules, and call LLM APIs (450) to produce partial scores and explanations regarding suitability, accuracy, and reliability. The LLM automatic scoring module (440) may verify consistency by performing identical query re-evaluation, cross-reference of ground documents, and contradiction detection checks, and may apply a penalty to the reliability score if the discrepancy exceeds a threshold. For user queries for verifying the reflection of the latest document generated by the scheduler module (425), the LLM automatic scoring module (440) may separately include in the explanation information whether the content of the latest document has actually been reflected in the search results and generated responses.
[0087] In one embodiment, LLM APIs (450) may be call interfaces to an on-premises deployment model or an external LLM service. LLM APIs (450) may apply call policies, authentication, quotas, and rate limits, and may failover to a pre-configured alternate path if depletion of availability or a policy violation is detected. LLM APIs (450) may return scoring results and a rationale summary in a standard JSON schema.
[0088] In one embodiment, the performance indicator module (460) may be an aggregation layer that calculates a quality index and a confidence interval by performing normalization and weighted summation of step-by-step partial scores. The performance indicator module (460) may calculate Recall series indicators, Accuracy series indicators, Rank series indicators, Cost and latency indicators, and Alert indicators, and may dynamically load user-defined indicator plugins to reflect them in multidimensional evaluation indicators. The performance indicator module (460) may perform error type classification, source diversity evaluation, and citation range analysis to generate a cause analysis report. The performance indicator module (460) may aggregate the real-time evaluation status by calculating together the search reflection latency value, response reflection latency value, change trends by time point, and patterns of change compared to past history. Based on the aggregation results, the performance indicator module (460) may calculate the performance status, abnormal signs, and candidate causes of performance degradation.
[0089] In one embodiment, the multilayer caching module (470) may be a performance optimization layer that manages search result caches, generated response caches, and evaluation result caches in a hierarchical manner. The multilayer caching module (470) may use content hash-based keys, scenario parameters, model versions, and prompt versions as key components, and may support expiration policies, invalidation policies, or partial match lookups. The multilayer caching module (470) may omit the original call upon a cache hit and reduce delays and costs in the evaluation processing flow. The multilayer caching module (470) may control duplicate executions for the same latest document identifier or the same user query identifier in conjunction with the user query execution history determined by the scheduler module (425).
[0090] In one embodiment, the database (480) may be a storage layer that stores correct answer data, prompt versions, indicator history, and evaluation results. To ensure audit traceability, the database (480) may label each record unit with an electronic signature value and a timestamp, and apply masking, encrypted storage, encrypted communication, and role-based access control to personal information items. The database (480) may manage data versions, schema versions, and experiment setting parameters to ensure that results can be consistently reproduced when executed repeatedly under the same conditions. The database (480) may store call schedule identifiers, trigger types, user query identifiers, latest document identifiers, search reflection delay values, response reflection delay values, cause analysis identifiers, and feedback setting identifiers in a linked manner. The database (480) may use the linked storage structure to manage the execution conditions, evaluation results, cause determination, setting changes, or re-evaluation results of the scheduler module (425) in a traceable manner.
[0091] FIG. 5 is a flowchart showing the operation of the RAG evaluation device (300) of the present invention.
[0092] Referring to FIG. 5, the RAG evaluation device (300) of the present invention collects search result and generated response data by automatically calling the search API and generation API of the RAG system for a user query using a scheduler (operation 510), analyzes performance evaluation elements including accuracy, similarity, reliability, and whether hallucination occurs by comparing the search result and generated response data with a correct answer standard or a search context (operation 520), quantitatively scores a performance indicator including at least one of an accuracy score, a similarity score, a reliability score, and a hallucination warning indicator based on the analysis result (operation 530), evaluates the performance status, abnormal signs, and causes of performance degradation of the RAG system based on whether the reference threshold of the performance indicator is met, the trend of change over time, and the pattern of change compared to past history (operation 540), and dynamically feeds system settings including at least one of search settings, generation settings, index settings, and alarm settings to the RAG system along with the evaluation result for the RAG system to improve the performance of the RAG system (operation 550) Can do.
[0093] According to one example, in operation 510, the RAG evaluation device (300) can collect search result and generated response data by automatically calling the search API and generation API of the RAG system for a user query using a scheduler.
[0094] Here, the scheduler may be an execution control means that determines the RAG evaluation point by detecting at least one of a predefined time condition, event condition, and state change condition. A user query may be a query sentence or a set of queries input to the search API and the generation API to check the performance of the RAG system. A user query for verifying the reflection of the latest document may be a query generated to include at least two items among the title, date, and changes of a new document or an updated document. The search API may be a call interface for searching for a document or part of a document corresponding to the user query. The generation API may be a call interface for producing generated response data by referring to the call result of the search API. The search result may be information including at least one of identification information of the searched document, top exposure order, relevance information, whether the search is reflected, and the search execution time. The generated response data may be information including at least one of a response sentence, response generation time, response delay time, whether the latest document is reflected, and source document link information. A new document or an updated document may be a document newly loaded into the document set of the RAG system or a document whose existing content has been modified. Time information may be information including at least one of document loading time, indexing completion time, search reflection time, and generation response time. User query candidates may be a set of multiple user queries that the scheduler can select as execution targets. The priority value may be a value calculated based on the recent document update frequency U, document importance C, and historical performance degradation score E.
[0095] FIG. 6 is a block diagram (600) showing the operation of the RAG evaluation device (300) of the present invention collecting search results and generated response data using a scheduler, and FIG. 7 is a flowchart showing the operation of the RAG evaluation device (300) of the present invention collecting search results and generated response data by automatically calling the search API and generation API of the RAG system for a user query using a scheduler.
[0096] Referring to FIGS. 6 and 7, the RAG evaluation device (300) can detect at least one of a predefined time condition, event condition, and state change condition using a scheduler (operation 710), determine a time for RAG evaluation using a scheduler (operation 720), automatically execute the search API and the generation API of the RAG system at the determined time for RAG evaluation (operation 730), and collect the search result and the generation response data based on the call results of the search API and the generation API (operation 740).
[0097] In one embodiment, as in operation 710, the RAG evaluation device (300) can detect at least one of a predefined time condition, event condition, and state change condition using a scheduler.
[0098] Time conditions may be conditions for preparing the RAG evaluation point based on preset time intervals, specific times, specific days of the week, or the elapsed time since a document update. Event conditions may be conditions detected based on the occurrence of new documents, updated documents, completion of document loading, completion of indexing, or configuration changes. State change conditions may be conditions detected based on recent performance degradation history, changes in query throughput, changes in error frequency, or a state of document update concentration.
[0099] The RAG evaluation device (300) can monitor the database status of the RAG system through the scheduler. The RAG evaluation device (300) can determine whether the event condition is satisfied when a new document is loaded into the database or an existing document is updated. The RAG evaluation device (300) can check the loading time of the new document or the updated document, the latest document identification information, and the document change item. The RAG evaluation device (300) can determine whether the document change item includes at least one of a title change, a date change, an addition of changes, a change in numerical information, or a change in application conditions.
[0100] The RAG evaluation device (300) can compare the current time with a preset evaluation cycle when detecting the time condition. When detecting the event condition, the RAG evaluation device (300) can check the document loading status, indexing execution status, and whether a new document has occurred in the RAG system. When detecting the state change condition, the RAG evaluation device (300) can compare at least one of the number of recent errors, call delay time, query throughput, and performance degradation history score. The RAG evaluation device (300) can determine that an evaluation of the user query is necessary if any one of the time condition, the event condition, and the state change condition is satisfied above a threshold value.
[0101] The RAG evaluation device (300) can store the detection result together with call schedule identification information. The RAG evaluation device (300) can store the detection result in conjunction with a trigger type, latest document identification information, detection time, and a set of related user query candidates. The RAG evaluation device (300) can use the detection result as basis information for determining the RAG evaluation time point in a subsequent step.
[0102] In one embodiment, as in operation 720, the RAG evaluation device (300) can determine the time for RAG evaluation using a scheduler.
[0103] The RAG evaluation point may be the time or point in time when the search API and the generation API must be executed. The user query candidate may be a set of multiple user queries that the scheduler can select as an execution target. The priority value P may be a value calculated based on the recent document update frequency U, document importance C, and historical performance degradation score E.
[0104] The RAG evaluation device (300) may determine the RAG evaluation time point as an immediate execution time point, a delayed execution time point, or a periodic execution time point depending on the type and intensity of the detected condition. When the occurrence of a new document or an updated document is detected, the RAG evaluation device (300) may advance the RAG evaluation time point to verify early whether the latest document has been reflected. When an indexing completion event is detected, the RAG evaluation device (300) may determine the RAG evaluation time point immediately after the indexing is completed. When a setting change event is detected, the RAG evaluation device (300) may separately determine an evaluation time point to check the impact of the changed system settings.
[0105] The RAG evaluation device (300) can calculate the execution priority of the user query candidates when there are multiple user query candidates. For example, the RAG evaluation device (300) can calculate the priority value P using the following [Equation 1].
[0106] [Formula 1]
[0107] P = αU + βC + γE
[0108] Here, the priority value P may be a numerical value representing the execution priority of multiple user query candidates, the recent document update frequency U may be a value representing the number of occurrences or update density of new or updated documents that occurred in the corresponding document group over a certain period, the document importance C may be a value reflecting the business impact, service importance, or recency requirement level of the document, and the historical performance degradation score E may be a value representing the cumulative level of history of accuracy degradation, non-reflection of recent documents, reliability degradation, or hallucination warnings that occurred in the past in the same user query or related document group. Additionally, the weights α, β, and γ may be coefficients for adjusting the reflection ratio of the recent document update frequency U, document importance C, and historical performance degradation score E, respectively.
[0109] Specifically, the RAG evaluation device (300) can collect a recent document update frequency U, a document importance C, and a past performance degradation history score E for each user query candidate. The RAG evaluation device (300) can calculate a priority value P for each user query candidate using the recent document update frequency U, the document importance C, and the past performance degradation history score E. The RAG evaluation device (300) can determine the execution order of the user queries and the RAG evaluation time so that user query candidates whose priority value P is greater than or equal to a reference value are executed preferentially.
[0110] The RAG evaluation device (300) may prioritize the placement of the user query for verifying the reflection of the latest document when such a user query is required. The RAG evaluation device (300) may generate the user query for verifying the reflection of the latest document by combining at least two items among the title, date, and changes of a new document or an updated document. The RAG evaluation device (300) may adjust the order of the user queries to execute first the user query corresponding to an area with a high risk of document update or high document importance. The RAG evaluation device (300) may store the determined RAG evaluation time and the user query identification information together.
[0111] In one embodiment, as in operation 730, the RAG evaluation device (300) can automatically execute the search API and the generation API of the RAG system at the determined time of the RAG evaluation.
[0112] A search API may be a call interface for searching for a document or part of a document corresponding to a user query. A generation API may be a call interface for producing generation response data by referencing the call result of the search API. User query identification information may be identification information for linking the call result of the search API and the call result of the generation API into a single evaluation set corresponding to the same user query.
[0113] The RAG evaluation device (300) may first execute the search API corresponding to the user query when the determined RAG evaluation time arrives. After executing the search API, the RAG evaluation device (300) may execute the generation API that includes the search result as a reference target. The RAG evaluation device (300) may assign the same user query identification information to the execution of the search API and the execution of the generation API. By maintaining the call order of the search API and the generation API, the RAG evaluation device (300) can clearly maintain the correspondence between the search result and the generation response data.
[0114] The RAG evaluation device (300) can check whether the call was successful during the execution process of the search API and the generation API. The RAG evaluation device (300) can determine whether to re-call the search API or the generation API if a call failure occurs. The RAG evaluation device (300) can store the call failure history together with user query identification information. The RAG evaluation device (300) can use the call failure history as reference information for determining abnormal signs in a subsequent step. The RAG evaluation device (300) can store the response delay time together if the response time of the search API or the generation API exceeds a reference time.
[0115] The RAG evaluation device (300) can link the latest document identification information and time information together at the time of execution of the search API and the generation API. The RAG evaluation device (300) can use the linkage structure to verify in what latest document situation the search result and the generation response data were obtained.
[0116] In one embodiment, as in operation 740, the RAG evaluation device (300) can collect the search results and the generated response data based on the results of calling the search API and the generated API.
[0117] Search results may be information including at least one of identification information of the searched document, top exposure order, relevance information, whether the search is reflected, search execution time, whether the latest document is included, and whether the search failed. Generated response data may be information including at least one of a response sentence, response generation time, response delay time, whether the latest document is reflected, source document linkage information, and key fact information included in the response. An evaluation set may be a data set for storing the search results and the generated response data together in combination, corresponding to the same user query. Time information may be information including at least one of document loading time, indexing completion time, search reflection time, and generation response time.
[0118] The RAG evaluation device (300) can collect the search results from the results of the call to the search API. In the process of collecting the search results, the RAG evaluation device (300) can collect at least one of identification information of the search target document, top exposure order, relevance information, search execution time, whether the latest document is included, and whether the search failed. The RAG evaluation device (300) can collect the generated response data from the results of the call to the generation API. In the process of collecting the generated response data, the RAG evaluation device (300) can collect at least one of the response sentence, response generation time, response delay time, whether the latest document is reflected, connection information between the response and the source document, and key fact information included in the response.
[0119] The RAG evaluation device (300) can store the search results and the generated response data individually, while simultaneously grouping them into a single evaluation set for the same user query. The RAG evaluation device (300) may include user query identification information, the execution time of the user query, the call sequence number of the search API, and the call sequence number of the generation API in the evaluation set. The RAG evaluation device (300) may include the latest document identification information and time information together in the evaluation set. The RAG evaluation device (300) may include call schedule identification information and trigger type together in the evaluation set.
[0120] The RAG evaluation device (300) can extract time information for a new document or an updated document and store it in conjunction with the search results and the generated response data. The RAG evaluation device (300) can extract at least one of the document loading time, indexing completion time, search reflection time, and generation response time for the new document or the updated document. The RAG evaluation device (300) can link the time information with the latest document identification information. The RAG evaluation device (300) can link the time information with the document identification information included in the search results. The RAG evaluation device (300) can store the time information in conjunction with the response generation time of the generated response data. The RAG evaluation device (300) can determine a search reflection delay or a generation reflection delay in a subsequent step using the linked storage structure of the time information.
[0121] The RAG evaluation device (300) can accumulate and store the call results of the search API and the call results of the generation API for each user query. The RAG evaluation device (300) can store the search results and generation response data at multiple points in time for the same user query in a time-series arrangement. The RAG evaluation device (300) can use the time-series storage structure to track whether the latest documents regarding the user query are reflected, changes in the top exposure order, changes in the response generation time, and changes in the response content. The RAG evaluation device (300) can use the time-series storage structure to evaluate the trend of change by point in time and the pattern of change compared to past history in a subsequent step.
[0122] The RAG evaluation device (300) can store at least one of the following for each user query execution: call schedule identification information, trigger type, user query identification information, latest document identification information, search reflection delay value, response reflection delay value, cause analysis identification information, and feedback setting identification information. The RAG evaluation device (300) can use the connection storage structure to verify whether the search result and the generated response data were collected according to a certain judgment of the scheduler. The RAG evaluation device (300) can use the connection storage structure to consistently track the reflection status of a specific latest document, the cause analysis of subsequent steps, and the history of system setting changes.
[0123] The RAG evaluation device (300) can store whether the call failed, whether it was re-called, and the response delay time together when the call to the search API or the generation API fails or is delayed. The RAG evaluation device (300) can use the call failure history and the response delay time as reference information for determining abnormal signs in a subsequent step. By securing the latest document reflection status, time information, and execution history together in the collection step of the search results and the generation response data, the RAG evaluation device (300) can obtain data for performance evaluation element analysis, performance indicator scoring, performance status evaluation, and system setting feedback in a subsequent step.
[0124] According to one example, in operation 520, the RAG evaluation device (300) can analyze performance evaluation factors including accuracy, similarity, reliability, and whether hallucination occurs by comparing the search result and the generated response data with a correct answer standard or search context.
[0125] Here, the correct answer criterion may be a sentence, information item, document set, or reference information with added version information that can be recognized as the correct answer to a user query. The search context may be document content, latest document content, supporting sentence, or a set of document parts that the generated response data must refer to in response to the user query.
[0126] Here, accuracy may be a performance evaluation factor indicating how correctly the generated response data reflects the key facts, numerical information, date information, and latest changes of the correct answer criteria. Similarity may be a performance evaluation factor indicating how semantically close the search result or the generated response data is to the correct answer criteria or the search context. Reliability may be a performance evaluation factor indicating how consistent the generated response data is with the grounds included in the search result, whether contradictions exist within the generated response data, and how consistent the execution results are across multiple time points for the same user query. The occurrence of hallucination may be a performance evaluation factor indicating whether the generated response data adds facts that do not exist in the correct answer criteria or the search context, or distorts existing facts.
[0127] Here, the latest change may be at least one of the title, date, change item, numeric change, or application condition change included in the new document or updated document. Time information may be information including at least one of the time of execution of the user query, the document loading time, the indexing completion time, the search reflection time, and the generation response time.
[0128] FIG. 8 is a drawing showing an example (800) of a multidimensional evaluation index according to automatic scoring of the RAG evaluation device (300) of the present invention.
[0129] Referring to FIG. 8, the RAG evaluation device (300) can perform accuracy evaluation, similarity analysis, reliability index calculation, and hallucination detection on the search results and the generated response data.
[0130] For example, the accuracy evaluation may be an item for evaluating the degree of agreement with the correct answer standard, and may be performed using at least one of BLEU, the correct answer inclusion rate, or whether the core facts match. For example, the similarity analysis may be an item for evaluating semantic proximity with the correct answer standard or the search context, and may be performed using at least one of Embedding similarity, BERTScore, or Sentence Transformer-based similarity. For example, the reliability index may be an item reflecting the confidence level, probability-based judgment signal, or level of evidence consistency of the generated response data. For example, hallucination detection may be an item for analyzing whether incorrect information is included, whether facts are distorted, or whether unfounded descriptions occur, and for making this into a warning indicator. That is, the RAG evaluation device (300) can evaluate the response quality of the RAG system in a multidimensional way by reflecting the accuracy evaluation, the similarity analysis, the reliability index, and the hallucination detection together.
[0131] The RAG evaluation device (300) can select a correct answer criterion or search context that is valid at the time of execution of the user query. When selecting the correct answer criterion or search context, the RAG evaluation device (300) may consider at least one of the document version, the document update time, the document application start time, and the document validity end time together. If the user query is executed at a specific time, the RAG evaluation device (300) may adopt only the correct answer criterion or search context that is valid at that specific time as a comparison target. The RAG evaluation device (300) may exclude from the comparison target document content that was valid at a past time but is not valid at the time of execution of the user query. By ensuring the time consistency of the correct answer criterion or search context, the RAG evaluation device (300) can reliably analyze performance evaluation factors even in a document environment that is updated in real time.
[0132] The RAG evaluation device (300) can decompose the search results and the generated response data into at least one of a core fact unit, an attribute item unit, and a sentence unit. The RAG evaluation device (300) can set the core fact unit as a judgment unit including at least one of title information, implementation date information, change content information, numerical information, target condition information, and exception condition information. The RAG evaluation device (300) can set the attribute item unit as an individual field or individual semantic item included in the correct answer criterion or search context. The RAG evaluation device (300) can partition the sentence unit according to the sentence boundary, phrase boundary, or basis connection section of the generated response data. By aligning the search results and the generated response data into the same comparison unit, the RAG evaluation device (300) can analyze the accuracy, similarity, reliability, and whether hallucination occurs for each comparison unit.
[0133] The RAG evaluation device (300) can first verify the correspondence between the search result and the correct answer criteria or search context. Additionally, the RAG evaluation device (300) can verify the correspondence between the generated response data and the correct answer criteria or search context. The RAG evaluation device (300) can identify candidate grounds corresponding to the user query by utilizing document identification information, document ranking, relevance information, and whether the latest documents are included in the search result. The RAG evaluation device (300) can track which document or sentence of the search result is connected to each sentence or each key fact of the generated response data. The RAG evaluation device (300) can analyze whether the generated response data appropriately reflects the search result by utilizing the connection relationships. The RAG evaluation device (300) can identify the latest changes that exist in the search result but are missing from the generated response data by utilizing the connection relationships. The RAG evaluation device (300) can identify data that exists in the generated response data but is not present in the search result by utilizing the connection relationships.
[0134] The RAG evaluation device (300) can determine whether the generated response data accurately includes the core facts of the correct answer criteria when analyzing accuracy. The RAG evaluation device (300) can determine whether the generated response data correctly reflects the numerical value, date value, name value, and condition value of the correct answer criteria when analyzing accuracy. When analyzing accuracy, the RAG evaluation device (300) can include whether the generated response data reflects the latest changes as a separate judgment factor. If the latest changes exist, the RAG evaluation device (300) can process whether the latest changes are reflected as a core accuracy item. If the latest changes are not reflected, the RAG evaluation device (300) can determine that there is a decrease in accuracy even if past information is naturally included in the sentence structure. The RAG evaluation device (300) can determine that there is a partial decrease in accuracy if some of the core facts are omitted, and can determine that there is a significant decrease in accuracy if the core facts are incorrectly written.
[0135] When analyzing similarity, the RAG evaluation device (300) can determine whether the generated response data conveys semantically identical content even if it does not use the same words as the correct answer criterion or the search context. When analyzing similarity, the RAG evaluation device (300) can also consider how closely the search result places the core document related to the user query in the top rank. When analyzing similarity, the RAG evaluation device (300) can separately calculate the semantic proximity between the generated response data and the correct answer criterion, and the semantic proximity between the search result and the search context. The RAG evaluation device (300) can determine high similarity if the core meaning is maintained even if the expression of the generated response data is abbreviated or the sentence order is changed. The RAG evaluation device (300) can determine low similarity if the generated response data omits core conditions or includes opposite meanings.
[0136] The RAG evaluation device (300) can determine whether each sentence of the generated response data is directly connected to the grounds included in the search results when analyzing reliability. The RAG evaluation device (300) can determine whether there is mutual contradiction between the sentences within the generated response data when analyzing reliability. The RAG evaluation device (300) can determine whether the execution results of multiple times for the same user query maintain a consistent response structure under the same time conditions when analyzing reliability. When analyzing reliability, the RAG evaluation device (300) can additionally determine whether the generated response data preferentially reflects the content of the latest document when the latest document is included in the search results. The RAG evaluation device (300) can determine reliability as low if the generated response data definitively states content unrelated to the search results. The RAG evaluation device (300) can determine reliability as low if the generated response data mixes the content of multiple documents included in the search results while maintaining conflicting information as is.
[0137] When analyzing whether hallucination has occurred, the RAG evaluation device (300) can determine whether the generated response data adds new facts that do not exist in the correct answer criteria or search context. When analyzing whether hallucination has occurred, the RAG evaluation device (300) can determine whether the generated response data modifies or exaggerates facts existing in the search results. When analyzing whether hallucination has occurred, the RAG evaluation device (300) can determine whether the generated response data maintains past information that conflicts with the latest changes. When analyzing whether hallucination has occurred, the RAG evaluation device (300) can determine whether the generated response data includes numbers, dates, or conditions that are not in the search results. The RAG evaluation device (300) can determine that hallucination has occurred if the generated response data includes content without grounds. The RAG evaluation device (300) can determine that hallucination has occurred even if the generated response data includes ground sentences, but presents a conclusion that deviates from the meaning of the ground sentences.
[0138] The RAG evaluation device (300) may treat whether the latest changes are reflected in the analysis process as an independent comparison factor. The RAG evaluation device (300) may determine whether at least two items among the title, date, and changes of the new document or the updated document are included in the generated response data when a new document or an updated document exists. If the search result does not include the new document or the updated document, the RAG evaluation device (300) may store the possibility of a problem related to the search step as reference information for a subsequent step. If the search result includes the new document or the updated document but the generated response data does not reflect the core content of the new document or the updated document, the RAG evaluation device (300) may store the possibility of a problem related to the generation step as reference information for a subsequent step. The RAG evaluation device (300) may store the analysis results for user queries for verifying the reflection of the latest document separately from the analysis results for user queries for general evaluation.
[0139] The RAG evaluation device (300) may refer to time information when selecting a valid correct answer criterion or search context at the time of execution of the user query. The RAG evaluation device (300) may determine the valid time point of the correct answer criterion or search context based on at least one of the document loading time, indexing completion time, search reflection time, and generation response time. The RAG evaluation device (300) may store the possibility of a lack of up-to-dateness of the search results as analysis reference information if the search reflection time is delayed after the document loading time. The RAG evaluation device (300) may store the possibility of a lack of up-to-dateness of the generation response data as analysis reference information if the generation response time is further delayed after the search reflection time. By storing the time information and the comparison result together, the RAG evaluation device (300) may enable the evaluation of the cause of performance degradation using the search reflection delay value or the response reflection delay value in a subsequent step.
[0140] The RAG evaluation device (300) can structure and store the analysis results by comparison unit. The RAG evaluation device (300) may include at least one of the following in the analysis results: whether core facts match, whether attribute items match, whether sentence meanings match, whether evidence is linked, whether the latest changes are reflected, and whether hallucination occurs. The RAG evaluation device (300) can store user query identification information, search result identification information, generated response data identification information, document identification information, and time information corresponding to the analysis results together. By storing the analysis results, the RAG evaluation device (300) can quantitatively calculate accuracy scores, similarity scores, reliability scores, and hallucination warning indicators in a subsequent step. By storing the analysis results, the RAG evaluation device (300) can more objectively evaluate performance status, abnormal signs, and causes of performance degradation in a subsequent step.
[0141] The RAG evaluation device (300) can accumulate and store analysis results for the search results and the generated response data for each user query. The RAG evaluation device (300) can perform time-series data analysis using the accumulated stored results. For example, the RAG evaluation device (300) can check changes in comparison results for the same user query over time using the accumulated stored results. For example, the RAG evaluation device (300) can check whether the reflection of the latest changes in a specific user query is repeatedly degraded using the accumulated stored results. For example, the RAG evaluation device (300) can check whether the consistency of the basis of the generated response data is repeatedly degraded using the accumulated stored results. For example, the RAG evaluation device (300) can evaluate the trend of change over time and the pattern of fluctuation compared to past history in a subsequent step using the accumulated stored results.
[0142] The RAG evaluation device (300) can store the analysis results and the execution history for each user query as a set by mutually labeling them. The RAG evaluation device (300) may include user query identification information, recent document identification information, time information, and evaluation result identification information for subsequent steps in the labeling set. The RAG evaluation device (300) can use the labeling set to track which search results and which generated response data were affected by changes to a specific recent document. The RAG evaluation device (300) can use the labeling set to consistently link the basis for re-evaluating system setting changes and performance improvements performed in subsequent steps.
[0143] The RAG evaluation device (300) may apply multiple comparison criteria in parallel to ensure objectivity in the analysis step of comparing the search results and the generated response data with the correct answer standard or search context. The RAG evaluation device (300) may store each comparison result separately when the core fact unit comparison result, the attribute item unit comparison result, and the sentence unit comparison result are different from each other. The RAG evaluation device (300) may enable normalization and weighting of the performance indicators in a subsequent step by utilizing the separate storage structure. The RAG evaluation device (300) may specifically identify quality degradation occurring only in a specific comparison unit by utilizing the separate storage structure.
[0144] According to one example, in operation 530, the RAG evaluation device (300) can quantitatively score performance indicators including at least one of an accuracy score, a similarity score, a reliability score, and a hallucination warning indicator based on the analysis results.
[0145] Here, the accuracy score may be a numerical value indicating how accurately the generated response data reflects the key facts, numerical information, date information, and latest changes of the correct answer criteria. The similarity score may be a numerical value indicating how semantically close the search result or the generated response data is to the correct answer criteria or search context. The confidence score may be a numerical value indicating how consistent the generated response data is with the grounds included in the search result, whether contradictions exist within the generated response data, and how consistent the results of multiple executions for the same user query are. The hallucination warning indicator may be a value indicating the level of warning or risk regarding the extent to which the generated response data contains unfounded content, distorted facts, or descriptions that conflict with the search result. The performance indicator may be quantitative evaluation information comprising at least one of the accuracy score, the similarity score, the confidence score, and the hallucination warning indicator. Normalization may be a process of converting multiple scores having different ranges or distributions into a common comparison range. The weight may be a coefficient for reflecting the importance of at least one of whether the latest information is reflected, time accuracy, and reasoning accuracy. Time accuracy may be a criterion indicating how well the search results and the generated response data match the correct answer criteria or search context valid at the time of execution of the user query. Reasoning accuracy may be a criterion indicating how well each sentence or each key fact of the generated response data is connected to the evidence included in the search results.
[0146] The RAG evaluation device (300) can receive judgment values for each comparison unit included in the analysis result. The RAG evaluation device (300) can verify from the judgment values for each comparison unit whether core facts match, whether attribute items match, whether sentence meanings match, whether evidence is linked, whether the latest changes are reflected, and whether hallucination occurs. The RAG evaluation device (300) can input the judgment values for each comparison unit into a scoring module. The scoring module may be configured to convert the judgment values for each comparison unit into numerical performance indicators.
[0147] In one embodiment, when calculating the accuracy score, the RAG evaluation device (300) may first determine whether there is a match for each key fact of the correct answer criteria. The RAG evaluation device (300) may classify at least one of the title information, date information, change information, numerical information, applicable target information, and exception condition information included in the correct answer criteria as an individual evaluation item. For each evaluation item, the RAG evaluation device (300) may quantify whether the generated response data is included, whether it is accurately expressed, and whether the latest changes are reflected. The RAG evaluation device (300) may assign a high accuracy score if all key facts are accurately reflected. The RAG evaluation device (300) may assign a medium accuracy score if only some key facts are reflected. The RAG evaluation device (300) may assign a low accuracy score if key facts are omitted or incorrectly stated. The RAG evaluation device (300) may assign a larger deduction value than for general omission items if the latest changes are omitted. Since the latest changes may be key information included in documents added or updated in real time, whether the latest changes are reflected may be a key factor in reflecting the real-time nature of the present invention in performance indicators.
[0148] In one embodiment, the RAG evaluation device (300) may evaluate semantic proximity between the generated response data and the correct answer criterion when calculating a similarity score. The RAG evaluation device (300) may additionally evaluate semantic proximity between the search result and the search context when calculating a similarity score. The RAG evaluation device (300) may assign a high similarity score if the core meaning is the same even if the sentence expression is different. The RAG evaluation device (300) may assign a low similarity score if the meaning is different even if a core word is included. The RAG evaluation device (300) may separately calculate the similarity between the generated response data and the correct answer criterion and the similarity between the search result and the search context, and then combine them. The separate calculation structure may distinguish and reflect the degree of relevant information selection in the search stage and the degree of semantic retention in the generation stage. The separate calculation structure may be used as basic information to evaluate the cause of performance degradation as a cause related to search settings or a cause related to generation settings in a subsequent stage.
[0149] In one embodiment, when calculating a reliability score, the RAG evaluation device (300) can verify whether each sentence or each key fact of the generated response data is connected to which document or which basis sentence of the search result. The RAG evaluation device (300) may assign a higher reliability score as the basis connection ratio is higher. The RAG evaluation device (300) may apply a deduction value if there are mutually contradictory sentences within the generated response data. The RAG evaluation device (300) may assign a high reliability score if the execution results of multiple times for the same user query maintain a consistent response structure under the same time condition. The RAG evaluation device (300) may apply a deduction value if the execution results of multiple times for the same user query fluctuate unnecessarily significantly. The RAG evaluation device (300) may apply an additional deduction value to the reliability score if the generated response data does not reflect the latest document even though the latest document is included in the search result. Since evidence consistency can indicate how faithfully the generated response data follows the actual search evidence, evidence consistency can be a key criterion for calculating the confidence score.
[0150] In one embodiment, when calculating a hallucination warning indicator, the RAG evaluation device (300) can identify at least one of groundless additional sentences, groundless distorted sentences, sentences not reflecting the latest information, and sentences maintaining conflicting information present in the generated response data. The RAG evaluation device (300) can assign different risk weights according to each error type. The RAG evaluation device (300) can assign a high risk level if groundless additional sentences are included. The RAG evaluation device (300) can assign a low risk level if they correspond to simple differences in expression. The RAG evaluation device (300) can raise the warning level if multiple risk sentences exist simultaneously. The RAG evaluation device (300) can convert the occurrence of hallucination not only into binary values but also into multiple warning values or continuous risk values. The staged structure of the hallucination warning indicator allows for more precise evaluation of the intensity of abnormal signs and the severity of the cause of performance degradation in subsequent stages.
[0151] The RAG evaluation device (300) can perform normalization processing on the accuracy score, the similarity score, the reliability score, and the hallucination warning indicator. For example, the RAG evaluation device (300) can perform normalization processing because the accuracy score, the similarity score, the reliability score, and the hallucination warning indicator may have different value ranges. The RAG evaluation device (300) can convert each score into a common comparison range through the normalization processing. The RAG evaluation device (300) can reduce distortion caused by simple range differences between scores through the normalization processing. The RAG evaluation device (300) can prepare multiple performance indicators in a state where they can be combined through the normalization processing. The normalization range may be a range of 0 to 100, a range of 0 to 1, or a pre-set score interval.
[0152] The RAG evaluation device (300) may assign weights to the normalized performance indicators. The RAG evaluation device (300) may use at least one of the following as a weight calculation criterion: whether the latest information is reflected, time-span consistency, and evidence consistency. The RAG evaluation device (300) may set a high weight corresponding to whether the latest information is reflected for user queries used to verify the reflection of the latest document. The RAG evaluation device (300) may set a high weight corresponding to time-span consistency for user queries executed immediately after a document update. The RAG evaluation device (300) may set a high weight corresponding to evidence consistency for query types where the connectivity of the evidence sentence is important. Whether the latest information is reflected, time-span consistency, and evidence consistency may be judgment factors for reflecting the real-time nature and operational environment suitability of the present invention within a numerical system.
[0153] The RAG evaluation device (300) can calculate an overall performance value using the normalized score and the weight. The RAG evaluation device (300) can calculate the overall performance value in the form of a single total score. The RAG evaluation device (300) can store the overall performance value in parallel with individual indicators. When calculating the overall performance value, the RAG evaluation device (300) can reflect deduction values for items with missing recent changes, deduction values for additional sentences without basis, and deduction values for the ratio of unconnected basis. When calculating the overall performance value, the RAG evaluation device (300) can reflect additional addition values if the latest document is accurately searched and the generated response data faithfully reflects the latest document. The overall performance value may serve as a basic indicator for evaluating the performance status, abnormal signs, and causes of performance degradation in a subsequent step.
[0154] The RAG evaluation device (300) can store the performance indicators for each user query. The RAG evaluation device (300) can store the performance indicators in conjunction with search result identification information, generated response data identification information, latest document identification information, and time information. The RAG evaluation device (300) can store the performance indicators cumulatively by time point. The RAG evaluation device (300) can track the trend of change in accuracy score, similarity score, reliability score, and hallucination warning indicator for the same user query using the cumulative storage structure by time point. The RAG evaluation device (300) can evaluate the pattern of change compared to past history in a subsequent step using the cumulative storage structure by time point.
[0155] FIG. 9 is a drawing showing an example (900) in which the RAG evaluation device (300) of the present invention outputs a performance evaluation indicator.
[0156] Referring to FIG. 9, the RAG evaluation device (300) can visually output an accuracy score, a similarity score, and a reliability score through a screen in the form of a performance dashboard.
[0157] For example, the RAG evaluation device (300) can display the accuracy score in the form of a circular indicator. Additionally, the RAG evaluation device (300) can display the accuracy score as a percentage value.
[0158] For example, the RAG evaluation device (300) can display the similarity score in the form of a circular indicator. Additionally, the RAG evaluation device (300) can display the similarity score as a numerical value.
[0159] For example, the RAG evaluation device (300) can display the reliability score in the form of a circular indicator. Additionally, the RAG evaluation device (300) can display the reliability score as a percentage value.
[0160] Additionally, the RAG evaluation device (300) can output the pattern of change of performance indicators for multiple points in time in the form of a line graph through a performance trend area. The RAG evaluation device (300) can also output the current status corresponding to accuracy, similarity, and reliability together in the form of bar indicators.
[0161] The RAG evaluation device (300) can intuitively provide the current performance level and the trend of change compared to the past using the performance dashboard, and can provide visual reference information for determining the performance status, abnormal signs, and causes of performance degradation in subsequent steps.
[0162] In one embodiment, the RAG evaluation device (300) may cross-label the performance indicators together with user query identification information when storing the performance indicators. When storing the performance indicators, the RAG evaluation device (300) may link the performance indicators with the latest document identification information, document loading time, search reflection time, generation response time, and subsequent cause evaluation identification information. The RAG evaluation device (300) may use the linked storage structure to determine in a subsequent step whether a decline in the performance indicator of a specific user query is related to a delay in reflecting a specific latest document. The RAG evaluation device (300) may use the linked storage structure to compare changes in performance indicators before and after a system setting change. The linked storage structure may be used as base information for linking and storing re-evaluation results and feedback settings.
[0163] The RAG evaluation device (300) may apply correction rules when missing values or error values occur during the process of calculating the performance indicators. If there is no supporting sentence corresponding to a specific comparison unit, the RAG evaluation device (300) may assign a pre-set minimum score or risk weight to the comparison unit. If the search results are not collected, the RAG evaluation device (300) may apply a conservative deduction value to the similarity score or the reliability score. If the generated response data is abnormally short or abnormally long, the RAG evaluation device (300) may apply a correction value that reflects the possibility of quality degradation. The correction rules may reduce excessive distortion of the performance indicators and increase the stability of subsequent judgments.
[0164] The RAG evaluation device (300) can calculate the above performance indicators by distinguishing between general user queries and user queries for verifying the reflection of the latest documents. The RAG evaluation device (300) can apply weights centered on overall response quality to general user queries. The RAG evaluation device (300) can apply weights centered on whether the latest information is reflected and time consistency to user queries for verifying the reflection of the latest documents. The RAG evaluation device (300) can separately store the above performance indicators by query type. The RAG evaluation device (300) can enable subsequent evaluations that distinguish between general quality degradation and degradation of the reflection of the latest documents using the above performance indicators by query type.
[0165] The RAG evaluation device (300) can convert the quality of the search results and the generated response data into a storable and comparable form by quantitatively scoring a performance indicator including at least one of an accuracy score, a similarity score, a reliability score, and a hallucination warning indicator based on the analysis results. The RAG evaluation device (300) can use the performance indicator to evaluate, in a subsequent step, whether the standard threshold of the performance indicator is met, the trend of change over time, and the pattern of change compared to past history. The RAG evaluation device (300) can use the performance indicator to more objectively evaluate the performance status, abnormal signs, and causes of performance degradation of the RAG system.
[0166] According to one example, in operation 540, the RAG evaluation device (300) can evaluate the performance status, abnormal signs, and causes of performance degradation of the RAG system based on whether the reference threshold of the performance indicator is met, the trend of change over time, and the pattern of change compared to past history.
[0167] Here, the performance indicator may be quantitative evaluation information including at least one of the accuracy score, similarity score, confidence score, and hallucination warning indicator. The threshold may be a standard value that must be met for each performance indicator to be maintained at a normal level. The trend of change over time may be information indicating the direction of change, the speed of change, and the magnitude of fluctuation of the performance indicator accumulated over multiple time points. The pattern of change compared to historical data may be information indicating the difference between past and current performance indicators for the same user query, the same query type, the same document group, or the same configuration conditions. The performance status may be a judgment result indicating whether the RAG system corresponds to a normal state, a borderline state, or a degraded state. An anomaly may be a signal indicating the possibility of future performance degradation even if the current performance indicator meets the threshold. The cause of performance degradation may be a type of cause estimated based on the search results, the generated response data, the performance indicator, and time information. The search reflection delay value Dr may be a value based on the difference between the search reflection time and the document loading time. The response reflection delay value Da may be a value based on the difference between the generation response time and the document loading time mentioned above. The reference search delay value τr may be a reference value for determining the search reflection delay. The reference response delay value τa may be a reference value for determining the generation reflection delay. The latest document may be a document that is valid at the current time among new documents or updated documents.
[0168] The RAG evaluation device (300) can calculate a search reflection delay value Dr based on the difference between the search reflection time and the document loading time. For example, the search reflection delay value Dr can be calculated using the following [Equation 2].
[0169] [Formula 2]
[0170] Dr = t_retrieval - t_ingest
[0171] The RAG evaluation device (300) can calculate a response reflection delay value Da based on the difference between the generation response time and the document loading time. For example, the response reflection delay value Da can be calculated using the following [Equation 3].
[0172] [Formula 3]
[0173] Da = t_answer - t_ingest
[0174] The RAG evaluation device (300) can determine that indexing delay is the cause of performance degradation if the search reflection delay value Dr exceeds the reference search delay value τr. The RAG evaluation device (300) can determine that creation reflection delay is the cause of performance degradation if the search reflection delay value Dr is less than or equal to the reference search delay value τr and the response reflection delay value Da exceeds the reference response delay value τa. By considering the search reflection delay value Dr and the response reflection delay value Da together with whether the latest document is included in the search or whether the latest document is reflected in the creation response data, the RAG evaluation device (300) can distinguish and evaluate the delay of the indexing stage and the delay of the creation stage in an operating environment that is updated in real time.
[0175] The RAG evaluation device (300) can determine whether the performance indicator meets the reference threshold. For example, the RAG evaluation device (300) can compare the accuracy score, similarity score, reliability score, and hallucination warning indicator with their respective reference thresholds. The RAG evaluation device (300) can classify the performance state of the RAG system as normal if the accuracy score and similarity score are above the reference threshold, the reliability score is maintained stably, and the hallucination warning indicator is within the allowable range. The RAG evaluation device (300) can classify the performance state of the RAG system as a borderline state if at least one of the accuracy score or reliability score repeatedly fluctuates near the reference threshold or the hallucination warning indicator increases intermittently. The RAG evaluation device (300) can classify the performance state of the RAG system as degraded if at least one of the accuracy score, similarity score, and reliability score drops below the reference threshold or the hallucination warning indicator repeatedly rises.
[0176] The RAG evaluation device (300) can analyze the trend of change by time point. The RAG evaluation device (300) can sort performance indicators stored for each user query in chronological order and calculate the amount of change, the number of consecutive drops, the point at which a sudden change occurs, and whether recovery has occurred at each time point. The RAG evaluation device (300) can determine an abnormal sign regarding the accuracy score if it continuously declines over multiple recent time points, even if the current accuracy score is above a reference threshold. The RAG evaluation device (300) can determine an abnormal sign related to the deterioration of the consistency of the basis of the generated response data if the similarity score is maintained but the reliability score continuously declines. The RAG evaluation device (300) can determine an abnormal sign indicating the possibility of distortion or lack of basis of the generated response data if the hallucination warning indicator repeatedly increases over a certain period. The RAG evaluation device (300) can determine the possibility of deterioration in the configuration stability of the RAG system as an abnormal sign if the performance indicators for the same user query at multiple time points fluctuate rapidly.
[0177] The RAG evaluation device (300) can further analyze the pattern of change relative to past history. The RAG evaluation device (300) can set the past average accuracy score, past average similarity score, past average confidence score, and past average hallucination warning indicator level for the same user query as reference values. The RAG evaluation device (300) can calculate the difference between the current performance indicator and the reference values. If the current performance indicator deviates from the reference values by more than a preset range, the RAG evaluation device (300) can determine that a significant change relative to past history has occurred. The RAG evaluation device (300) can determine whether a score drop is repeated only in a specific area by comparing it with the past history for the same query type or the same document group. If the failure to reflect the latest changes is repeated only in a specific document group, the RAG evaluation device (300) can determine whether there is a possibility of a problem in an area related to index settings or search settings. The RAG evaluation device (300) can determine whether a problem has occurred in the area related to the generation settings when a decrease in the reliability score and an increase in the hallucination warning indicator appear together in the overall type of query.
[0178] The RAG evaluation device (300) can evaluate the cause of performance degradation based on a combination of search results, generated response data, performance indicators, and time information. The RAG evaluation device (300) can evaluate the cause of performance degradation related to search settings or index settings if the search results do not include the latest documents. The RAG evaluation device (300) can evaluate the cause of performance degradation related to generation settings if the search results include the latest documents but the generated response data does not reflect the latest documents. The RAG evaluation device (300) can evaluate the cause of performance degradation related to lack of evidence consistency or response distortion if the reliability score of the generated response data decreases and the hallucination warning indicator increases. The RAG evaluation device (300) can evaluate the cause of performance degradation related to search range, relevance criteria, or top exposure order if a decrease in relevance and a decrease in similarity score of the search results occur together. The RAG evaluation device (300) can distinguish between search stage problems and generation stage problems by separately determining whether the latest documents are included in the search and whether the latest documents are reflected in the response.
[0179] The RAG evaluation device (300) can evaluate performance status and anomalies at at least one level among the single user query level, the query type level, and the document group level. The RAG evaluation device (300) can determine anomalies by comparing the results of repeated execution of a specific user query at the single user query level. The RAG evaluation device (300) can determine anomalies related to the reflection of the latest information by aggregating performance indicators of a set of user queries for verifying the reflection of the latest documents at the query type level. The RAG evaluation device (300) can evaluate the cause of performance degradation in a specific area by aggregating performance indicators of user queries corresponding to a specific document group at the document group level. The RAG evaluation device (300) can distinguish between global performance degradation and local performance degradation using the above multi-level evaluation structure.
[0180] The RAG evaluation device (300) can structure and store evaluation results regarding performance status, abnormal signs, and causes of performance degradation. The RAG evaluation device (300) may include user query identification information, latest document identification information, performance status classification value, abnormal sign type, candidate causes of performance degradation, and time-based judgment results in the evaluation results. The RAG evaluation device (300) may include at least one of search setting related possibility, creation setting related possibility, and index setting related possibility in the evaluation results. The RAG evaluation device (300) can store the evaluation results by mutually labeling them with the search results and the creation response data. The RAG evaluation device (300) can specifically determine the dynamic feedback target setting for a subsequent step using the labeling structure.
[0181] The RAG evaluation device (300) may store the evaluation results as re-evaluation criteria information in the step prior to re-executing the same user query or a user query under the same conditions. The RAG evaluation device (300) may include the current performance status, the current level of abnormal signs, the cause of the current performance degradation, and the current performance indicator value in the re-evaluation criteria information. The RAG evaluation device (300) may use the re-evaluation criteria information to compare whether performance has improved before and after a change in system settings in a subsequent step. The RAG evaluation device (300) may use the re-evaluation criteria information to track whether the same type of abnormality is repeated. The RAG evaluation device (300) may use the re-evaluation criteria information to determine whether to strengthen alarm settings.
[0182] The RAG evaluation device (300) can comprehensively diagnose the current state of the RAG system by evaluating the performance status, abnormal signs, and causes of performance degradation. The RAG evaluation device (300) can distinguish between one-time errors and continuous performance degradation by considering together whether the reference threshold is met, the trend of change by time point, and the pattern of change compared to the past history. The RAG evaluation device (300) can specifically evaluate problems related to search settings, problems related to creation settings, and problems related to index settings by reflecting the difference in roles between the search results and the generated response data. In this way, by reflecting the time information and the performance indicators together, the RAG evaluation device (300) can stably evaluate the performance status of the RAG system even in an operating environment where documents are added or updated in real time.
[0183] According to one example, in operation 550, the RAG evaluation device (300) can dynamically feed back to the RAG system a system setting including at least one of a search setting, a creation setting, an index setting, and an alarm setting, along with an evaluation result for the RAG system, in order to improve the performance of the RAG system.
[0184] Here, the evaluation result may be judgment information including the performance status, anomalies, causes of performance degradation, performance indicators, time information, and user query identification information of the RAG system. System settings may be setting information for changing the operating conditions of the RAG system. Search settings may be setting information including at least one of a search range, relevance criteria, number of top exposures, reordering criteria, filter conditions, and conditions for prioritizing the latest documents. Generation settings may be setting information including at least one of prompt conditions, context reflection criteria, response length conditions, response filter conditions, priority for reflecting the latest information, and grounds reflection criteria. Indexing settings may be setting information including at least one of index execution time, index priority, index update conditions, embedding regeneration conditions, and index re-execution conditions. Alarm settings may be setting information including at least one of a warning generation threshold, warning repetition count conditions, transmission time, transmission target, warning grade, and additional inspection execution conditions. Dynamic feedback may be a process that transmits system settings corresponding to the evaluation results to the RAG system in the form of control signals or policy information, and causes the RAG system to reflect the transmitted setting information. The control signal may be information including at least one of the type of setting to be changed, the setting value before change, the setting value after change or candidate setting value, the time of application, the condition for maintaining application, and the user query to be re-evaluated. The feedback setting identification information may be identification information for tracking the history of changes to the system settings.
[0185] The RAG evaluation device (300) can determine the type of system setting corresponding to the cause of performance degradation after receiving the evaluation result. The RAG evaluation device (300) can interpret the performance status, abnormal signs, cause of performance degradation, and time information included in the evaluation result to select the setting that should be adjusted first among the search setting, creation setting, index setting, and alarm setting. If the evaluation result includes multiple causes of performance degradation, the RAG evaluation device (300) can calculate the priority for each cause of performance degradation. The RAG evaluation device (300) can change multiple system settings sequentially or simultaneously according to the priority.
[0186] The RAG evaluation device (300) may prioritize feedback on the search settings or index settings when the search results do not include the latest documents. If the cause of the search results not including the latest documents is related to search range limitation, non-compliance with top exposure criteria, or index reflection delay, the RAG evaluation device (300) may include at least one of the following in the control signal: search range expansion, relevance criteria adjustment, top exposure count adjustment, index priority increase, or index re-execution request. If the search reflection delay value Dr exceeds the reference search delay value τr, the RAG evaluation device (300) may set the priority of index setting changes higher than the priority of search setting changes. If the phenomenon of the search results not including the latest documents is concentrated in a specific document group or a specific user query type, the RAG evaluation device (300) may perform setting changes limited to the specific document group or the specific user query type.
[0187] The RAG evaluation device (300) may prioritize feedback on the generation settings when the search results contain the latest documents but the generated response data does not reflect the latest documents. The RAG evaluation device (300) may generate a control signal including at least one of strengthening the context reflection criteria, raising the priority of reflecting the latest information, adjusting the response filter conditions, or correcting the prompt conditions based on a decrease in the reliability score of the generated response data, an increase in the hallucination warning indicator, or results of not reflecting the latest changes. The RAG evaluation device (300) may raise the priority of changing the generation settings when the response reflection delay value Da exceeds the reference response delay value τa. The RAG evaluation device (300) may perform a setting change including strengthening the basis reflection criteria or adding response generation restriction conditions when the generated response data does not match the search results.
[0188] The RAG evaluation device (300) may consider the document loading time, indexing completion time, search reflection time, and creation response time together when providing feedback on the index settings. Based on the time information, the RAG evaluation device (300) may perform at least one of adjusting the index execution cycle, raising the index priority, prioritizing indexing of a specific document group, or requesting embedding regeneration. The RAG evaluation device (300) may increase the adjustment intensity for the index settings if the failure to reflect the latest changes is repeatedly detected in the user query for verifying the reflection of the latest documents. The RAG evaluation device (300) may track the reflection time for the same document group again after the change in the index settings.
[0189] The RAG evaluation device (300) may consider the number of repetitions of the abnormal signs, the duration of consecutive occurrences, and the importance of the query when providing feedback on the alarm settings. The RAG evaluation device (300) may lower the warning threshold or raise the warning level if the same type of abnormal sign is repeated within a short period. The RAG evaluation device (300) may expand the target for sending to the administrator if the performance status is classified as degraded and the degraded status is maintained even after automatic adjustment. By changing the alarm settings, the RAG evaluation device (300) may apply at least one warning policy among general warnings, intensive inspection warnings, and immediate response warnings.
[0190] The RAG evaluation device (300) can adjust the system settings within a predefined allowable range without changing them indefinitely. For example, the RAG evaluation device (300) may refer to the minimum and maximum values of the top exposure count, the allowable range of the relevance criteria, and the allowable list of filter conditions for the search settings. The RAG evaluation device (300) may refer to the allowable range of the response length condition, the candidate value of the priority for reflecting the latest information, and the allowable list of the response filter conditions for the generation settings. The RAG evaluation device (300) may refer to the upper limit of the number of re-runs, the priority adjustment limit, and the allowable criteria for the embedding regeneration condition for the index settings. The RAG evaluation device (300) can reduce instability caused by excessive setting changes by utilizing the allowable range structure.
[0191] The RAG evaluation device (300) can store a rule for the correspondence relationship between the evaluation result and the system setting. The RAG evaluation device (300) may include the correspondence between the exclusion of the latest document in the search result and the change in index setting, the correspondence between the exclusion of the latest document in the generated response data and the change in generation setting, and the correspondence between repeated abnormal signs and the strengthening of alarm setting in the correspondence relationship. The RAG evaluation device (300) can store the correspondence relationship as control policy information. The RAG evaluation device (300) can automatically apply the same system setting change or a similar system setting change when the same type of evaluation result occurs again according to the control policy information.
[0192] The RAG evaluation device (300) can specify the user query to be changed or the document group to be changed together when changing the system settings. The RAG evaluation device (300) can perform a global setting change that is applied collectively to the entire RAG system using the specified structure. The RAG evaluation device (300) can perform a restriction setting change that is applied only to a specific user query type or a specific document group using the specified structure.
[0193] The RAG evaluation device (300) may transmit re-execution information to the scheduler to re-execute the same user query or a user query under the same conditions after the system setting change. The RAG evaluation device (300) may store the re-execution information together with the user query identification information, feedback setting identification information, and re-evaluation time information. The RAG evaluation device (300) may check whether the accuracy score, similarity score, reliability score, and hallucination warning indicator have been recovered based on the re-execution result. If the re-execution result is above the reference performance, the RAG evaluation device (300) may maintain the changed system setting. If the re-execution result is below the reference performance, the RAG evaluation device (300) may apply other candidate settings or restore to the previous setting value. If the re-execution result is below the reference performance and abnormal signs are repeated, the RAG evaluation device (300) may strengthen the alarm setting.
[0194] The RAG evaluation device (300) can record the state before and after the system setting change. The RAG evaluation device (300) may include the setting value before the change, the setting value after the change, the time of application, the reason for application, the user query to which the application was made, and the re-evaluation result in the recorded information. The RAG evaluation device (300) may include information identifying the cause of performance degradation and information identifying the feedback setting together in the recorded information. The RAG evaluation device (300) may store the recorded information by mutually labeling it with search results and generated response data stored for each user query. The RAG evaluation device (300) can track which setting change a specific evaluation result led to using the mutual labeling structure.
[0195] The RAG evaluation device (300) can store the feedback setting identification information and the cause analysis identification information in a linked manner. The RAG evaluation device (300) may additionally include user query identification information, latest document identification information, search reflection delay value Dr, response reflection delay value Da, performance status classification value, and re-evaluation results in the linked storage structure. The RAG evaluation device (300) can use the linked storage structure to refer to which system setting change was effective for a specific cause type at a subsequent time. The RAG evaluation device (300) can use the linked storage structure to determine the system setting that should be applied first when the same cause type occurs repeatedly.
[0196] The RAG evaluation device (300) can change actual operating conditions beyond merely detecting performance degradation of the RAG system by dynamically feeding back to the RAG system system settings that include at least one of search settings, creation settings, index settings, and alarm settings along with the evaluation results. By specifically defining the correspondence between the evaluation results and the system settings, the RAG evaluation device (300) can distinguish and adjust problems in the search stage, creation stage, and index reflection. By utilizing the dynamic feedback structure, the RAG evaluation device (300) can maintain the reflectivity of the latest documents, the stability of creation response data, and operational reliability even in an environment where documents are added or updated in real time. By utilizing the dynamic feedback structure, the RAG evaluation device (300) can link subsequent re-evaluation with the provision of administrator notifications.
[0197] In one embodiment, the RAG evaluation device (300) may analyze the evaluation results for the RAG system according to predefined meta-evaluation criteria, calculate a self-evaluation score, diagnose the evaluation performance of the evaluation method used based on the self-evaluation score, and update the evaluation method based on the diagnosis result of the evaluation performance. The meta-evaluation criteria may include at least one of scoring accuracy, scoring consistency, evaluation latency, evaluation cost, false alarm rate, false alarm rate, and distribution drift sensitivity. The self-evaluation score may be a numerical value indicating how stably and reliably the evaluation method operates in the current operating environment. Evaluation performance may be a quality level indicating how appropriately the evaluation method reflects the performance status, abnormal signs, and causes of performance degradation of the RAG system.
[0198] In one embodiment, the RAG evaluation device (300) collects the evaluation results, evaluation history, log data, and performance indicator trends, and can quantitatively analyze the collected information according to the meta-evaluation criteria. In the analysis process, the RAG evaluation device (300) can evaluate the scoring accuracy by using the degree of match between the scoring results and the correct answer criteria or search context, and can evaluate the scoring consistency by using the consistency of repeated evaluation results for the same user query or the same conditions. In the analysis process, the RAG evaluation device (300) can evaluate the evaluation delay time and evaluation cost by using the time and resource usage required to perform the evaluation, and can evaluate the false alarm rate and false alarm rate by comparing the warning occurrence results with whether actual performance degradation occurred. In the analysis process, the RAG evaluation device (300) can evaluate the distribution drift sensitivity by using the degree of responsiveness to changes in the input document set, user query type, or the latest document update pattern.
[0199] In one embodiment, the RAG evaluation device (300) can calculate the self-evaluation score by normalizing the analysis results for each meta-evaluation criterion and assigning a predefined weight to the normalized result. The RAG evaluation device (300) can diagnose the evaluation performance of the evaluation method based on the self-evaluation score. If the self-evaluation score is below a reference value, the RAG evaluation device (300) can determine that the reliability or operational suitability of the evaluation method has decreased. If the deviation from a specific meta-evaluation criterion repeatedly increases or a sudden fluctuation occurs in a specific section, the RAG evaluation device (300) can identify a weak section of the evaluation method or the need for structural adjustment.
[0200] In one embodiment, the RAG evaluation device (300) may update the evaluation method by adjusting at least one of the evaluation criteria weight, evaluation threshold, scoring prompt template, selection of LLM for evaluation, batch size, sampling strategy, cache expiration time, worker parallelism, reporting period, and feedback policy based on the diagnostic results of the evaluation performance. The RAG evaluation device (300) may adjust the evaluation criteria weight or scoring prompt template if the scoring accuracy or scoring consistency is degraded. The RAG evaluation device (300) may adjust the batch size, cache expiration time, or worker parallelism if the evaluation delay time increases or the evaluation cost rises excessively. The RAG evaluation device (300) may improve the stability and suitability of the evaluation method by adjusting the evaluation threshold, sampling strategy, selection of LLM for evaluation, reporting period, or feedback policy if the alarm false positive rate, alarm false negative rate, or distribution drift sensitivity falls outside the reference range.
[0201] In this way, the RAG evaluation device (300) of the present disclosure and the RAG system evaluation method using the same can automatically call the search API and generation API of the RAG system based on a scheduler and collect search result and generation response data together with time information. Accordingly, the RAG evaluation device (300) of the present disclosure and the RAG system evaluation method using the same can continuously check whether the latest documents are reflected even in an operating environment where documents are added or updated in real time, and can stably manage the performance status of the RAG system.
[0202] In addition, the RAG evaluation device (300) of the present disclosure and the RAG system evaluation method using the same can multi-dimensionally analyze performance evaluation elements including accuracy, similarity, reliability, and whether hallucination occurs by comparing the search results and the generated response data with a correct answer standard or a search context, and can quantify the analysis results into performance indicators including an accuracy score, a similarity score, a reliability score, and a hallucination warning indicator. Accordingly, the RAG evaluation device (300) of the present disclosure and the RAG system evaluation method using the same can perform an objective and precise performance diagnosis that comprehensively reflects the mutual influence relationship between the search stage and the generation stage, and can identify abnormal signs early and distinguish between one-time errors and structural performance degradation by considering together whether the standard threshold of the performance indicator is met, the trend of change by time, and the pattern of change compared to past history.
[0203] In addition, the RAG evaluation device (300) of the present disclosure and the RAG system evaluation method using the same can evaluate the cause of performance degradation by distinguishing between search reflection delay, creation reflection delay, non-reflection of the latest document, and non-reflection of the latest document in the creation response data, and can dynamically feed back to the RAG system a system setting including at least one of a search setting, a creation setting, an index setting, and an alarm setting along with the evaluation result. Accordingly, the RAG evaluation device (300) of the present disclosure and the RAG system evaluation method using the same can re-evaluate whether performance has been improved by re-executing the same user query or a user query under the same conditions, and can improve operational reliability, maintenance efficiency, and overall system performance management efficiency while reducing administrator intervention.
[0204] However, as this has been explained above, a redundant explanation thereof will be omitted.
[0205] Using the embodiments of the present invention described above, those skilled in the art will be able to easily make various changes and modifications within the scope of the essential characteristics of the present invention. The content of each claim of the patent claims may be combined with other claims that are not related by reference within the scope of what can be understood from this specification.
Claims
Claim 1 A RAG evaluation device for diagnosing the performance of a RAG system in real time based on a scheduler, comprising: a memory for storing at least one instruction;and includes at least one processor that executes the above at least one instruction, wherein the at least one processor collects search result and generated response data by automatically calling the search API and generation API of the RAG system for a user query using a scheduler, and analyzes performance evaluation factors including accuracy, similarity, reliability, and whether hallucination occurs by comparing the search result and generated response data with a correct answer standard or a search context, and quantitatively scores a performance indicator including at least one of an accuracy score, a similarity score, a reliability score, and a hallucination warning indicator based on the analysis result, and evaluates the performance status, abnormal signs, and causes of performance degradation of the RAG system based on whether the reference threshold of the performance indicator is met, the trend of change over time, and the pattern of change compared to past history, and dynamically feeds back to the RAG system a system setting including at least one of a search setting, a generation setting, an index setting, and an alarm setting along with the evaluation result for the RAG system for performance improvement of the RAG system, and the at least one processor monitors the database status of the RAG system through the scheduler to load new documents or generate updated documents A RAG evaluation device that detects, generates a user query for verifying the reflection of the latest document by combining at least two items among the title, date, and changes of the new document or the updated document, inputs the generated user query for verifying the reflection of the latest document as the user query into the search API and the generation API, transmits the system settings to the RAG system in the form of a control signal including the type of setting to be changed, the setting value before change, and the setting value after change when dynamically feeding back the system settings, and re-evaluates whether performance has been improved by executing the same user query or a user query under the same conditions again after the change of the system settings. Claim 2 A RAG evaluation device according to claim 1, wherein the at least one processor detects at least one of a predefined time condition, an event condition, and a state change condition using the scheduler, determines a RAG evaluation time using the scheduler, automatically executes the search API and the generation API of the RAG system at the determined RAG evaluation time, and collects the search result and the generation response data based on the call results of the search API and the generation API. Claim 3 delete Claim 4 A RAG evaluation device according to paragraph 2, wherein at least one processor calculates a priority value P using a recent document update frequency U, document importance C, and a past performance degradation history score E for a plurality of user query candidates, and determines the execution order of the user queries and the RAG evaluation time so that the user queries having a priority value P greater than or equal to a reference value are executed preferentially. Claim 5 A RAG evaluation device according to claim 1, wherein the at least one processor extracts time information including at least one of a document loading time, indexing completion time, search reflection time, and generation response time for the new document or the updated document, and stores the time information in conjunction with the search result and the generation response data. Claim 6 A RAG evaluation device according to claim 1, wherein at least one processor selects a valid correct answer criterion or search context at the time of execution of the user query, and analyzes the accuracy, similarity, reliability, and hallucination occurrence, including whether the latest changes are reflected, by comparing the search result and the generated response data with the correct answer criterion or search context in at least one of a core fact unit, an attribute item unit, and a sentence unit. Claim 7 A RAG evaluation device according to claim 1, wherein at least one processor normalizes the accuracy score, the similarity score, the reliability score, and the hallucination warning indicator, and assigns weights by reflecting at least one of whether the latest information is reflected, time-matching, and evidence-matching, thereby quantitatively scoring the performance indicator. Claim 8 A RAG evaluation device according to claim 1, wherein at least one processor calculates a search reflection delay value (Dr) based on the difference between a search reflection time and a document loading time, and a response reflection delay value (Da) based on the difference between a generation response time and the document loading time, and if the search reflection delay value (Dr) exceeds a preset reference search delay value (τr), determines that an indexing delay is the cause of performance degradation, and if the search reflection delay value (Dr) is less than or equal to the reference search delay value (τr) and the response reflection delay value (Da) exceeds the reference response delay value (τa), determines that a generation reflection delay is the cause of performance degradation. Claim 9 A RAG evaluation device according to claim 1, wherein at least one processor classifies the performance state of the RAG system into a normal state, a borderline state, or a degraded state based on whether the reference threshold of the performance indicator is met, the trend of change over time, and the pattern of change compared to past history, and evaluates the cause of performance degradation related to the search setting or the index setting when the latest document including the new document or the updated document is not included in the search result, and evaluates the cause of performance degradation related to the generation setting when the latest document is included in the search result but the generated response data does not reflect the latest document. Claim 10 A RAG evaluation device according to claim 1, wherein at least one processor stores the search results and generated response data for each user query, and stores the evaluation results regarding the cause of performance degradation and the system settings dynamically fed back to the RAG system by mutually labeling them. Claim 11 A RAG evaluation device according to claim 1, wherein the at least one processor analyzes the evaluation results for the RAG system according to a predefined meta-evaluation criterion, calculates a self-evaluation score regarding at least one of scoring accuracy, scoring consistency, evaluation latency, evaluation cost, false alarm rate, false alarm rate, and distribution drift sensitivity, and diagnoses the evaluation performance of the evaluation method used based on the self-evaluation score. Claim 12 A RAG evaluation device according to claim 11, wherein the at least one processor updates the evaluation method by adjusting at least one of an evaluation criterion weight, an evaluation threshold, a scoring prompt template, an LLM selection for evaluation, a batch size, a sampling strategy, a cache expiration time, a worker parallelism, a reporting period, and a feedback policy based on the diagnostic result of the evaluation performance. Claim 13 A method for evaluating a RAG system that diagnoses the performance of the RAG system in real time based on a scheduler comprises: a step of collecting search result and generated response data by automatically calling the search API and generation API of the RAG system in response to a user query using a scheduler; a step of analyzing performance evaluation elements including accuracy, similarity, reliability, and whether hallucination occurs by comparing the search result and generated response data with a correct answer standard or a search context; a step of quantitatively scoring a performance indicator including at least one of an accuracy score, a similarity score, a reliability score, and a hallucination warning indicator based on the analysis result; and a step of evaluating the performance status, abnormal signs, and causes of performance degradation of the RAG system based on whether the reference threshold of the performance indicator is met, the trend of change over time, and the pattern of change compared to past history. The method comprises the step of dynamically feeding back to the RAG system a system setting including at least one of a search setting, a creation setting, an index setting, and an alarm setting, along with an evaluation result of the RAG system, in order to improve the performance of the RAG system; the step of detecting the loading of a new document or the occurrence of an updated document by monitoring the database status of the RAG system through the scheduler; the step of generating a user query for verifying the reflection of the latest document by combining at least two items among the title, date, and changes of the new document or the updated document; the step of inputting the generated user query for verifying the reflection of the latest document as the user query into the search API and the creation API; and the step of transmitting the system setting to the RAG system in the form of a control signal including the type of setting to be changed, the setting value before change, and the setting value after change when dynamically feeding back the system setting.A RAG system evaluation method further comprising the step of re-evaluating whether performance has been improved by re-executing the same user query or a user query under the same conditions after the change in the system settings.
Citation Information
Patent Citations
Autonomous simulated testing and benchmarking framework for agentic AI systems
US20260099653A1
Method for generating answer based on advanced retrieval augmented generation and system thereof
KR1020260012105A
Knowledge Graph Generator Enabled by Diagonal Search
US20150095303A1
Anomaly detection and classification using telemetry data
US20190116131A1
Copilot architecture: network of microservices including specialized machine learning tools
US20250231960A1
