RAG application self-optimization method, system and device and storage medium

Abstract

The invention relates to the technical field of RAG application, in particular to a self-optimization method, system and device of RAG application and a storage medium. The method aims at solving the technical problem that in the prior art, an RAG model has certain limitation in the aspect of self-optimization. Comprising the steps of creating an original RAG application, and performing question answering on user questions to obtain original evaluation data; creating a sandbox RAG application with the same configuration in the sandbox environment; real answers are obtained based on the user questions and the sandbox RAG application, data updating is carried out through the real answers, and perfect evaluation data are obtained; performing performance evaluation on the perfect evaluation data to obtain performance evaluation data; and based on the performance evaluation data, registering a corresponding optimizer into the sandbox RAG application to obtain an optimized RAG application, updating the optimized RAG application to obtain the configuration of the optimizer in the updated optimized RAG application, and optimizing the original RAG application according to the configuration of the optimizer.

Description

Technical Field

[0001] The present invention belongs to the technical field of RAG applications, and in particular, relates to a self-optimization method, system, device and storage medium for RAG applications. Background Art

[0002] Large Language Model (LLM) refers to a natural language processing model based on deep learning, which is designed to understand and generate human language. LLM is usually based on a neural network model and trained using a large-scale corpus. These models usually have billions to trillions of parameters and are able to handle a variety of natural language processing tasks. LLMs are characterized by their large scale and billions of parameters, which help them learn complex patterns in language data. Since these models are based on deep learning architectures, they help them achieve impressive performance on various NLP tasks.

[0003] In the prior art, Retrieval Augmented Generation (RAG) is used to retrieve information from data sources to assist LLM in generating answers. When a query or question is given, RAG retrieves relevant information from the knowledge base or data source, integrates the retrieved information into prompts, and provides them to LLM, which generates responses based on the prompts and the retrieved information.

[0004] The prior art has the following technical problems:

[0005] Existing RAG models have certain limitations in self-optimization and often require manual intervention to adjust model parameters or optimization strategies, which limits the degree of automation and adaptability of RAG applications. Summary of the invention

[0006] The present invention provides a self-optimization method, system, device and storage medium for RAG applications, aiming to solve the technical problem that the existing RAG model in the above-mentioned prior art has certain limitations in self-optimization, and often requires manual intervention to adjust model parameters or optimization strategies, which limits the degree of automation and adaptability of RAG applications.

[0007] The technical solution of the present invention to solve the above technical problem is as follows: a self-optimization method for RAG application, comprising:

[0008] Create original RAG application based on the knowledge base;

[0009] Using the original RAG application to answer user questions and obtain original evaluation data;

[0010] Based on the original RAG application, creating a sandbox RAG application with the same configuration in a sandbox environment;

[0011] Obtaining a true answer based on the user question and the sandbox RAG application, and updating the original evaluation data with the true answer to obtain improved evaluation data;

[0012] Performing performance evaluation on the improved evaluation data to obtain a performance evaluation result, and updating the performance evaluation result to the improved evaluation data to obtain performance evaluation data;

[0013] Based on the performance evaluation data, the corresponding optimizer is registered to the sandbox RAG application to obtain the optimized RAG application, and the optimized RAG application is updated based on the user problem to obtain the configuration of the optimizer in the updated optimized RAG application, and the original RAG application is optimized according to the configuration of the optimizer. After the optimization is completed, the optimized RAG application is deleted.

[0014] Furthermore, the above-mentioned creation of the original RAG application based on the knowledge base specifically includes:

[0015] Establishing a knowledge base, and using the knowledge base to obtain knowledge documents;

[0016] Preprocessing and splitting the knowledge document to obtain split documents, converting the split documents into vector form using an embedding model, and storing them in a vector database;

[0017] Based on the knowledge base and the vector database, an original RAG application is established, and configuration parameters of the original RAG application are configured.

[0018] Furthermore, the above-mentioned creation of a sandbox RAG application with the same configuration in the sandbox environment specifically includes:

[0019] According to all configuration parameters of the original RAG application, a sandbox RAG application applied to the sandbox environment is cloned, and the sandbox RAG application remains invisible to the user.

[0020] Furthermore, the above-mentioned original evaluation data specifically includes:

[0021] Obtaining a user question, searching the user question using the original RAG application to obtain multiple search documents, and concatenating all the search documents to obtain original context information;

[0022] Concatenate the user question and the original context information to obtain a prompt word, and send the prompt word to a large language model;

[0023] The original answer generated by the large language model according to the prompt word is obtained, and the prompt word and the original answer are packaged and stored; wherein the original evaluation data is the packaged prompt word and the original answer.

[0024] Further, the above-mentioned obtaining of the real answer based on the original evaluation data and the sandbox evaluation data specifically includes:

[0025] Get the knowledge document of the knowledge base of the sandbox RAG application;

[0026] Generate answers to the user questions using the large language model based on the knowledge documents of the knowledge base of the sandbox RAG application to obtain real answers;

[0027] The true answer is updated to the original evaluation data to obtain perfect evaluation data.

[0028] Furthermore, the above-mentioned performance evaluation data specifically includes:

[0029] Using the RAG application evaluation model to evaluate the multiple performance indicators of the improved evaluation data, and correspondingly obtaining multiple performance evaluation items;

[0030] Determine whether the performance evaluation item is successfully evaluated, and obtain a performance evaluation result; wherein determining whether the performance evaluation item is successfully evaluated is specifically:

[0031] If yes, then update the corresponding evaluation item to the performance evaluation result;

[0032] If not, re-evaluate the corresponding performance indicator to obtain a new performance evaluation item, and then determine whether the new performance evaluation item is successfully evaluated;

[0033] If the number of times a performance indicator generates a performance evaluation entry reaches a maximum failure threshold, the performance evaluation entry will no longer be regenerated, the performance indicator will be marked as abandoned, and the performance evaluation result will be updated;

[0034] The performance evaluation result is updated into the improved evaluation data to obtain performance evaluation data.

[0035] Further, the above-mentioned optimizing the original RAG application according to the configuration of the optimizer specifically includes:

[0036] S1: Build an optimizer registration center and classify the optimizers to be registered according to the RAG application pipeline and the performance indicators;

[0037] S2: Obtain a performance indicator marked as abandoned, search the optimizer registration center for a first optimizer that matches the performance indicator marked as abandoned, and a RAG application pipeline corresponding to the first optimizer, obtain the working phase of the sandbox RAG application according to the RAG application pipeline, register the first optimizer with the working phase, and obtain an optimized RAG application;

[0038] S3: Obtain the user question, use the user question to ask and answer the optimized RAG application, obtain optimization context information and optimization answer, and update the improvement evaluation data with the optimization context information and the optimization answer to obtain optimization evaluation data;

[0039] S4: According to all the performance indicators, the optimized evaluation data is evaluated using the RAG application evaluation model to obtain new performance evaluation data;

[0040] S5: Determine whether there is a performance indicator marked as abandoned in the new performance evaluation data. If yes, repeat S2-S4; if no, execute S6;

[0041] S6: copy the configuration of the optimizer in the optimized RAG application, use the configuration of the optimizer to optimize the original RAG application, and delete the optimized RAG application after the optimization is completed.

[0042] In a second aspect, in order to solve the above technical problems, the present invention further provides a self-optimization system for RAG applications, comprising:

[0043] The application creation component is used to create the original RAG application based on the knowledge base;

[0044] A question-and-answer component, used to use the original RAG application to answer user questions and obtain original evaluation data;

[0045] A sandbox application creation component, used to create a sandbox RAG application with the same configuration in a sandbox environment based on the original RAG application;

[0046] An evaluation data collection component, used to obtain real answers based on the user questions and the sandbox RAG application, and update the original evaluation data with the real answers to obtain improved evaluation data;

[0047] A performance evaluation component is used to perform a performance evaluation on the improved evaluation data to obtain a performance evaluation result, and update the performance evaluation result to the improved evaluation data to obtain performance evaluation data;

[0048] The optimizer registration component is used to register the corresponding optimizer to the sandbox RAG application based on the performance evaluation data, obtain the optimized RAG application, and update the optimized RAG application based on the user problem to obtain the configuration of the optimizer in the updated optimized RAG application, optimize the original RAG application according to the configuration of the optimizer, and delete the optimized RAG application after the optimization is completed.

[0049] In a third aspect, in order to solve the above-mentioned technical problem, the present invention also provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the self-optimization method of the RAG application of the present application is implemented.

[0050] In a fourth aspect, in order to solve the above-mentioned technical problem, the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the self-optimization method of the RAG application of the present application is implemented.

[0051] Compared with the prior art, the present invention has the following advantages:

[0052] 1. The system of the present invention automatically collects user question and answer data, generates answers, evaluates performance, and dynamically selects and registers the most appropriate optimizer to the corresponding RAG pipeline stage, thereby realizing self-optimization of the RAG model.

[0053] 2. The entire process of the present invention does not require human intervention, and is automatically completed from data collection, answer generation, performance evaluation to optimizer registration, which greatly improves the automation of the model. This not only saves labor costs, but also ensures the continuity and efficiency of the optimization process.

[0054] 3. The present invention dynamically registers the optimizer, and the system can select the most appropriate optimization strategy according to different performance evaluation results, thereby improving the adaptability of the model.

[0055] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0057] Figure 1 A schematic flow chart of a self-optimization method for RAG application according to an embodiment of the present invention is shown;

[0058] Figure 2 A flowchart of a self-optimization method for RAG application according to an embodiment of the present invention is shown;

[0059] Figure 3 A schematic diagram of a process of creating an original RAG application according to an embodiment of the present invention is shown;

[0060] Figure 4 A schematic diagram of the process of creating a sandbox RAG application according to an embodiment of the present invention is shown;

[0061] Figure 5 A schematic diagram of a process of performance evaluation according to an embodiment of the present invention is shown;

[0062] Figure 6 A schematic diagram of the process of optimizer registration according to an embodiment of the present invention is shown;

[0063] Figure 7 A self-optimization system block diagram of a RAG application according to an embodiment of the present invention is shown;

[0064] Figure 8 A schematic structural diagram of an electronic device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0066] Figure 1 FIG. 4 shows a flow chart of a self-optimization method for RAG application according to an embodiment of the present invention. Figure 1 and Figure 2 As shown, a self-optimization method for RAG application according to an embodiment of the present invention includes:

[0067] Create original RAG application based on the knowledge base;

[0068] Using the original RAG application to answer user questions and obtain original evaluation data;

[0069] Based on the original RAG application, creating a sandbox RAG application with the same configuration in a sandbox environment;

[0070] Obtaining a true answer based on the user question and the sandbox RAG application, and updating the original evaluation data with the true answer to obtain improved evaluation data;

[0071] Performing performance evaluation on the improved evaluation data to obtain a performance evaluation result, and updating the performance evaluation result to the improved evaluation data to obtain performance evaluation data;

[0072] Based on the performance evaluation data, the corresponding optimizer is registered to the sandbox RAG application to obtain the optimized RAG application, and the optimized RAG application is updated based on the user problem to obtain the configuration of the optimizer in the updated optimized RAG application, and the original RAG application is optimized according to the configuration of the optimizer. After the optimization is completed, the optimized RAG application is deleted.

[0073] Furthermore, the above-mentioned creation of the original RAG application based on the knowledge base specifically includes:

[0074] Establishing a knowledge base, and using the knowledge base to obtain knowledge documents;

[0075] Preprocessing and splitting the knowledge document to obtain split documents, converting the split documents into vector form using an embedding model, and storing them in a vector database;

[0076] Based on the knowledge base and the vector database, an original RAG application is established, and configuration parameters of the original RAG application are configured.

[0077] like Figure 3 As shown, in this embodiment, the knowledge base provides a broad and in-depth knowledge base for the original RAG application, ensuring that the system can access comprehensive and up-to-date information resources, while supporting the continuous learning and development of the system. The vector library enhances the retrieval performance and generation quality of the system through efficient similarity calculation and semantic understanding, enabling the original RAG application to provide accurate and in-depth answers in a short time. In this embodiment, the configuration parameters include top_k, temperature, top_n, score_threshold, etc.

[0078] Furthermore, the above-mentioned creation of a sandbox RAG application with the same configuration in the sandbox environment specifically includes:

[0079] According to all configuration parameters of the original RAG application, a sandbox RAG application applied to the sandbox environment is cloned, and the sandbox RAG application remains invisible to the user.

[0080] like Figure 4As shown, in this embodiment, the creation of the sandbox RAG application is implemented through the sandbox application creation component.

[0081] Among them, the RAG application in the figure is the original RAG application in this embodiment, and the sandbox application is the sandbox RAG application in this embodiment.

[0082] Furthermore, the above-mentioned original evaluation data specifically includes:

[0083] Obtaining a user question, searching the user question using the original RAG application to obtain multiple search documents, and concatenating all the search documents to obtain original context information;

[0084] Concatenate the user question and the original context information to obtain a prompt word, and send the prompt word to a large language model;

[0085] In this embodiment, the structure of the prompt word is as follows:

[0086]

[0087] The original answer generated by the large language model according to the prompt word is obtained, and the prompt word and the original answer are packaged and stored; as shown in the following structure:

[0088]

[0089] The structure of the initial evaluation data is as follows:

[0090]

[0091]

[0092] In this embodiment, the original RAG application uses the langchai n and other frameworks to call the dialogue service of the large voice model, splices the user question and the retrieved related search document blocks, and sends them to the large voice model as a complete prompt word, and the large voice model answers the question.

[0093] Further, the above-mentioned obtaining of the real answer based on the original evaluation data and the sandbox evaluation data specifically includes:

[0094] Get the knowledge document of the knowledge base of the sandbox RAG application;

[0095] Generate answers to the user questions using the large language model based on the knowledge documents of the knowledge base of the sandbox RAG application to obtain real answers;

[0096] The true answer is updated to the original evaluation data to obtain perfect evaluation data.

[0097] In this embodiment, the prompt word structure sent to the large language model to generate the answer to the specified question is as follows:

[0098]

[0099] The structure of the refined evaluation data with updated true answers is as follows:

[0100]

[0101]

[0102] Furthermore, the above-mentioned performance evaluation data specifically includes:

[0103] Using the RAG application evaluation model to evaluate the multiple performance indicators of the improved evaluation data, and correspondingly obtaining multiple performance evaluation items;

[0104] Determine whether the performance evaluation item is successfully evaluated, and obtain a performance evaluation result; wherein determining whether the performance evaluation item is successfully evaluated is specifically:

[0105] If yes, then update the corresponding evaluation item to the performance evaluation result;

[0106] If not, re-evaluate the corresponding performance indicator to obtain a new performance evaluation item, and then determine whether the new performance evaluation item is successfully evaluated;

[0107] If the number of times a performance indicator generates a performance evaluation entry reaches a maximum failure threshold, the performance evaluation entry will no longer be regenerated, the performance indicator will be marked as abandoned, and the performance evaluation result will be updated;

[0108] The performance evaluation result is updated into the improved evaluation data to obtain performance evaluation data.

[0109] like Figure 5 As shown, in this embodiment, the maximum failure threshold is 3 times. The structure of the performance evaluation data is as follows:

[0110]

[0111]

[0112] Further, the above-mentioned optimizing the original RAG application according to the configuration of the optimizer specifically includes:

[0113] S1: Build an optimizer registration center and classify the optimizers to be registered according to the RAG application pipeline and the performance indicators;

[0114] In this embodiment, the structure of the two-dimensional table [performance index, RAG pipeline] when designing the optimizer is as follows:

[0115]

[0116]

[0117] S2: Obtain a performance indicator marked as abandoned, search the optimizer registration center for a first optimizer that matches the performance indicator marked as abandoned, and a RAG application pipeline corresponding to the first optimizer, obtain the working phase of the sandbox RAG application according to the RAG application pipeline, register the first optimizer with the working phase, and obtain an optimized RAG application;

[0118] like Figure 6 As shown, in this embodiment, different RAG pipeline stages (such as retrieval, generation, fusion, etc.) have different task requirements and challenges. By registering the optimizer to a specific processing step, it can be ensured that each optimizer works in the environment that best suits it. Among them, the retrieval stage: optimize retrieval efficiency and accuracy, such as improving the selection logic of document blocks or adjusting retrieval algorithm parameters. Generation stage: improve the quality of generated answers, such as fine-tuning the output strategy of the language model or introducing more contextual information. Fusion stage: enhance the answer integration effect, such as optimizing the way to merge information from multiple sources or improving the formatting of the final answer. Suppose in a RAG application, we find that the answer relevance in the retrieval stage is not strong enough, and the answer length in the generation stage is too long. Then you can make a targeted selection: select an optimizer specifically for improving retrieval accuracy (such as improving the similarity calculation method) and register it to the retrieval stage. Select another optimizer for controlling the length of generated answers (such as setting a maximum generation length limit) and register it to the generation stage.

[0119] The configuration information for optimizing the RAG application is as follows:

[0120]

[0121]

[0122] S3: Obtain the user question, use the user question to ask and answer the optimized RAG application, obtain optimization context information and optimization answer, and update the improvement evaluation data with the optimization context information and the optimization answer to obtain optimization evaluation data;

[0123] In this embodiment, after completing the optimizer registration of the sandbox RAG application, based on the collected initial evaluation data, the user questions of the initial evaluation data are automatically used one by one to ask and answer questions for the optimized RAG application (the sandbox RAG application needs to sort the execution order of the optimizer according to the configured optimizer information, and register it to the corresponding RAG pipeline after instantiation), and collect and save the answers and context information answered by the optimized RAG application, wherein the real answers and user questions are saved unchanged, and the original context and original answers in the improved evaluation data are updated as a new round of evaluation data; the structure of the obtained optimized evaluation data is as follows:

[0124]

[0125]

[0126] S4: According to all the performance indicators, the optimized evaluation data is evaluated using the RAG application evaluation model to obtain new performance evaluation data;

[0127] In this embodiment, the method of using the RAG application evaluation model to perform performance evaluation on the optimized evaluation data is the same as the method of using the RAG application evaluation model to perform performance evaluation on the perfect evaluation data, which will not be repeated here.

[0128] S5: Determine whether there is a performance indicator marked as abandoned in the new performance evaluation data. If yes, repeat S2-S4; if no, execute S6;

[0129] S6: copy the configuration of the optimizer in the optimized RAG application, use the configuration of the optimizer to optimize the original RAG application, and delete the optimized RAG application after the optimization is completed.

[0130] In this embodiment, the optimized RAG application in the sandbox environment is deleted, all evaluation data is cleared, and the performance optimization of the original RAG application is completed.

[0131] Based on Figure 1 Based on the same principle as the method shown in , the embodiment of the present invention also provides a self-optimization system for RAG applications, such as Figure 7 As shown in, including:

[0132] The application creation component is used to create the original RAG application based on the knowledge base;

[0133] A question-and-answer component, used to use the original RAG application to answer user questions and obtain original evaluation data;

[0134] A sandbox application creation component, used to create a sandbox RAG application with the same configuration in a sandbox environment based on the original RAG application;

[0135] An evaluation data collection component, used to obtain real answers based on the user questions and the sandbox RAG application, and update the original evaluation data with the real answers to obtain improved evaluation data;

[0136] A performance evaluation component is used to perform a performance evaluation on the improved evaluation data to obtain a performance evaluation result, and update the performance evaluation result to the improved evaluation data to obtain performance evaluation data;

[0137] The optimizer registration component is used to register the corresponding optimizer to the sandbox RAG application based on the performance evaluation data, obtain the optimized RAG application, and update the optimized RAG application based on the user problem to obtain the configuration of the optimizer in the updated optimized RAG application, optimize the original RAG application according to the configuration of the optimizer, and delete the optimized RAG application after the optimization is completed.

[0138] The self-optimization system of the RAG application in the embodiment of the present invention can execute the self-optimization method of the RAG application provided in the embodiment of the present invention. The implementation principle is similar. The actions performed by each module and unit in the self-optimization system of the RAG application in each embodiment of the present invention correspond to the steps in the self-optimization method of the RAG application in each embodiment of the present invention. For the detailed functional description of each module of the self-optimization system of the RAG application, please refer to the description in the corresponding self-optimization method of the RAG application shown in the previous text, which will not be repeated here.

[0139] Among them, the self-optimization system of the above-mentioned RAG application can be a computer program (including program code) running in a computer device, for example, the self-optimization system of the RAG application is an application software; the application software can be used to execute the corresponding steps in the method provided in an embodiment of the present invention.

[0140] The modules involved in the embodiments of the present invention may be implemented by software or hardware, wherein the name of a module does not limit the module itself in some cases.

[0141] Based on the same principle as the method shown in the embodiments of the present invention, an electronic device is also provided in the embodiments of the present invention, which may include but is not limited to: a processor and a memory; the memory is used to store a computer program; the processor is used to execute the method shown in any embodiment of the present invention by calling the computer program.

[0142] In an alternative embodiment, an electronic device is provided, such as Figure 8 As shown, Figure 8The electronic device shown includes: a processor and a memory. The processor and the memory are connected, such as through a bus. Optionally, the electronic device may also include a transceiver, which may be used for data exchange between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver is not limited to one, and the structure of the electronic device does not constitute a limitation on the embodiments of the present invention.

[0143] The memory is used to store the application code (computer program) for executing the solution of the present invention, and the execution is controlled by the processor. The processor is used to execute the application code stored in the memory to implement the content shown in the above method embodiment.

[0144] The electronic device may also be a terminal device. Figure 8 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0145] An embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer can execute the corresponding contents of the aforementioned method embodiment.

[0146] According to another aspect of the present invention, a computer program product or a computer program is also provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in the implementation modes of the above-mentioned various embodiments.

[0147] It should be understood that the flow charts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the module, the program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0148] The computer-readable storage medium provided by the embodiment of the present invention may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device, or device.

[0149] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0150] The above description is only a preferred embodiment of the present invention and an explanation of the technical principles used. Those skilled in the art should understand that the disclosure scope involved in the present invention is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present invention (but not limited to) by each other.

Claims

1. A self-optimization method for RAG application, characterized in that: The method comprises: Create original RAG application based on the knowledge base; Using the original RAG application to answer user questions and obtain original evaluation data; Based on the original RAG application, creating a sandbox RAG application with the same configuration in a sandbox environment; Obtaining a true answer based on the user question and the sandbox RAG application, and updating the original evaluation data with the true answer to obtain improved evaluation data; Performing performance evaluation on the improved evaluation data to obtain a performance evaluation result, and updating the performance evaluation result to the improved evaluation data to obtain performance evaluation data; Based on the performance evaluation data, the corresponding optimizer is registered to the sandbox RAG application to obtain the optimized RAG application, and the optimized RAG application is updated based on the user problem to obtain the configuration of the optimizer in the updated optimized RAG application, and the original RAG application is optimized according to the configuration of the optimizer. After the optimization is completed, the optimized RAG application is deleted.

2. A self-optimization method for RAG application according to claim 1, characterized in that: According to the knowledge base, creating an original RAG application specifically includes: Establishing a knowledge base, and using the knowledge base to obtain knowledge documents; Preprocessing and splitting the knowledge document to obtain split documents, converting the split documents into vector form using an embedding model, and storing them in a vector database; Based on the knowledge base and the vector database, an original RAG application is established, and configuration parameters of the original RAG application are configured.

3. A self-optimization method for RAG application according to claim 2, characterized in that: Creating a sandbox RAG application with the same configuration in a sandbox environment specifically includes: According to all configuration parameters of the original RAG application, a sandbox RAG application applied to the sandbox environment is cloned, and the sandbox RAG application remains invisible to the user.

4. The self-optimization method for RAG application according to claim 1, characterized in that: The original evaluation data include: Obtaining a user question, searching the user question using the original RAG application to obtain multiple search documents, and concatenating all the search documents to obtain original context information; The user question and the original context information are concatenated to obtain a prompt word, and the prompt word is sent to a large language model; The original answer generated by the large language model according to the prompt word is obtained, and the prompt word and the original answer are packaged and stored; wherein the original evaluation data is the packaged prompt word and the original answer.

5. The self-optimization method for RAG application according to claim 4, characterized in that: Acquiring a true answer based on the original evaluation data and the sandbox evaluation data specifically includes: Get the knowledge document of the knowledge base of the sandbox RAG application; Generate answers to the user questions using the large language model based on the knowledge documents of the knowledge base of the sandbox RAG application to obtain real answers; The true answer is updated to the original evaluation data to obtain perfect evaluation data.

6. The self-optimization method for RAG application according to claim 1, characterized in that: The performance evaluation data obtained specifically include: Using the RAG application evaluation model to evaluate the multiple performance indicators of the improved evaluation data, and correspondingly obtaining multiple performance evaluation items; Determine whether the performance evaluation item is successfully evaluated, and obtain a performance evaluation result; wherein determining whether the performance evaluation item is successfully evaluated is specifically: If yes, then update the corresponding evaluation item to the performance evaluation result; If not, re-evaluate the corresponding performance indicator to obtain a new performance evaluation item, and then determine whether the new performance evaluation item is successfully evaluated; If the number of times a performance indicator generates a performance evaluation entry reaches a maximum failure threshold, the performance evaluation entry will no longer be regenerated, the performance indicator will be marked as abandoned, and the performance evaluation result will be updated; The performance evaluation result is updated into the improved evaluation data to obtain performance evaluation data.

7. A self-optimization method for RAG application according to claim 6, characterized in that: Optimizing the original RAG application according to the configuration of the optimizer specifically includes: S1: Build an optimizer registration center and classify the optimizers to be registered according to the RAG application pipeline and the performance indicators; S2: Obtain a performance indicator marked as abandoned, search the optimizer registration center for a first optimizer that matches the performance indicator marked as abandoned, and a RAG application pipeline corresponding to the first optimizer, obtain the working phase of the sandbox RAG application according to the RAG application pipeline, register the first optimizer with the working phase, and obtain an optimized RAG application; S3: Obtain the user question, use the user question to ask and answer the optimized RAG application, obtain optimization context information and optimization answer, and update the improvement evaluation data with the optimization context information and the optimization answer to obtain optimization evaluation data; S4: According to all the performance indicators, the optimized evaluation data is evaluated using the RAG application evaluation model to obtain new performance evaluation data; S5: Determine whether there is a performance indicator marked as abandoned in the new performance evaluation data. If yes, repeat S2-S4; if no, execute S6; S6: copy the configuration of the optimizer in the optimized RAG application, use the configuration of the optimizer to optimize the original RAG application, and delete the optimized RAG application after the optimization is completed.

8. A self-optimizing system for RAG applications, characterized in that: include: The application creation component is used to create the original RAG application based on the knowledge base; A question-and-answer component, used to use the original RAG application to answer user questions and obtain original evaluation data; A sandbox application creation component, used to create a sandbox RAG application with the same configuration in a sandbox environment based on the original RAG application; An evaluation data collection component, used to obtain real answers based on the user questions and the sandbox RAG application, and update the original evaluation data with the real answers to obtain improved evaluation data; A performance evaluation component is used to perform a performance evaluation on the improved evaluation data to obtain a performance evaluation result, and update the performance evaluation result to the improved evaluation data to obtain performance evaluation data; The optimizer registration component is used to register the corresponding optimizer to the sandbox RAG application based on the performance evaluation data, obtain the optimized RAG application, and update the optimized RAG application based on the user problem to obtain the configuration of the optimizer in the updated optimized RAG application, optimize the original RAG application according to the configuration of the optimizer, and delete the optimized RAG application after the optimization is completed.

9. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer storage medium, characterized in that: The computer storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images