Retrieval enhancement generation distributed experiment method, system and equipment and storage medium

Abstract

The invention discloses a retrieval enhancement generation distributed experiment method, system and equipment and a storage medium. The method comprises the following steps: acquiring original data; classifying the original data including FAQ data, text data and problem data; processing and marking different types of data to obtain evaluation data; performing fine tuning training on the vector model, the rearrangement model and the large language model based on the evaluation data; after training is completed, the vector model, the rearrangement model and the language large model are evaluated; if the evaluation does not reach the standard, performing fine tuning training on the test data and the evaluation data obtained by the test again; and if the evaluation reaches the standard, stopping training. According to the method, the research and development efficiency and the analysis accuracy of the retrieval enhancement generation technology are improved, RAG experiment, evaluation and tracking tools are created for developers, and the efficiency of subsequent AI application operation iteration is improved.

Description

Technical Field

[0001] The present invention relates to the field of retrieval technology, and in particular to a retrieval enhancement generation distributed experiment method, system, device and storage medium. Background Art

[0002] Retrieval-augmented generation (RAG) is a natural language processing technology that combines retrieval and generation strategies to enhance the performance of language models through domain knowledge / classic cases. In the research and development practice of retrieval-augmented generation technology, it is necessary to combine dozens of module technologies such as document parsing, text segmentation, retrieval strategy, semantic search, and large model reasoning strategy to build retrieval-augmented generation applications, and implement performance evaluation of retrieval-augmented generation pipelines through manual / intelligent methods, so as to build the most suitable retrieval-augmented generation application for this field.

[0003] The difference between the retrieval enhancement generation method in the present invention and the "A Dynamic Adaptive Question and Answer System and Method Based on Hierarchical Structure and Retrieval Enhancement" (Patent No.: CN202410611948.4) is as follows: This patent is a dynamic adaptive question and answer system based on hierarchical structure and retrieval enhancement. The core idea is to combine multiple modules such as knowledge representation, retrieval enhancement generation, dynamic learning and adaptation, and represent entity relationships through a hierarchical structure, improve the accuracy of question and answer through RAG technology and fine-tuning large language models, and realize dynamic learning and cross-domain knowledge integration, thereby improving the performance of the overall RAG pipeline, and more emphasis is placed on interactive module design and model fine-tuning. The present invention is a method and system for retrieval enhancement generation of distributed experiments, and pays more attention to automatically finding the optimal RAG pipeline hyperparameters and strategies according to the set parameters during the development and construction of RAG applications, to achieve domain adaptation and improve the development efficiency of RAG applications. The retrieval enhancement generation in "A retrieval enhancement generation method based on text rewriting" (patent number: CN202410561083.5) is different from that in the present invention: the patent is a retrieval enhancement generation method based on text rewriting, which is a technical point of retrieval enhancement generation optimization. This invention uses a large model to rewrite user questions to generate multiple similar questions, and obtains the feature vectors of these multiple questions for mixed search, and inputs the user question text and the reference text as prompt text into the large model to obtain the answer output. It is mainly an optimization technique for user question understanding. The present invention is a method and system for distributed experiments on retrieval enhancement generation. User question understanding optimization is only one of the module functions, and it pays more attention to improving the performance of the retrieval enhancement generation pipeline by adjusting the "hyperparameters" in the retrieval enhancement generation pipeline and adopting different tuning strategies.

[0004] The differences between "Data governance system, method, equipment and medium based on natural language processing" (patent number: CN202410465219.2) and the present invention in terms of intelligent data processing are as follows: This patent is a technology for medical data governance, which aims to improve data reading capabilities and reduce misrecognition of NLP solutions when processing medical texts. It includes four trained natural language processing models: part-of-speech recognition, entity similarity calculation, medical unified coding and variable extraction. After reading the original medical data, these models are used to obtain specific results of word segmentation data sets and various types of data. The present invention is a method and system for intelligent processing of enterprise data based on a large language model and a series of data intelligent processing algorithm strategies. It also has the characteristics of process orchestration, dynamic expansion, etc., and is more scalable.

[0005] Existing experimental methods and systems for retrieval-enhanced generation have some defects:

[0006] The experimental setting is not flexible enough, the experimental cost is high, it is difficult to efficiently design and control the experimental factors and response variables, the experimental process is not unified, and the single experimental cycle is long. The effect of the retrieval enhancement generation pipeline is jointly affected by dozens of models and application components. The experimental efficiency and accuracy of traditional methods are low, and it is difficult to adapt to the changing experimental needs and environmental changes.

[0007] The performance evaluation process is cumbersome and mostly relies on manual subjective evaluation, which is inefficient and difficult to manually evaluate for some tasks, making it difficult to quantify the results.

[0008] The lack of effective data management and analysis tools has led to insufficient utilization of experimental data, a lack of detail and transparency in the process, and an inability to efficiently record experimental process data to guide subsequent R&D practices.

[0009] The lack of a user-friendly interactive interface makes it difficult for non-experts to operate and manage the experimental process of the retrieval enhancement generation pipeline. Summary of the invention

[0010] In view of the shortcomings existing in the above problems, the present invention provides a retrieval enhanced generation distributed experiment method, system, device and storage medium.

[0011] To achieve the above object, the present invention provides a distributed experiment method for retrieval enhancement generation, comprising:

[0012] Get the original data;

[0013] Classifying the original data into FAQ data, text data and question data;

[0014] Processing and labeling the data of different categories to obtain evaluation data;

[0015] Based on the evaluation data, performing combination experiments on different indexing strategies, search strategies, vector models, and rearrangement models, and calculating evaluation scores of different combinations;

[0016] If the evaluation is not satisfactory, the experiment is repeated; if the evaluation is satisfactory, the experiment ends.

[0017] Preferably, processing the data of different categories comprises:

[0018] Directly extract the FAQ data;

[0019] Performing data synthesis on the text data and the question data to obtain a question-answer pair;

[0020] After merging the text data to obtain questions and question-answer pairs;

[0021] The text data is simply collected and edited to obtain text blocks.

[0022] Preferably, the evaluation data is converted into vector data and then fine-tuned for training.

[0023] Preferably, the combined experiment process includes fine-tuning of design patterns, index relationship, retrieval, hyperparameters and rearrangement strategies.

[0024] The present invention also provides a retrieval enhancement generation distributed experiment system, comprising:

[0025] An acquisition module is used to obtain raw data;

[0026] A classification module, used to classify the original data into FAQ data, text data and question data;

[0027] A labeling module, used for processing and labeling the data of different categories to obtain evaluation data;

[0028] An experiment module, used to conduct combination experiments on different indexing strategies, search strategies, vector models, and rearrangement models based on the evaluation data, and calculate evaluation scores of different combinations;

[0029] The scoring module is used to repeat the experiment if the evaluation fails to meet the standard; if the evaluation meets the standard, the experiment ends.

[0030] The present invention also provides a device, comprising at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the program is executed by the processing unit, the processing unit executes the above method.

[0031] The present invention also provides a storage medium storing a computer program executable by an electronic device, and when the program runs on the electronic device, the electronic device executes the above method.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] This invention not only improves the R&D efficiency and analysis accuracy of retrieval enhancement generation technology, but also creates RAG experiment, evaluation and tracking tools for developers, improving the efficiency of subsequent AI application operation iterations. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a flow chart of the distributed experimental method for retrieval enhancement generation of the present invention. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] Reference Figure 1 The present invention provides a distributed experiment method for retrieval enhancement generation, comprising:

[0037] Get the original data;

[0038] Specifically, through strategies such as problem generation, data noise, self-instruction construction, data reflow, and RAG data intelligent labeling, a data intelligent generation link is built to obtain original data through hierarchical management and control based on the task types in RAG.

[0039] Classify the original data into FAQ data, text data and question data; process and annotate the data of different categories to obtain evaluation data;

[0040] Specifically, processing the data of different categories includes:

[0041] Directly extract the FAQ data;

[0042] Performing data synthesis on the text data and the question data to obtain a question-answer pair;

[0043] After merging the text data to obtain questions and question-answer pairs;

[0044] The text data is simply collected and edited to obtain text blocks.

[0045] Based on the evaluation data, conduct combination experiments on different indexing strategies, search strategies, vector models, and rearrangement models, and calculate the evaluation scores of different combinations;

[0046] Specifically, the evaluation data is converted into vector data for fine-tuning training; the data set and fine-tuning algorithm obtained by intelligent data processing are used, combined with DOE experimental design, with data, algorithms, parameters, etc. as experimental factors, and the intelligent evaluation system constructed with high-quality evaluation data sets as response variables, to achieve a detailed, transparent, and scientific training process, and efficiently achieve fine-tuning of large language models, vector models, rearrangement models, etc. The fine-tuning training process includes fine-tuning of design patterns, index relationships, retrieval, hyperparameters, and rearrangement strategies.

[0047] If the evaluation results in failure to meet the standard, the experiment is repeated; if the evaluation results in meeting the standard, the experiment ends.

[0048] Specifically, combining the technical characteristics of retrieval enhancement generation, from the dimensions of retrieval recall capability, generation reasoning capability, retrieval enhancement generation pipeline performance, etc., combined with high-quality evaluation datasets, an intelligent evaluation system is established. Through strategies such as ablation experiments / ABTest, it is possible to analyze and evaluate the impact of each component on the RAG pipeline.

[0049] In this embodiment, the present invention makes up for the shortcomings and pain points of the existing retrieval enhancement generation research and development and implementation process, creates a retrieval enhancement generation development experimental platform centered on "data science", analyzes and guides the optimization of the retrieval enhancement generation (RAG) pipeline from the perspective of data science, and realizes automated and intelligent experimental verification. The core processing modules of the development experimental platform include: intelligent data processing, model fine-tuning, model selection / version arrangement, application development arrangement, and intelligent evaluation modules.

[0050] Data processing - intelligent data processing pipeline: Through strategies such as problem generation, data noise, self-instruction construction, data reflow, and RAG data intelligent labeling, we build a data intelligent generation link, and implement hierarchical management and control for the task types in RAG to build high-quality training and evaluation data sets for multiple tasks.

[0051] Model training - standardized training platform: using the data set and fine-tuning algorithm obtained by intelligent data processing, combined with DOE experimental design, taking data, algorithms, parameters, etc. as experimental factors, and the intelligent evaluation system constructed with high-quality evaluation data sets as response variables, to achieve a detailed, transparent and scientific training process, and efficiently achieve fine-tuning of large language models, vector models, rearrangement models and other models.

[0052] Model selection / version orchestration: Through the model version management tool, different models after model fine-tuning are uniformly managed, and model capability service development is implemented as a model component for retrieval enhancement generation.

[0053] Application Development - Graphical RAG Orchestration: Integrate multiple types of AI models and technologies, combine multimodal document parsing, text semantic segmentation, query conversion / routing, hybrid search, large language models, hypothetical questions and HyDE and other AI technologies to achieve design and orchestration between retrieval enhancement generation modules and build a variety of retrieval enhancement generation pipelines.

[0054] Application evaluation - intelligent hierarchical evaluation and analysis: Combining the characteristics of retrieval enhancement generation technology, from the dimensions of retrieval recall capability, generation reasoning capability, retrieval enhancement generation pipeline performance, etc., combined with high-quality evaluation data sets, an intelligent evaluation system is established. Through strategies such as ablation experiments / AB Tests, the impact of each component on the RAG pipeline can be analyzed and evaluated.

[0055] The above process combines ray, MLflow, Kubernetes, JiuSi-RAG and other technologies to realize the distributed experiment and tracking method of retrieval enhancement generation, which makes up for the shortcomings and pain points of existing retrieval enhancement generation technology. This innovative method not only improves the research and development efficiency and analysis accuracy of retrieval enhancement generation technology, but also creates RAG experiment, evaluation and tracking tools for developers, improving the efficiency of subsequent AI application operation iterations.

[0056] Example

[0057] This application is based on a customer service knowledge base practice scenario combined with the self-developed RAG component for verification. Nearly 1,000 knowledge documents were experimented, 2,000 test cases were constructed, and nearly 100 groups of experiments were designed through a distributed experiment framework. The application was evaluated and analyzed in terms of retrieval recall capability, generation reasoning capability, and retrieval enhancement generation pipeline performance. The results are as follows:

[0058] Improved RAG application experiment and optimization efficiency: Combined with the retrieval enhancement generation distributed experiment method, nearly 100 groups of experiments were set up in batches, and experiments, evaluations and tracking were carried out simultaneously in a distributed manner. Compared with the traditional iteration cost, it was reduced from 1 person * day / experiment to 0.01 person * day / experiment. This automated and intelligent approach has improved the RAG application iteration efficiency by nearly 100 times and resource utilization by nearly 40%.

[0059] Intelligent evaluation system: Combining big model technology and automation strategies, we build an intelligent evaluation system for the customer service knowledge field. We also use data intelligence to generate high-quality evaluation data sets for multiple tasks. For the RAG overall link, we use strategies such as ablation experiments / AB Tests to achieve quantifiable evaluation and analysis of each component in each link, making the overall RAG experimental process more detailed and scientific, providing data support for component research and development and application optimization, and achieving more efficient and accurate performance analysis.

[0060] Improved experimental data management and analysis efficiency: Experimental data management and experimental tracking are implemented in each link of the complete iteration chain of the RAG application, which can support multi-tenant and multi-scenario experimental iterations. The parameter combination of historical cases is combined to guide the RAG application modeling of other similar scenarios, thereby improving the efficiency of scenario migration.

[0061] This experiment adopted a comparative test method, based on a customer service knowledge base practice scenario combined with self-developed RAG components for verification, conducted experiments on nearly 1,000 knowledge documents, constructed 2,000 test cases for experiments, and constructed nearly 256 experimental schemes. The experimental analysis results are shown in Table 1:

[0062] Table 1

[0063]

[0064] After the test evaluation, the test effect evaluation comparison table in Table 1 was obtained, and the analysis showed that:

[0065] Through the solution of the present invention, the experimental efficiency and resource utilization rate are greatly improved. When it is implemented in a certain field or enterprise, an efficient intelligent experimental tool can be used to achieve hyperparameter optimization, which greatly improves the optimization efficiency of RAG applications and project support efficiency.

[0066] Through the scheme of the present invention, the evaluation of experimental results can be realized intelligently, a transparent, detailed and scientific experimental process can be realized, and the abnormality of each component can be quickly tracked and located.

[0067] Combined with RAG distributed link, vector search model test comparison is carried out, as shown in Table 2: Combined with the large model, intelligent annotation of retrieval data is realized, and nearly 100,000 high-quality data sets are constructed for training. Compared with traditional annotation solutions, the cost is saved by more than 80%. In the domain knowledge scenario, the retrieval accuracy is improved by about 25% compared with the open source TOP1 model.

[0068] Table 2

[0069]

[0070] Combined with RAG distributed link, the document parsing and cutting strategy experiments are compared, as shown in Table 3: Combined with document understanding, paragraph parsing and OCR capabilities, the large model technology is integrated for information extraction to achieve document parsing and cutting intelligent cutting strategy. Compared with the traditional sliding window and small2big, the effect is superior and supports table and graphic parsing functions.

[0071] Table 3

[0072]

[0073] The present invention also provides a retrieval enhancement generation distributed experiment system, comprising:

[0074] An acquisition module is used to obtain raw data;

[0075] Classification module, used to classify the original data including FAQ data, text data and question data;

[0076] The labeling module is used to process and label different categories of data to obtain evaluation data;

[0077] The experimental module is used to conduct combined experiments on different indexing strategies, search strategies, vector models, and rearrangement models based on the evaluation data, and calculate the evaluation scores of different combinations;

[0078] The scoring module is used to repeat the experiment if the evaluation result is not up to standard; if the evaluation result is up to standard, the experiment ends.

[0079] The present invention also provides a device, comprising at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the program is executed by the processing unit, the processing unit executes the above method.

[0080] The present invention also provides a storage medium storing a computer program executable by an electronic device. When the program runs on the electronic device, the electronic device executes the above method.

[0081] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A distributed experimental method for retrieval enhancement generation, characterized in that: include: Get the original data; Classifying the original data into FAQ data, text data and question data; Processing and labeling the data of different categories to obtain evaluation data; Based on the evaluation data, performing combination experiments on different indexing strategies, search strategies, vector models, and rearrangement models, and calculating evaluation scores of different combinations; If the evaluation is not satisfactory, the experiment is repeated; if the evaluation is satisfactory, the experiment ends.

2. The distributed experiment method for retrieval enhancement generation according to claim 1, characterized in that: The different categories of data processed include: Directly extract the FAQ data; Performing data synthesis on the text data and the question data to obtain a question-answer pair; After merging the text data to obtain questions and question-answer pairs; The text data is simply collected and edited to obtain text blocks.

3. The retrieval enhancement generation distributed experiment method according to claim 2 is characterized in that: The evaluation data is converted into vector data and then a combination experiment is performed.

4. The retrieval enhancement generation distributed experiment method according to claim 3 is characterized in that: The combined experiment process includes fine-tuning of design patterns, index relationship, retrieval, hyperparameters, and rearrangement strategies.

5. A distributed experiment system for retrieval enhancement generation, characterized in that: include: An acquisition module is used to obtain raw data; A classification module, used to classify the original data into FAQ data, text data and question data; A labeling module, used for processing and labeling the data of different categories to obtain evaluation data; An experiment module, used to conduct combination experiments on different indexing strategies, search strategies, vector models, and rearrangement models based on the evaluation data, and calculate evaluation scores of different combinations; The scoring module is used to repeat the experiment if the evaluation fails to meet the standard; if the evaluation meets the standard, the experiment ends.

6. A device, characterized in that: The method comprises at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the program is executed by the processing unit, the processing unit executes the method according to any one of claims 1 to 4.

7. A storage medium, characterized in that: It stores a computer program executable by an electronic device. When the program runs on the electronic device, the electronic device executes the method according to any one of claims 1 to 4.

Images