Improved method for retrieval-augmented generation based on embedding-fine-tuning

Abstract

The application provides an improved retrieval enhancement generation method based on Embedding-FineTuning, pre-processes and cleans large-scale multi-source heterogeneous data of a power grid, ensures consistency and quality of the data, dynamically divides the processed text data based on a LumberChunker technology, divides text blocks according to semantic turning points, and generates text blocks suitable for fine-tuning training; then, a T5 model is used to generate a question and answer pair based on the text blocks in multiple rounds of question and answer, to form a question and answer pair data set for fine-tuning; a customized prompt template is designed according to an actual application scenario in the power grid field, to guide the model generation process; and Embedding-FineTuning fine-tuning training is performed on the question and answer pair data set, model parameters are optimized, and the embedding representation performance and accuracy of the model in the power grid field are improved.

Description

Technical Field

[0001] This invention relates to the field of power grid knowledge question answering technology, and in particular to an improved retrieval generation method based on Embedding-FineTuning. Background Technology

[0002] Large models are a highly regarded technique in the field of deep learning in recent years. Large pre-trained models such as BERT and GPT are trained on massive amounts of data, master rich language structure and contextual information, and have powerful general language processing capabilities.

[0003] However, large language models may suffer from the "illusion" problem when generating responses based on a lack of information, i.e., generating responses that appear reasonable but are actually incorrect. To address this issue, existing technologies have proposed Retrieval Augmentation (RAG) systems, which improve the precision and accuracy of information by generating model-generated content based on context-relevant documents. Retrieval Augmentation systems enhance information accuracy by combining model-generated content with context-relevant documents, where how the text content is segmented into "chunks" has a significant impact on retrieval quality.

[0004] Traditional fixed-length text segmentation methods lack an understanding of the text's semantic structure, easily disrupting semantic integrity and leading to information loss or redundancy. They also fail to adapt to the complex structures of different document types, particularly when dealing with cross-chapter content, which can cause semantic confusion, reducing processing efficiency and accuracy. This results in significant limitations in complex text processing. In contrast, dynamic text segmentation technology can flexibly adjust the segmentation based on natural semantic transitions. The length and content of corresponding text blocks are more flexible, better adapting to different semantic scenarios and ensuring that each text block is semantically independent while maintaining document coherence.

[0005] The LumberChunker technology utilizes dynamic text segmentation based on the principle that retrieval efficiency is improved when the retrieved text blocks are as semantically independent as possible. This allows for dynamic block size to better capture the semantic independence of the content. The technology leverages a large language model to dynamically segment documents, iteratively prompting the model to identify points where content transitions begin within a series of consecutive paragraphs. This ensures that each text block is contextually coherent but distinct from adjacent blocks. (See attached manual.) Figure 1 As shown.

[0006] The LumberChunker technique proceeds as follows: First, the document is segmented into paragraphs, each with a unique ID. The paragraphs are then sequentially grouped until a predefined token count threshold θ is exceeded. These groups are then fed as context into a large language model, which is guided to identify specific paragraphs within a group whose content significantly deviates from the preceding text, marking the end of a text block. This process is repeated throughout the entire document to segment it.

[0007] However, while pre-trained models perform well on many general tasks, they may not always be directly applicable to specific downstream tasks, such as sentiment analysis, text summarization, question answering systems, or domain-specific tasks. Fine-tuning becomes a key technical strategy to better adapt these models to specific tasks. The fine-tuning process involves using the weights of the pre-trained model as a starting point and then continuing training on a task-specific dataset. This process can be seen as a refinement process; through additional training steps, the model learns the data features and patterns specific to the task, thus better adapting to its requirements. Fine-tuning not only improves the model's performance on specific tasks but can sometimes even be achieved on smaller datasets because pre-trained models already possess strong language understanding capabilities. The principles are detailed in the attached manual. Figure 2 As shown.

[0008] Fine-tuning is a complex process involving multiple aspects such as dataset preparation, model selection, optimization strategy formulation, hyperparameter tuning, and task-specific adjustments. Its success largely depends on the quality of the dataset and the quality of the model's pre-training. An excellent pre-trained model has already been trained on large-scale text data and has learned the basic structure and semantics of the language, which provides a solid foundation for fine-tuning. On the other hand, a task-specific dataset should contain rich, diverse, and accurately labeled samples, covering all key aspects and edge cases of the task, so that the model can learn all aspects of the task. Summary of the Invention

[0009] Based on the aforementioned advancements in existing technologies, and in order to alleviate the model illusion problem in the field of power grid knowledge question answering, this invention proposes an improved retrieval enhancement generation method based on Embedding-FineTuning. This method aims to enhance the performance and accuracy of large language models in power grid question answering scenarios, thereby achieving reliable reasoning in the power grid knowledge question answering field.

[0010] This solution first preprocesses and cleans large-scale, multi-source, heterogeneous data in the power grid field to ensure data consistency and quality. Then, based on LumberChunker technology, the processed text data is dynamically partitioned to generate text blocks suitable for fine-tuning training. Next, the T5 model is used to generate question-answer pairs based on these text blocks through multiple rounds, forming a question-answer pair dataset for fine-tuning. Following this, a customized prompt template is designed to guide the model generation process, considering the practical application scenarios in the power grid field. Then, fine-tuning training is performed on the question-answer pair dataset, optimizing model parameters to improve the model's embedding representation performance and accuracy in the power grid field. Finally, the fine-tuned model is integrated into the retrieval system, enabling the system to perform similarity matching on user queries based on the model-generated embedding representations, thereby improving the relevance and accuracy of retrieval results.

[0011] The present invention specifically adopts the following technical solution:

[0012] An improved retrieval enhancement generation method based on Embedding-FineTuning is characterized by: preprocessing and cleaning large-scale, multi-source, heterogeneous data of the power grid to ensure data consistency and quality; dynamically partitioning the processed text data based on LumberChunker technology, dividing text blocks according to semantic inflection points to generate text blocks suitable for fine-tuning training; then, using the T5 model to generate question-answer pairs based on the text blocks in multiple rounds, forming a question-answer pair dataset for fine-tuning; designing customized prompt templates for practical application scenarios in the power grid field to guide the model generation process; and performing Embedding-FineTuning fine-tuning training on the question-answer pair dataset to improve the model's embedding representation performance and accuracy in the power grid field by optimizing model parameters.

[0013] Furthermore, the BGE embedding model is used for fine-tuning training via embedding-fine-tuning.

[0014] Furthermore, the fine-tuned BGE embedding model is integrated into the retrieval system based on the workflow of retrieval enhancement generation. The embedding representation generated by the BGE embedding model is used to match user queries, and highly similar text segments are obtained from the text block knowledge base that has been processed by block embedding as the basis for answering, thereby improving the relevance and accuracy of question answering results. The embedding representation is to map user queries and document representations to the same vector space by calculating the user query vector.

[0015] Furthermore, the preprocessing and cleaning include cleaning irrelevant or redundant information in the document, correcting typos, grammatical errors and logical conflicts, and standardizing technical terms and units to ensure data consistency and reduce noise; parsing structured and unstructured data separately to extract key information and unify data formats for subsequent processing; and employing desensitization strategies for different types of sensitive information, including generalization, perturbation or masking, and verifying through random sampling to ensure data privacy protection.

[0016] Furthermore, the dynamic segmentation of the processed text data based on LumberChunker technology specifically involves: using an iterative prompting model to dynamically identify semantic turning points in consecutive paragraphs, ensuring that each segmented content block maintains semantic coherence in the context, while distinguishing it from adjacent content blocks, and dividing semantically independent parts into relatively independent blocks to reduce semantic confusion caused by fixed-length segmentation.

[0017] Furthermore, the method of using the T5 model to generate question-answer pairs based on text blocks through multi-turn question-answering, forming a question-answer pair dataset for fine-tuning, specifically involves: enabling the T5 model to deeply understand the context, semantics, and syntactic structure of each text block, identifying valuable question points, forming core questions and preliminary answers focusing on the topic or key information of the text block, and then further guiding the model to generate subsequent questions. For each question point, the model deeply analyzes the document content, filters out the most relevant and accurate information, and then generates related questions. Through this multi-turn dialogue generation method, each text block forms a question-answer chain containing a core question and multiple subsequent questions, forming a question-answer pair dataset that meets the specific needs of the power grid field for fine-tuning.

[0018] Furthermore, in the design process of the prompt template, data is integrated into the prompt through direct embedding, context building, and interactive guidance to help the model accurately parse and reason about data in specific contexts.

[0019] Furthermore, the Embedding-FineTuning training on the question-answering pair dataset specifically involves: dividing the generated question-answering pair dataset into training, validation, and test sets according to a certain ratio to monitor the model's performance during training and evaluate its performance on unseen data; during the fine-tuning phase, adjusting the parameters of the BGE model using the training set data, with the goal of adapting the model to question-answering tasks in the power grid domain; through repeated iterations, enabling the model to learn a pattern of producing accurate output given input; and using Hite Rate and MMR as test metrics during training.

[0020] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method described above.

[0021] A non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0022] Compared with existing technologies, the present invention and its preferred solutions can not only achieve high-accuracy search results, but also provide the source of the search results, providing valuable reference information for practitioners in the power grid field, and enhancing the practicality and reliability of the model. Attached Figure Description

[0023] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0024] Figure 1 This is a schematic diagram of LumberChunker technology;

[0025] Figure 2 This is a schematic diagram of the fine-tuning principle;

[0026] Figure 3 This is a schematic diagram of the model framework of the inference model in an embodiment of the present invention. Detailed Implementation

[0027] In the following, specific embodiments of this application will be described in detail with reference to the accompanying drawings. Based on these detailed descriptions, those skilled in the art will be able to clearly understand and implement this application. Without departing from the principles of this application, features from various embodiments can be combined to obtain new implementations, or certain features from some embodiments can be substituted to obtain other preferred implementations.

[0028] To make the features and advantages of the present invention more apparent and understandable, specific embodiments are described below in conjunction with the accompanying drawings:

[0029] like Figure 3 As shown, the overall improvement and design process of the retrieval enhancement generation improvement method based on Embedding-FineTuning in this embodiment of the invention includes the following steps;

[0030] Step 1: Benchmark Performance Testing and Comparative Evaluation. This step involves conducting comprehensive and in-depth benchmark performance testing on the established fine-tuning strategy. The strategy is placed within a broad comparative framework, scientifically comparing it with various advanced fine-tuning methods in the field. This rigorously verifies the effectiveness of the Embedding-FineTuning method in enhancing the performance of BGE models for specific tasks in the power grid, while accurately evaluating its performance in domain adaptability and task customization. This lays the theoretical foundation and data support for subsequent fine-tuning optimization strategies. A comprehensive performance evaluation of existing fine-tuning methods is conducted, covering accuracy, computational efficiency, and scalability tests on standard datasets, serving as a reference benchmark for future work.

[0031] Step Two: Data Preprocessing and Cleaning. The power grid sector involves numerous and complex documents, including a large amount of unstructured text and structured data, such as requirements analysis reports, business requirements reports, and requirements submission forms. To ensure data usability and consistency, data preprocessing is essential. In this embodiment, irrelevant or redundant information is removed from the reports. Algorithms are used to detect and correct data errors, including typos, grammatical errors, and logical inconsistencies. Technical terms and units in the text are standardized to ensure data consistency, reduce noise, and lay the foundation for subsequent analysis and processing.

[0032] For structured data such as tables and charts, specialized parsing tools are used to extract the data content. For natural language text, natural language processing technology is used to parse it and extract key information. Different documents are then converted into a unified data format for storage. Finally, different de-identification strategies are selected for different types of sensitive information. For example, generalization is used for geographic locations, while data perturbation or masking is used for financial data. Appropriate de-identification strategies are selected based on the content of the document, and a random sample of data is manually verified to ensure data privacy protection.

[0033] This embodiment targets large-scale, multi-source, heterogeneous data in the power grid field, and performs data preprocessing and cleaning operations to ensure data quality and consistency. At the same time, it collects relevant text data to construct a fine-tuning dataset, including business requirement reports, requirement specifications, and feasibility study reports.

[0034] Step 3: Dynamically partitioning text data. In text analysis within the power grid sector, considering the complexity of documents, which contain multiple chapters, sub-chaps, tables, and charts, traditional text processing methods may fail to effectively capture key information, thus affecting the accuracy of the analysis.

[0035] To address this issue, this step employs a dynamic text segmentation method based on LumberChunker technology. This method dynamically divides the document according to its semantic content, forming semantically relatively independent text blocks. In this way, the model can more accurately understand the text content and quickly locate key information, thereby improving the quality of the analysis results. LumberChunker technology utilizes long-text narrative document segmentation to dynamically divide the document into multiple semantic blocks, ensuring that each block contains as complete and independent information as possible. Compared to traditional fixed-length segmentation, this semantically based dynamic segmentation method can better handle complex content across chapters and topics, and adapts to the actual document structure, meeting the needs of multi-source heterogeneous documents. This method effectively reduces semantic confusion caused by fixed-length segmentation, improves the model's ability to understand and process text, and ensures the accuracy and effectiveness of text segmentation.

[0036] This step uses LumberChunker to dynamically divide the text data obtained in step two, thereby dividing the text into blocks based on semantic turning points and maintaining the independence of contextual semantics.

[0037] Step Four: Question-Answer Pair Dataset Generation. In this step, the T5 model is used to perform in-depth analysis of text blocks to generate a high-quality question-answer pair dataset for subsequent model fine-tuning. The T5 model is based on a "text-to-text" framework, unifying complex natural language processing tasks into text conversion problems, and can effectively handle question-answer pair generation tasks. In this embodiment, the T5 model first performs in-depth analysis of each text block to understand its context, semantics, and syntactic structure, while automatically identifying key question points in the text block, such as document topics and core viewpoints. Then, based on these key points, the T5 model further integrates this information into clear and accurate answers using prompt templates for the question-answer pair generation task, forming question-answer pairs. Finally, a question-answer pair dataset that meets the specific needs of the power grid field is generated.

[0038] This step uses the T5 model to generate question-answer pairs based on multi-turn dialogue from the text blocks obtained in step three, forming a question-answer pair dataset for fine-tuning.

[0039] Step 5: Model deployment and initialization configuration. Deploy the BGE embedding model locally to prepare for Embedding-FineTuning fine-tuning training.

[0040] This step involves deploying the BGE model and its initial configuration on a local server. The aim is to build a stable and controllable training environment, laying a solid foundation for fine-tuning. The use of a local server not only ensures sufficient and efficient utilization of computing resources but also guarantees uninterrupted training and data security. Furthermore, the introduction of the Embedding-FineTuning method further enhances the model's ability to quickly adapt to and efficiently learn within the power grid domain.

[0041] Step Six: Customized Prompt Template Design. Prompt templates are designed based on actual application scenarios in the power grid field to guide the model generation process and improve the accuracy and rationality of model reasoning. Prompt template design is a crucial step in the fine-tuning process; its quality directly affects the model's in-depth understanding of domain information and its efficient reasoning ability. Therefore, considering the unique application scenarios and complex needs of the power grid field, a series of highly customized prompt templates are designed to accurately capture the core requirements of the task and incorporate professional terminology and logical structures from the power grid field. The aim is to guide the model to generate answers that are both accurate and conform to domain standards. For example, a basic knowledge base question-answering prompt could be: "[Instruction] Based on the known information, answer the question concisely and professionally, and provide the thought process behind the answer. [Known Information]... . [Question]... ."

[0042] Step 7: Fine-tuning and performance optimization of the model. Based on the designed prompt, fine-tuning and training are performed on the dataset obtained in Step 4. By optimizing the model parameters, the performance and accuracy of the model's embedded representation in the power grid field are improved.

[0043] This step focuses on effectively applying the question-answering pair dataset to the fine-tuning phase of the BGE model. Through a series of iterative training processes, the model parameters are continuously adjusted and optimized, aiming to significantly improve the model's performance accuracy and generalization ability in knowledge-based question-answering tasks in the power grid field. In this embodiment, new task-specific layers are first added to the model to output the most relevant document content. The parameters of these task-specific layers are randomly initialized at the start of fine-tuning and updated throughout the process to optimize the model's performance on specific tasks. When new data is introduced into the model being fine-tuned, it is first passed through a frozen layer of the pre-trained model. The frozen layer performs a series of nonlinear transformations on the input data to extract task-related feature representations. Subsequently, these feature representations are passed to the newly added task-specific layers. In the task layers, the feature representations undergo linear transformations and the application of activation functions, ultimately producing the model's prediction results. Finally, the model's predictions are evaluated by comparing them with ground-based labels and based on the difference between the prediction results and the actual results. During the fine-tuning process, Hite Rate and MMR are used as performance metrics to ensure the stability and effectiveness of the training process, while accurately monitoring changes in the model's learning process to avoid overfitting.

[0044] Step 8: System Integration and Retrieval Mechanism Enhancement: Integrate the fine-tuned BGE embedding model into the retrieval system. Based on the workflow generated by the retrieval enhancement, the system will perform similarity matching on user queries according to the embedding representation generated by the model, and obtain fragments with high similarity as the basis for the answer, thereby improving the relevance and accuracy of the retrieval results.

[0045] In this step, the finely tuned and optimized BGE model is integrated into the retrieval system framework, fully leveraging its enhanced generative capabilities to implement a more precise and efficient matching strategy for user queries. In this process, this embodiment innovatively introduces model-generated embedding representations as a core optimization method for the retrieval algorithm. By calculating the similarity between the query and the document in the embedding space, the relevance and accuracy of the retrieval results are significantly improved. This measure not only optimizes the user experience but also promotes the further development of retrieval technology towards intelligence and precision.

[0046] As a preferred embodiment, in step one, a specific Embedding-FineTuning method is selected and its performance is compared with other methods to evaluate its performance in improving the text embedding model for similarity analysis needs in the power grid field. By comparing different fine-tuning methods, the adaptability, task customization, and interpretability of the reasoning results for power grid data in this field can be verified, thereby determining its superiority in similarity matching tasks. The performance comparison results provide a basis for model optimization and fine-tuning strategy selection, helping to meet the practical needs of complex data retrieval and analysis in the power grid field.

[0047] As a preferred embodiment, in step two, firstly, irrelevant or redundant information in the document is cleaned, typos, grammatical errors, and logical conflicts are corrected, and professional terms and units are standardized to ensure data consistency and reduce noise. Secondly, structured data (such as tables and charts) and unstructured data (such as natural language text) are parsed separately to extract key information and unify the data format for subsequent processing. Finally, appropriate desensitization strategies are adopted for different types of sensitive information, such as using generalization for geographical location and perturbation or masking for financial data, and random sampling is used for verification to ensure data privacy protection.

[0048] As a preferred embodiment, in step three, the document in the power grid field is dynamically segmented based on LumberChunker technology. Through an iterative prompting model, it is able to dynamically identify semantic turning points in continuous paragraphs, ensuring that each segmented content block maintains semantic coherence in the context, while being distinguished from adjacent content blocks. Semantically independent parts are divided into relatively independent blocks, reducing semantic confusion caused by fixed-length segmentation, and effectively handling complex document content across chapters and topics.

[0049] As a preferred embodiment, in step four, the T5 model is used to perform in-depth analysis of the text blocks to generate a high-quality question-answer pair dataset for subsequent model fine-tuning. The T5 model, based on a "text-to-text" framework, unifies complex natural language processing tasks into text conversion problems, effectively handling question-answer pair generation tasks. First, the T5 model deeply understands the context, semantics, and syntactic structure of each text block, automatically identifying valuable question points, such as document topics and core viewpoints, forming core questions and preliminary answers focusing on the topic or key information of the text block. Then, based on this, the model is further guided to generate subsequent questions. For each question point, the model deeply analyzes the document content, filters out the most relevant and accurate information, and then generates related "follow-up question-answer pairs," such as "What is the relevant background of this question?" and "What are the advantages of this solution?" This process moves from basic knowledge learning to in-depth detail, further analysis, and comparison, making the dataset more challenging and comprehensive for model training. By generating this multi-turn dialogue, each text block can form a question-and-answer chain containing a core question and multiple subsequent questions, which increases the richness of the data and forms a question-and-answer pair dataset that meets the specific needs of the power grid field. These question-and-answer pair datasets will be directly used for fine-tuning to improve the performance and accuracy of the model in actual question-and-answer tasks.

[0050] As a preferred embodiment, in step five, the BGE text embedding model is deployed locally. This method allows for model training and testing without network latency, which helps improve overall work efficiency and provides control over the training environment, effectively ensuring data security and privacy and preventing information leakage.

[0051] As a preferred embodiment, in step six, to improve the accuracy and rationality of the question-answering system in the power grid field, based on principles such as domain relevance, task clarity, data fusion, flexibility, and simplicity, and combined with the experience of domain experts, a suitable prompt template is designed to guide the model generation process, ensuring that it can provide effective decision support. Furthermore, data is integrated into the prompt through direct embedding, context building, and interactive guidance to help the model accurately parse and reason about data in specific contexts. For example, a suitable prompt could be: "[Instruction] Based on the known information, answer the question concisely and professionally, and provide the thought process for the answer. [Known Information]... . [Question]... ."

[0052] In a preferred embodiment, in step seven, the generated question-and-answer pair dataset is divided into a training set, a validation set, and a test set according to a certain ratio. 80% of the data is used as the training set, 10% as the validation set, and 10% as the test set to monitor the model's performance during training and evaluate its performance on unseen data. In the fine-tuning phase, the parameters of the BGE model are adjusted using the training set data. The goal of fine-tuning is to better adapt the model to question-and-answer tasks in the power grid domain. Through repeated iterations, the model will gradually learn a pattern that produces accurate output given an input. During training, Hite Rate and MMR were used as testing metrics. Hite Rate evaluates a recall algorithm, representing the probability that the algorithm selects the top k texts from a large corpus as recall results and includes the expected target text. This probability indicates whether the algorithm can accurately locate and merge the required text, while limiting the number of returned results. When a recall algorithm achieves high accuracy, it means it frequently includes the expected text, and its performance is generally considered satisfactory. MMR, on the other hand, is a widely used retrieval effectiveness evaluation metric, focusing specifically on the ranking of relevant documents or information returned by the system for each query. Specifically, MMR measures the average reciprocal rank of the first relevant document or information returned by the system in a series of queries. A higher MMR value indicates a higher ranking of the relevant documents or information returned by the system, thus indicating better retrieval performance. The formula is as follows:

[0053]

[0054] Where is the set to be queried, rank i It is the ranking of the first relevant item in the i-th query.

[0055] The following is a comparison of the test results before and after model fine-tuning:

[0056]

[0057] As a preferred embodiment, in step eight, after model fine-tuning, the fine-tuned model is integrated into the retrieval system. The system uses the embedded representation generated by the model to match user queries, using highly similar text segments as the basis for answers, thereby improving the relevance and accuracy of the question-and-answer results. Embedded representation maps user queries and document representations to the same vector space. By calculating the user query vector, compared to keyword search, it can match more comprehensive relevant content while understanding the user's intent, thus improving the accuracy of the retrieval system.

[0058] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0059] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0060] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0061] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0062] It should be noted that, unless otherwise defined, the technical or scientific terms used in this invention should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0063] Based on the same inventive concept, this invention also provides a computer device, comprising: one or more processors, and a memory for storing one or more computer programs; the programs include program instructions, and the processor executes the program instructions stored in the memory. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, used to implement one or more instructions, specifically for loading and executing one or more instructions stored in a computer storage medium to implement the above-described method.

[0064] It should be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium storing a computer program, which, when executed by a processor, performs the above-described method. This storage medium can be any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0065] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0066] The foregoing has shown and described the basic principles, main features, and advantages of this disclosure. Those skilled in the art should understand that this disclosure is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of this disclosure. Various changes and modifications can be made to this disclosure without departing from its spirit and scope, and all such changes and modifications fall within the scope of this disclosure as claimed.

[0067] This invention is not limited to the preferred embodiment described above. Anyone inspired by this invention can derive other forms of improved search generation methods based on Embedding-FineTuning. All equivalent variations and modifications made within the scope of the claims of this invention should be included in the scope of this invention.

Claims

1. An improved retrieval enhancement generation method based on Embedding-FineTuning, characterized in that: Large-scale, multi-source, heterogeneous data from the power grid are preprocessed and cleaned to ensure data consistency and quality. The processed text data is dynamically divided based on LumberChunker technology, and text blocks are generated according to semantic inflection points to generate text blocks suitable for fine-tuning training. Then, the T5 model is used to generate question-answer pairs based on the text blocks in multiple rounds, forming a question-answer pair dataset for fine-tuning. Customized prompt templates were designed to guide the model generation process in practical application scenarios in the power grid field. Embedding-FineTuning was performed on the question-answering dataset to improve the model's embedding representation performance and accuracy in the power grid field by optimizing the model parameters; Embedding-FineTuning fine-tuning training was performed using the BGE embedding model; The method of using the T5 model to generate question-answer pairs based on text blocks through multi-turn question answering, forming a question-answer pair dataset for fine-tuning, specifically involves: enabling the T5 model to deeply understand the context, semantics, and syntactic structure of each text block, identifying valuable question points, forming core questions and preliminary answers focusing on the topic or key information of the text block, and then further guiding the model to generate subsequent questions. For each question point, the model deeply analyzes the document content, filters out the most relevant and accurate information, and then generates related questions. Through this multi-turn dialogue generation method, each text block forms a question-answer chain containing a core question and multiple subsequent questions, forming a question-answer pair dataset that meets the specific needs of the power grid field for fine-tuning. In the design process of the prompt template, data is also incorporated into the prompt through direct embedding, context building, and interactive guidance to help the model accurately parse and reason about data in specific contexts.

2. The improved retrieval enhancement generation method based on Embedding-FineTuning according to claim 1, characterized in that: The fine-tuned BGE embedding model is integrated into the retrieval system based on the workflow of retrieval enhancement generation. The embedding representation generated by the BGE embedding model is used to match user queries. Highly similar text segments are obtained from the text block knowledge base that has been processed by block embedding as the basis for answering, thereby improving the relevance and accuracy of question answering results. The embedding representation is to map user queries and document representations to the same vector space and calculate the user query vector.

3. The improved retrieval enhancement generation method based on Embedding-FineTuning according to claim 1, characterized in that: The preprocessing and cleaning include cleaning irrelevant or redundant information in the document, correcting typos, grammatical errors and logical conflicts, and standardizing technical terms and units to ensure data consistency and reduce noise. The system parses structured and unstructured data separately, extracts key information, and standardizes data formats for subsequent processing. For different types of sensitive information, it employs desensitization strategies, including generalization, perturbation, or masking, and verifies the data through random sampling to ensure data privacy protection.

4. The improved retrieval enhancement generation method based on Embedding-FineTuning according to claim 1, characterized in that: The specific method of dynamically dividing the processed text data based on LumberChunker technology is as follows: through an iterative prompting model, semantic turning points in continuous paragraphs are dynamically identified to ensure that each divided content block maintains semantic coherence in the context, while distinguishing it from adjacent content blocks. Parts with semantic independence are divided into relatively independent blocks to reduce semantic confusion caused by fixed-length division.

5. The improved retrieval enhancement generation method based on Embedding-FineTuning according to claim 1, characterized in that: The Embedding-FineTuning fine-tuning training on the question-answer pair dataset specifically involves dividing the generated question-answer pair dataset into a training set, a validation set, and a test set according to a certain ratio to monitor the model's performance during training and evaluate its performance on unseen data. During the fine-tuning phase, the parameters of the BGE model are adjusted using data from the training set. The goal of fine-tuning is to adapt the model to question-answering tasks in the power grid domain. Through repeated iterations, the model learns a pattern of producing accurate outputs given inputs. During training, Hite Rate and MMR are used as test metrics.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the improved retrieval enhancement generation method based on Embedding-FineTuning as described in any one of claims 1-5.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the retrieval enhancement generation improvement method based on Embedding-FineTuning as described in any one of claims 1-5.

Images