Adaptive RAG memory enhancement method

Abstract

The invention belongs to the technical field of RAG memory enhancement, and particularly relates to a self-adaptive RAG memory enhancement method, which comprises the following steps of: 1, determining an application scene and task requirements; 2, a backbone LLM is initialized and configured; step 3, attribute mining and identification; step 4, attribute labeling granularity selection and labeling; step 5, evaluating and sorting attribute priorities; step 6, constructing memory instance embedding representation; step 7, implementing a retrieval strategy based on attributes and embedding; step 8, integrating and optimizing retrieval results; and step 9, performing experimental evaluation and continuous improvement, namely determining the specific application scene of the self-adaptive RAG memory enhancement technology, retrieving appropriate text content for recommendation according to the theme and requirements input by the user, and determining the specific requirements of the task according to different scenes. According to the method, memory information is enriched by deeply mining and marking key attributes, and meanwhile, two efficient retrieval means are adopted to ensure quick positioning of related information.

Description

Technical Field

[0001] This invention belongs to the field of RAG memory enhancement technology, and specifically relates to an adaptive RAG memory enhancement method. Background Technology

[0002] The core idea of ​​RAG memory augmentation is to introduce external knowledge sources or memory mechanisms to provide additional information when a language model generates text. Traditional language models mainly rely on their pre-trained parameters and contextual information when generating text, but this approach can be limited by model capacity and training data. RAG memory augmentation enriches the knowledge sources in the generation process by retrieving relevant external knowledge or obtaining specific information from memory, thereby improving the quality, accuracy, and relevance of the generated text and integrating the language model with external knowledge bases or corpora. When generating text, information retrieval techniques are first used to find relevant documents, facts, or information from external knowledge sources based on the input question or context. Then, the retrieved information is provided as additional input to the language model to help it generate more accurate and targeted answers. For example, when answering scientific questions, scientific literature databases can be searched to obtain the latest research results and relevant knowledge.

[0003] This invention addresses the challenges of memory modules in large-scale language model (RAG) systems, particularly in terms of retrieval efficiency, memory management, knowledge retrieval, and knowledge integration. To this end, we propose an adaptive RAG memory enhancement method to solve these problems. Summary of the Invention

[0004] The purpose of this invention is to provide an adaptive RAG memory enhancement method that can enrich memory information by deeply mining and annotating key attributes, while employing two efficient retrieval methods to ensure the rapid location of relevant information.

[0005] The specific technical solution adopted by this invention is as follows:

[0006] An adaptive RAG memory enhancement method includes the following steps:

[0007] Step 1. Determine the application scenario and task requirements;

[0008] Step 2. Backbone LLM initialization and configuration;

[0009] Step 3. Attribute mining and identification;

[0010] Step 4. Attribute annotation granularity selection and annotation;

[0011] Step 5. Attribute priority evaluation and sorting;

[0012] Step 6. Construct the memory instance embedding representation;

[0013] Step 7. Implement the attribute-based and embedding-based retrieval strategy;

[0014] Step 8. Integrate and optimize search results;

[0015] Step 9. Experimental evaluation and continuous improvement.

[0016] In a preferred embodiment, determining the application scenario and task requirements involves clarifying the specific application scenario of the adaptive RAG memory enhancement technology. Based on the user's input topic and requirements, appropriate text content is retrieved and recommended. For different scenarios, specific task requirements are determined, including requirements for retrieval speed, accuracy, and relevance. In the intelligent customer service scenario, a high requirement for retrieval speed is placed on providing accurate answers in a short time. In the intelligent writing assistance scenario, more emphasis is placed on the relevance of the retrieved content to the user's needs.

[0017] In a preferred embodiment, the backbone LLM initialization and configuration involves selecting a high-performance backbone LLM suitable for the task, such as GPT-3 or Wenxin Yiyan, and setting model parameters based on task characteristics and data scale. These model parameters include the learning rate η, the number of layers L, and the hidden layer dimension d. h In text classification tasks, if the amount of data is small, reduce the number of layers and the dimension of hidden layers to prevent overfitting. When the amount of data is large, appropriately increase the parameters to improve the model's performance. The goal of model training is to minimize the loss function. The formula for the cross-entropy loss function is:

[0018]

[0019] Where N is the number of samples, C is the number of categories, and y ij p is the true label of sample i belonging to category j. ij It is the probability that sample i belongs to category j as predicted by the model.

[0020] In a preferred embodiment, the attribute mining and identification involves mining attributes from two perspectives: entity-centric and dialogue-centric. The entity-centric perspective focuses on specific entities, while the dialogue-centric perspective focuses on user interactions. Attributes are identified using the natural language processing capabilities of a backbone LLM (Local Language Management) system. Named entity recognition technology is used to extract entities and attributes from the text. Assuming the text is T, after entity recognition processing, a set of entities is obtained:

[0021] E = {e1, e2, ..., e} n}

[0022] Attribute set:

[0023] A = {a1, a2, ..., a} m}

[0024] Among them, ei Let a represent the i-th entity. j This represents the j-th attribute.

[0025] In a preferred embodiment, the granularity of attribute annotation is selected and annotated based on task requirements and data characteristics, determining whether to use round-level or session-level annotation. Round-level annotation is performed independently for each dialogue round, while session-level annotation starts from the overall dialogue, grasping the broad context. During annotation, the attributes and their corresponding values ​​are recorded in the memory of the RAG. If round-level annotation is used, the movie type and actor preference mentioned by the user are annotated in each dialogue round. If session-level annotation is used, the user's overall movie preference and viewing scene attributes are annotated throughout the entire dialogue.

[0026] In a preferred embodiment, the attribute priority evaluation and ranking is based on a comprehensive consideration of the relevance of the attribute to the task, its frequency of occurrence, and its importance. Relevance is measured by calculating the similarity between the attribute and task keywords, using the cosine similarity formula:

[0027]

[0028] Where A and B represent the vector representations of attributes and task keywords, respectively;

[0029] Frequency is the number of times an attribute appears in a dialogue or dataset, f. Importance is assigned a weight w based on domain knowledge or expert experience, using the formula:

[0030] P = w × Sim(A,B) + f

[0031] Calculate the attribute priority score P, and sort the attributes according to the score, placing the more relevant and important attributes first.

[0032] In a preferred embodiment, the construction of memory instance embeddings is represented by using memory enhancement technology to transform each memory instance into an embedding vector. The embedding methods are word embedding and sentence embedding. The memory instance is text S. After processing by the embedding model, the embedding vector v = Embedding(S) is obtained. Semantically similar memory instances are closer together, and subsequent retrieval is performed using similarity search technology.

[0033] In a preferred embodiment, the attribute-based and embedding-based retrieval strategy is implemented as follows: Attribute-based retrieval generates attributes specific to the current context and retrieves memory instances with similar attributes; embedding-based retrieval transforms the current problem or requirement into an embedding vector and performs retrieval by calculating the similarity with the embedding vector of the memory instance. The similarity calculation uses the Euclidean distance formula.

[0034]

[0035] Where v1 and v2 represent the embedding vectors of the question and the memory instance, respectively, n is the vector dimension, and the memory instance with the highest similarity is selected as the retrieval result.

[0036] In a preferred embodiment, the search result integration and optimization involves integrating the results obtained from attribute-based and embedding-based searches, removing duplicate information, and ranking the results according to relevance and reliability. Relevance is determined by referring to the previously calculated similarity, and reliability is determined by a weight r based on the source and credibility factors of the remembered instances, using the formula:

[0037] Score = r × Sim

[0038] Calculate the overall score, where Score is the overall score of the search results and Sim is the similarity. Sort the search results according to the overall score. If the search results are not ideal, adjust the search parameters, such as the attribute matching threshold, embedding similarity measurement methods, or re-evaluating attribute priorities.

[0039] In a preferred embodiment, the experimental evaluation and continuous improvement involves conducting experimental evaluations of the method on multiple tasks and datasets, using accuracy, recall, and F1 score metrics to measure performance, analyzing the advantages and disadvantages of the method based on the evaluation results, and making improvements to address the existing problems.

[0040] The technical effects achieved by this invention are as follows:

[0041] The core advantage of this invention lies in its autonomy. It can automatically identify and define key attributes without manual design, thereby reducing human intervention and costs.

[0042] Experimental results show that this invention effectively improves the performance of LLM agents in multiple tasks, including dialogue recommendation, question answering, and event summarization. This demonstrates the effectiveness of this invention in practical applications.

[0043] Attribute filtering and embedded retrieval enable the rapid retrieval of relevant memories, improving search efficiency. This is crucial for handling large amounts of historical data and complex tasks.

[0044] It can be applied to various downstream tasks, such as dialogue recommendation, question answering, and event summarization. This demonstrates the good versatility of the invention, making it applicable to different scenarios.

[0045] The design of this invention makes it easy to expand. With the development of LLM technology, this invention can easily integrate new methods and models to further improve its performance. Attached Figure Description

[0046] Figure 1 This is a flowchart illustrating an adaptive RAG memory enhancement method according to the present invention;

[0047] Figure 2 This is a schematic diagram of an adaptive RAG memory enhancement method according to the present invention. Detailed Implementation

[0048] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0049] Please see Figure 1-2 As shown, this invention provides an adaptive RAG memory enhancement method, comprising the following steps:

[0050] Step 1. Determine the application scenario and task requirements;

[0051] Step 2. Backbone LLM initialization and configuration;

[0052] Step 3. Attribute mining and identification;

[0053] Step 4. Attribute annotation granularity selection and annotation;

[0054] Step 5. Attribute priority evaluation and sorting;

[0055] Step 6. Construct the memory instance embedding representation;

[0056] Step 7. Implement the attribute-based and embedding-based retrieval strategy;

[0057] Step 8. Integrate and optimize search results;

[0058] Step 9. Experimental evaluation and continuous improvement.

[0059] To clarify the specific application scenarios and task requirements of adaptive RAG memory enhancement technology, the system retrieves and recommends appropriate text content based on the user's input topic and requirements. For different scenarios, the specific task requirements are determined, including requirements for retrieval speed, accuracy, and relevance. In the intelligent customer service scenario, high retrieval speed is required, with the expectation of providing accurate answers in a short time. In the intelligent writing assistance scenario, more emphasis is placed on the relevance of the retrieved content to the user's needs.

[0060] The backbone LLM initialization and configuration involves selecting high-performance, task-adaptive backbone LLMs, such as GPT-3 and Wenxin Yiyan. Model parameters are set based on task characteristics and data scale, including the learning rate η, number of layers L, and hidden layer dimension d. h In text classification tasks, if the amount of data is small, reduce the number of layers and the dimension of hidden layers to prevent overfitting. When the amount of data is large, appropriately increase the parameters to improve the model's performance. The goal of model training is to minimize the loss function. The formula for the cross-entropy loss function is:

[0061]

[0062] Where N is the number of samples, C is the number of categories, and y ij p is the true label of sample i belonging to category j. ij It is the probability that sample i belongs to category j as predicted by the model.

[0063] Attribute mining and recognition involves extracting attributes from two perspectives: entity-centric and dialogue-centric. The entity-centric perspective focuses on specific entities, while the dialogue-centric perspective focuses on user interactions. Attributes are identified using the natural language processing capabilities of a core LLM (Local Language Management) architecture. Named entity recognition technology is used to extract entities and attributes from the text. Assuming the text is T, after entity recognition processing, a set of entities is obtained:

[0064] E = {e1, e2, ..., e} n}

[0065] Attribute set:

[0066] A = {a1, a2, ..., a} m}

[0067] Among them, e i Let a represent the i-th entity. j This represents the j-th attribute.

[0068] The granularity of attribute annotation is selected and labeled based on task requirements and data characteristics. It is determined whether to use round-level or session-level annotation. Round-level annotation is performed independently for each dialogue round, while session-level annotation starts from the overall dialogue and grasps the broad context. During annotation, the attributes and their corresponding values ​​are recorded in the memory of RAG. If round-level annotation is used, the movie type and actor preference mentioned by the user are annotated in each dialogue round. If session-level annotation is used, the user's overall movie preference and viewing scene attributes are annotated throughout the dialogue.

[0069] Attribute priority evaluation and ranking considers the relevance of attributes to the task, frequency of occurrence, and importance factors to assess priority. Relevance is measured by calculating the similarity between the attribute and task keywords, using the cosine similarity formula:

[0070]

[0071] Where A and B represent the vector representations of attributes and task keywords, respectively;

[0072] Frequency is the number of times an attribute appears in a dialogue or dataset, f. Importance is assigned a weight w based on domain knowledge or expert experience, using the formula:

[0073] P = w × Sim(A,B) + f

[0074] Calculate the attribute priority score P, and sort the attributes according to the score, placing the more relevant and important attributes first.

[0075] Constructing a memory instance embedding representation involves using memory enhancement techniques to transform each memory instance into an embedding vector. The embedding methods are word embedding and sentence embedding. The memory instance is text S. After processing by the embedding model, the embedding vector v = Embedding(S) is obtained. Semantically similar memory instances are closer together, and subsequent retrieval is performed using similarity search techniques.

[0076] The attribute-based and embedding-based retrieval strategies are implemented as follows: Attribute-based retrieval generates task-specific attributes based on the current context and retrieves memory instances with similar attributes; Embedding-based retrieval transforms the current question or requirement into an embedding vector and retrieves the memory instance by calculating the similarity to the embedding vector of the memory instance. The similarity calculation uses the Euclidean distance formula.

[0077]

[0078] Where v1 and v2 represent the embedding vectors of the question and the memory instance, respectively, n is the vector dimension, and the memory instance with the highest similarity is selected as the retrieval result.

[0079] The search results integration and optimization involves combining the results obtained from attribute-based and embedding-based searches, removing duplicate information, and ranking the results according to relevance and reliability. Relevance is determined by the previously calculated similarity, while reliability is weighted by a weight r based on the source and credibility factors of the remembered instances, using the formula:

[0080] Score = r × Sim

[0081] Calculate the overall score, where Score is the overall score of the search results and Sim is the similarity. Sort the search results according to the overall score. If the search results are not ideal, adjust the search parameters, such as the attribute matching threshold, embedding similarity measurement methods, or re-evaluating attribute priorities.

[0082] Experimental evaluation and continuous improvement involve conducting experimental evaluations of the method on multiple tasks and datasets, using accuracy, recall, and F1 score metrics to measure performance, analyzing the advantages and disadvantages of the method based on the evaluation results, and making improvements to address the existing problems.

[0083] The core advantage of this invention lies in its autonomy. It can automatically identify and define key attributes without manual design, thereby reducing human intervention and costs.

[0084] Experimental results show that this invention effectively improves the performance of LLMagent in multiple tasks, including dialogue recommendation, question answering, and event summarization. This demonstrates the effectiveness of this invention in practical applications.

[0085] Attribute filtering and embedded retrieval enable the rapid retrieval of relevant memories, improving search efficiency. This is crucial for handling large amounts of historical data and complex tasks.

[0086] It can be applied to various downstream tasks, such as dialogue recommendation, question answering, and event summarization. This demonstrates the good versatility of the invention, making it applicable to different scenarios.

[0087] The design of this invention makes it easy to expand. With the development of LLM technology, this invention can easily integrate new methods and models to further improve its performance.

[0088] The above description is merely a preferred embodiment of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention. Structures, devices, and operating methods not specifically described or explained in this invention are implemented according to conventional methods in the art unless otherwise specified or limited.

Claims

1. An adaptive RAG memory enhancement method, characterized by: Comprising the following steps: Step 1. Determine the application scenario and task requirements; Step 2. Backbone LLM initialization and configuration; Step 3. Attribute mining and identification; Step 4. Attribute annotation granularity selection and annotation; Step 5. Attribute priority evaluation and sorting; Step 6. Constructing memory instance embedding representation; Step 7. Implementing retrieval strategy based on attributes and embeddings; Step 8. Retrieval result integration and optimization; Step 9. Experimental evaluation and continuous improvement.

2. The adaptive RAG memory enhancement method of claim 1, wherein: The determination of application scenario and task requirement is to determine the specific application scenario of adaptive RAG memory enhancement technology. According to the theme and requirements input by the user, the appropriate text content is retrieved for recommendation. For different scenarios, the specific requirements of the task are determined, including the requirements for retrieval speed, accuracy and relevance. In the intelligent customer service scenario, the retrieval speed is required to be high, and accurate answers are expected to be given within a short time. In the intelligent writing assistance scenario, more attention is paid to the relevance of retrieval content and user requirements.

3. The adaptive RAG memory enhancement method of claim 1, wherein: The backbone LLM initialization and configuration is to select a backbone LLM with excellent performance and adapt to the task, including GPT-3 and Ernie, and model parameters are set according to task characteristics and data size, including learning rate η, layer number L and hidden layer dimension d h In the text classification task, if the data volume is small, the layer number and the hidden layer dimension are reduced to prevent overfitting, and when the data volume is large, the parameters are appropriately increased to improve the model performance. The goal of model training is to minimize the loss function. The cross-entropy loss function formula is: where N is the number of samples, C is the number of classes, y ij is the true label of sample i belonging to class j, p ij is the probability of the model predicting that sample i belongs to class j.

4. The adaptive RAG memory enhancement method of claim 1, wherein: The attribute mining and identification is to mine attributes from the perspectives of entity center and dialogue center. From the perspective of entity center, the attributes of specific entities are mined, and from the perspective of dialogue center, the user interaction is focused. The natural language processing capability of backbone LLM is used to identify attributes. Through named entity recognition technology, entities and attributes in the text are extracted. Assuming that the text is T, after entity recognition processing, the entity set is obtained: E = {e1, e2,..., e n} Attribute set: A = {a1, a2,..., a m} wherein e i represents the ith entity, a j represents the jth attribute.

5. The adaptive RAG memory enhancement method of claim 1, wherein: The attribute annotation granularity selection and annotation is to determine whether to use round level or session level annotation according to task requirements and data characteristics. Round level annotation is independent for each dialogue round, and session level annotation is based on the overall dialogue to grasp the extensive context. When annotating, the attributes and their corresponding values are recorded in the memory of RAG. If round level annotation is used, the movie type and actor preference mentioned by the user are annotated in each dialogue round. If session level annotation is used, the overall preference of the user for movies and the viewing scene attribute in the whole dialogue process are annotated.

6. The adaptive RAG memory enhancement method of claim 1, wherein: The attribute priority evaluation and sorting is to evaluate the priority by considering the relevance, frequency of occurrence and importance of attributes and tasks. The relevance is measured by calculating the similarity between the attribute and the task keyword using the cosine similarity formula: Where A and B represent the vector representation of the attribute and the task keyword, respectively; The frequency of occurrence f can be calculated by counting the number of times the attribute appears in the dialogue or data set. The importance is set according to the domain knowledge or expert experience with a weight w. The attribute priority score P is calculated by the formula: P = w x Sim(A, B) + f The attributes are sorted according to the score, with more relevant and important attributes ranked first.

7. The adaptive RAG memory enhancement method of claim 1, wherein: The construction of memory instance embedding representation is to convert each memory instance into an embedding vector using memory enhancement technology. The embedding method is word embedding and sentence embedding. The memory instance is text S, which is processed by the embedding model to obtain the embedding vector v = Embedding(S). Memory instances with similar semantics are closer in distance, and subsequent retrieval is performed through similarity search technology.

8. The adaptive RAG memory enhancement method of claim 1, wherein: The attribute and embedding-based retrieval strategy is implemented as attribute-based retrieval, generates attributes of a specific task according to the current context, retrieves memories with similar attributes, embedding-based retrieval, converts the current question or demand into an embedding vector, and retrieves by calculating the similarity of the embedding vector with the memory instance, the similarity calculation uses the Euclidean distance formula: Where v1, v2 represent the embedding vectors of the question and memory instance respectively, n is the vector dimension, and the memory instance with the highest similarity is selected as the retrieval result.

9. The adaptive RAG memory enhancement method of claim 1, wherein: The retrieval result integration and optimization integrates the results obtained by attribute and embedding-based retrieval, removes duplicate information, sorts the results according to relevance and reliability, the relevance refers to the similarity calculated above, the reliability can determine the weight r according to the source and credibility of the memory instance, and the comprehensive score is calculated by the formula: Score = r x Sim The comprehensive score is calculated, where Score is the comprehensive score of the retrieval result and Sim is the similarity, the retrieval results are sorted according to the comprehensive score, if the retrieval result is not ideal, adjust the retrieval parameters, attribute matching threshold, embedding similarity measurement method, or re-evaluate the attribute priority.

10. The adaptive RAG memory enhancement method of claim 1, wherein: The experimental evaluation and continuous improvement is to evaluate the method on multiple tasks and datasets, use accuracy, recall, and F1 value indicators to measure performance, analyze the advantages and disadvantages of the method according to the evaluation results, and improve the existing problems.

Images