Enhanced retrieval generation method based on latent fusion LoRA

Abstract

This invention relates to the field of natural language processing technology, specifically to an enhanced retrieval and generation method based on latent similarity fusion LoRA. The method includes: a three-level architecture of retrieval layer, adaptation layer, and output layer to achieve knowledge fusion and dynamic optimization; the retrieval layer uses a pre-trained model to encode the input text into query vectors, and selects and concatenates Top-n relevant vectors from a knowledge base based on hybrid similarity; the adaptation layer filters low-relevance retrieval results through a dynamic gating mechanism, and combines LoRA for low-rank decomposition and feature fusion to reduce computation and optimize knowledge injection efficiency; the output layer freezes the parameters of the large language model, retaining its general generation capabilities to output text, avoiding knowledge forgetting caused by fine-tuning. This invention solves the problems of noise interference, high computational cost, and knowledge forgetting in existing methods through hybrid similarity scoring, dynamic gating, and LoRA collaborative design and freezing strategy, and is applicable to question answering systems and text generation scenarios.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and more specifically to an enhanced retrieval generation method based on Latent Fusion LoRA. Background Technology

[0002] With the increasing demand for knowledge integration and generation capabilities in natural language processing technology, retrieval augmented generation (RAG) methods have become the mainstream technical approach to improve the performance of tasks such as question answering systems and text generation by integrating external knowledge bases to enhance the model's reasoning ability. Existing research is mostly based on directly concatenating retrieval results or fine-tuning all parameters of pre-trained language models. Although this can inject domain knowledge in specific scenarios, there is still room for optimization in terms of dynamic knowledge screening efficiency and model stability.

[0003] Chinese invention application CN119578557A discloses a hybrid enhancement strategy for LLM capability enhancement. This method involves acquiring corpus data from the target application domain and external literature data, performing denoising, word segmentation, and semantic indexing to establish a searchable heterogeneous knowledge base. Then, topic analysis and word vector clustering are used to classify the corpus data, forming a basic corpus set. Based on this basic corpus set, a large-scale language model undergoes multiple rounds of parameter updates using a domain-specific fine-tuning algorithm, with a fusion threshold set between the new and original parameters. A matching algorithm retrieves relevant literature fragments from the heterogeneous knowledge base, and semantic similarity assessment is used to filter content that matches domain relevance. This content is then fused with the fine-tuned model, and the fusion results are processed using a context fusion algorithm and entity linking method to complete syntactic analysis and generate accurate answers for the target application domain.

[0004] Low-rank adaptation (LoRA) technology achieves lightweight model adaptation through parameter decomposition and has proven its efficiency in single-task fine-tuning scenarios. However, it has not yet been deeply integrated with retrieval and generation frameworks. Against this backdrop, there is an urgent need for a technical solution that balances knowledge fusion accuracy, computational efficiency, and model generalization ability to meet the needs of dynamic integration of multi-source knowledge in complex scenarios and promote the practical application of retrieval and generation technology. Summary of the Invention

[0005] The purpose of this invention is to address the problems existing in the background technology by proposing an enhanced retrieval generation method based on potential fusion LoRA.

[0006] The technical solution of this invention: an enhanced retrieval generation method based on latent fusion LoRA, comprising:

[0007] Retrieval layer: Encodes the input text into a query vector Q, and retrieves the top-n relevant vectors from the vector knowledge base using a hybrid similarity scoring method;

[0008] Adaptation layer: Highly relevant vectors are selected through dynamic gating, and features are fused using the low-rank adaptation component LoRA. Residual features are generated through low-rank decomposition matrix, and the weight distribution of Top-n retrieval results is dynamically adjusted in conjunction with the fully connected layer.

[0009] Output layer: The fused features output from the adaptation layer are input into the large language model with frozen parameters to generate the final text. The parameters of the large language model remain fixed during both the training and inference phases.

[0010] The preferred hybrid similarity scoring method is as follows:

[0011]

[0012] Among them, Score i This represents the overall score that measures the degree of matching between the query vector and the knowledge base vector; 'a' represents the weighting coefficient; and 'K' represents the weighting coefficient. i This represents the vectorized representation of the i-th fragment in the knowledge base; cos(Q,K) i () represents cosine similarity, measuring the similarity between Q and K. i The similarity in the vector direction ranges from [-1, 1]; ||QK i ||2 represents the Euclidean distance, i.e., the distance between vectors Q and K. i Absolute distance in space.

[0013] Preferably, the activation condition for the dynamic gating method is:

[0014]

[0015] Where τ represents the hyperparameter threshold, i.e. the threshold τ of the dynamic gating mechanism; g(x) represents the dynamic gating coefficient; x represents the semantic information being processed; W represents the trainable weight matrix; and σ(·) represents the Sigmoid activation function.

[0016] Preferably, the threshold τ of the dynamic gating mechanism is dynamically adjusted by the ratio of positive to negative samples in the training dataset.

[0017] Preferably, the LoRA component achieves feature dimensionality reduction and mapping through low-rank decomposition matrices A and B;

[0018] The feature fusion formula is as follows:

[0019] h lora =h base +g(x)·LoRA([h base ;k1;...;k n ]);

[0020] Among them, h lora The hidden layer representation after fusion; h baseLoRA(·) represents the hidden layer output of the base model; LoRA(·) represents the low-rank adaptive component; k1,…,k n Represents the retrieved knowledge fragment vector; [h base ;k1;...;k n ] indicates a feature splicing operation.

[0021] Preferably, the training loss function for the adaptation layer includes:

[0022] LoRA loss function L lora This is used to monitor the accuracy of knowledge fusion;

[0023] Gating loss function L gate , used to optimize the decision weights of dynamic gating.

[0024] Preferably, the LoRA loss function L lora Cross-entropy loss:

[0025]

[0026] Gating loss function L gate For binary cross-entropy loss:

[0027]

[0028] Where N represents the number of samples; y i The output logits score for the correct token; x i represents the output logits score of the current token; c represents the binary label value.

[0029] Preferably, the knowledge base retrieval in the retrieval layer uses the hierarchical-navigable-small-world-graph HNSW algorithm to accelerate vector matching.

[0030] Preferably, in the training process of the adaptation layer, only the parameters of the dynamic gating network, the fully connected layer, and the LoRA component are optimized, while the parameters of the embedding model of the retrieval layer and the large language model of the output layer are kept frozen.

[0031] Preferably, the weight allocation is implemented through a fully connected layer, which specifically includes: reducing the dimensionality of the concatenated matrix of the Top-n search results, generating attention weights based on the contextual semantics, and adjusting the contribution ratio of different knowledge sources.

[0032] Compared with the prior art, the above-mentioned technical solution of the present invention has the following beneficial technical effects:

[0033] This invention designs an enhanced retrieval generation method based on latent fusion LoRA. Through three key technologies—gated LoRA update, dynamic weight allocation, and efficient knowledge injection—it achieves the following comprehensive technical effects: while maintaining a lightweight design, it improves the model's knowledge utilization efficiency, inference accuracy, and scenario applicability, providing an efficient and reliable solution for dynamic knowledge enhancement generation of large language models.

[0034] The retrieval layer employs a hybrid similarity scoring formula that combines cosine similarity with weighted calculation of Euclidean distance to comprehensively evaluate the semantic relevance and spatial proximity between the query vector and knowledge base fragments, significantly improving the accuracy and robustness of retrieval results.

[0035] The adaptation layer filters low-relevance search content through a dynamic gating mechanism and combines low-rank adaptation technology to decompose and fuse features, thereby reducing the number of parameters while retaining the core knowledge injection capability, reducing invalid computation and improving training efficiency.

[0036] The output layer adopts a large language model parameter freezing strategy to avoid the risk of knowledge coverage during fine-tuning and ensure that the model can stably maintain its general generation capability when incorporating external knowledge.

[0037] This invention, while ensuring the quality of generated data, achieves efficient execution of retrieval and generation tasks through a hierarchical architecture design and a lightweight adaptation mechanism. It is applicable to scenarios such as intelligent question answering and text summarization, and has significant advantages in deployment cost and technical practicality. Attached Figure Description

[0038] Figure 1 This is a flowchart of an enhanced retrieval generation method based on potential fusion LoRA proposed in this invention;

[0039] Figure Labels: Decoder Input: The initial input vector of the decoder, representing the preprocessed original user text; A: Low-rank matrix, projecting the input features into a low-dimensional latent space; B: Low-rank matrix, mapping the low-dimensional features back to the original model dimension, forming a low-rank decomposition with matrix A; h lora : The fused hidden layer representation; h base : Hidden layer output of the base model; Top-n vectors: The top n knowledge fragments most relevant to the input query Q, retrieved from the vector knowledge base. Detailed Implementation

[0040] Example 1, as Figure 1 As shown, the present invention proposes an enhanced retrieval generation method based on latent fusion LoRA, which constructs a three-level coupled architecture of retrieval layer, adaptation layer and output layer. Its specific implementation steps include:

[0041] Level 1, Retrieval Layer: Responsible for vectorizing the input text and retrieving data from the vector knowledge base, specifically:

[0042] Input text encoding, that is, the input text sequence is encoded into a query vector Q through a pre-trained Embedding model;

[0043] It should be noted that the core function of the Embedding model is to map high-dimensional, discrete text data (such as words, sentences, and documents) into low-dimensional, continuous real vectors (i.e., embedding vectors).

[0044] Knowledge base retrieval: The query vector Q obtains the top-n vectors from a vector knowledge base (such as HNSW (Hierarchical Navigable Small World graphs) used by Faiss), and then reorders them by combining cosine similarity and Euclidean distance through similarity calculation, using a weighted formula:

[0045]

[0046] Among them, Score i This represents the overall score that measures the degree of matching between the query vector and the knowledge base vector; 'a' represents the weighting coefficient, used to balance the contribution ratio of cosine similarity and Euclidean distance. In this embodiment, 'a' is a hyperparameter, with a default value of 0.7; K i This represents the vectorized representation of the i-th fragment in the knowledge base; cos(Q,K) i () represents cosine similarity, measuring the similarity between Q and K. i The similarity in the vector direction ranges from [-1, 1]; ||QK i ||2 represents the Euclidean distance (L2 norm), i.e., the distance between vectors Q and K. i In space, the smaller the value, the closer the two people are.

[0047] Vector concatenation: Retrieve the top-n relevant vectors with the highest cosine similarity to Q, and concatenate the query vector Q with the retrieved top-n vectors using tensors according to the following rules:

[0048] Step 1: Construct a fusion matrix with dimensions (n+1)×dim, where dim represents the vector dimension;

[0049] Step 2: Use Q as the first row vector, and arrange the Top-n vectors in descending order of similarity to form the subsequent matrix;

[0050] Level 2, Adaptation Layer: This layer optimizes knowledge fusion through gating mechanisms, dynamic weight allocation, and LoRA components. It includes dynamic gating coefficients, a fully connected layer, and LoRA components. The weight matrix of the dynamic gating coefficient g(x) is initialized using Xavier and L2 regularization is applied. Specifically:

[0051] Dynamic gating, where a gating network activates or inhibits knowledge fusion paths based on the relevance of the input data:

[0052] Where τ represents the hyperparameter threshold, i.e., the threshold of the dynamic gating mechanism, which is dynamically adjusted by the ratio of positive and negative samples in the training dataset. If the current gating value is less than τ, the LoRA component for latent representation fusion will not be invoked; g(x) represents the dynamic gating coefficient, whose core function is to determine whether to activate a specific module (LoRA component) based on the threshold, thereby flexibly adjusting the knowledge injection strategy; x represents the input feature vector, usually the query vector Q or knowledge vector K output by the retrieval layer. i , representing the semantic information being processed; W represents the trainable weight matrix, which performs a linear transformation on the input x to generate scalar values ​​for gating decisions; through training optimization, it learns knowledge relevance weights for different scenarios; σ(·) represents the Sigmoid activation function;

[0053] It's important to note that LoRA (Low-Rank Adaptation) is a technique for model fine-tuning, designed to reduce training time and computational resource requirements for specific tasks while maintaining model performance. LoRA works by decomposing the model's parameters into the product of two low-rank matrices, representing the model's weights and the adaptation layer, respectively. This allows the model to adapt to new tasks or datasets while maintaining the low-rank structure of its original parameters. Specifically, LoRA reduces the total number of parameters and computational cost by decomposing the parameters of each layer into two smaller (lower-rank) matrices, multiplying them, and then adding the product to a smaller adaptation layer.

[0054] Dynamic weight allocation and dimensionality reduction: The LoRA component module uses a fully connected layer to dynamically reduce the dimensionality of the input fused Q and top-n vector matrix to a specified dimension. Low-rank adaptation is then performed using matrices A and B, and the output residual is updated. The retrieved vector-related information is then compared with the original model input h. base Perform calculations to generate h lora h lora By h base The gating and LoRA components are obtained as follows:

[0055] h lora =h base +g(x)·LoRA([h base ;k1;...;k n ]);

[0056] Among them, h lora This represents the fused hidden layer representation, combined with the output of the base model (h). base ) and external knowledge (k1,…,k n Enhanced features generated through the LoRA component; h base LoRA(·) represents the hidden layer output of the base model; LoRA(·) represents the low-rank adaptation component; k1,…,k n This represents the retrieved knowledge fragment vector, obtained from an external knowledge base through a vector retrieval engine (such as Faiss); [h base ;k1;...;k n ] indicates a feature concatenation operation;

[0057] Loss functions, including LoRA loss L lora and gated loss function:

[0058] LoRA loss function (cross-entropy loss) L lora To ensure that the model answers questions correctly in the dataset task:

[0059]

[0060] Where N represents the number of samples; y i The output logits score for the correct token; x i This indicates the output logits score of the current token;

[0061] Gated loss function (binary cross-entropy) L gate Supervise the gating network's decision-making to ensure that only highly relevant search content triggers knowledge fusion:

[0062] Where 'c' represents the binary label value, i.e., whether the current label is on or off;

[0063] Accordingly, in the adaptation layer, the following adaptive processing is achieved through gating and a fully connected network:

[0064] (1) When the similarity between the Top-n vector and Q is detected to be lower than the preset threshold, the gating unit automatically suppresses the information transmission of low-relevance vectors;

[0065] (2) For highly correlated subset vectors, feature enhancement is performed using the attention weights learned by the fully connected layer;

[0066] (3) The mechanism is automatically optimized through backpropagation and finally outputs a weighted fusion feature representation;

[0067] Level three involves generating the final text based on the large language model with frozen parameters. This means inputting the information processed by the adaptation layer into the large language model structure. Leveraging the powerful language understanding and generation capabilities of the large language model, the final output is generated based on the previously fused information without altering its original parameters. This ensures that the model maintains its stable language processing paradigm while utilizing new knowledge. Specifically:

[0068] C1. Freeze the large language model generation. The output layer adopts the large language model with parameter freezing. The fusion features output by the adaptation layer are input into the model to generate the final text. The freezing strategy preserves the general language ability of the model and avoids the problem of knowledge forgetting caused by fine-tuning.

[0069] C2. The model training process is as follows:

[0070] D1. Dataset Construction: Label the relevance of the input question Q and the search results (positive and negative sample ratio 1:1) and construct the training data for the adaptation layer;

[0071] D2. Model Construction: Freeze the large language model and the embedding model, and add a gated network, a fully connected layer, and a LoRA component to the adaptation layer. The loss function of the gated network is L. gate The loss function of the fully connected layer and the LoRA network is L. lora ;

[0072] That is, only the gating network, fully connected layer and LoRA module of the adaptation layer are optimized, while the parameters of the embedding model and the large language model are fixed;

[0073] D3. Iterative optimization: The network model takes text as input, obtains Q through word embedding, retrieves Top-n from the knowledge base through the retrieval tool, and concatenates them. The gating network is trained based on whether the labels in the dataset are related. If the labels are related, the subsequent fully connected layers and LoRA components are trained and backpropagation is performed until convergence.

[0074] If the search result is relevant to the input (label is 1), the LoRA component is activated to update the parameters; otherwise, LoRA training is skipped to reduce invalid computation.

[0075] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited thereto. Various changes can be made within the scope of knowledge possessed by those skilled in the art without departing from the spirit of the present invention.

Claims

1. A latent fusion LoRA-based enhanced retrieval generation method, characterized in that, The application comprises: a retrieval layer: encode the input text into a query vector Q, and retrieve top-n relevant vectors from the vector knowledge base through a hybrid similarity scoring method; an adaptation layer: filter high correlation vectors through a dynamic gating method, and fuse features using a low-rank adaptation component LoRA, generate residual features through low-rank decomposition matrix, and dynamically adjust the weight distribution of top-n retrieval results through a full connection layer; the activation condition of the dynamic gating method is: ； wherein, represents a hyper-parameter threshold, i.e. a threshold for the dynamic gating mechanism ; g(x) represents a dynamic gating coefficient; x represents the semantic information currently processed; W represents a trainable weight matrix; represents a Sigmoid activation function; Threshold of dynamic gating mechanism By dynamically adjusting the proportion of positive and negative samples in the training dataset; the LoRA component realizes feature dimension reduction and mapping through low-rank decomposition matrices A and B; the feature fusion formula is: ； where h lora denotes the fused hidden layer representation; h base denotes the hidden layer output of the base model; denotes the low-rank adaptation component; k1,..., k n denotes the retrieved knowledge snippet vector; denotes the feature concatenation operation; the weight distribution is realized through a full connection layer, specifically including: dimension reduction processing on the splicing matrix of top-n retrieval results, and generating attention weight according to context semantics to adjust the contribution proportion of different knowledge sources; an output layer: input the fusion features output by the adaptation layer into a large language model with frozen parameters to generate the final text, and the parameters of the large language model remain fixed in the training and inference stages.

2. The method of claim 1, wherein, the hybrid similarity scoring method is: ； wherein, represents a comprehensive score measuring the matching degree of the query vector and the knowledge base vector; a represents a weight coefficient; K i represents the vectorized representation of the i-th segment in the knowledge base; represents the cosine similarity, measuring the similarity of Q and K i in the vector direction, with a value range of [-1, 1]; represents the Euclidean distance, i.e., the absolute distance of the vector Q and K i in space.

3. The method of claim 1, wherein, the training loss function of the adaptation layer includes: LoRA loss function L lora accuracy for supervised knowledge distillation; Gating loss function L gate , for optimizing the decision weights of dynamic gating.

4. The method of claim 3, wherein, LoRA loss function L lora is the cross-entropy loss: ； Gating loss function L gate is the binary cross-entropy loss: ； where N denotes the number of samples; y i denotes the output logits score for the correct token; x i denotes the output logits score for the current token; c denotes the binary label value.

5. The method of claim 1, wherein, the knowledge base retrieval of the retrieval layer adopts the hierarchical-navigable-small-world-graph HNSW algorithm to accelerate vector matching.

6. The method of claim 1, wherein, In the training process of the adaptation layer, only the parameters of the dynamic gating network, the full connection layer and the LoRA component are optimized, and the embedding model of the retrieval layer and the parameters of the large language model of the output layer remain frozen.

Images