Knowledge-enhanced generation method, system, and readable storage medium

Abstract

The application provides a knowledge-enhanced generation method, system and readable storage medium, the method comprises the following steps: encoding input text containing user questions or instructions into a query sequence and inputting a large language model comprising a front decoding layer and a rear decoding layer; generating a shallow semantic vector using the front decoding layer, and extracting the corresponding part of the user questions or instructions as a query feature; calculating the probability distribution of the query feature in the preset knowledge field, selecting the two fields with the highest probability and normalizing the probability as weights; inputting the query feature into the memory sub-module corresponding to the two fields respectively, and performing weighted summation according to the weights to obtain a memory vector; generating low-rank adaptive parameters according to the memory vector, and injecting the rear decoding layer; inputting the shallow semantic vector into the rear decoding layer after the parameter injection, and obtaining a prediction result. The method of the application does not need external retrieval, stores knowledge through a memory sub-module, injects a model with dynamically generated parameters, and improves the accuracy and efficiency of the model.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a knowledge augmentation generation method, system, and readable storage medium. Background Technology

[0002] Large Language Models (LLMs) have demonstrated powerful capabilities in natural language understanding and generation tasks. However, the knowledge stored in their parameters is static and limited, often leading to factual errors (illusions) or outdated knowledge when faced with knowledge-intensive tasks. To address these issues, academia and industry have proposed various knowledge augmentation schemes, with Retrieval-Augmented Generation (RAG) and Parametric Memory Augmentation (PMA) being two main technical approaches.

[0003] Traditional RAG methods retrieve relevant document fragments from a document database during inference using an external retrieval tool and append them as context to the input sequence to enhance the knowledge coverage of LLM. However, this method has the following shortcomings:

[0004] The limited length of the context window means that concatenating multiple retrieved documents into the input will significantly increase the sequence length, leading to a sharp increase in the time complexity and memory usage of attention computation, which limits the amount of knowledge available.

[0005] Information utilization in long contexts is inefficient. LLM suffers from the "Lostin the Middle" phenomenon when processing long contexts, meaning that the model tends to focus on the beginning and end of the context while ignoring the middle content, resulting in the ineffective use of the retrieved key information.

[0006] Retrieval latency and system complexity: The RAG method relies on an online retrieval system, which introduces additional network I / O overhead and system maintenance costs, making it difficult to deploy in offline or low-latency scenarios.

[0007] To alleviate the aforementioned problems, researchers in related fields have proposed the Dynamic Parametric Retrieval-Augmented Generation (DyPRAG) method. DyPRAG utilizes a lightweight parametric translator model to convert retrieved documents online into parameters of the Low-Rank Adaptation (LoRA) module of an LLM feedforward network, thereby injecting document knowledge into the LLM in a parametric form and avoiding the length limitations imposed by context concatenation. However, DyPRAG still has the following drawbacks:

[0008] The retrieval dependency has not been eliminated. DyPRAG's reasoning process still requires the support of external search engines and document databases, and it has not been able to fundamentally get rid of its dependence on online retrieval systems. Its latency and availability are still constrained by the retrieval process.

[0009] Document-level knowledge fragmentation: DyPRAG generates LoRA parameters independently for each retrieved document and aggregates the LoRA of multiple documents through simple averaging. This "single-document independent encoding - multi-document averaging fusion" strategy undermines the ability for cross-document associative reasoning. When the answer requires integrating information from multiple documents, the averaging operation causes significant information loss.

[0010] The knowledge capacity is limited by the number of searches per search. The amount of knowledge that DyPRAG can utilize is limited by the number of documents returned by the search (usually 3 to 5). If the relevant knowledge is scattered across a large number of documents or the search engine fails to effectively recall key documents, the system performance will degrade significantly.

[0011] Parametric memory augmentation methods replace external retrieval with parameterized modules. For example, the MLP Memory method pre-trains a multi-layer perceptron (MLP) to simulate the retrieval behavior of a k-nearest neighbor (kNN) retrieval system across the entire pre-trained corpus, thus compressing retrieval-based knowledge access capabilities into a differentiable parameterized memory component. During inference, the output of the MLP Memory (a probability distribution the size of a vocabulary) is linearly interpolated with the probability distribution of the original LLM output to obtain the final prediction distribution. While MLP Memory successfully eliminates dependence on external retrieval systems, it still has the following limitations:

[0012] The knowledge fusion layer is shallow. MLP Memory adopts a fusion strategy of output layer probabilistic interpolation, that is, the output of MLP is only weighted and mixed with the output of LLM after the LLM has completed all forward propagation. This strategy means that the memorized knowledge cannot participate in the layer-by-layer reasoning process of the multi-layer Transformer blocks inside the LLM, and the depth and precision of knowledge utilization are fundamentally limited.

[0013] The high dimensionality of the memory output space means that MLP Memory needs to directly output a probability distribution vector the size of a vocabulary (with dimensions typically ranging from tens of thousands to hundreds of thousands). This requires the memory module to simultaneously handle both knowledge storage and token-level prediction within a limited parameter capacity, increasing the learning difficulty and limiting the compactness of knowledge representation.

[0014] Lacking a domain-aware routing mechanism, MLP Memory uses a single, monolithic MLP structure to store knowledge from all domains. This mixed storage of knowledge from different domains in the same parameter space may lead to knowledge interference between domains and cannot support independent updates of knowledge from specific domains. Summary of the Invention

[0015] The purpose of this invention is to provide a knowledge enhancement generation method, system, and readable storage medium to solve the problems of retrieval dependence, knowledge fragmentation, and limited knowledge capacity in existing retrieval-based enhancement methods, and shallow knowledge fusion and poor domain adaptability in parameterized memory-based enhancement methods.

[0016] To address the aforementioned technical problems, this invention provides a knowledge augmentation generation method, comprising the following steps:

[0017] The input text, including user questions and / or instructions, is encoded into a query sequence and input into a large language model; wherein the large language model includes a front-end decoding layer and a back-end decoding layer, the front-end decoding layer is used to generate a shallow semantic vector based on the query sequence, and the back-end decoding layer is used to output a prediction result for the input text based on the shallow semantic vector;

[0018] The front-end decoding layer is used to generate a shallow semantic vector, and the part corresponding to the user's question and / or instruction is extracted from the shallow semantic vector as a query feature;

[0019] Calculate the probability distribution of the query feature in several preset knowledge domains, select the two knowledge domains with the highest probability as activation domains, and normalize the corresponding probabilities as output weights; wherein, each knowledge domain is constructed through different memory sub-modules, and the memory sub-modules are used to generate domain feature vectors based on the query feature;

[0020] The query features are input into the two memory sub-modules respectively, and the domain feature vectors output by the two memory sub-modules are weighted and summed according to the output weights to obtain the memory vector;

[0021] Low-rank adaptation parameters are generated based on the memory vector, and the low-rank adaptation parameters are injected into the back-end decoding layer to enhance the knowledge generation capability of the back-end decoding layer.

[0022] The shallow semantic vector is input into the subsequent decoding layer after parameter injection to obtain the prediction result.

[0023] Optionally, calculating the probability distribution of the query features in several preset knowledge domains includes the following steps:

[0024] The vectors at each position in the query features are averaged and pooled to obtain the average feature vector.

[0025] The average feature vector is mapped to a vector with the same dimension as the number of knowledge domains through a linear transformation, and then activated by a Softmax function to output the probability distribution of the query feature in several knowledge domains.

[0026] Optionally, the memory submodule includes two layers of multilayer perceptrons for performing two-layer mapping on the query features to generate the domain feature vector; wherein the parameters of the multilayer perceptrons in different memory submodules are independent of each other.

[0027] Optionally, generating low-rank adaptation parameters based on the memory vector includes the following steps:

[0028] Attention pooling is performed on the memory vector to obtain the global memory vector;

[0029] Based on the global memory vector, corresponding low-rank adaptation parameters are generated for each layer in the back-end decoding layer.

[0030] Optionally, attention pooling is performed on the memory vector to obtain a global memory vector, including the following steps:

[0031] The attention pooling uses a trainable attention query vector to calculate the importance of each position in the memory vector to obtain a weighting coefficient, and then performs a weighted summation on the vectors corresponding to each position in the memory vector based on the weighting coefficient.

[0032] Optionally, before injecting the low-rank adaptation parameters into the back-end decoding layer, the method further includes the following steps:

[0033] A gating signal is calculated based on the global memory vector; wherein the gating signal is used to adjust the injection intensity of the low-rank adaptation parameter.

[0034] Based on the same inventive concept, the present invention also provides a knowledge enhancement generation system, comprising:

[0035] The basic large language model module, including a front-end decoding layer and a back-end decoding layer, is configured to encode input text including user questions and / or instructions into a query sequence, use the front-end decoding layer to generate shallow semantic vectors based on the query sequence, and use the back-end decoding layer to output the corresponding prediction results based on the shallow semantic vectors.

[0036] The feature extraction module is configured to extract the portion of the shallow semantic vector that corresponds to the user's question and / or instruction as query features;

[0037] The parameterized memory module includes several preset knowledge domains and corresponding memory sub-modules. It is configured to calculate the probability distribution of the query feature in each preset knowledge domain, select the two highest knowledge domains as activation domains, normalize the corresponding probabilities as output weights, map the query feature using the two corresponding memory sub-modules respectively, and perform a weighted summation of the mapping results according to the output weights to obtain a memory vector.

[0038] The parameter generation module is configured to generate low-rank adaptation parameters based on the memory vector;

[0039] The parameter injection module is configured to inject the low-rank adaptation parameters into the back-end decoding layer to enhance the knowledge representation capability of the basic large language model.

[0040] Optionally, the parameter generation module is further configured to:

[0041] Attention pooling is performed on the memory vector to obtain the global memory vector;

[0042] Based on the global memory vector, corresponding low-rank adaptation parameters are generated for each layer in the back-end decoding layer.

[0043] Optionally, the system further includes:

[0044] A gating module is configured to calculate a gating signal based on the global memory vector, the gating signal being configured to adjust the injection intensity of the low-rank adaptation parameter.

[0045] Based on the same inventive concept, the present invention also provides a readable storage medium on which a computer program is stored, which, when executed, implements the knowledge enhancement generation method as described above.

[0046] In summary, this invention provides a knowledge augmentation generation method, system, and readable storage medium, comprising the following steps: encoding input text including user questions and / or instructions into a query sequence and inputting it into a large language model; wherein the large language model includes a front-end decoding layer and a back-end decoding layer, the front-end decoding layer being used to generate shallow semantic vectors based on the query sequence, and the back-end decoding layer being used to output a prediction result for the input text based on the shallow semantic vectors; using the shallow semantic vectors generated by the front-end decoding layer, extracting the portion corresponding to the user questions and / or instructions from the shallow semantic vectors as query features; calculating the probability distribution of the query features in several preset knowledge domains, and selecting the feature with the highest probability... The two knowledge domains with higher probabilities are used as activation domains, and their corresponding probabilities are normalized as output weights. Each knowledge domain is constructed using a different memory submodule, which generates a domain feature vector based on the query features. The query features are input into the two memory submodules, and the domain feature vectors output by the two memory submodules are weighted and summed according to the output weights to obtain a memory vector. Low-rank adaptation parameters are generated based on the memory vectors and injected into the back-end decoding layer to enhance its knowledge generation capability. The shallow semantic vector is input into the back-end decoding layer after parameter injection to obtain the prediction result.

[0047] Compared with the prior art, the technical solution provided by the present invention has the following beneficial effects:

[0048] By replacing the external retrieval tool and document database with a memory submodule, the reliance on external retrieval during reasoning is eliminated, knowledge fragmentation is avoided, and knowledge capacity is increased.

[0049] By performing domain discrimination on query features and using the Top-2 strategy to select the two most relevant domains, this technology achieves on-demand knowledge activation and selective fusion compared to existing technologies, effectively avoiding cross-domain knowledge interference.

[0050] Converting memory vectors into low-rank adaptive parameters allows memorized knowledge to participate in the reasoning process of each layer within the model, thereby improving the depth of knowledge fusion. Attached Figure Description

[0051] Those skilled in the art will understand that the accompanying drawings are provided to better understand the invention and do not constitute any limitation on the scope of the invention.

[0052] Figure 1 A flowchart illustrating the knowledge enhancement method provided in an embodiment of the present invention;

[0053] Figure 2 This is a comparison chart of question-answering performance provided in an embodiment of the present invention;

[0054] Figure 3 A comparison chart of inference delay and inference speed provided for embodiments of the present invention. Detailed Implementation

[0055] To make the objectives, advantages, and features of the present invention clearer, the knowledge enhancement generation method, system, and readable storage medium provided by the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the drawings are all in a very simplified form and use non-precise proportions, and are only used to facilitate and clearly illustrate the objectives of the embodiments of the present invention. Furthermore, the structures shown in the drawings are often part of the actual structure. In particular, different proportions may be used in different drawings to show different focuses. It should also be understood that, unless specifically stated or indicated, the terms "first," "second," "third," etc., in the specification are only used to distinguish the various components, elements, steps, etc., in the specification, and are not used to indicate the logical or sequential relationships between the various components, elements, steps, etc.

[0056] refer to Figure 1 As shown, the knowledge enhancement generation method provided in this embodiment of the invention includes the following steps:

[0057] S1, the input text including user questions and / or instructions is encoded into a query sequence and input into a large language model; wherein, the large language model includes a front-end decoding layer and a back-end decoding layer, the front-end decoding layer is used to generate a shallow semantic vector based on the query sequence, and the back-end decoding layer is used to output a prediction result for the input text based on the shallow semantic vector;

[0058] S2, using the front-end decoding layer to generate a shallow semantic vector, and extracting the part corresponding to the user's question and / or instruction from the shallow semantic vector as a query feature;

[0059] S3, calculate the probability distribution of the query feature in several preset knowledge domains, select the two knowledge domains with the highest probability as activation domains, and normalize the corresponding probabilities as output weights; wherein, each knowledge domain is constructed through different memory sub-modules, and the memory sub-modules are used to generate domain feature vectors based on the query feature;

[0060] S4, input the query features into the two memory sub-modules respectively, and sum the domain feature vectors output by the two memory sub-modules according to the output weights to obtain the memory vector;

[0061] S5, generate low-rank adaptation parameters based on the memory vector, and inject the low-rank adaptation parameters into the back-end decoding layer to enhance the knowledge generation capability of the back-end decoding layer;

[0062] S6, the shallow semantic vector is input into the post-decoding layer after parameter injection to obtain the prediction result.

[0063] The knowledge enhancement generation method provided by this invention eliminates the dependence on external retrieval by replacing the external retrieval tool and document database with a memory submodule. Furthermore, the method provides that by converting the memory vector into low-rank adaptive parameters and injecting them into the back-end decoding layer, knowledge memory is deeply coupled with the model and participates in the reasoning process, which greatly improves the knowledge fusion degree. At the same time, by performing domain discrimination on query features and adopting the Top-2 strategy, knowledge is activated on demand, avoiding cross-domain knowledge interference.

[0064] The knowledge enhancement generation method provided by this invention will be further described below.

[0065] First, step S1 is executed to provide a large language model, which includes N decoding layers. Layers 1 to L are divided into front-end decoding layers, and layers L+1 to N are divided into back-end decoding layers. Each decoding layer is constructed from different Transformer decoding blocks.

[0066] The input text, including user questions and / or instructions, is encoded into a query sequence, for example, the query sequence being of the form: x = (x1, x2, …, x…). T ), where x represents the query sequence, and T represents the length of the query sequence, i.e., the number of tokens obtained after encoding the input text, each x T This represents the encoded vector of a token in the sequence.

[0067] Next, step S2 is executed, where the query sequence is input into the front-end decoding layer to obtain the shallow semantic vector h. L full ∈ Q T×d , where h L full d represents the shallow semantic vector, i.e., the hidden state of all tokens in the query sequence; T represents the length of the query sequence (i.e., the number of tokens); d represents the hidden vector dimension of each token (i.e., the hidden dimension of the large language model); Q represents the hidden state of all tokens in the query sequence. T×d This represents a T-row, d-column matrix, where each row corresponds to a vector representation of a token. Then, the portion corresponding to the user's question and / or instruction is extracted from the shallow semantic vector and used as the query feature h. L prompt ∈ Q T p ×d = h Lfull [1:T p ], h L prompt Represents query characteristics, h L prompt ∈ Q T p ×d This indicates that the query feature is a T. p A matrix with d rows and d columns, h L full [1:T p ] represents the shallow semantic vector h L full Extract the first 1 to T p Line, where the first T p Each token corresponds to the part of the user's question and / or instruction (i.e., the prompt area).

[0068] Then, step S3 is executed to calculate the query feature h. L prompt Preferably, the query feature h is calculated based on the probability distribution across several predefined knowledge domains. L prompt The steps for determining the probability distribution include: providing a domain router to distribute the query features h L prompt The input is sent to the domain router. Since the domain router only needs coarse-grained neighborhood discrimination, to reduce computational overhead, as a preferred embodiment of the present invention, the domain router is used to process the query feature h. L prompt Mean pooling is performed on the vectors of each position (token) in the query feature to obtain an average feature vector. The average feature vector is then mapped to a vector with the same dimension as the number of knowledge domains through a linear transformation, and activated by the Softmax function to output the probability distribution of the query feature in several knowledge domains.

[0069] In this embodiment of the invention, K knowledge domains are preset. As a preferred solution, this embodiment of the invention adopts a Top-2 routing strategy, that is, selecting the two knowledge domains with the highest probabilities as activation domains, and normalizing the corresponding probabilities as output weights. Each knowledge domain is constructed through different memory submodules. The memory submodule includes two layers of multilayer perceptron (MLP). The two MLP layers are used to process the query feature h in a position-independent and parameter-shared manner. L prompt Perform a two-layer MLP mapping, that is, each layer of MLP maps the query feature h. L promptThe positions of each token are mapped using the same set of parameters to generate a domain feature vector; wherein the parameters of the multilayer perceptron in different memory submodules are independent of each other.

[0070] After selecting the domain, proceed to step S4, which involves setting the query feature h. L prompt The inputs are fed into the two memory submodules respectively. The neighborhood feature vectors output by the two memory submodules are then weighted and summed according to the output weights to obtain the memory vector m. L ∈ Q T p ×d The memory vector m L For a T p A matrix with d rows and d columns.

[0071] Then, step S5 is executed to process the memory vector m. L ∈ Q T p ×d Attention pooling is performed to compress the data into a d-dimensional global memory vector m. 1 L Specifically, in this embodiment of the invention, a learnable d-dimensional attention query vector q is introduced, and the attention weights at each position t in the memory vector are calculated using the following formula: w t = Softmax t (q) T · m L [t] / √d), w t This represents the attention weight at position t in the memory vector. It should be noted that this pooling range only covers the cue region (i.e., t ∈ {1, …, T}). p}); where q T Let m be the transpose of the attention query vector q. L [t] represents the memory vector m L The vector at position t, with dimension d, corresponds to the query feature h. L prompt The representation of the t-th token obtained after being mapped by the memory submodule MLP, q T · m L [t] represents the attention query vector q and the vector m at the t-th position in the memory vector. L The similarity of [t] (i.e., calculating the importance of the t-th position in the memory vector). Further, through the attention weight w t The global memory vector m is obtained by weighted summation. 1 L = Σt=1 Tp w t · m L [t]. Then, based on the global memory vector m... 1 L The corresponding low-rank adaptation (LoRA) parameters are generated for each layer in the back-end decoding layer.

[0072] To transfer the global memory vector m 1 L To convert the parameters into LoRA parameters, this embodiment of the invention provides a parameter translator, which includes a layer index embedding module, a shared bottleneck module, and an independent parameter generation module, used to convert the global memory vector m into LoRA parameters. 1 L For each target layer l in the subsequent decoding layer, a set of low-rank adaptation matrix pairs (composed of low-rank adaptation matrix A and low-rank adaptation matrix B) is generated for the three projection targets proj (including gate, up, down) of the feedforward network (FFN) of each Transformer decoding block. Here, l ∈ {L+1, …, N}. The specific implementation is as follows:

[0073] The global memory vector m 1 L The input is given to the parameter translator, and a learnable p-dimensional layer index embedding e is introduced into each target layer l in the subsequent decoding layer through the layer index embedding module. l And embed the layer index into e l With the global memory vector m 1 L Concatenating the elements yields a d+p dimensional index memory vector c. l = [m 1 L e l ].

[0074] The index memory vector c l The input is given to the shared bottleneck module, and the index memory vector c is weighted by a preset weight matrix. l Perform a linear transformation (reduce the dimension from d+p to d) b Then, the result of the linear transformation is activated by the ReLU function to obtain a compact vector h. tr (l) It should be noted that all target layers l use the same parameters when performing linear transformations in the shared bottleneck module.

[0075] Then the compact vector h tr (l)The input is sent to the parameter generation module, which includes three parameter generation heads, each corresponding to a projection target, for generating low-rank matrix pairs (i.e., LoRA parameters) adapted to the projection target. Each parameter generation head transforms the compact vector h using two independent linear transformations (to form low-rank adaptation matrix A and low-rank adaptation matrix B, respectively). tr (l) The process involves mapping the vector to an intermediate vector and then reshaping it using the Reshape function to obtain a low-rank matrix pair. This can be represented as: h tr (l) = ReLU(W tr down · [m 1 L e l ] + b tr down A proj (l) = Reshape(W tr A,proj · h tr (l) + b tr A,proj ), B proj (l) = Reshape(W tr B,proj · h tr (l) + b tr B,proj ), proj ∈{gate, up, down}, l ∈ {L+1, …, N}, A proj (l) ∈ Q r×d B proj (l) ∈ Q d×r Among them, W tr down and b tr down The weight matrix and bias parameters, W, are respectively used to perform linear transformation on the shared bottleneck module. tr A,proj b tr A,proj W tr B,proj and b tr B,proj These represent the weight matrix and bias parameters used by each parameter generation head to form low-rank adaptation matrices A and B, respectively, where r is the low-rank dimension. Through the above steps, the parameter translator utilizes the global memory vector m... 1L For each target layer l and each projected target proj in the subsequent decoding layer, a set of low-rank adaptation matrix pairs (B) are generated. proj (l) A proj (l) It should be noted that the dimensions of the three low-rank adaptation matrix pairs generated in each layer of the subsequent decoding layer are consistent with the dimensions of the original projection matrices of the three projection targets (gate, up, down) in the corresponding FFN of the layer. Furthermore, the value of the low-rank dimension affects the amount of parameter injection and expressive power; a value of 8 has been experimentally verified as a generally effective configuration that balances the amount of parameters and expressive power, and this invention does not impose any limitations on this.

[0076] To further prevent the model from generating hallucinations, as a preferred embodiment, after obtaining the global memory vector, the method provided in this embodiment of the invention further includes the following steps: calculating a gating signal based on the global memory vector; wherein, the gating signal is used to adjust the injection intensity of the low-rank adaptation parameter. Specifically, based on the global memory vector m obtained in step S5... 1 L The gating signal is calculated using the following formula: g = σ(w g T · m 1 L + b g ), g ∈ (0, 1), w g T For the training obtained with m 1 L w of weight vectors with the same dimension g transpose, b g The bias parameters are obtained through training, and σ(·) is the Sigmoid function used to control the output within (0, 1). The gating signal g reflects the global memory vector m. 1 L Regarding the contribution of the large language model to inference, when the memory submodule cannot provide useful knowledge for the current input text, the gating signal g will approach 0 to reduce the injection intensity of the low-rank adaptation parameter, which can effectively prevent invalid knowledge from affecting model inference.

[0077] Continuing with step S5, after obtaining the low-rank adaptation matrix pair, it is injected into the subsequent decoding layer (specifically, the feedforward network of the Transformer decoding block in each layer), and the enhanced projection matrix is ​​obtained using the following formula: W 1 proj (l) = W proj (l)+ g · B proj (l) A proj (l) ,proj ∈ {gate, up, down}, where W 1 proj (l) W represents the enhanced projection matrix of the l-th layer in the subsequent decoding layer. proj (l) Let g represent the original projection matrix, g represent the gate signal, and B represent the gate signal. proj (l) and A proj (l) This represents the low-rank adaptation matrix pair generated for the l-th layer in step S5. During subsequent inference, the original projection matrix is ​​replaced with the enhanced projection matrix, while other calculations in the feedforward network remain unchanged.

[0078] Finally, step S6 is executed to process the shallow semantic vector h. L full The input is the back-end decoding layer after parameter injection, and the output is the final prediction result of the input text.

[0079] It should be noted that the parameters and weights involved in the above method can be obtained through end-to-end training using gradient descent. The specific training strategy can be set according to the actual task requirements, which will not be elaborated here.

[0080] Furthermore, using the knowledge augmentation generation method provided in this embodiment, experiments were conducted using LLaMA-3-8B-Instruct as the base model. This model includes 32 Transformer decoding blocks, with a hidden dimension (d) of 4096, a feedforward network intermediate dimension (dff) of 14336, and a Swiglu activation function (gated FFN). Layers 1 to 22 were configured as front-end decoding layers, and layers 23 to 32 were configured as back-end decoding layers. The layer index embedding dimension p = 32, the preset knowledge domain number K = 8 (total parameters approximately 1B), the shared bottleneck module dimension db = 384, and the low-rank dimension r = 8.

[0081] refer to Figure 2As shown, this embodiment of the invention uses the following four question-answering benchmarks as test sets: 2WikiMultihopQA (2WQA, multi-hop reasoning), HotpotQA (HQA, multi-hop reasoning), PopQA (PQA, fact-based question answering), and ComplexWebQuestions (CWQ, complex multi-step question answering), with F1 score as the evaluation metric. The method provided by this embodiment is compared with the following prior art methods: LLaMA-3-8B (closed-book), Standard RAG (BM25, Top-3), kNN-LM, DyPRAG, and MLP Memory (1B). Reference Figure 2 The comparison results show that, without relying on external retrieval, the technical solution provided by this invention achieves an average F1 score of 32.90%, which is 3.92 percentage points higher than MLP Memory (a relative improvement of +13.5%), 2.44 percentage points higher than DyPRAG, and 5.31 percentage points higher than Standard RAG. On 2WQA, this invention achieved 40.15%. Because it injects LoRA parameters, the memorized knowledge can participate in chained reasoning layer by layer in the decoding layer of the large language model, improving the accuracy of multi-hop reasoning. On HotpotQA, the improvement effect of deep fusion of memorized knowledge on multi-hop reasoning was further verified. On PopQA, this invention achieved 15.87%, close to Standard RAG's 16.13% and significantly better than DyPRAG's 13.60%, indicating that the memorized submodule in this invention can be effectively stored and activated. On ComplexWebQuestions, this invention achieved the highest 44.31%, significantly better than other baselines. Standard RAG's performance dropped to 35.45% on this dataset (retrieval noise interfered with reasoning), lower than LLaMA-3-8B which directly uses the base model. This invention uses a gating mechanism to avoid the above problems. Furthermore, while MLP Memory, as a shallow fusion baseline (performing probabilistic interpolation at the output layer), improves performance compared to directly using the pedestal model, it is significantly inferior to the solution provided in this invention on all datasets due to its limited fusion depth, thus verifying the essential advantages of deep parameter injection over shallow probabilistic interpolation.

[0082] refer to Figure 3As shown, this embodiment of the invention evaluated the inference latency of various methods on a single NVIDIA A100 80GB GPU. The evaluation metrics included first-token latency (TTFT, in ms) and token generation speed (TPS, in tokens / s). Analysis of the latency comparison results shows that the solution provided by this invention has a TTFT latency of 48ms, which is approximately 2.8 times faster than Standard RAG and only 14% more latency than the pedestal model. This latency stems from the computational overhead of knowledge domain discrimination and layer-by-layer LoRA parameter generation. Since this invention does not involve online external retrieval, its inference speed and latency are significantly faster than all methods that rely on external retrieval.

[0083] Based on the same inventive concept, embodiments of the present invention also provide a knowledge enhancement generation system, including:

[0084] The basic large language model module, including a front-end decoding layer and a back-end decoding layer, is configured to encode input text including user questions and / or instructions into a query sequence, use the front-end decoding layer to generate shallow semantic vectors based on the query sequence, and use the back-end decoding layer to output the corresponding prediction results based on the shallow semantic vectors.

[0085] The feature extraction module is configured to extract the portion of the shallow semantic vector that corresponds to the user's question and / or instruction as query features;

[0086] The parameterized memory module includes several preset knowledge domains and corresponding memory sub-modules. It is configured to calculate the probability distribution of the query feature in each preset knowledge domain, select the two highest knowledge domains as activation domains, normalize the corresponding probabilities as output weights, map the query feature using the two corresponding memory sub-modules respectively, and perform a weighted summation of the mapping results according to the output weights to obtain a memory vector.

[0087] The parameter generation module is configured to generate low-rank adaptation parameters based on the memory vector;

[0088] The parameter injection module is configured to inject the low-rank adaptation parameters into the back-end decoding layer to enhance the knowledge representation capability of the basic large language model.

[0089] Specifically, the parameter generation module is configured to: perform attention pooling on the memory vector to obtain a global memory vector; and generate corresponding low-rank adaptation parameters for each layer in the back-end decoding layer based on the global memory vector.

[0090] Preferably, the knowledge enhancement generation system further includes a gating module, which is configured to calculate a gating signal based on the global memory vector, and the gating signal is configured to adjust the injection intensity of the low-rank adaptation parameter.

[0091] It should be noted that each module in the knowledge enhancement generation system provided in this embodiment is used to implement each step of the knowledge enhancement generation method provided in this embodiment. Therefore, the specific description of the functions that each module can achieve can be found in the relevant descriptions of the corresponding steps of the knowledge enhancement generation method described above, and repeated descriptions will not be repeated here. In addition, the knowledge enhancement generation system can achieve the same technical effects as the knowledge enhancement generation method described above, and will not be repeated here either.

[0092] It is understood that the basic large language model module, feature extraction module, parameterization memory module, parameter generation module, parameter injection module, and gating module can be implemented in a single device, or any one of these modules can be split into multiple sub-modules, or at least some of the functions of one or more of the above modules can be combined with at least some of the functions of other modules and implemented in a single functional module. According to embodiments of the present invention, at least one of the basic large language model module, feature extraction module, parameterization memory module, parameter generation module, parameter injection module, and gating module can be at least partially implemented as a hardware circuit, such as a Field Programmable Gate Array (FPGA), Programmable Logic Array (PLA), System-on-a-Chip, System-on-a-Substrate, System-on-Package, Application-Specific Integrated Circuit (ASIC), or can be implemented in hardware or firmware in any other reasonable manner by integrating or packaging the circuit, or implemented in a suitable combination of software, hardware, and firmware implementations. Alternatively, at least one of the basic large language model module, feature extraction module, parameterization memory module, parameter generation module, parameter injection module, and gating module can be at least partially implemented as a computer program module, which, when run by a computer, can execute the functions of the corresponding module.

[0093] Based on the same inventive concept, embodiments of the present invention also provide a readable storage medium storing a computer program thereon, which, when executed, implements the knowledge enhancement generation method provided in embodiments of the present invention.

[0094] The readable storage medium can be a tangible device capable of holding and storing instructions used by an instruction execution device, such as, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, and any suitable combination thereof. The computer programs described herein can be downloaded from the readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. Furthermore, the computer program used to perform the operations of this invention can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages. The computer program may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information of the computer program. This electronic circuitry can execute computer-readable program instructions to implement various aspects of the invention.

[0095] In summary, embodiments of the present invention provide a knowledge augmentation generation method, system, and readable storage medium. The method includes: encoding input text containing user questions and / or instructions into a query sequence and inputting it into a large language model, the model including a front-end decoding layer and a back-end decoding layer; generating shallow semantic vectors using the front-end decoding layer and extracting the parts corresponding to user questions and / or instructions as query features; calculating the probability distribution of query features in a preset knowledge domain, selecting the two domains with the highest probabilities as activation domains, and normalizing their probabilities as output weights; inputting the query features into memory submodules corresponding to the two activation domains respectively to obtain domain feature vectors and weighting and summing them according to the output weights to obtain memory vectors; generating low-rank adaptation parameters based on the memory vectors and injecting them into the back-end decoding layer; inputting the shallow semantic vectors into the back-end decoding layer after the parameters are injected to obtain prediction results. The technical solution provided by the embodiments of the present invention can eliminate the dependence on external retrieval, improve inference speed and reduce inference latency. By storing knowledge through the memory submodule and injecting LoRA parameters layer by layer, the knowledge fusion degree is improved. In addition, the domain discrimination mechanism and gating mechanism effectively avoid interference of invalid knowledge on inference, thus effectively improving the accuracy of the model.

[0096] It should be noted that the above description is only a description of the preferred embodiment of the present invention and is not intended to limit the scope of the present invention in any way. Any changes or modifications made by those skilled in the art based on the above disclosure shall fall within the protection scope of the present invention.

Claims

1. A knowledge augmentation generation method, characterized in that, Includes the following steps: The input text, including user questions and / or instructions, is encoded into a query sequence and input into a large language model; wherein the large language model includes a front-end decoding layer and a back-end decoding layer, the front-end decoding layer is used to generate a shallow semantic vector based on the query sequence, and the back-end decoding layer is used to output a prediction result for the input text based on the shallow semantic vector; The front-end decoding layer is used to generate a shallow semantic vector, and the part corresponding to the user's question and / or instruction is extracted from the shallow semantic vector as a query feature; Calculate the probability distribution of the query feature in several preset knowledge domains, select the two knowledge domains with the highest probability as activation domains, and normalize the corresponding probabilities as output weights; wherein, each knowledge domain is constructed through different memory sub-modules, and the memory sub-modules are used to generate domain feature vectors based on the query feature; The query features are input into the two memory sub-modules respectively, and the domain feature vectors output by the two memory sub-modules are weighted and summed according to the output weights to obtain the memory vector; Low-rank adaptation parameters are generated based on the memory vector, and the low-rank adaptation parameters are injected into the back-end decoding layer to enhance the knowledge generation capability of the back-end decoding layer. The shallow semantic vector is input into the subsequent decoding layer after parameter injection to obtain the prediction result.

2. The knowledge enhancement generation method according to claim 1, characterized in that, Calculating the probability distribution of the query features across several preset knowledge domains includes the following steps: The vectors at each position in the query features are averaged and pooled to obtain the average feature vector. The average feature vector is mapped to a vector with the same dimension as the number of knowledge domains through a linear transformation, and then activated by a Softmax function to output the probability distribution of the query feature in several knowledge domains.

3. The knowledge enhancement generation method according to claim 1, characterized in that, The memory submodule includes two layers of multilayer perceptrons for performing two-layer mapping on the query features to generate the domain feature vector; wherein the parameters of the multilayer perceptrons in different memory submodules are independent of each other.

4. The knowledge enhancement generation method according to claim 1, characterized in that, Generating low-rank adaptation parameters based on the memory vector includes the following steps: Attention pooling is performed on the memory vector to obtain the global memory vector; Based on the global memory vector, corresponding low-rank adaptation parameters are generated for each layer in the back-end decoding layer.

5. The knowledge enhancement generation method according to claim 4, characterized in that, The global memory vector is obtained by performing attention pooling on the memory vector, including the following steps: The attention pooling uses a trainable attention query vector to calculate the importance of each position in the memory vector to obtain a weighting coefficient, and then performs a weighted summation on the vectors corresponding to each position in the memory vector based on the weighting coefficient.

6. The knowledge enhancement generation method according to claim 4, characterized in that, Before injecting the low-rank adaptation parameters into the back-end decoding layer, the method further includes the following steps: A gating signal is calculated based on the global memory vector; wherein the gating signal is used to adjust the injection intensity of the low-rank adaptation parameter.

7. A knowledge enhancement and generation system, characterized in that, include: The basic large language model module, including a front-end decoding layer and a back-end decoding layer, is configured to encode input text including user questions and / or instructions into a query sequence, use the front-end decoding layer to generate shallow semantic vectors based on the query sequence, and use the back-end decoding layer to output the corresponding prediction results based on the shallow semantic vectors. The feature extraction module is configured to extract the portion of the shallow semantic vector that corresponds to the user's question and / or instruction as query features; The parameterized memory module includes several preset knowledge domains and corresponding memory sub-modules. It is configured to calculate the probability distribution of the query feature in each preset knowledge domain, select the two highest knowledge domains as activation domains, normalize the corresponding probabilities as output weights, map the query feature using the two corresponding memory sub-modules respectively, and perform a weighted summation of the mapping results according to the output weights to obtain a memory vector. The parameter generation module is configured to generate low-rank adaptation parameters based on the memory vector; The parameter injection module is configured to inject the low-rank adaptive parameters into the back-end decoding layer to enhance the knowledge representation capability of the basic large language model; wherein, the language model module is further configured to use the back-end decoding layer after the low-rank adaptive parameters are injected to output the prediction result based on the shallow semantic vector.

8. The knowledge enhancement generation system according to claim 7, characterized in that, The parameter generation module is also configured to: Attention pooling is performed on the memory vector to obtain the global memory vector; Based on the global memory vector, corresponding low-rank adaptation parameters are generated for each layer in the back-end decoding layer.

9. The knowledge enhancement generation system according to claim 8, characterized in that, The system also includes: A gating module is configured to calculate a gating signal based on the global memory vector, the gating signal being configured to adjust the injection intensity of the low-rank adaptation parameter.

10. A readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed, it implements the knowledge enhancement generation method as described in any one of claims 1 to 6.

Images