Method for injecting external knowledge base into language model with loop architecture

Abstract

The invention discloses a method for injecting an external knowledge base into a language model with a loop architecture, and the method comprises the steps: firstly carrying out the intelligent segmentation and three-wheel semantic processing of the external knowledge base, extracting and screening key proper nouns, and generating a structured triple; converting the triple into a key / value basic vector by using a pre-training sentence encoder, and mapping the key / value basic vector to an embedding space of an RWKV model through a parameter efficient adapter to form a read-only knowledge key value tensor; in the reasoning stage, a cyclic rectangular attention mechanism is innovatively provided, the mechanism serves as a bridge between a knowledge token of KBLAM and a cyclic state of an RWKV model, and an external knowledge base can be injected into a language model with a cyclic architecture; in the training stage, the illusion is effectively inhibited by adopting a synthetic knowledge base and an instruction fine tuning strategy containing a rejection sample. According to the method, the defects of the large-scale language model of the loop architecture in the aspects of knowledge timeliness, accuracy and vertical field adaptation can be overcome.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and natural language processing technology. Specifically, it relates to knowledge augmentation technology for large language models (LLMs), which is a method for injecting external knowledge bases into language models with a circular architecture (such as RWKV). Background Technology

[0002] In recent years, large language models (LLMs) have made significant progress in the field of natural language processing. However, existing LLMs still have limitations, such as insufficient knowledge timeliness (unable to know new knowledge after the training deadline), susceptibility to "illusions" (generating content that does not conform to the facts), and difficulties in knowledge adaptation in vertical professional fields.

[0003] To address the aforementioned limitations, external knowledge base (KB) enhancement techniques have become an important research direction. By combining external KBs with LLM (Local Level Management), dynamic knowledge updates can be achieved, the accuracy of responses can be improved, and the cost of domain adaptation can be reduced.

[0004] Currently, mainstream knowledge augmentation techniques include Retrieval Augmentation (RAG), Contextual Learning (ICL), and Knowledge Injection. Among them, KBLAM (Knowledge Base Augmented Language Model) is an advanced knowledge injection technique that achieves efficient and interpretable knowledge augmentation by converting KB triples into "knowledge tokens" and injecting them into the attention layer of LLMs using a rectangular attention mechanism.

[0005] Meanwhile, to address the high computational complexity of traditional Transformer models, novel language model architectures such as RWKV (Receptance Weighted Key Value) have been proposed. RWKV employs a linear approximate self-attention mechanism and a recurrent architecture, optimizing computational complexity from quadratic growth to linear growth, significantly reducing reliance on computational resources.

[0006] However, existing technologies generally lack effective external knowledge base enhancement methods for efficient recurrent architecture models like RWKV. Knowledge injection methods like KBLAM are primarily designed for Transformer architectures and cannot be directly applied to the recurrent architecture of RWKV, making it difficult for RWKV models to obtain external knowledge support for specialized domain tasks. Summary of the Invention

[0007] To address, or at least partially address, the technical problems existing in the background art, the present invention provides a method for injecting an external knowledge base into a language model with a cyclic architecture. This method designs a "Cyclic Rectangular Attention" mechanism, which acts as a bridge between the knowledge tokens of KBLAM and the cyclic state (WKV State) of the RWKV model. This mechanism can inject an external knowledge base into a language model with a cyclic architecture, helping to solve problems such as insufficient knowledge timeliness and inadequate domain-specific capabilities in traditional large models.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows: In a first aspect, the present invention provides a method for injecting an external knowledge base into a language model with a circular architecture, the method comprising the following steps: S1. Preprocess and encode the external knowledge base to generate a knowledge token, wherein the knowledge token includes a knowledge key tensor and a knowledge value tensor; In the inference process of the S2. recurrent architecture language model, at each decoding step of the temporal loop, for each layer of the model, the standard recurrent state evolution path and the recurrent rectangular attention path are executed in parallel, where: The standard cyclic state evolution path outputs a model based on the representation of the internal cyclic memory. The circular rectangular attention path uses the model's receptance vector as the query vector, calculates the attention weight with the knowledge key tokens of the knowledge key tensor, and then sums the knowledge value tokens of the knowledge value tensor to obtain the knowledge context vector. The outputs of the standard cyclic state evolution path and the cyclic rectangular attention path are then fused to achieve knowledge injection.

[0009] Furthermore, in S1, structured triples are generated from the external knowledge base through intelligent segmentation, data extraction and filtering. The triples are converted into key / value basis vectors using a sentence encoder and mapped to the model embedding space through a parameter-efficient adapter to generate read-only knowledge key tensors and value tensors.

[0010] Furthermore, the data extraction and filtering process includes three rounds of processing: the first round uses a large language model to tag proper nouns; the second round extracts the tagged words to form a dictionary; and the third round compares the semantics of RAG with a general large model to eliminate words with the same semantics.

[0011] Furthermore, the parameter-efficient adapter is constructed using low-rank adaptive LoRA, which decomposes the projection matrix into the product of two low-rank matrices.

[0012] Furthermore, in S2, the cyclic architecture language model is the RWKV model, in which the standard cyclic state evolution path and the cyclic rectangular attention path are executed in parallel within its temporal mixing block.

[0013] Furthermore, in S2, the specific calculation process of the knowledge context vector in the cyclic rectangular attention path includes: (a) Knowledge Query Vector Generation: It is used for querying Knowledge Key Token A D-dimensional vector, reusing the Receptance vector As a query vector, that is: ; (b) Knowledge Attention Calculation: calculate With the knowledge key tensor stored in VRAM at the 1st Layer Knowledge Key Token By applying the knowledge base scaling factor, we obtain Unnormalized attention score

[0014]

[0015] in, For hyperparameters; (c) Knowledge Context Vector Generation: Use the Softmax function to fractions Normalized to attention weights

[0016]

[0017] Use weights For the knowledge-valued tensor in VRAM at the th Layer Knowledge Value Token By performing a weighted summation, we obtain a D-dimensional knowledge context vector that incorporates information from M knowledge triples. :

[0018] Knowledge Context Vector Representative model in Content retrieved from external knowledge bases at all times.

[0019] Furthermore, in S2, the outputs of the standard cyclic state evolution path and the cyclic rectangular attention path are fused through trainable scalar gating to achieve non-intrusive knowledge injection.

[0020] Secondly, the present invention also provides a knowledge enhancement system, which includes a knowledge encoding module and an inference engine. By applying the above-described method, an external knowledge base is injected into a language model with a cyclic architecture to achieve knowledge enhancement.

[0021] Compared with the prior art, the present invention has at least the following beneficial technical effects: 1. The method of this invention can inject external knowledge bases into language models with a cyclical architecture, which helps to solve problems such as insufficient timeliness of knowledge and lack of professional domain capabilities in traditional large models.

[0022] 2. This method first performs intelligent segmentation and three-round semantic processing on an external knowledge base to extract and filter key proper nouns, generating structured triples. Then, a pre-trained sentence encoder is used to convert the triples into key / value vectors, which are then mapped to the embedding space of the RWKV model through a parameter-efficient adapter, forming read-only knowledge key-value tensors. In the inference stage, an innovative "cyclic rectangle attention" mechanism is proposed, which acts as a bridge between the knowledge tokens of KBLAM and the cyclic states of the RWKV model, allowing the injection of external knowledge bases into language models with cyclic architectures. During the training stage, a synthetic knowledge base and instruction fine-tuning strategy containing rejection samples are employed to effectively suppress illusions. This invention addresses the shortcomings of large language models with cyclic architectures in terms of knowledge timeliness, accuracy, and vertical domain adaptability.

[0023] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.

[0024] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0025] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0026] Figure 1 This is a schematic diagram illustrating the method for injecting an external knowledge base into a language model with a cyclic architecture, as provided by the present invention.

[0027] Figure 2 This is a schematic diagram illustrating the data extraction and filtering process provided by the present invention.

[0028] Figure 3 A schematic diagram of the cyclic rectangular attention mechanism provided by the present invention. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0030] In the description of this invention, it should be noted that some processes described in this application specification and accompanying drawings include multiple operations that appear in a specific order. However, it should be clearly understood that these operations may be performed in any order or in parallel. Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention.

[0031] See Figure 1 As shown, this invention proposes a method for injecting external knowledge bases into a language model with a cyclical architecture, addressing the problems of insufficient timeliness and inadequate domain-specific capabilities in traditional large-scale models. The main steps of this method include: S1. Preprocess and encode the external knowledge base to generate knowledge tokens, which include knowledge key tensors and knowledge value tensors; In the inference process of the S2. recurrent architecture language model, at each decoding step of the temporal loop, for each layer of the model, the standard recurrent state evolution path and the recurrent rectangular attention path are executed in parallel, where: The standard cyclic state evolution path outputs a model based on the representation of the internal cyclic memory. The cyclic rectangular attention path uses the model's receptance vector as the query vector, calculates the attention weights with the knowledge key tokens of the knowledge key tensor, and then sums the weighted sums of the knowledge value tokens of the knowledge value tensor to obtain the knowledge context vector. The outputs of the standard cyclic state evolution path and the cyclic rectangular attention path are then merged to achieve knowledge injection.

[0032] The specific implementation methods and working principles of the present invention are described in detail below: This invention provides a knowledge augmentation method that integrates KBLAM and RWKV architectures. The method designs a "Cyclic Rectangular Attention" mechanism, which acts as a bridge between the knowledge tokens of KBLAM and the cyclic state (WKV State) of the RWKV model.

[0033] The method specifically includes the following detailed steps: Step 1: Knowledge base preprocessing and temporal knowledge token encoding: Before model inference or training, the external knowledge base (KB) is preprocessed and encoded once. The purpose of this step is to transform heterogeneous, string-based knowledge into a D-dimensional continuous vector representation that the RWKV model can understand.

[0034] 1. Structured data preparation and generation: The goal of this step is to obtain a dictionary.

[0035] If the input source is unstructured text (such as PDF reports, web pages, emails, or long documents), this invention will employ a data extraction technique based on a large language model.

[0036] (a) Intelligent Segmentation: For text exceeding the LLM context window, this invention employs a semantic-based intelligent segmentation strategy instead of fixed-length segmentation. For example, paragraph or chapter titles, or specific semantic tags can be used as boundaries. (b) Data extraction and filtering: First round (marking): For the long text blocks obtained in (a), use a large model to mark the proper nouns in the text with special symbols to avoid illusions.

[0037] Second round (extraction): Extract the words marked in the first round to form a dictionary.

[0038] The third round (screening): Use the rag search library to assign explanation 1 to the nouns, then use a general large model to assign explanation 2, and compare the two explanations using the large model, removing words whose explanations have the same semantic meaning. See details for further information. Figure 2 As shown.

[0039] 2. Structured data generation: The large model used is generated from the dictionary. A set of knowledge triples .

[0040] 3. Basic Vector Encoding: The goal of this step is to Convert the string triples to right A fundamental vector of dimension.

[0041] 3.1. Encoder Choice: Initialize a pre-trained, black-box sentence encoder model. Its function is to map text strings to... A continuous semantic vector space.

[0042] In a preferred embodiment, the OpenAI ada-002 model can be used ( It has a powerful semantic capture capability.

[0043] In another embodiment, to ensure that the entire system is open source and controllable, all-mpnet-base-v2 can be selected. ) or bge-large-en-v1.5 ( Sentence-BERT model, etc. The choice is a trade-off between the richness of semantic representation and the computational / storage overhead of subsequent steps.

[0044] 3.2. Key / Value Vector Generation: Iterate through each (m-th) triplet in the knowledge base KB, using the encoder. Generate a pair of fundamental vectors: Base Key Vector This vector is used for "querying" in subsequent steps. Its design goal is to simulate the "questions" a user might ask. Therefore, a templated string is used. This asymmetric structure (unlike) KBLAM has been shown to be more effective because it is closer to the form of natural language queries (e.g., "What is the description of Posh Poodle?").

[0045] Base Value Vector This vector represents the "answer" or "content" of the triple. It is directly related to the triple. <value>Encoding strings: .

[0046] 3.3 Adapting to RWKV's knowledge token mapping: The goal of this step is to dimensional fundamental vectors ( Projected onto the interlayer of the RWKV model Internal embedded space.

[0047] Introduction of Adapters: Two sets of trainable linear adapter parameters are introduced as projection matrices between the two heterogeneous spaces: Knowledge Key Adapter:

[0048] Knowledge Value Adapter:

[0049] The total number of parameters for these adapters is .

[0050] 3.4 Parameter-efficient adapter embodiment (preferred): To address the parameter explosion problem, this invention preferably employs Parameter-Efficient Fine-Tuning (PEFT) technology to construct the adapter.

[0051] In one embodiment, low-rank adaptation (LoRA) is employed: Will Decomposed into the product of two low-rank matrices: a reduced-dimensional matrix and an upgraded matrix , where rank much smaller and (For example ).

[0052] The same decomposition is also performed.

[0053] At this point, the total number of trainable parameters decreases to The number of parameters has been reduced by several orders of magnitude.

[0054] 3.5 Final Generation of Knowledge Tokens: For all M triples in KB, their final representation in the RWKV model is... Knowledge tokens on the layer:

[0055]

[0056] Output: The final product of this step is two huge tensors that can be loaded into GPU video memory (VRAM): Knowledge Key Tensor:

[0057] Knowledge value tensor:

[0058] These tensors are treated as a read-only memory library in step two.

[0059] Step 2: Knowledge Injection Based on Cyclic Rectangle Attention (Online / Online Stage): This step is executed when the user inputs a prompt during model runtime. The core of this step is to inject the M sets of knowledge tokens generated in step one into the L time-mixing blocks of the RWKV model in real time through the "cyclic rectangular attention" mechanism.

[0060] 1. System Initialization: (a) Load the parameters of a pre-trained RWKV model (e.g., RWKV-7 "Goose", with 7B parameters, L=32, D=4096). Go to the processor (GPU) and set it to evaluation mode.

[0061] (b) Load the knowledge key tensor generated in step one (3.5) and knowledge value tensor It is stored in the GPU's video memory (VRAM).

[0062] (c) Initialize an empty RWKV loop state (WKV State). (L) (a zero matrix) and an empty Token Shift state (L zero vectors of D dimensions).

[0063] 2. Process the input prompt sequence (Pre-fill & Decode): Receive a length of User input prompts ,in It is a token.

[0064] This invention can be processed using Decode mode: from The model begins by entering RNN inference mode. Each step... : Receive previous step Output As current input .

[0065] Receive the status from the previous step and .

[0066] Perform step 3 (improved RWKV time mixing block) to calculate Operations with high complexity.

[0067] Output (Predicted next lexicon) and (Updated status), and send it to step.

[0068] The specific workflow is as follows: Figure 3 As shown below, its application will be explained in detail. The internal workings of the moment.

[0069] 3. Improved RWKV time mixing block (the core of this invention) In each step of the time loop (In decoding mode), for each layer of RWKV (From 1 to L), the following two computation paths are executed in parallel: 3.1. Standard RWKV state evolution path (circular memory path): (a) Input: Receive the current token embedding from the previous layer (or the embedding layer). And the previous time step stored in the loop state. WKV state .

[0070] (b) Internal parameter calculation (Weight Prepare): The RWKV model first calculates based on... A series of internal control parameters for the current step are calculated (using a combination of TokenShift and lerp), all of which are D-dimensional vectors (using RWKV-7 as an example): Receptance (R) Used for "querying" and "receiving" historical status.

[0071] Key (K) : The value used to "write" new information.

[0072] Value (V) : Content used to "write" new information.

[0073] Decay (W) : Timing decay vector, controlling The forgetting rate of each channel.

[0074] In-context Learning Rate (A) and Removal Key ) : A component of the generalized Delta rule, allowing the model to adapt to the current input Selectively erase (remove) Specific information in (by) Decide).

[0075] Gating (G) Output gate: Controls how much information flows to the next module.

[0076] (c) WKV State Evolution: The core iterative step of RWKV. The WKV state matrix is ​​updated according to the Generalized Delta Rule. :

[0077] The meaning of this formula is: First, for state of time Apply a "forget" operation (by) (jointly decided), then added to the current situation. New information at any time (by) Decide).

[0078] (d) Standard Readout: RWKV uses its Receptance Come to "Query" the WKV status that has just been updated. and add a value to the current token. The "bonus" item To obtain the output of the standard path :

[0079] This represents the model in Time is based on its internal circular memory (i.e. The content that is "thought out".

[0080] 3.2. Circular Rectangular Attention Path (External Knowledge Path): This path is computed in parallel with path 3.1. Its purpose is to achieve the desired result at the current time step. From VRAM The knowledge token is used to "retrieve" the information most relevant to the current context.

[0081] (a) Knowledge Query Vector Generation: It is used for querying Knowledge Key The D-dimensional vector. This invention will directly reuse the Receptance vector calculated in 3.1(b). As a query vector: . reason: It is trained in RWKV to "query" and "receive" historical information (WKV state), therefore it already contains the current time. The semantic representation of the required information is an ideal, inexpensive (zero additional computation) vector for querying external knowledge bases.

[0082] (b) Knowledge Attention Computation (Rectangular Attention): This step is a calculation. With stored in VRAM Tensor in the first Layer Key vectors The similarity.

[0083] This is a parallel matrix-vector multiplication: Calculate Knowledge Key Token .

[0084] Applying the knowledge base scaling (KB Scaling) factor, we obtain Unnormalized attention score

[0085] reason: This item (of which) For example, the largest KB-sized hyperparameter seen during training. Ensure that the knowledge base size is appropriate. As the size increases, the total contribution of the knowledge part (KB part) remains stable, preventing it from "overwhelming" the information from the standard RWKV path (prompt / question part).

[0086] (c) Knowledge Context Vector Generation: Use the Softmax function to fractions Normalized to attention weights

[0087]

[0088] Use weight (one (dimensional vector) in VRAM Tensor in the first Layer Knowledge Value Token (one We perform a weighted summation on the matrix to obtain a D-dimensional "knowledge context vector" that incorporates information from M knowledge triples. :

[0089] This represents the model in Content that is constantly "queried" and "retrieved" from external knowledge bases.

[0090] 3.3. Knowledge Fusion Mechanism: This step will output the path from 3.1. (Representing the circular memory of RWKV) and the output of path 3.2 (This represents the fusion of external knowledge retrieved).

[0091] knowledge vector Injected into the RWKV readout phase, that is, in Afterwards, in Previously, a trainable scalar gating system was introduced. (One per layer, trained in step three), used to balance the contributions of the two paths:

[0092] Reason: This method is non-invasive and does not interfere with the above 3.1(c). The state evolution is "attached" only at the final output, ensuring the stability of the RWKV core loop (Delta rule). It can be initialized to 0, which closes the knowledge path in the early stages of training and ensures training stability.

[0093] 3.4. Standard RWKV Subsequent Path: Will and Added via residual connection .

[0094] here This represents the output vector after processing by the time mixing block at time step t and the l-th layer. This represents the original input feature vector before entering the current l-layer temporal mixing block.

[0095] Will Passed to the channel-mixing block. The channel-mixing block is essentially a 1x1 convolution or equivalent MLP used to mix information along the channel dimension.

[0096] The output of the channel mixing block (also via residual connection) Passed to the next layer of the model .

[0097] At the top level The final output Passed to LM Head to predict the next token .

[0098] Step 3: Adapter Training and Optimization If it is necessary to train the adapter parameters introduced in this invention Then, the following steps are performed. The purpose of this step is to teach the adapter how to correctly encode knowledge (Step 1) and how the model can correctly query and fuse knowledge (Step 2).

[0099] ①. Freeze the base model: Maintain the RWKV basic model All parameters (including) (etc.) are in a frozen state (requires_grad=False) to prevent catastrophic forgetting. Only adapter parameters are... Set it to trainable (requires_grad=True).

[0100] ②. Construct training data (fine-tuning instructions) (a) Synthetic Knowledge Base (KB): Generate a large synthetic knowledge base (e.g., 100K to 200K triples) using GPT-4 or similar methods. A key step in this process is decoupling: For example, first generate 50 random <name>(such as "Posh Poodle", "Bohr's Bookshelf", "Samosa Sage") and 3 <property>(such as "description", "purpose", "objectives").

[0101] Then, generate combinations that are independent of the literal meaning of their names. <value>(For example, generate "A line of organic teas with health-boosting properties" for "Posh Poodle").

[0102] Reason: This forces the model to look up the answer through the "circular rectangle attention" mechanism of this invention, rather than relying on the knowledge already in its parameters (e.g., it does not know what "Posh Poodle" is).

[0103] (b) Generate instructions (Q&A pairs): Based on this synthesized KB, generate at least four types of Q&A training samples: Type 1: Simple Q&A (Search): Question: "What is the description of Posh Poodle?"; Answer: "A line of organic teas with health-boosting properties."; Objective: To train the model to have accurate key-value lookup capabilities.

[0104] Type 2: Multi-entity Q&A (parallel search): Question: "Tell me the purpose of Samosa Sage and the description of TempuraTitan."; Answer: "The purpose of Samosa Sage is to provoke thought and discussion and the description of Tempura Titan is a bodybuilder competing in international competitions."; Objective: To train a model to query multiple different pieces of knowledge in a single step (or in consecutive steps).

[0105] Type 3: Open-ended / Inference Q&A (Find and reason): Question: "What is the objective of Pythia's Plates and what do you think ofits social impacts?"; A: (The first half of the sentence comes from KB) "The objective of Pythia's Plates is to promote holistic health and relaxation," (The second half of the sentence is inferred by LLM based on the first half of the sentence) "which could potentially lead to improved physical and mental wellbeing in their community..."; Objective: To train the model to retrieve knowledge The supplementary context allows for further complex reasoning.

[0106] Type 4: Unanswerable Q&A (Hallucination Suppression): Q: "What is the description of Pesto Paladin?" (Assuming "Pesto Paladin" is not in KB); Answer: "Sorry, I cannot find relevant information in the KB."; Objective: To train the model on attention weights When the maximum value (max-softmax-score) is below a certain threshold, the system learns to refuse to answer, which is the core of reducing hallucinations.

[0107] ③. Construct training batches: During training, each training sample consists of (Q, A, ...) )constitute.

[0108] It is a subset randomly sampled from a large synthetic KB, the size of which is It also varies randomly within a range, for example If Q is answerable (type 1, 2, 3), then The answer must contain one or more triples corresponding to that answer. The rest Each triple acts as a distractor. This forces the attention mechanism to... We must learn from The correct item is precisely identified from the list of items (i.e., assigned a high weight), while all distractors are assigned weights close to 0.

[0109] ④. Optimization process: Optimizer: Select AdamW optimizer, use default settings. .

[0110] Learning rate scheduling: A cosine decay scheduler with warmup is used. The total training steps are 20,000, with a 1,000-step warmup, and the peak learning rate is set to... .

[0111] Loss function: The optimization objective is a standard autoregressive loss function (cross-entropy loss), which maximizes the log-likelihood probability of each word in the model's generated correct answer A.

[0112] in, This represents the value of the loss function to be optimized. Indicates the length of the answer sequence. This represents the set of trainable adapter parameters introduced in this invention. Indicates pre-training The fixed parameters of the basic model Represents the first in the answer sequence i Each word element, This indicates an input problem. This represents a subset of triples randomly sampled from the complete knowledge base.

[0113] Step 4: System Implementation and Deployment The present invention also provides a knowledge enhancement system, which includes a knowledge encoding module and an inference engine. By applying the above-described method, an external knowledge base is injected into a language model with a cyclic architecture to achieve knowledge enhancement.

[0114] Furthermore, the present invention also provides a knowledge enhancement device, which is a complete and operable computing device, comprising: 1. Storage units (non-volatile storage such as SSDs, volatile storage such as CPU RAM and GPU VRAM): Non-volatile storage (disk): (a) Parameters of the pre-trained RWKV model (For example, the rwkv-7b.pth file).

[0115] (b) External knowledge base KB (e.g., enron_kb.json, stored as a triple string).

[0116] (c) Pre-trained sentence encoder model (For example, the model file or API interface information of ada-002).

[0117] (d) Parameters of the pre-trained knowledge adapter reconstructed for RWKV (For example, the rwkv_kblam_adapter.pth file).

[0118] Volatile memory (VRAM / RAM, during system runtime): (e) RWKV model parameters (Loaded into VRAM).

[0119] (f) Adapter parameters (Loaded into VRAM).

[0120] (g) Knowledge key-value tensor (Generated after encoding in step one, it is the main component occupying video memory and must be loaded into VRAM to function) (Quick search).

[0121] (h) RWKV cycle state (Dynamically updated in VRAM).

[0122] 2. Processor Unit: This unit is configured to execute the software logic of the present invention: (1) Knowledge Coding Module: This module is responsible for executing "Step One". It reads the KB from (b) and calls the function in (c). and (d) Perform encoding and adapter mapping to ultimately generate (g) Tensors are loaded into VRAM.

[0123] This module also includes KB update logic: when the KB file in (b) changes, this module is triggered, recalculating only the changed entries. The corresponding row is then processed, and the data in the VRAM is hot-swaped to achieve dynamic updates.

[0124] (2) Model loading module (executed at system startup): Responsible for (e) and (f) Load from disk into VRAM.

[0125] (3) Inference Engine (Executed Online): This module is the core of the system and is responsible for executing "Step Two".

[0126] When user input is received At this time, the engine first performs a pre-fill operation: leveraging the parallelism of RWKV, it performs computation on the GPU in one go. All operations, get The final loop state at time step .

[0127] Then, the engine enters the decoding loop: For : 1. Call LM Head for prediction Lexicons of time .

[0128] 2. If Not a terminator (EOS): 3. state of time As input, the "improved RWKV time mixing block" is fed in.

[0129] 4. Execute the "standard RWKV state evolution path" (3.1) and the "cyclic rectangular attention path" (3.2) in parallel, where path 3.2 will start from VRAM. Search for knowledge in the middle.

[0130] 5. Merge the results of the two paths through the knowledge fusion mechanism (3.3).

[0131] 6. Pass the result through the "subsequent path" (3.4) to obtain... Output at time and new status .

[0132] Return to step 1 to prepare for prediction. The lexical unit of time.

[0133] The computational complexity of each step in this loop (steps 3-6) is O(n). ,and The length of the generated text is irrelevant, thus achieving a constant generation delay.

[0134] The system and apparatus provided in this embodiment of the invention have the same implementation principle and technical effects as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in this embodiment can be referred to the corresponding content in the aforementioned method embodiment, and will not be repeated here.

[0135] Experimental verification: 1. Experimental setup: Basic language model: (a) RWKV model: The open-source RWKV-7 7B (7 billion parameters) model is selected as the basic model of this invention. .

[0136] (b) Transformer model: The open-source Llama-3 8B (8 billion parameters) model was selected as the baseline model for KBLAM.

[0137] Knowledge Base (KB): This invention uses GPT-4 to generate 135,000 triples as a synthetic knowledge base and the Enron dataset as a real knowledge base for training and evaluation. Sentence Encoder OpenAI ada-002 (P=1536) was selected uniformly.

[0138] Comparison method: Baseline-1 (Zero-shot): The RWKV-7 7B model does not load any KB and directly answers the question.

[0139] Baseline-2 (ICL, Contextual Learning): The RWKV-7 7B model concatenates KB triples as plain text strings into the input prompt.

[0140] Baseline-3 (Transformer+KBLAM): Llama-3 8B model + original KBLAM implementation (rectangular attention based on Transformer).

[0141] This invention (RWKV+KBLAM): RWKV-7 7B model + this invention; Training: Both Baseline-3 and "this invention" use the first 120K data points from Synthetic-KB to train the adapter as described in step three.

[0142] 2. Evaluation Metrics and Tasks: Task 1: Knowledge Q&A Settings: Randomly select from KB Triples as context ( (Varies between 100 and 10,000), and then a question is posed about one of the triples.

[0143] Metric: BERT Score (F1). Measures the difference between the generated answer and the standard answer (i.e., the triplet). <value>semantic similarity of the answer to the question.

[0144] Task 2: Hallucination: Setting: Provide triples, but raise a question (unanswerable Q&A) that cannot be answered from these triples.

[0145] Metrics: Refusal Accuracy. Measures the percentage of times the model answers a refusal phrase such as "I cannot find relevant information in the knowledge base."

[0146] Task 3: Performance: Setting: Fix , change the length of the input prompt (from 100 to 8000 tokens).

[0147] Metrics: Average token generation delay (ms / token). Measures the average time it takes for the model to generate the next token after processing a prompt of length.

[0148] 3. Experimental Results: The experimental results are shown in Tables 1-3: Table 1: Accuracy

[0149] Table 2: Refusal Rate

[0150] Table 3: Performance

[0151] Table 1 shows the accuracy of different models in the knowledge question answering task, where Synthetic represents synthetic data and Enron is a database released by the Enron Corporation. Table 2 compares the refusal accuracy of the models when faced with unanswerable questions, with M representing the number of knowledge entries. Table 3 compares the average token generation delay and computational complexity, with T representing the context length.

[0152] ​As described in the above embodiments, those skilled in the art will understand that the present invention proposes a method, system, and apparatus for injecting an external knowledge base into a language model with a cyclic architecture. The method first performs intelligent segmentation and three-round semantic processing on the external knowledge base, extracting and filtering key proper nouns to generate structured triples. Then, a pre-trained sentence encoder is used to convert the triples into key / value basis vectors, and a parameter-efficient adapter is used to map them to the embedding space of the RWKV model, forming a read-only knowledge key-value tensor. In the inference stage, an innovative "cyclic rectangle attention" mechanism is proposed, which acts as a bridge between the knowledge tokens of KBLAM and the cyclic state of the RWKV model, enabling the injection of the external knowledge base into the language model with a cyclic architecture. During the training stage, a synthetic knowledge base and instruction fine-tuning strategy containing rejection samples are employed to effectively suppress illusions. This invention can address the shortcomings of large-scale language models with cyclic architectures in terms of knowledge timeliness, accuracy, and vertical domain adaptability. Experiments show that this invention significantly improves the performance of RWKV in professional domain tasks while maintaining constant-level generation latency. It is suitable for vertical domains such as finance, healthcare, and law, where high accuracy and timeliness of knowledge are required.

[0153] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, devices, or computer program products, etc. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0154] It should be noted that the word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The present invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer.

[0155] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0156] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention.< / value> < / value> < / property> < / name> < / value>

Claims

1. A method for injecting an external knowledge base into a language model with a circular architecture, characterized in that, The method includes the following steps: S1. Preprocess and encode the external knowledge base to generate a knowledge token, wherein the knowledge token includes a knowledge key tensor and a knowledge value tensor; In the inference process of the S2. recurrent architecture language model, at each decoding step of the temporal loop, for each layer of the model, the standard recurrent state evolution path and the recurrent rectangular attention path are executed in parallel, where: The standard cyclic state evolution path outputs a model based on the representation of the internal cyclic memory. The circular rectangular attention path uses the model's receptance vector as the query vector, calculates the attention weight with the knowledge key tokens of the knowledge key tensor, and then sums the knowledge value tokens of the knowledge value tensor to obtain the knowledge context vector. The outputs of the standard cyclic state evolution path and the cyclic rectangular attention path are then fused to achieve knowledge injection.

2. The method according to claim 1, characterized in that, In step S1, structured triples are generated from the external knowledge base through intelligent segmentation, data extraction and filtering. The triples are then converted into key / value vectors using a sentence encoder and mapped to the model embedding space through a parameter-efficient adapter, generating read-only knowledge key tensors and value tensors.

3. The method according to claim 2, characterized in that, The data extraction and filtering process includes three rounds of processing: the first round uses a large language model to tag proper nouns; the second round extracts the tagged words to form a dictionary; and the third round compares the semantics of RAG with a general large model to eliminate words with the same semantics.

4. The method according to claim 2, characterized in that, The parameter-efficient adapter is constructed using low-rank adaptive LoRA, which decomposes the projection matrix into the product of two low-rank matrices.

5. The method according to claim 1, characterized in that, In S2, the cyclic architecture language model is the RWKV model, in which the standard cyclic state evolution path and the cyclic rectangular attention path are executed in parallel within its temporal mixing block.

6. The method according to claim 1, characterized in that, In S2, the specific calculation process of the knowledge context vector in the cyclic rectangular attention path includes: (a) Knowledge Query Vector Generation: It is used for querying Knowledge Key Token A D-dimensional vector, reusing the Receptance vector As a query vector, that is: ; (b) Knowledge Attention Calculation: calculate With the knowledge key tensor stored in VRAM at the 1st Layer Knowledge Key Token By applying the knowledge base scaling factor, we obtain Unnormalized attention score in, For hyperparameters; (c) Knowledge Context Vector Generation: Use the Softmax function to fractions Normalized to attention weights Use weights For the knowledge-valued tensor in VRAM at the th Layer Knowledge Value Token By performing a weighted summation, we obtain a D-dimensional knowledge context vector that incorporates information from M knowledge triples. : Knowledge Context Vector Representative model in Content retrieved from external knowledge bases at all times.

7. The method according to claim 1, characterized in that, In S2, the outputs of the standard cyclic state evolution path and the cyclic rectangular attention path are fused through trainable scalar gating to achieve non-intrusive knowledge injection.

8. A knowledge enhancement system, characterized in that, The system includes a knowledge encoding module and an inference engine, which uses the method described in any one of claims 1–7 to inject an external knowledge base into a language model with a cyclic architecture to achieve knowledge enhancement.

Images