Large language model construction method based on KV cache and expert network parallel computing
By combining KV caching, MoE architecture, RMSnorm, and RoPE encoding, a hybrid expert large language model construction method is developed, which solves the resource consumption and stability problems of large language models during training and inference, and achieves efficient and stable language model construction.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-25
- Publication Date
- 2026-04-21
AI Technical Summary
Existing large language models face problems such as a sharp increase in the number of parameters, high memory consumption, high computational cost, and poor training stability during training and inference, making it difficult to balance the model's expressive power and inference efficiency.
We adopt a hybrid expert large language model construction method based on KV caching and expert network parallel computing. We combine MoE architecture, RMSNorm normalization, RoPE encoding and gating network to optimize the model architecture and perform pre-training and fine-tuning.
It achieves a balance between high expressive power and efficient inference performance, reduces computational costs, improves training stability and inference efficiency, and provides a complete technical solution.
Smart Images
Figure CN121902864A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of artificial intelligence and natural language processing, specifically relating to a method for constructing a large language model based on key-value caching and parallel computation of expert networks. Background Technology
[0002] In recent years, large language models based on the Transformer architecture have made revolutionary progress in the field of natural language processing, demonstrating outstanding performance in various tasks such as text generation, dialogue systems, and code completion. Its core component—the multi-head self-attention mechanism—can effectively capture long-range dependencies between tokens in a sequence, thereby achieving a deep understanding of contextual information.
[0003] However, as model sizes continue to increase in pursuit of higher performance, traditional intensive Transformer models face significant challenges during training and inference. First, the number of model parameters grows dramatically, leading to an exponential increase in the demand for computing hardware (such as GPU memory), resulting in extremely high training and deployment costs. Second, during the autoregressive generation process, the key-value vectors of all previous tokens need to be repeatedly computed, increasing inference latency and making it difficult to meet the demands of low latency and high throughput in practical applications.
[0004] To address these challenges, various improvement schemes have been proposed in the industry. For example, key-value caching technology is widely used in the inference stage to avoid redundant calculations by caching already computed key-value pairs, thus significantly accelerating the generation process. However, this technology also introduces new problems, namely, caching requires a large amount of dynamic memory, especially when generating long sequences, where memory bottlenecks become a key factor restricting model efficiency. On the other hand, the MoE architecture introduces a sparse activation mechanism, which greatly increases the total number of model parameters while activating only a portion of the expert network for computation, thereby improving the model's capacity and expressive power without significantly increasing computational cost. However, how to efficiently integrate the MoE structure with the attention mechanism and optimize its training stability remains a topic that requires further research.
[0005] Furthermore, in terms of model normalization, Root Mean Square Normalization (RMSNorm) is gradually becoming an alternative to traditional LayerNorm due to its computational simplicity and stable performance. Regarding positional encoding, Rotated Position Encoding (RoPE) can provide better positional information representation for attention mechanisms, improving the model's ability to process long sequences. Although these techniques each have their advantages, existing technologies have yet to provide a complete solution for how to systematically integrate them into a unified and efficient model framework and address the resulting memory, computation, and stability issues.
[0006] Therefore, there is an urgent need in the current technological field for an innovative method for constructing large language models that can effectively integrate advanced technologies such as KV caching, MoE, and RoPE, while ensuring the model's powerful generation capabilities and significantly optimizing its training efficiency and inference performance, while reducing resource consumption. This invention aims to address these and other technical challenges. Summary of the Invention
[0007] (a) Technical problems to be solved The technical problem to be solved by this invention is how to provide a method for constructing a large language model based on KV caching and parallel computing of expert networks, so as to solve the core problem in the prior art that it is difficult to balance model expressiveness, training stability and inference efficiency.
[0008] (II) Technical Solution To address the aforementioned technical problems, this invention proposes a method for constructing a large language model based on KV caching and parallel computation of expert networks. This method includes the following steps: Step 1: Data Preprocessing First, the input text data is preprocessed; Step 2: Constructing the Model Architecture The hybrid expert big language model adopts an improved Transformer decoder architecture, which includes an embedding layer, multiple Transformer blocks, and an output layer. The embedding layer uses the discrete data labels of the input to perform digitization and vectorization representation. Each Transformer block contains two core sub-modules: a multi-head self-attention mechanism and a hybrid expert feedforward network, which are used for deep contextual feature extraction and information fusion. The output layer is used to realize autoregressive text generation. Step 3: Model Pre-training and Full Fine-tuning The model was unsupervised pre-trained using the binary pre-training data prepared in step one, and then supervised fine-tuning was performed using the instruction fine-tuning dataset released by BELLEGroup.
[0009] (III) Beneficial Effects This invention proposes a method for constructing large language models based on key-value caching and parallel computation of expert networks. Compared with existing technologies, the hybrid expert large language model construction method based on key-value caching and multi-head attention mechanism provided by this invention has the following significant advantages: 1. Achieving an excellent balance between performance and efficiency: By introducing the MoE architecture as a feedforward network, this invention enables the model to significantly increase the total number of parameters (thus possessing extremely strong expressive power) while activating only a portion of experts in each forward computation, keeping the computational cost (FLOPs) essentially unchanged, and efficiently resolving the contradiction between model capacity and computational cost.
[0010] 2. Significantly improved inference efficiency and optimized memory usage: This invention deeply integrates a KV caching mechanism, effectively avoiding redundant calculations of key-value vectors during the inference phase, greatly accelerating the autoregressive text generation process. Simultaneously, through overall optimization of the model architecture, it alleviates the memory consumption pressure of a pure KV caching scheme, achieving lower latency and higher throughput inference performance.
[0011] 3. Enhanced model training stability and expressiveness: This invention employs a pre-implemented RMSnorm normalization strategy, combined with residual connections, effectively mitigating gradient problems in deep networks and improving training stability. Combined with RoPE, the model can more accurately understand token location information, performing exceptionally well when processing long text sequences. The dynamic routing mechanism of the gating network also ensures efficient and balanced utilization of experts.
[0012] 4. A complete and implementable technical solution is provided: This invention is not a simple application of a single technology, but rather provides a complete workflow from data preprocessing, word embedding, model architecture design, pre-training to instruction fine-tuning. This solution systematically integrates multiple cutting-edge technologies, and each step has been carefully designed and verified, exhibiting high feasibility and repeatability, providing a clear blueprint for building high-performance, high-efficiency large language models. Attached Figure Description
[0013] Figure 1 This is a flowchart of the preprocessing process of the present invention; Figure 2 This is a diagram of the hybrid expert large language model of the present invention. Detailed Implementation
[0014] To make the objectives, contents, and advantages of the present invention clearer, the specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.
[0015] This invention relates to the fields of artificial intelligence and natural language processing (NLP), and particularly to a method for the structural design and optimization of large-scale deep learning models. Specifically, this invention provides a method for constructing a Mixture of Experts (MoE) Large Language Model (LLM) based on a key-value cache (KV Cache) and implementing a multi-head attention mechanism.
[0016] The core technical problem solved by this invention lies in how to efficiently construct and train a generative large language model that combines powerful expressive capabilities with efficient reasoning performance. Its technical solution deeply integrates the Transformer decoder architecture, hybrid expert system, Rotation Position Encoding (RoPE), and key-value caching optimization strategies, and is mainly applied to natural language processing tasks such as text generation, dialogue systems, machine translation, and code generation.
[0017] The purpose of this invention is to provide a method for constructing a hybrid expert large language model based on a multi-head attention mechanism using a key-value (KV) cache, thereby addressing the core problem in existing technologies where it is difficult to simultaneously achieve model expressiveness, training stability, and inference efficiency. Specifically, this method aims to overcome the shortcomings of large language models, such as the dramatic increase in computational resource consumption due to the expansion of parameter scale, excessive KV cache memory usage, and unstable training of hybrid expert systems, through systematic architectural integration, thus achieving efficient, stable, and high-performance model training and inference.
[0018] This invention proposes a method for constructing a hybrid expert large language model based on a KV cache and a multi-head attention mechanism. The method includes the following steps: Step 1: Data Preprocessing like Figure 1 As shown, the present invention first preprocesses the input text data. The preprocessing process includes the following sub-steps: S11. Training the word segmenter: A dedicated word segmenter is built and trained based on the Byte-Pair Encoding (BPE) algorithm, resulting in a vocabulary size of 6400, which efficiently converts the original text into a discrete token sequence that the model can process.
[0019] S12. Data Cleaning and Formatting: Based on the mobvoi_seq_monkey general_open_corpus dataset provided by "Sequence Monkey", a rigorous data cleaning process is performed, including verifying the encoding format, removing invalid characters, and standardizing the text structure to ensure data quality.
[0020] S13. Serialization Processing: The cleaned text data is divided into blocks of fixed length, processed by the aforementioned word segmenter, converted into an integer index sequence, and BOS (Begin of Sentence) and EOS (End of Sentence) special markers are added to the beginning and end of the sequence. Finally, the blocks are merged and saved as a binary file for efficient reading.
[0021] Step 2: Constructing the Model Architecture The hybrid expert large language model constructed in this invention employs an improved Transformer decoder architecture. For example... Figure 2As shown, this model mainly includes an embedding layer, multiple Transformer blocks, and an output layer. The model contains the following core processing modules: 1. Embedded layer First, the embedding layer digitizes and vectorizes the input discrete data tags. After being processed by a tokenizer, the input data is transformed into a sequence of integer indices. This index sequence is then searched through a learnable embedding layer, which retrieves the input integer indices. It is mapped to a continuous high-dimensional vector space. The output tensor of the embedding layer is passed as input to the RMSNorm layer for processing.
[0022] 2. Transformer layer After obtaining the embedded representation of the input sequence, this tensor is used as input for deep contextual feature extraction and information fusion through a series of stacked Transformer blocks. This invention employs eight structurally identical Transformer blocks stacked together. Each Transformer block contains two core sub-modules: a multi-head self-attention mechanism and a hybrid expert feedforward network. Each sub-module uses residual connections and undergoes pre-processing RMSnorm normalization. The detailed processing flow is as follows: like Figure 1 As shown, each block first applies RMSNorm normalization to the input tensor.
[0023] The normalized tensor is input into the multi-head attention module. This module uses query (Q), key (K), and value (V) projection matrices (respectively...) , , This linearly transforms the input to different representation spaces.
[0024] After applying RoPE encoding to the Q and K vectors, KV caching optimization is performed. Let the current generation step number be t, and the caching mechanism is represented as follows: Attention computation based on caching: The output is the same as the original input. After performing residual connections, Dropout (p=0.1) is applied to obtain the final output of the multi-head attention module. .
[0025] in, Let K be the dimension.
[0026] Hybrid Expert Feedforward Network (MoE-FFN): Will After RMSnorm normalization, the input is fed into the MoE-FFN layer. The MoE-FFN layer consists of multiple expert networks and a gating network, which can dynamically select the most suitable expert for processing based on the input, significantly improving the model's expressive power. Its core expression formula is as follows: The output of the i-th expert model The weights calculated by the router determine which experts should be activated and to what extent each expert should be activated.
[0027] N: The total number of expert models.
[0028] This output, after being residually concatenated with the output of the attention submodule, forms the final output of the Transformer block. .
[0029] 3. Output layer The output of the last Transformer block is obtained by RMSnorm normalization. By mapping the hidden vectors to the vocabulary space through a linear transformation layer and applying the Softmax function to calculate the probability distribution of the next word, autoregressive text generation is achieved.
[0030] Step 3: Model Pre-training and Full Fine-tuning Model training is divided into two stages: 1. Pre-training: Use the binary pre-training data prepared in step one to perform unsupervised pre-training on the model. The training epoch is set to 15, so that the model learns general language representations.
[0031] 2. Supervised Fine-Tuning (SFT): Supervised fine-tuning of the pre-trained model is performed using the Supervised Fine-Tuning dataset released by the BELLE Group, with a training cycle of 5, to align the model output with the intent of human commands.
[0032] Step 4: Experiment and Verification 1. Experimental Environment Configuration The experiment was conducted on a virtualization platform, with the following specific hardware configuration: Computing Unit: Two NVIDIA GeForce RTX 3090 GPUs (24GB of video memory each) provide powerful computing support for model inference and evaluation.
[0033] Key parameters: The total number of parameters in the model constructed in this validation is 26.878 million (26.878M). Thanks to the sparse activation characteristics of the hybrid expert (MoE) architecture, the number of parameters that are effectively involved in the calculation during inference is much lower than this value, which reflects its advantage in computational efficiency.
[0034] 2. Evaluation Methods To closely align with real-world application scenarios, this invention employs manual evaluation to assess model performance. We have meticulously designed a test question bank covering multiple capabilities, with reviewers providing comprehensive scores (1-5 points) for the model's generated results across dimensions such as coherence, accuracy, relevance, and logicality. This verifies the model's overall capabilities in language understanding and generation tasks.
[0035] Scoring Criteria Explanation: 5 points (Excellent): The answer is completely accurate, informative, logically clear, and fully in line with the intent of the question.
[0036] 4 points (Good): The answer is basically correct and coherent, but there may be slight incompleteness or redundancy.
[0037] 3 points (average): The answer is partially correct, but there are obvious errors, missing information, or unclear expression.
[0038] 2 points (poor): The answer is relevant to the question, but the core content contains major factual errors or logical inconsistencies.
[0039] 1 point (poor): The answer is completely irrelevant or incomprehensible.
[0040] Some sample test questions and answers are shown below: 3. Results Analysis and Discussion Based on the analysis of the table above and other extensive test results, we draw the following conclusions: Excellent overall performance: In most cases, especially in daily conversations, task following, and concept explanation, the model constructed by this invention can generate coherent, relevant, and accurate text responses, effectively verifying the powerful capabilities and design superiority of its core architecture based on KV caching and hybrid experts in language understanding and generation tasks.
[0041] Limitations Analysis and Attribution: Testing also revealed that the model may exhibit factual bias on a very small number of specific scientific fact-related questions. It is important to note that this is primarily due to the relatively limited total number of parameters (26.878 million) in the current validation model, which restricts the capacity and memory density of the model's internal knowledge base, rather than a flaw in its architectural design. The proposed architecture integrating KV caching and MoE has demonstrated extremely high computational efficiency and expressive potential; its performance ceiling can be fully realized by further increasing the number of experts and the total number of model parameters.
[0042] Future optimization directions: The aforementioned limitations point to a clear path for future continuous iterations. Building upon the existing efficient architecture, this deficiency can be rapidly addressed and model performance comprehensively improved by expanding the model parameter scale, introducing higher-quality training data, or incorporating techniques such as Retrieval Augmentation (RAG).
[0043] Overall Conclusion: Experimental results demonstrate that the model design based on KV caching and a hybrid expert architecture successfully achieves the expected design goals, effectively balancing model performance and inference efficiency. Any limitations observed are direct consequences of model scale rather than the architectural methodology, fully proving that this approach provides a solid and scalable underlying foundation for developing larger-scale, higher-performance language models in the future.
[0044] Compared with existing technologies, the hybrid expert large language model construction method based on KV caching and multi-head attention mechanism provided by this invention has the following significant advantages: 1. Achieving an excellent balance between performance and efficiency: By introducing the MoE architecture as a feedforward network, this invention enables the model to significantly increase the total number of parameters (thus possessing extremely strong expressive power) while activating only a portion of experts in each forward computation, keeping the computational cost (FLOPs) essentially unchanged, and efficiently resolving the contradiction between model capacity and computational cost.
[0045] 2. Significantly improved inference efficiency and optimized memory usage: This invention deeply integrates a KV caching mechanism, effectively avoiding redundant calculations of key-value vectors during the inference phase, greatly accelerating the autoregressive text generation process. Simultaneously, through overall optimization of the model architecture, it alleviates the memory consumption pressure of a pure KV caching scheme, achieving lower latency and higher throughput inference performance.
[0046] 3. Enhanced model training stability and expressiveness: This invention employs a pre-implemented RMSnorm normalization strategy, combined with residual connections, effectively mitigating gradient problems in deep networks and improving training stability. Combined with RoPE, the model can more accurately understand token location information, performing exceptionally well when processing long text sequences. The dynamic routing mechanism of the gating network also ensures efficient and balanced utilization of experts.
[0047] 4. A complete and implementable technical solution is provided: This invention is not a simple application of a single technology, but rather provides a complete workflow from data preprocessing, word embedding, model architecture design, pre-training to instruction fine-tuning. This solution systematically integrates multiple cutting-edge technologies, and each step has been carefully designed and verified, exhibiting high feasibility and repeatability, providing a clear blueprint for building high-performance, high-efficiency large language models.
[0048] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. A method for constructing a large language model based on KV caching and parallel computation of expert networks, characterized in that, The method includes the following steps: Step 1: Data Preprocessing First, the input text data is preprocessed; Step 2: Constructing the Model Architecture The hybrid expert big language model adopts an improved Transformer decoder architecture, which includes an embedding layer, multiple Transformer blocks, and an output layer. The embedding layer uses the discrete data labels of the input to perform digitization and vectorization representation. Each Transformer block contains two core sub-modules: a multi-head self-attention mechanism and a hybrid expert feedforward network, which are used for deep contextual feature extraction and information fusion. The output layer is used to realize autoregressive text generation. Step 3: Model Pre-training and Full Fine-tuning The model was unsupervised pre-trained using the binary pre-training data prepared in step one, and then supervised fine-tuning was performed using the instruction fine-tuning dataset released by the BELLE Group.
2. The method for constructing a large language model based on KV caching and parallel computation of expert networks as described in claim 1, characterized in that, Step one specifically includes: S11. Training the word segmenter: A dedicated word segmenter is built and trained based on the BPE algorithm, and the resulting vocabulary is used to efficiently convert the original text into a discrete token sequence that the model can process. S12. Data Cleaning and Formatting: Based on the mobvoi_seq_monkey general_open_corpus dataset provided by "Sequence Monkey", a strict data cleaning process is performed; S13. Serialization processing: The cleaned text data is divided into blocks of fixed length, processed by the above-mentioned word segmenter, converted into an integer index sequence, and BOS and EOS special markers are added to the beginning and end of the sequence. Finally, the blocks are merged and saved as a binary file for efficient reading.
3. The method for constructing a large language model based on KV caching and parallel computation of expert networks as described in claim 2, characterized in that, S12 includes: verifying the encoding format, removing invalid characters, and standardizing the text structure to ensure data quality.
4. The method for constructing a large language model based on KV caching and parallel computation of expert networks as described in claim 1, characterized in that, In step two, the input data, after being processed by the tokenizer, will be converted into an integer index sequence. This index sequence is then searched through a learnable embedding layer to retrieve the input integer index. Mapped to a continuous high-dimensional vector space; The output tensor of the embedding layer is passed as input to the RMSNorm layer for processing. 。 5. The method for constructing a large language model based on KV caching and parallel computation of expert networks as described in claim 4, characterized in that, In step two, eight Transformer blocks with the same structure are stacked.
6. The method for constructing a large language model based on KV caching and parallel computation of expert networks as described in claim 4, characterized in that, In step two, the multi-head self-attention mechanism includes: Each block first applies RMSNorm normalization to the input tensor; The normalized tensor is input into the multi-head attention module; this module uses the query Q, key K, and value V projection matrix to linearly transform the input to different representation spaces. After applying RoPE encoding to the Q and K vectors, KV caching optimization is performed. Attention scores are calculated based on the caching mechanism, and then the attention scores are projected onto the projection matrix. The output after linear transformation; this output is the same as the original input. After performing residual connections, Dropout is applied to obtain the final output of the multi-head attention module. ; in, Let K be the dimension.
7. The method for constructing a large language model based on KV caching and parallel computation of expert networks as described in claim 6, characterized in that, In step two, the hybrid expert feedforward network includes: Will After RMSnorm normalization, the input is fed into the MoE-FFN layer. The MoE-FFN layer consists of multiple expert networks and a gating network, which can dynamically select the most suitable expert for processing based on the input, significantly improving the model's expressive power. Its core expression formula is as follows: The output of the i-th expert model The weights calculated by the router determine which experts should be activated and to what extent each expert should be activated. N: The total number of expert models; This output, after being residually concatenated with the output of the attention submodule, forms the final output of the Transformer block. ; 。 8. The method for constructing a large language model based on KV caching and parallel computation of expert networks as described in claim 7, characterized in that, In step two, the output layer includes: The output of the last Transformer block is obtained by RMSnorm normalization. By mapping the hidden vectors to the vocabulary space through a linear transformation layer and applying the Softmax function to calculate the probability distribution of the next word, autoregressive text generation is achieved. 。 9. The method for constructing a large language model based on KV caching and parallel computation of expert networks as described in claim 8, characterized in that, In step three, the binary pre-training data prepared in step one is used to perform unsupervised pre-training on the model, with the training epoch set to 15, so that the model learns general language representations; the instruction fine-tuning dataset released by BELLE Group is used to perform supervised fine-tuning on the pre-trained model, with the training epoch set to 5, so as to align the model output with the intent of human instructions.
10. The method for constructing a large language model based on KV caching and parallel computation of expert networks as described in claim 8, characterized in that, Step three is followed by: evaluating the model's performance using manual assessment; designing a test question bank covering multiple capabilities, with reviewers comprehensively scoring the model's generated results in terms of coherence, accuracy, relevance, and logic, thereby verifying the model's comprehensive capabilities in language understanding and generation tasks.