Model long text processing method and system based on mixed precision third-order cache

Abstract

The application discloses a large model long text processing method and system based on a mixed precision third-order cache, relates to the technical field of language text processing, and divides the total space of the history key value state of a large language model into three physically continuous storage areas, namely, a fidelity head anchor point area, a high-density continuous quantization area and a real-time fidelity tail area; in the generation of a streaming long text, when the tail area overflows, a degradation compression flow is triggered, an asymmetric quantization algorithm is used to compress and store the oldest batch of Token states in the quantization area; when the quantization area overflows, the oldest states are physically erased, and the total video memory is constant; in the reconstruction of the history states, the features of the quantization area are extracted for inverse quantization, and the states of the three areas are physically and continuously spliced in the sequence dimension, and finally, attention calculation is completed under the condition that the rotary position encoding is complete and continuous, so that the quality and stability of long text generation are ensured.

Description

Technical Field

[0001] This invention relates to the field of language and text processing technology, specifically to a method and system for processing large-scale long texts based on a mixed-precision third-order cache. Background Technology

[0002] Large language models (such as GPT-4, PaLM, and LLaMA) have demonstrated superior performance in streaming tasks such as natural language generation and multi-turn dialogue due to their hundreds of billions of parameters and rich semantic understanding capabilities. However, the reasoning process of long sequences requires caching the previously generated key-value states (KV Cache) to avoid repeated forward computations. As the length of the generated text increases, the memory usage of the KV Cache becomes linear. Even the quadratic level The rapid increase in memory usage can easily exceed the physical memory limits of a single GPU or edge device, leading to OutOfMemory (OOM) errors or triggering extremely slow virtual memory swapping, causing inference interruptions. Especially in edge devices, low-latency scenarios, and resource-constrained environments, the high cost of GPU memory severely restricts practical applications. To overcome this bottleneck, cache compression and truncation techniques have become key methods for reducing model computation and storage overhead. Their core objective is to effectively map the historical context generated by the model to the limited GPU memory space, thereby improving the efficiency of long text inference.

[0003] Existing large language models generally rely on Rotated Position Encoding (RoPE) to perceive relative word order, which requires the underlying cached sequences to maintain strict physical continuity. Physical discontinuity can directly cause RoPE breaks, leading to model collapse. This misalignment of physical sequences can completely destroy the model's spatial cognition, making inference extremely unstable and even resulting in garbled output. Summary of the Invention

[0004] To address the shortcomings mentioned in the background section, the present invention aims to provide a method and system for processing large-scale long text based on a mixed-precision third-order cache.

[0005] Firstly, the objective of this invention can be achieved through the following technical solution: a method for processing large-model long texts based on a mixed-precision third-order cache, the method comprising the following steps: Obtain a newly generated token and input the newly generated token into a pre-established large language model. Perform long text processing based on the historical key-value state space of the large language model. The historical key-value state space includes a first-order fidelity header anchor region, a second-order high-density continuous quantization region, and a third-order real-time fidelity tail region. The newly generated token is generated based on a set streaming long text generation task. When a newly generated token causes the tail region of the third-order real-time fidelity to overflow, a downgrade compression flow is triggered. The token that first entered the tail region of the third-order real-time fidelity is removed and its key-value state is compressed into a low-bit format and stored in the tail of the second-order high-density continuous quantization region. When the number of tokens in the second-order high-density continuous quantization region reaches the preset limit, the token that first entered the second-order high-density continuous quantization region is eliminated. Based on the processing of the historical key-value state space, the key matrix and value matrix are reconstructed before the autoregressive generation of each frame prediction. The key matrix and value matrix are then input into the large language model to output the long text prediction processing result.

[0006] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: setting a fixed capacity for the first-order fidelity header anchor area, storing it in a native high-precision format, and using it to anchor the key-value state of the starting token of the text sequence without being evicted; The third-order real-time fidelity tail region is set with a fixed capacity as a sliding window and uses a native high-precision format to store the key-value state of the latest generated token of the model. The second-order high-density continuous quantization region is set with a fixed capacity as a sliding window, and uses a low-bit format to quantize and store the key-value state of the token that is eliminated by the third-order overflow.

[0007] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the downgraded compression stream uses an asymmetric channel-by-channel or marker-by-mark quantization algorithm to dynamically extract extreme values ​​from the key matrix and value matrix respectively, calculate their respective quantization step size and zero offset, and map and truncate to a low-bit format, the calculation formula of which is: in, and These are the high-precision key matrix and value matrix to be compressed, respectively. Quantize the bit width to the target. and To quantize the step size, and This is the zero-point offset. and For the low-bit key matrix and value matrix generated by compression, max() is the maximum value function, min() is the minimum value function, round() is the rounding function, and clip() is the numerical truncation function.

[0008] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: before generating each frame prediction through autoregression, restoring the pseudo-floating-point format based on the quantization features extracted by the second-order high-density continuous quantization region and the instantaneous dequantization using the synchronously saved quantization parameters.

[0009] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the calculation formula for the instantaneous inverse quantization, as follows: in, and These are the pseudo-floating-point format key matrix and value matrix after dequantization recovery, respectively.

[0010] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the pseudo-floating-point format key matrix and value matrix are independently assembled based on the low-level memory concatenation operator of the deep learning framework, and the concatenation formula is as follows: in, This represents the state of the first-order fidelity head anchor point region. The second-order region state is recovered by dequantization. This represents the third-order real-time fidelity tail region state. and To ensure strict alignment of the concatenated key and value matrices along the sequence dimension.

[0011] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the rotation position encoding operation at the bottom layer of the large language model is applied only to the complete concatenated key matrix and the current query vector, injecting relative position information into them, and finally the current query vector performs a dot product attention operation with the complete continuous key matrix and value matrix.

[0012] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the calculation formula for the dot product attention operation, as follows: In the formula, The feature dimension of the attention head.

[0013] Secondly, in order to achieve the above objectives, this invention discloses a large-model long text processing system based on a mixed-precision third-order cache, comprising: The data processing module is used to obtain the newly generated token, input the newly generated token into the pre-established large language model, and perform long text processing based on the historical key-value state space of the large language model. The historical key-value state space includes a first-order fidelity header anchor region, a second-order high-density continuous quantization region, and a third-order real-time fidelity tail region. The newly generated token is generated based on the set streaming long text generation task. The data optimization module is used to trigger a downgrade compression flow when the newly generated token causes the third-order real-time fidelity tail area to overflow. The earliest token to enter the third-order real-time fidelity tail area is removed and its key-value state is compressed into a low-bit format and stored in the tail of the second-order high-density continuous quantization area. When the number of tokens in the second-order high-density continuous quantization area reaches the preset limit, the earliest token to enter the second-order high-density continuous quantization area is eliminated. The prediction output module is used for processing based on the total space of historical key-value states. Before generating predictions for each frame through autoregression, it reconstructs the key matrix and value matrix, inputs the key matrix and value matrix into the large language model, and outputs the long text prediction processing results.

[0014] The beneficial effects of this invention are: This invention achieves a smooth transition and space compression of historical context between native high precision and low bit quantization precision by constructing a three-order sliding cache architecture based on continuous mixed precision. It integrates an infinite streaming generation mechanism with sunken anchors and an absolutely continuous physical memory splicing technique, effectively avoiding the problem of rotation position encoding fragmentation caused by discrete token discarding, and preserving key intermediate context as much as possible within extremely demanding memory boundaries, thus solving the bottleneck of increased perplexity caused by severe amnesia. Finally, this invention reduces memory complexity to constant level without significantly increasing memory consumption or modifying the underlying model architecture, and in practice, it significantly reduces model perplexity compared to the baseline solution. It provides high stability and low deployment cost for achieving high-quality streaming generation of infinite length in resource-constrained environments, and improves the stability of inference. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Figure 1 This is a schematic diagram of the method flow of the present invention; Figure 2 This is a schematic diagram of the overall architecture of the present invention; Figure 3 This is a schematic diagram of the dynamic state transition process of the present invention; Figure 4 This is a schematic diagram of the asymmetric feature quantization compression calculation logic of the present invention; Figure 5 This is a schematic diagram of the physical continuous tensor splicing and relative position encoding smooth mapping of the present invention; Figure 6 This is a schematic diagram of the forward attention calculation process of the large model of this invention; Figure 7 This is a schematic diagram of the system structure of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] Example 1: like Figure 1 As shown, a method for processing large-scale long text based on a mixed-precision third-order cache includes the following steps: S101: Obtain the newly generated token, input the newly generated token into the pre-established large language model, and perform long text processing based on the historical key-value state space of the large language model. The historical key-value state space includes the first-order fidelity header anchor region, the second-order high-density continuous quantization region, and the third-order real-time fidelity tail region. The newly generated token is generated based on the set streaming long text generation task. like Figure 2 The allocation process of the total space for historical key-value states is as follows: Before the streaming long text generation task begins, the system needs to pre-allocate static memory blocks in the cache architecture in the underlying hardware (such as GPU memory).

[0018] Define the cache lifecycle parameters for each stage. It should be noted that the capacity thresholds for each physical block in this invention are not absolutely fixed. In actual large-scale long text processing tasks, the system will dynamically evaluate and adaptively configure the capacity ratio of the three-level cache based on the total length of the user's initial input text, the expected span of the generated task, and the upper limit of available video memory in the underlying hardware. The following settings, value ranges, and specific values ​​selected in this embodiment are only illustrative examples of this invention under specific video memory constraints, test models, and test datasets: (1) Capacity of the head anchor point area for fidelity The threshold is set based on the "attention sink" phenomenon inherent in the attention mechanism of large language models, where a very small number of tokens at the beginning of the sequence absorb an extremely large amount of attention score in the underlying Softmax calculation. Its value is typically set between 1 and 8 tokens. In this embodiment, it is preferably set to 4 tokens because extensive empirical calculations show that retaining the high-precision state of the first 4 tokens is sufficient as the sink anchor point for global attention calculation, ensuring absolute stability of the generation process. Increasing the capacity further does not significantly improve the quality of long text generation and instead wastes valuable non-quantized GPU memory.

[0019] (2) Real-time fidelity tail area capacity The threshold is set based on the strong and precise dependence of autoregressive language generation tasks on local context. The newly generated token chunk has the most direct semantic and syntactic determining effect on predicting the next adjacent word. This capacity can be dynamically scaled according to the local semantic complexity of the input text, and its value is usually between 64 and 512 tokens. In this embodiment, it is preferably set to 255 tokens. The reason is that this value (together with the 1 newly generated prediction token, it forms the underlying GPU thread block / memory alignment standard length of 256 tokens) can provide enough high-precision local memory to cover the entire short sentence or paragraph without triggering high-precision memory warnings, achieving the best balance between semantic accuracy and hardware resources.

[0020] (3) Capacity of high-density continuous quantization region As the core region receiving intermediate memory, this threshold has extremely strong dynamic scalability. In practical applications, the system will allocate as much of the remaining available physical video memory as possible to this region after deducting model weights, instantaneous activation values, and the high-precision occupancy of the aforementioned T1 and T3 regions, based on the total length of the extremely long input text. Its value range can dynamically extend from hundreds of tokens to tens of thousands or even hundreds of thousands of tokens, depending on the device's video memory limit. In the specific video memory-constrained environment test of this embodiment, it is set to 255 tokens for example, because it is compared with... Maintaining a consistent window size allows for a complete and intuitive verification of the effectiveness of feature decompression from high precision to low bit depth, under the stringent constraint of minimal memory overhead, as well as the system's real-world effect in mitigating model perplexity (PPL) fluctuations. The total cache window length is strictly locked to the sum of the three dynamically configured parameters mentioned above, i.e. .

[0021] Static memory block pre-allocation. A tensor space with absolutely contiguous physical addresses is allocated in the video memory to store the KVCache. This space is logically and strictly divided into three adjacent blocks, T1, T2, and T3, to ensure that memory addresses will not jump discretely during subsequent tensor slicing and splicing operations, thus preventing model inference interruptions or anomalies.

[0022] The first-order fidelity header anchor area has a fixed capacity and is stored in a native high-precision format. It is used to anchor the key-value state of the starting token of the text sequence and is not eliminated. The third-order real-time fidelity tail area is set with a fixed capacity as a sliding window, and the key-value state of the latest generated token of the model is stored in the native high-precision format. The second-order high-density continuous quantization region is set with a fixed capacity as a sliding window, and uses a low-bit format to quantize and store the key-value state of the tokens that are eliminated by the third-order overflow.

[0023] S102: When a newly generated token causes the tail area of ​​the third-order real-time fidelity region to overflow, a downgrade compression flow is triggered. The token that entered the tail area of ​​the third-order real-time fidelity region earliest is removed and its key-value state is compressed into a low-bit format and stored in the tail of the second-order high-density continuous quantization region. When the number of tokens in the second-order high-density continuous quantization region reaches the preset limit, the token that entered the second-order high-density continuous quantization region earliest is eliminated. Specifically, such as Figure 3 As shown, with the large language model generating words by word (Autoregressive Generation), the system continuously captures and processes newly generated contextual features.

[0024] New Token State Calculation. For each new Token generated by the model, its corresponding Query, Key, and Value vectors are calculated in the current layer using native high-precision floating-point format (such as FP16 or BF16).

[0025] Static anchoring in area T1 (fidelity header anchor area). For the very beginning of the text sequence... Each token, whose high-precision key-value state is written into the T1 area, is permanently "physically frozen". The data in this area will never participate in subsequent elimination and circulation, and is dedicated to absorbing redundant attention scores that have no actual semantic meaning generated in the underlying Softmax normalization operation of the large model. This part is used to ensure the global stability of model generation.

[0026] Dynamic updates to the T3 region (real-time fidelity tail region). For the generated sequence ranked... Subsequent tokens will have their high-precision key-value states appended to the end of the T3 region in real time. The T3 region, as a high-precision sliding window, consistently maintains the coherence of the model's short-term language and the rigor of its local logic.

[0027] Specifically, T3 overflow detection. Each time new data is injected into the T3 area, the system determines whether the total number of tokens in the current T3 area exceeds [the limit]. If the limit is not exceeded, the iteration continues; if the limit is exceeded, the downgrade compression process is triggered.

[0028] Asymmetric feature quantization compression (T3 to T2) (e.g.) Figure 4 (As shown). The oldest (or a batch of consecutive) high-precision token states in region T3 are extracted. Instead of discarding them directly as usual, the system sends them to the low-bit quantization engine. In large language models, historical states are composed of a key matrix (K matrix) and a value matrix (V matrix). To address the common outlier problem in large language model inference quantization, this invention employs asymmetric per-channel or per-token quantization algorithms to compress the K and V matrices respectively. Let the high-precision key matrix to be compressed be... The value matrix is The target quantization bit width is (For example The specific calculation process is as follows: First, for high precision Matrix and The matrix dynamically extracts extreme values ​​and calculates the quantization step size (ScalingFactor). And zero point offset : Subsequently, high precision Matrix and Matrix mapping and truncation into 4-bit pseudo-quantized integer format and : in, and These are the high-precision key matrix and value matrix to be compressed, respectively. Quantize the bit width to the target. and To quantize the step size, and This is the zero-point offset. and For the low-bit key matrix and value matrix generated by compression, max() is the maximum value function, min() is the minimum value function, round() is the rounding function, and clip() is the numerical truncation function.

[0029] Finally, the low-bit key matrix generated by compression Sum matrix and its corresponding lightweight metadata (quantization parameters) and ), which are jointly appended to the end of T2 (high-density continuous quantization region) in physically contiguous memory space.

[0030] When T2 receives new quantified data, the system determines whether its total token count exceeds [a certain threshold]. If this is exceeded, it means that the most front-end quantified token has reached the set physical lifespan limit (i.e., it has exceeded the distance from the current generation position). At this point, this portion of data will be physically erased directly from the video memory. Through this closed-loop control, regardless of the length of the generated text, the system's space complexity remains constant. .

[0031] S103: Based on the processing of the total space of historical key-value states, before generating predictions for each frame through autoregression, the key matrix and value matrix are reconstructed. The key matrix and value matrix are then input into the large language model, and the long text prediction processing results are output.

[0032] The rotation position encoding operation at the bottom layer of the large language model only applies to the complete concatenated key matrix and the current query vector, injecting relative position information into them. Finally, the current query vector performs a dot product attention operation with the complete and continuous key matrix and value matrix.

[0033] like Figure 5 As shown, before generating predictions for each frame via autoregression, the system must reconstruct the complete historical K and V matrices from the third-order buffer to complete the standard multi-head attention calculation.

[0034] Instantaneous dequantization of the quantization features in region T2. ​​Extract the 4-bit bond tensor stored in region T2. AND-value tensor Using the synchronously saved quantization parameters respectively and Perform real-time dequantization to restore to pseudo-floating-point format. and : The physical sequences of the K and V matrices are concatenated in an absolutely continuous manner. Using low-level memory concatenation operators in deep learning frameworks (such as `torch.cat` in PyTorch), the key and value states are assembled independently. This ensures that the T1 region maintains high precision throughout. T2 region recovered after dequantization And the recently developed high-precision T3 area Strict alignment and splicing are performed along the sequence dimension: Relative Position Encoding (RoPE) Smooth Mapping and Lossless Reasoning. Because the aforementioned state transition and concatenation mechanisms maintain absolute continuity in the physical temporal sequence of the vocabulary (i.e., from position...),... arrive There are no index gaps caused by manual discarding; the Rotation Position Encoding (RoPE) operation at the bottom layer of the large model can naturally and correctly operate only on the fully concatenated key matrix. (and the current query vector Query), injecting relative position information into it.

[0035] Ultimately, the current query vector is connected to the complete continuous... and Perform standard dot product attention operations: In the formula, This refers to the feature dimension of the attention head. This continuous attention calculation process eliminates positional fragmentation and model collapse caused by discrete filtering, thus ensuring high-quality long text generation with extremely low perplexity. (The complete attention calculation and forward inference process is as follows...) Figure 6 (as shown) Example 2: To achieve the above objective, such as Figure 7 As shown, based on Embodiment 1, this invention discloses a large-model long text processing system based on a hybrid precision third-order cache, comprising: Data processing module 11 is used to obtain a newly generated token, input the newly generated token into a pre-established large language model, and perform long text processing based on the historical key-value state space of the large language model. The historical key-value state space includes a first-order fidelity header anchor region, a second-order high-density continuous quantization region, and a third-order real-time fidelity tail region. The newly generated token is generated based on a set streaming long text generation task. Data optimization module 12 is used to trigger a downgrade compression flow when the newly generated token causes the third-order real-time fidelity tail area to overflow. The token that first entered the third-order real-time fidelity tail area is removed and its key-value state is compressed into a low-bit format and stored in the tail of the second-order high-density continuous quantization area. When the number of tokens in the second-order high-density continuous quantization area reaches the preset limit, the token that first entered the second-order high-density continuous quantization area is eliminated. The prediction output module 13 is used for processing based on the total space of historical key-value states. Before generating predictions for each frame through autoregression, it reconstructs the key matrix and value matrix, inputs the key matrix and value matrix into the large language model, and outputs the long text prediction processing result.

[0036] Based on the same inventive concept, this invention also provides a computer device, comprising: one or more processors, and a memory for storing one or more computer programs; the programs include program instructions, and the processor executes the program instructions stored in the memory. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, used to implement one or more instructions, specifically for loading and executing one or more instructions stored in a computer storage medium to implement the above-described method.

[0037] It should be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium storing a computer program, which, when executed by a processor, performs the above-described method. This storage medium can be any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0038] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0039] The foregoing has shown and described the basic principles, main features, and advantages of this disclosure. Those skilled in the art should understand that this disclosure is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of this disclosure. Various changes and modifications can be made to this disclosure without departing from its spirit and scope, and all such changes and modifications fall within the scope of this disclosure as claimed.

Claims

1. A method for processing large-scale long text based on mixed-precision third-order caching, characterized in that: The method includes the following steps: Obtain a newly generated token and input the newly generated token into a pre-established large language model. Perform long text processing based on the historical key-value state space of the large language model. The historical key-value state space includes a first-order fidelity header anchor region, a second-order high-density continuous quantization region, and a third-order real-time fidelity tail region. The newly generated token is generated based on a set streaming long text generation task. When a newly generated token causes the tail region of the third-order real-time fidelity to overflow, a downgrade compression flow is triggered. The token that first entered the tail region of the third-order real-time fidelity is removed and its key-value state is compressed into a low-bit format and stored in the tail of the second-order high-density continuous quantization region. When the number of tokens in the second-order high-density continuous quantization region reaches the preset limit, the token that first entered the second-order high-density continuous quantization region is eliminated. Based on the processing of the historical key-value state space, the key matrix and value matrix are reconstructed before the autoregressive generation of each frame prediction. The key matrix and value matrix are then input into the large language model to output the long text prediction processing result.

2. The method for processing large-scale long text based on mixed-precision third-order caching according to claim 1, characterized in that, The first-order fidelity header anchor area has a fixed capacity and is stored in a native high-precision format. It is used to anchor the key-value state of the starting token of the text sequence and is not evicted. The third-order real-time fidelity tail region is set with a fixed capacity as a sliding window and uses a native high-precision format to store the key-value state of the latest generated token of the model. The second-order high-density continuous quantization region is set with a fixed capacity as a sliding window, and uses a low-bit format to quantize and store the key-value state of the token that is eliminated by the third-order overflow.

3. The method for processing large-scale long text based on a hybrid precision third-order cache according to claim 1, characterized in that, The downgraded compression stream employs an asymmetric channel-by-channel or marker-by-mark quantization algorithm to dynamically extract extreme values ​​from the key and value matrices, calculate their respective quantization step size and zero offset, and map and truncate them to a low-bit format. The calculation formula is as follows: in, and These are the high-precision key matrix and value matrix to be compressed, respectively. Quantize the bit width to the target. and To quantize the step size, and This is the zero-point offset. and For the low-bit key matrix and value matrix generated by compression, max() is the maximum value function, min() is the minimum value function, round() is the rounding function, and clip() is the numerical truncation function.

4. The method for processing large-scale long text based on a hybrid precision third-order cache according to claim 1, characterized in that, Before generating predictions for each frame through autoregression, the pseudo-floating-point format is restored by using the quantization features extracted from the second-order high-density continuous quantization region and the instantaneous dequantization based on the synchronously saved quantization parameters.

5. The method for processing large-model long texts based on mixed-precision third-order caching according to claim 4, characterized in that, The calculation formula for instantaneous inverse quantization is as follows: in, and These are the pseudo-floating-point format key matrix and value matrix after dequantization recovery, respectively.

6. The method for processing large-scale long text based on mixed-precision third-order caching according to claim 5, characterized in that, The pseudo-floating-point format key matrix and value matrix are assembled independently of the key state and value state based on the low-level memory concatenation operator of the deep learning framework. The concatenation formula is as follows: in, This represents the state of the first-order fidelity head anchor point region. The second-order region state is recovered by dequantization. This represents the third-order real-time fidelity tail region state. and To ensure strict alignment of the concatenated key and value matrices along the sequence dimension.

7. The method for processing large-scale long text based on mixed-precision third-order caching according to claim 1, characterized in that, The rotation position encoding operation at the bottom layer of the large language model only applies to the complete concatenated key matrix and the current query vector, injecting relative position information into them. Finally, the current query vector performs a dot product attention operation with the complete and continuous key matrix and value matrix.

8. The method for processing large-model long texts based on mixed-precision third-order caching according to claim 7, characterized in that, The calculation formula for the dot product attention operation is as follows: In the formula, The feature dimension of the attention head.

9. A large-model long text processing system based on mixed-precision third-order caching, employing the large-model long text processing method based on mixed-precision third-order caching as described in any one of claims 1 to 8, characterized in that, include: The data processing module is used to obtain the newly generated token, input the newly generated token into the pre-established large language model, and perform long text processing based on the historical key-value state space of the large language model. The historical key-value state space includes a first-order fidelity header anchor region, a second-order high-density continuous quantization region, and a third-order real-time fidelity tail region. The newly generated token is generated based on the set streaming long text generation task. The data optimization module is used to trigger a downgrade compression flow when the newly generated token causes the tail area of ​​the third-order real-time fidelity to overflow. The earliest token to enter the tail area of ​​the third-order real-time fidelity is removed and its key-value state is compressed into a low-bit format and stored in the tail of the second-order high-density continuous quantization area. When the number of tokens in the second-order high-density continuous quantization area reaches the preset limit, the earliest token to enter the second-order high-density continuous quantization area is eliminated. The prediction output module is used for processing based on the total space of historical key-value states. Before generating predictions for each frame through autoregression, it reconstructs the key matrix and value matrix, inputs the key matrix and value matrix into the large language model, and outputs the long text prediction processing results.

10. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, The memory stores a computer program that can run on the processor. When the processor loads and executes the computer program, it employs the large-model long text processing method based on a mixed-precision third-order cache as described in any one of claims 1 to 8.

Images