A method and system for long context management based on a large language model with multi-level memory consolidation
By employing a multi-level memory consolidation method, the problem of managing long contexts in large language models with limited GPU memory was solved, enabling differentiated storage and consolidation of semantic units, thereby improving information retention accuracy and GPU memory utilization efficiency.
Patent Information
- Application Number
- CN202611507202.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-28
- Publication Date
- 2026-11-17
AI Technical Summary
Existing large language models struggle to effectively manage long contexts under limited memory conditions, leading to information loss and low memory efficiency. Current technologies lack differentiated processing for assessing the importance of semantic units and consolidating memory.
A multi-level memory consolidation method is adopted, which uses techniques such as semantic block partitioning, importance scoring, hierarchical slot management and dormant cycle correction to achieve hierarchical storage and differentiated consolidation of input tokens, forming a hierarchical memory structure.
It improves the efficiency of video memory utilization, enhances the accuracy of information retention and memory capacity, ensures the long-term retention and recoverability of critical information, reduces information loss, and optimizes the allocation of computing resources.
Smart Images

Figure CN123117355A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of large language model technology, specifically to a method and system for managing long contexts of large language models based on multi-level memory consolidation. Background Technology
[0002] Large language models, when handling tasks such as long document understanding, multi-turn dialogue, and long-chain reasoning, need to access long input sequences during forward computation. Models based on self-attention mechanisms typically cache the key and value vectors of historical tokens in GPU memory and reuse this cache in subsequent generation processes. As the input sequence grows, the GPU memory occupied by this cache continuously increases, while GPU memory capacity is limited. Therefore, the length of the processed context is often limited by the cache capacity, rather than by the expressive power of the model itself. How to preserve and utilize long historical information under limited GPU memory conditions has become a problem that needs to be solved in the application of large language models.
[0003] To alleviate the aforementioned capacity bottleneck, common practices in existing technologies include using sliding window or hard eviction strategies. This involves evicting the oldest token by position after the cache is full, or directly discarding inputs exceeding the window length. These approaches manage tokens as units and evict tokens based on their position, essentially employing an all-or-nothing management method. Their drawbacks include the fact that evicted content often contains crucial reasoning chain information or factual knowledge, which, once deleted, cannot be accessed in subsequent generation, leading to information loss in cross-window tasks. Furthermore, because they fail to differentiate between various types of information within the context, short-term details and long-term semantic information are processed at the same storage level, making it difficult to form a hierarchical memory structure.
[0004] Another type of existing technology expands the effective memory range by introducing compressed representations. For example, using fast weights, linear attention mechanisms, or state vectors with cyclic structures in state-space models, historical information is compressed into a fixed-size representation and continuously updated during the generation process. This type of method alleviates cache capacity pressure to some extent, but its update process is often continuous and indiscriminate. That is, each new token affects memory updates with similar intensity, and important and redundant information are not differentiated during memory formation, affecting the representation efficiency of the compressed representation. Furthermore, since the size of the compressed representation is fixed, there is an upper limit to the amount of information it can retain. When the cached content is completely cleared, details in the original context can only be accessed indirectly through the compressed representation, easily leading to a decrease in access accuracy.
[0005] Another type of existing technology employs external memory or retrieval enhancement, writing historical content into external storage and retrieving it back to the model when needed. These methods typically use text blocks as retrieval units, and the retrieval results are not in the same representation space as the hidden state within the model. This requires additional transformation and concatenation mechanisms for computation, and the processes of writing, updating, and discarding information are independent of the model generation process, lacking a unified consolidation and verification mechanism. Regarding memory discarding and updating, existing technologies often discard based on position, writing time, or simple access frequency, lacking an assessment of the importance of semantic units. This easily leads to the consumption of storage and computational resources on irrelevant information, while key information fails to receive appropriate retention strength. Furthermore, there is a lack of a progressive information transfer mechanism between different levels of memory, the memory consolidation process is disconnected from the reasoning process, the consolidation results lack verifiable criteria, and the determination of consolidation strength and the number of consolidation rounds is difficult to coordinate with the other parameters of the model for optimization.
[0006] In summary, how to assess the importance of contextual information on a semantic block basis, how to store and consolidate memories in a hierarchical manner, how to form a closed loop that connects memory clearing, memory retrieval, and memory consolidation, and how to enable end-to-end training of the above processes are technical problems that still need to be solved in this field. Summary of the Invention
[0007] The purpose of this invention is to provide a method and system for managing long contexts of large language models based on multi-level memory consolidation, which can improve effective memory capacity and information retention accuracy.
[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A long context management method for a large language model based on multi-level memory consolidation, wherein the large language model includes multiple memory injection layers, the method comprising: Step 1: Obtain the input token sequence, divide the input token sequence into semantic blocks according to the neighbor similarity of the hidden state of the aggregation layer, calculate the importance score and aggregation key vector of each semantic block, and settle the importance score when the settlement condition is met; Step 2: Write the key vector and value vector of the input token sequence into the primary memory, determine the ownership tag of the semantic block according to the importance score, and write the semantic blocks that meet the ownership conditions into the secondary memory slots and update the slot quality according to the slot quality level. Step 3: When the occupancy rate of the primary memory reaches the primary consolidation trigger occupancy rate threshold, the target representation is read from the primary memory using the aggregated bond vector. The writing intensity and the number of dormancy cycles are determined according to the importance score. The target representation after error feedback correction is written into the secondary memory slot, and the relative reconstruction error is verified on the writing result. Step 4: When the cumulative number of slot writes reaches the cumulative number of slot writes threshold, divide the secondary memory slots with slot quality not lower than the critical slot quality threshold into semantic categories, calculate the class prototype value vector according to the semantic category, and write the class prototype value vector into the tertiary memory slot. Step 5: During periods when the computing resource utilization rate is continuously lower than the idle threshold or during session switching, perform value vector re-anchoring, key vector orthogonalization, and slot quality decay on the level 3 memory slots, and persistently store the level 3 memory slots to non-volatile storage media. Step 6: When the occupancy rate of the first-level memory reaches the first-level consolidation trigger occupancy rate threshold, the semantic blocks that have passed the verification are removed from the first-level memory in ascending order of importance score. During the prediction phase, the result vectors are read from the first-level memory, second-level memory and third-level memory respectively and fused with confidence weight. The fused read vector is injected into the memory injection layer and the prediction distribution of the next token is output.
[0009] Further, in step 1, the input token sequence is divided into semantic blocks according to the neighbor similarity of the hidden states of the aggregation layer. The importance score and aggregation key vector of each semantic block are calculated, and the importance score is calculated when the settlement conditions are met. Specifically: The cosine similarity between two adjacent token positions in the hidden state of the aggregation layer is calculated sequentially. When the cosine similarity is less than the semantic boundary judgment threshold, a semantic boundary is set between the two adjacent token positions. At the same time as setting the semantic boundary, upper and lower bound constraints on block length are applied. The aggregation layer is a layer in the large language model that does not belong to the memory injection layer. The set of token positions obtained by the semantic boundary segmentation constitutes a semantic block. The importance score is obtained by multiplying the task relevance sub-score, novelty sub-score, inference chain criticality sub-score, and repetition sub-score by their respective normalized weights and then summing them. The task relevance sub-score represents the degree of relevance between the semantic block and the user instruction. The novelty sub-score represents the degree of unexpectedness of the semantic block relative to the existing information in the session. The inference chain criticality sub-score represents the intensity of attention the semantic block receives in subsequent generation. The repetition sub-score represents the degree of overlap between the semantic block and the existing slot key vector in secondary memory. All four sub-scores are in the range of 0 to 1. The key vectors of each token in the semantic block at the aggregation layer are weighted and summed according to the aggregation weight. The weighted sum is then reduced in dimension and normalized to obtain the aggregated key vector. The aggregation weight is obtained by normalizing the hidden state of each token in the semantic block at the aggregation layer. The length of the normalized aggregated key vector is 1. Set the settlement conditions as follows: the number of subsequent tokens of the semantic block reaches the length of the attention settlement window; the number of subsequent tokens of the semantic block reaches the minimum settlement window and the occupancy rate of the first-level memory reaches the threshold of the first-level consolidation trigger occupancy rate; a session switching event is detected; wherein the minimum settlement window is less than the length of the attention settlement window.
[0010] Further, in step 2, the key vector and value vector of the input token sequence are written into the primary memory, the ownership tag of the semantic block is determined according to the importance score, and the semantic blocks that meet the ownership conditions are written into the secondary memory slots and the slot quality is updated according to the slot quality classification. Specifically: Write the key vectors and value vectors of all token positions in the input token sequence into the first-level memory, where the writing is not affected by the importance score; The attribution tags are determined as high-importance semantic blocks, medium-importance semantic blocks, or low-importance semantic blocks according to the value range of the importance score. The high-importance semantic blocks enter the writing process of the secondary memory slot after the importance score is calculated. The medium-importance semantic blocks and low-importance semantic blocks enter the writing process of the secondary memory slot in batches during step 3. The three attribution tags are mutually exclusive. The set of writable slots for a semantic block is determined based on the condition that the slot quality is not higher than the importance score of the semantic block. The writable slots are selected from the set of writable slots in descending order of inner product. The inner product is the inner product of the aggregation key vector of the semantic block and the key vector of the secondary memory slot. When the set of writable slots is empty, the secondary memory slot with the smallest slot quality is released and used as the writing object. The key vector written to the slot is interpolated to the aggregate key vector of the semantic block and then normalized. The value vector written to the slot is interpolated to the target representation of the semantic block obtained in step 3. The slot quality of the slot is updated according to the importance score. The updated slot quality is monotonically non-decreasing and uses the importance score of the semantic block as the asymptotic value. The condition for writing a secondary memory slot to a tertiary memory slot in step 4 is that the slot quality of the secondary memory slot is not lower than the critical slot quality threshold.
[0011] Furthermore, in step 3, when the occupancy rate of the primary memory reaches the primary consolidation trigger occupancy rate threshold, the target representation is read from the primary memory using the aggregated bond vector. The writing intensity and the number of dormant cycles are determined according to the importance score. The target representation, corrected by error feedback, is written into the secondary memory slot, and the relative reconstruction error is verified on the writing result. Specifically: Using the aggregated key vector of the semantic block as the query, the normalized exponential operation is performed on the key vector of each token in the aggregate layer within the semantic block to obtain the readout weight. The value vectors of each token in the aggregate layer within the semantic block are weighted and summed according to the readout weight, and then the dimension is reduced to obtain the target representation. The hibernation cycle is executed round by round according to the number of hibernation cycles. In each hibernation cycle, a reconstruction read, a residual calculation, an error feedback correction, and a slot write are executed sequentially. The result of the reconstruction read is the result vector read from the secondary memory with the aggregation key vector of the semantic block as the query. The result of the residual calculation is the difference between the target representation and the result vector read from the reconstruction. The error feedback correction is to reduce the dimension of the residual vector and then increase the dimension to obtain the correction direction vector and add it to the target representation according to the correction amount. The slot write is to write the corrected target representation into the secondary memory slot according to the write intensity. The correction ratio is the smaller of the ratio of the norm of the target representation to the norm of the correction direction vector and the upper limit of the error feedback amplitude. The upper limit of the error feedback amplitude is the maximum multiple of the single-round correction amount relative to the norm of the target representation. The correction ratio is no greater than 1 so that the dormant loop is a compressed mapping. The write intensity is taken as the interpolation between the lower bound of the write intensity of the memory level and the upper bound of the write intensity of the memory level. The interpolation ratio is determined by the power of the importance response index of the importance score and the round decay factor. The write intensity remains monotonically constant with the importance score and decreases with the number of sleep cycle rounds. The upper bound of the write intensity is set to make the error compression factor of the sleep cycle positive. The importance response index of the memory level with the larger memory level number is greater than the importance response index of the memory level with the smaller memory level number. The number of dormant cycle rounds is taken as the interpolation between the lower bound of the number of dormant cycle rounds for that level of memory and the upper bound of the number of dormant cycle rounds for that level of memory, and then rounded up. The interpolation ratio is the importance score. The lower bound and upper bound of the number of dormant cycle rounds for each of the first-level memory consolidation, second-level memory consolidation and third-level memory consolidation increase sequentially and do not overlap. After each dormancy cycle, the relative reconstruction error is determined by the ratio of the norm of the difference between the target representation and the next level memory readout result vector to the norm of the target representation. The judgment threshold of the semantic block in the memory consolidation of that level is determined by the interpolation between the upper bound and the lower bound of the memory consolidation judgment threshold of that level. The interpolation ratio is the importance score, and the judgment threshold decreases as the importance score increases. When the relative reconstruction error is not greater than the determination threshold, the dormancy loop of the semantic block is terminated and the consolidation completion flag is set to consolidated; When the number of sleep cycles reaches the number of sleep cycle rounds and the relative reconstruction error is greater than the judgment threshold, the number of sleep cycle rounds is added by the increment of the number of supplementary sleep cycles. In the sleep cycle that is added, the round decay factor is set to 1. The number of additional executions does not exceed the upper limit of the number of supplementary sleep cycles. If the judgment threshold is still not met after the additional execution is completed, the consolidation completion flag remains as unconsolidated. The number of token positions provided by semantic blocks marked as consolidated in the primary memory is counted to obtain the number of token positions that can be cleared. When the number of token positions that can be cleared after a number of consecutive failure consolidation cycles is less than the number of token positions that need to be released in one clearing, the write strength of semantic blocks marked as not consolidated is taken as the upper bound of the write strength of that level of memory, and a sleep cycle with a forced consolidation round is executed with the round decay factor set to 1.
[0012] Further, in step 4, when the cumulative number of slot writes reaches the cumulative number of slot writes threshold, the secondary memory slots with slot quality not lower than the critical slot quality threshold are classified into semantic categories, class prototype value vectors are calculated according to semantic categories, and the class prototype value vectors are written into the tertiary memory slots, specifically as follows: After each primary memory consolidation is completed, the number of slots actually written to the secondary memory slots during this primary memory consolidation is added to the cumulative slot write count. When the cumulative slot write count reaches the cumulative slot write count threshold, secondary memory consolidation is triggered, and the cumulative slot write count is set to zero. For each secondary memory slot whose quality is not lower than the key slot quality threshold, calculate the inner product of the key vector of the secondary memory slot and the key vector of each tertiary memory slot. Take the tertiary memory slot with the largest inner product as the semantic category to which the secondary memory slot belongs. When the largest inner product is less than the slot creation similarity threshold and there are unoccupied slots in the tertiary memory, create a new slot in the tertiary memory and take the key vector, value vector and slot quality of the new slot as the key vector, value vector and slot quality of the secondary memory slot, respectively. Secondary memory slots whose quality is lower than the key slot quality threshold do not participate in step 4. The class prototype value vector is obtained by weighting the value vectors of all second-level memory slots under the same semantic category according to the slot quality of the second-level memory slots. The class prototype key vector is obtained by weighting and combining the key vectors of all second-level memory slots under the same semantic category according to the slot quality of the second-level memory slots and then normalizing them. Using class prototype key vectors as queries and class prototype value vectors as target representations, the hibernation cycle is executed on the tertiary memory slots according to the write intensity of the secondary memory consolidation and the number of hibernation cycle rounds, and verification is performed according to the relative reconstruction error. The judgment threshold of the secondary memory consolidation is greater than the judgment threshold of the primary memory consolidation, and the total number of slots of the tertiary memory slots is less than the dimension of the slot vector, so that the tertiary memory slots retain the semantic category skeleton. When all three-level memory slots are occupied and a new slot needs to be created, the two slots with the largest inner product among the occupied three-level memory slots are identified as slot pairs to be merged. When the inner product of the slot pairs to be merged is not less than the slot merging similarity threshold, the key vector of one slot in the pair is taken as the result of the normalized combination of the slot quality of the two slot key vectors, the value vector of the slot is taken as the slot quality weighted average of the two slot value vectors, the slot quality of the slot is taken as the smaller value between the sum of the two slot qualities and 1, and the key vector, value vector and slot quality of the other slot in the pair are set to zero. When the inner product of the slot pairs to be merged is less than the slot merging similarity threshold, the current slot creation request is postponed to the next second-level memory consolidation execution.
[0013] Furthermore, in step 5, during periods when the computing resource utilization rate is continuously below the idle threshold or at the time of session switching, value vector re-anchoring, key vector orthogonalization, and slot quality decay are performed on the level 3 memory slots, and the level 3 memory slots are persistently stored to non-volatile storage media, specifically: The computing resource utilization rate is taken as the graphics processor memory bandwidth utilization rate. When the continuous idle determination time of the memory bandwidth utilization rate is lower than the idle determination threshold, or when a session switching event is detected, the third-level memory consolidation is triggered. The execution object of the third-level memory consolidation is all occupied third-level memory slots. Using the class prototype key vector as a query, the anchor representation is read from the secondary memory, and the value vector of the tertiary memory slot is interpolated and updated to the anchor representation according to the writing intensity of the tertiary memory consolidation. The anchor representation is the current central representation of the semantic category in the secondary memory. The correction amount is determined by the inner product of the tertiary memory slot key vector and the key vectors of the other occupied tertiary memory slots, the slot quality of the other tertiary memory slots, and the orthogonalization step size. The correction amount is subtracted from the tertiary memory slot key vector and then normalized. The orthogonalization step size is less than 1 to orthogonalize the key vectors into a compressed mapping. The orthogonalization of the key vectors causes the key vectors of the tertiary memory slots to converge to an approximately orthogonal configuration. Multiply the slot quality of the unhit tertiary memory slots in this round by the slot quality decay coefficient. Increase the slot quality of the hit tertiary memory slots in this round by the slot quality hit increment, not exceeding 1. The hit determination condition is that the tertiary memory slot participates in the value vector re-anchoring write in this round of dormancy, or the read weight of the tertiary memory slot in tertiary memory read is greater than the slot hit weight threshold. When the slot quality of the tertiary memory slot decays to below the slot release quality threshold, the key vector, value vector and slot quality of the tertiary memory slot are not released and continue to participate in the key vector orthogonalization and slot merging. After the Level 3 memory consolidation is completed, the key vectors, value vectors, slot quality, class prototype key vectors, and class prototype value vectors of all Level 3 memory slots are written to the non-volatile storage medium. At the start of the session, the slot contents stored in the non-volatile storage medium are read and loaded into the Level 3 memory slots. The loaded slot key vectors and the newly generated slot key vectors in this session are weighted and summed according to the session load weight, and then normalized. The session load weight is the weight of the newly generated slot key vectors in this session in the weighted sum. The session load weight is less than 0.5, so that the historical persistence representation has a major weight in the weighted sum.
[0014] Further, in step 6, when the occupancy rate of the first-level memory reaches the first-level consolidation trigger occupancy rate threshold, the validated semantic blocks are removed from the first-level memory in ascending order of importance score. During the prediction phase, result vectors are read from the first-level memory, second-level memory, and third-level memory respectively, and fused with confidence weights. The fused read vector is then injected into the memory injection layer, and the prediction distribution of the next token is output, specifically as follows: Semantic blocks that have passed verification are selected from the primary memory, marked as having been consolidated and whose relative reconstruction error is no greater than the judgment threshold. They are then sorted by importance score from smallest to largest and removed sequentially. Removal involves deleting the key vector and value vector of the token position in the memory injection layer and aggregation layer of the semantic block. Removal stops when the cumulative number of removed token positions reaches the number of token positions required to be released in one clearing. If the number of token positions provided by the verified semantic blocks is less than the number of token positions required to be released in one clearing, the actual number of token positions are removed without removing unverified semantic blocks. Removal does not change the content of the secondary and tertiary memory slots, so that the information of the removed semantic blocks can still be read from the secondary and tertiary memories. After each semantic block is removed, the occupancy rate of the first-level memory is recalculated based on the ratio of the number of token positions currently residing in the first-level memory to the capacity of the first-level memory. The removal is terminated when the recalculated occupancy rate drops below the first-level consolidation trigger occupancy rate threshold. In each memory injection layer, the output of the self-attention sublayer of that memory injection layer is used as the basis for the query vector. The slot value vectors are read from the second-level memory and the third-level memory respectively according to the slot weights and weighted summation to obtain the result vector. The query vector is read from the first-level memory according to the normalized exponential weight of the key vector of each token in the aggregation layer in the semantic block and weighted summation of the value vector of each token in the aggregation layer in the semantic block, and then dimension reduction is performed to obtain the result vector. The query vector is obtained by dimension reduction and then dimension increase and normalization of the output of the self-attention sublayer. The retrieval temperature coefficient of the third-level memory is smaller than that of the second-level memory. The readout vectors of the three memory levels are weighted and summed after dimensionality increase according to the confidence weights to obtain the fused readout vector. The sum of the three confidence weights is 1. The confidence weights are jointly determined by the cosine similarity between the output of the self-attention sublayer and the dimensionality increase readout vector, as well as the hierarchical quality index of the memory level. The hierarchical quality index of the first-level memory is the average importance score of the settled semantic blocks in the first-level memory, and the hierarchical quality index of the second-level and third-level memories is the average quality of all slots in each memory level. The fused readout vector is normalized and transformed by layer, and then superimposed on the output of the self-attention sub-layer of the memory injection layer according to the injection gating coefficient. The result is input into the feedforward sub-layer of the memory injection layer. The output of the last layer of the large language model is projected by the output projection matrix and then the prediction distribution of the next token is obtained by normalization exponent operation. The initial value of the injection gating coefficient is 0, so that the memory injection layer is an identity mapping at the beginning of training. After each primary memory consolidation is completed, the slot quality of the secondary memory slots that were not hit in this round is multiplied by the slot quality decay coefficient. When the slot quality of the secondary memory slot is less than the slot release quality threshold, the key vector, value vector and slot quality of the secondary memory slot are set to zero and the slot is converted to an unoccupied state. Among them, the secondary memory slots that participated in the primary memory consolidation writing in this round are judged as hits and no slot quality decay is performed.
[0015] This invention also provides a long context management system for large language models based on multi-level memory consolidation, which is applied to the above-mentioned long context management method for large language models based on multi-level memory consolidation. It includes: a context receiving and preprocessing module, a three-level memory storage module, a hierarchical dormancy consolidation module, a memory retrieval and fusion module, an intelligent clearing strategy module, and an end-to-end training module. The context receiving and preprocessing module is used to obtain the input token sequence, divide the input token sequence into semantic blocks according to the neighbor similarity of the hidden state of the aggregation layer, calculate the importance score and aggregation key vector of each semantic block, and settle the importance score when the settlement conditions are met. The three-level memory storage module is used to write the key vector and value vector of the input token sequence into the first-level memory, determine the ownership tag of the semantic block according to the importance score, and write the semantic block that meets the ownership condition into the second-level memory slot and update the slot quality according to the slot quality level addressing. It is also used to store the second-level memory slot and the third-level memory slot and persistently store the third-level memory slot to a non-volatile storage medium. The hierarchical dormancy consolidation module is used to read the target representation from the primary memory using aggregated key vectors when the occupancy rate of the primary memory reaches the primary consolidation trigger occupancy rate threshold. It then determines the write intensity and number of dormancy cycles based on importance scores, writes the target representation corrected by error feedback into the secondary memory slot, and performs relative reconstruction error verification on the write results. Furthermore, when the cumulative number of slot writes reaches the cumulative number of slot writes threshold, it classifies secondary memory slots with slot quality not lower than the critical slot quality threshold into semantic categories, calculates class prototype value vectors according to semantic categories, and writes the class prototype value vectors into the tertiary memory slots. Finally, it performs value vector re-anchoring, key vector orthogonalization, and slot quality decay on the tertiary memory slots during periods when the computing resource occupancy rate is continuously lower than the idle determination threshold or during session switching. The intelligent clearing strategy module is used to remove verified semantic blocks from the first-level memory in ascending order of importance score when the occupancy rate of the first-level memory reaches the first-level consolidation trigger occupancy rate threshold. The memory retrieval and fusion module is used to read the result vectors from the first-level memory, second-level memory and third-level memory respectively in the prediction stage, and fuse them with confidence weight. After injecting the fused read vector into the memory injection layer, the prediction distribution of the next token is output. The end-to-end training module is connected to the context receiving and preprocessing module, the three-level memory storage module, the hierarchical dormancy consolidation module, the memory retrieval and fusion module, and the intelligent clearing strategy module, respectively, and is used to sequentially execute four training phases on the large language model, all memory slots, and the parameters on which the importance score is based.
[0016] In summary, the present invention has at least one of the following beneficial technical effects: Firstly, this invention uses semantic blocks rather than individual tokens as the basic unit of memory management, and determines the semantic boundary according to the adjacent similarity of the hidden state of the aggregation layer and the block length constraint. This allows the writing, consolidation, elimination and retrieval of memory to be carried out on semantically complete units, avoiding the situation of semantic fragmentation caused by segmentation by fixed windows, and also making the granularity of memory correspond to the granularity of semantics.
[0017] Secondly, this invention synthesizes an importance score by combining task relevance sub-scores, novelty sub-scores, reasoning chain criticality sub-scores, and repetition sub-scores. This allows the importance assessment to simultaneously consider the relevance of information to the current task, the unexpectedness of information relative to existing content in the conversation, the intensity of attention the information receives in subsequent generation, and the degree of overlap between the information and existing representations in secondary memory. As a result, the importance score can more comprehensively reflect the actual role of semantic blocks in long contexts.
[0018] Third, this invention determines the attribution marker of semantic blocks based on importance scores and determines the writing slots of secondary memory based on slot quality grading. Only slots with a quality no higher than the importance score of the semantic blocks can be written, thus forming a replacement order from low to high importance. This makes lower importance content tend to replace existing representations of equal or lower importance, and makes it more difficult to cover representations established by higher importance content. Combined with the monotonic update method of slot quality, the slot quality can gradually converge to the importance level of the content carried by the slot as it is written, thus simultaneously undertaking the functions of memory strength measurement and importance screening.
[0019] Fourth, in the consolidation of primary memory, the present invention reads the target representation from primary memory using aggregated bond vectors and introduces error feedback correction in the dormant cycle. That is, the written target is corrected along the residual direction between the target representation and the reconstruction result of secondary memory, and the correction amount is limited to a preset multiple of the norm of the target representation. This keeps the error compression factor of the dormant cycle positive and forms a compression mapping, thereby gradually approaching the high-precision content representation in primary memory in terms of compressed representation.
[0020] Fifth, this invention determines the write intensity and the number of sleep cycle rounds based on importance scores, and sets an upper bound on the write intensity to keep the sleep cycle process convergent, and allows semantic blocks with higher importance to obtain higher write intensity and more sleep cycle rounds, thereby realizing the differentiated allocation of computing resources according to the importance of information; since the write intensity has a lower bound, semantic blocks with lower importance can also obtain a write intensity sufficient for convergence, thereby avoiding the situation where low-importance semantic blocks cannot pass verification for a long time and primary memory is difficult to release.
[0021] Sixth, this invention uses relative reconstruction error as the criterion for consolidation verification, and the judgment threshold adapts to the importance score, so that semantic blocks with higher importance need to achieve higher reconstruction accuracy, while semantic blocks with lower importance can use a relatively broad criterion; at the same time, it sets up supplementary dormant cycles and fallback forced consolidation, so that the consolidation process is reachable within a given round budget, thereby ensuring the quality of consolidation while avoiding long-term occupation of primary memory without release.
[0022] Seventh, in the consolidation of secondary memory, this invention filters secondary memory slots based on slot quality and classifies them into semantic categories. The weighted average of the slot quality of the value vectors of each member in the same category is used as the class prototype value vector and written into the tertiary memory slot. Since the total number of slots in the tertiary memory is less than the dimension of the slot vector, the subspace spanned by its slot key vector set has a limited dimension. Using a single high-precision representation as the consolidation target makes it difficult to achieve high reconstruction accuracy in terms of representation capability. By using the class prototype as the consolidation target, the consolidation target itself is located in the central direction of the space spanned by the representations of each member, so that the tertiary memory retains the semantic skeleton of each category rather than the details of each member, which is compatible with the high compression rate setting of the tertiary memory.
[0023] Eighth, this invention performs value vector re-anchoring, key vector orthogonalization, and slot quality decay in the third-level memory consolidation process. This aligns the value vectors of the third-level memory slots with the newer class prototypes in the second-level memory during idle periods, and causes the key vectors of the third-level memory slots to converge to an approximately orthogonal configuration to improve their coverage efficiency for the target representation. It also causes the quality of slots that have not been hit for a long time to decrease by a decay coefficient, while the quality of hit slots is improved, thereby achieving slot differentiation and effective coverage. When the quality of the third-level memory slots decays to a low level, they are not released, so that cross-session knowledge can be preserved.
[0024] Ninth, this invention writes the key vector, value vector, slot quality and class prototype of the three-level memory into a non-volatile storage medium, and after loading at the beginning of the session, it merges with the slot key vector newly generated in this session according to the session load weight, so that the model can accumulate memory across multiple sessions, thereby realizing cross-session knowledge retention and accumulation.
[0025] Tenth, in the prediction stage, the present invention reads the result vectors from the first-level memory, second-level memory and third-level memory respectively, and injects them into the memory injection layer after weighted fusion according to the confidence weight determined by cosine similarity and hierarchical quality index. This allows memories with different compression levels to participate in the prediction as needed in different decoding steps. The first-level memory provides near-end high-precision content, while the second-level memory and third-level memory provide compressed mid-to-far-end content, thereby expanding the range of contexts that can participate in the prediction under limited video memory conditions.
[0026] Eleventh, when clearing primary memory, this invention removes verified semantic blocks in ascending order of importance score, and removes them one by one at the semantic block granularity, terminating the removal when the occupancy rate drops below the threshold. This ensures that the clearing process targets content already carried by secondary and tertiary memory, and the information of the removed semantic blocks can still be read from downstream memory, thereby reducing the irrecoverable loss of information caused by clearing. Furthermore, it enables primary memory to form a sliding window-like gradual clearing during continuous generation, avoiding performance fluctuations caused by a one-time overall clear.
[0027] Twelfthly, the present invention performs slot quality decay on slots in secondary memory that have not been hit for a long time and releases the slots when the quality falls below the release threshold, so that the slot resources can be reoccupied by subsequent semantic content, thereby realizing the reclamation of storage space of long-term unaccessed memory.
[0028] Thirteenth, this invention enables end-to-end training to collaboratively optimize the parameters used for importance scoring, the parameters used for memory injection layers, the matrix used for error feedback correction, and the write strength and importance response index. It also uses state transcription and replay mechanisms to backpropagate the segmented consolidation process, ensuring consistency between the consolidation and retrieval processes in the training and inference phases. Furthermore, by zeroing the gradients for index-based operations such as slot addressing, slot merging, and slot release, the training process can proceed stably. Attached Figure Description
[0029] Figure 1 This is a schematic diagram of the method flow of the present invention. Detailed Implementation
[0030] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.
[0031] like Figure 1 As shown, this invention provides a long context management method for large language models based on multi-level memory consolidation, including: Step 1: Obtain the input token sequence, divide the input token sequence into semantic blocks according to the neighbor similarity of the hidden state of the aggregation layer, calculate the importance score and aggregation key vector of each semantic block, and settle the importance score when the settlement condition is met; Step 2: Write the key vector and value vector of the input token sequence into the primary memory, determine the ownership tag of the semantic block according to the importance score, and write the semantic blocks that meet the ownership conditions into the secondary memory slots and update the slot quality according to the slot quality level. Step 3: When the occupancy rate of the primary memory reaches the primary consolidation trigger occupancy rate threshold, the target representation is read from the primary memory using the aggregated bond vector. The writing intensity and the number of dormancy cycles are determined according to the importance score. The target representation after error feedback correction is written into the secondary memory slot, and the relative reconstruction error is verified on the writing result. Step 4: When the cumulative number of slot writes reaches the cumulative number of slot writes threshold, divide the secondary memory slots with slot quality not lower than the critical slot quality threshold into semantic categories, calculate the class prototype value vector according to the semantic category, and write the class prototype value vector into the tertiary memory slot. Step 5: During periods when the computing resource utilization rate is continuously lower than the idle threshold or during session switching, perform value vector re-anchoring, key vector orthogonalization, and slot quality decay on the level 3 memory slots, and persistently store the level 3 memory slots to non-volatile storage media. Step 6: When the occupancy rate of the first-level memory reaches the first-level consolidation trigger occupancy rate threshold, the semantic blocks that have passed the verification are removed from the first-level memory in ascending order of importance score. During the prediction phase, the result vectors are read from the first-level memory, second-level memory and third-level memory respectively and fused with confidence weight. The fused read vector is injected into the memory injection layer and the prediction distribution of the next token is output.
[0032] The following sections will provide a detailed explanation of each step.
[0033] In step 1, the input token sequence is obtained, and it is divided into semantic blocks according to the neighbor similarity of the hidden states of the aggregation layer. The importance score and aggregation key vector of each semantic block are calculated, and the importance score is calculated when the settlement conditions are met. Specifically: The input to this step is the hidden state and key-value vector generated by the backbone network for the current input. The output is a set of semantic blocks, a set of token positions for each semantic block, an importance score, and an aggregate key vector. The importance score is passed down to step 2 to determine the memory level and slot writing object, down to steps 3, 4, and 5 to determine the write intensity and the number of sleep cycle rounds, and down to step 6 to determine the clearing order. The aggregate key vector is passed down to step 2 for secondary memory slot addressing, down to steps 3 and 4 for read queries, and down to step 5 for orthogonalization of the slot key vector. Step 1 includes the following five parts: 1. Perform semantic segmentation on the input token sequence. The semantic block unit receives the input token sequence ,in This represents the total number of tokens in the current input sequence. For the first One token, This refers to the token's position sequence number. Semantic chunking units are hidden in the aggregation layer state. To determine semantic boundaries, For token position The output vector at layer 15 of the backbone network.
[0034] When two adjacent token positions satisfy the following formula, at token position and Set semantic boundaries between them: ; in, The semantic boundary determination threshold is set to 0.72. The input token sequence is segmented by semantic boundaries to form a set of semantic blocks. , Semantic block number, semantic block The set of token locations contained therein is denoted as .
[0035] Imposing block length constraints while setting semantic boundaries: When the number of tokens in a candidate semantic block reaches the upper bound of the block length. Forced semantic boundaries to be set at times The value is 128; when the number of tokens in the candidate semantic block does not reach the lower bound of the block length. Ignore the boundary determination result at that point, and continue to incorporate the candidate semantic block into subsequent tokens. The value is 32. Under the above threshold and block length constraints, the average number of tokens in a semantic block is denoted as . The value is 64.
[0036] 2. Calculate the four sub-scores The importance assessment unit calculates task relevance sub-scores, novelty sub-scores, inference chain criticality sub-scores, and repetition sub-scores for each semantic block.
[0037] (1) The task relevance sub-score is calculated using the following formula: ; in, For semantic blocks Task relevance sub-score; For semantic blocks The mean vector of each token in the hidden state of the last layer of the backbone network; The task query vector is obtained from the mean vector of the hidden states of each token in the user instruction segment of the input sequence. The user instruction segment is defined by the role marker token in the input sequence. The value of is between 0 and 1, with 0.5 for orthogonal and 1 for in the same direction.
[0038] (2) The novelty score is calculated using the following formula: ; in, For semantic blocks Novelty score; For semantic blocks The average negative log-likelihood of each token within the backbone network; This is the exponential moving average of the average negative log-likelihood within the current session. equal hour Take 0.5, equal When it is twice the value, take 1.
[0039] Take the calibration value at the start of the session. , The value of 2.50 is obtained from the calibration corpus during the first training phase; thereafter, it is updated once for each semantic block calculated using the following formula: ; in, This is the sliding update coefficient, with a value of 0.99.
[0040] (3) The criticality score of the reasoning chain is calculated according to the following formula: ; in, For semantic blocks The criticality score of the reasoning chain; For semantic blocks Each token will be used in the following... The average of the attention weights received at each token location; It is the moving average of the same amount for all settled semantic blocks in the current session.
[0041] For semantic blocks The number of available subsequent tokens is determined by the following formula: ; in, The length of the attention calculation window, with a value of 64; For semantic blocks The position number of the last token. Token position during the autoregressive generation process. to The attention weights for each token within a semantic block are calculated and stored in the attention matrix when these tokens are generated. It can be obtained without additional forward computation. When it is 0, Pick ,at this time Take 0.5.
[0042] Take the calibration value at the start of the session. , The values were obtained from the calibration corpus during the training phase, and were taken as 5 × 10. -4 Subsequently, each semantic block is settled by AND. Update once using the same format.
[0043] (4) The repeatability sub-score is calculated using the following formula: ; in, For semantic blocks Repeatability sub-score; For the key vector and aggregate key vector in secondary memory The similarity is not lower than the slot hit similarity threshold. The number of slots, The value is 0.75; The value is 4, which serves as the benchmark for repeatability normalization. A score of 0 indicates that the semantic content has no corresponding representation in the established memory slots, and the sub-score is 0 in this case; An increase indicates that the semantic content has been represented, and the sub-score then approaches 1.
[0044] 3. Composite Importance Score The importance score is composed of four sub-scores as follows: ; in, For semantic blocks Importance rating; For sub-scoring dimension numbers, to The four sub-scores are listed above in order; For the first Unnormalized weight parameters; For the first Normalized weights, satisfying The sum is 1. Since all sub-ratings are in the range of 0 to 1 and the sum of the normalized weights is 1, It is located in the range of 0 to 1.
[0045] The scores are learned in training phase one. The initial values of the four sub-scores are all set to a normalized weight of 0.25 before training phase one, and are determined by the learned values after training phase one.
[0046] 4. Calculate the aggregate bond vector semantic blocks The aggregate bond vector is calculated using the following formula: ; in, For semantic blocks The aggregated key vector, with dimension ; The dimension reduction matrix of the aggregation bond has the following shape: OK List; For token position Normalized aggregate weights within semantic blocks; This is the aggregated weight vector, with dimension 1. ; The token position in the 15th layer attention module The key vector obtained by concatenating all key headers has a dimension of . ; For a normalized function, it is defined as follows: ,in This is a numerically stable term, taking a value of 10^-8. Normalization makes... The length of is 1, which is consistent with the length constraint of the key vectors of the second-level memory and the third-level memory slots. Therefore, the inner product between the two is the cosine similarity.
[0047] The reason for choosing the 15th layer as the aggregation layer instead of the memory injection layer is that the output of the 15th layer does not include the memory injection amount, so that the aggregation key vector is determined only by the backbone network itself, avoiding the interdependence between the aggregation key vector and the memory readout result.
[0048] 5. Determine the settlement time for importance scores. The importance score in the inference chain depends on the attention weights of the 64 tokens following the end of the semantic block; therefore, the calculation of the importance score requires waiting for the generation of these 64 tokens. Semantic block Importance scores are calculated when any of the following conditions are met: (1) The number of subsequent tokens in this semantic block reaches ; (2) The number of subsequent tokens in this semantic block reaches the minimum settlement window. Furthermore, the occupancy rate of Level 1 memory has reached the Level 1 consolidation trigger occupancy rate threshold. The value is 16; (3) A session switching event was detected.
[0049] Before settlement, semantic blocks The prewritten tags are marked as pending, and they do not participate in the memory hierarchy allocation in step 2, the first-level memory consolidation in step 3, or the selective clearing and sorting in step 6. After settlement, the semantic block The pre-written markers are converted to settled.
[0050] When the occupancy rate of Level 1 memory reaches the Level 1 consolidation trigger occupancy rate threshold, it corresponds to the 7373rd token position. The settlement waiting window is 64 tokens. Therefore, in actual operation, all semantic blocks participating in Level 1 memory consolidation have completed settlement.
[0051] In step 2, the key vector and value vector of the input token sequence are written into the primary memory. The ownership tag of the semantic block is determined according to the importance score. Then, the semantic blocks that meet the ownership criteria are written into the secondary memory slots and the slot quality is updated according to the slot quality classification. Specifically: The inputs to this step are the semantic block set output from step 1, the token location set for each semantic block, the importance score, the aggregate key vector and the write-ahead tag, and the current occupancy rate of the first-level memory. The outputs of this step are the updated first-level memory, the updated second-level memory slot key vector, slot value vector and slot quality, and the write condition determination result for the third-level memory. The updated first-level memory is passed down to steps 3 and 6, the updated second-level memory slots are passed down to steps 3 and 4, and the write condition for the third-level memory is passed down to step 4.
[0052] Step 2 includes the following five parts.
[0053] 1. Write the input token sequence into the first-level memory. The first-level memory consists of a key-value cache, containing the key and value vectors generated by each attention module in the memory injection layer set for all resident token positions. Layer, Level 1 memory The layer part is denoted as Its elements are and , respectively representing the first Token position in layer attention module The key vector and value vector, obtained by concatenating all key headers, both have a dimension of 1. .
[0054] All token positions in the input token sequence received by the token input unit are written into the first-level memory, and this writing is not affected by the importance score. The reason for not setting an importance threshold here is that the key-value vector in the first-level memory is the basis for all attention readings and all memory readings, and the absence of any token position will change the attention result.
[0055] The set of currently residing token locations in primary memory is denoted as Before outputting the predicted distribution of the next token in each prediction step, the key vector and value vector of the current token have already been written into the first-level memory. Each prediction step is non-empty.
[0056] The set of semantic blocks in primary memory that have completed importance scoring is denoted as The number of its semantic blocks is denoted as ,according to The calculation is performed, and the value is 128. The aggregated key vectors of all written semantic blocks residing in secondary memory are used as key vectors for secondary memory slots in all subsequent calculations.
[0057] 2. Memory hierarchy classification of semantic blocks The importance assessment unit categorizes semantic blocks into memory hierarchies based on their importance scores. The categorization marker has three possible values, and the rules for classification are as follows: (1) When At that time, semantic block The attribution marker is set as a high-importance semantic block. The threshold for determining high importance is set at 0.75. (2) When At that time, semantic block The attribution marker is a medium-importance semantic block. The threshold for determining low importance is set to 0.35. (3) When At that time, semantic block The attribution marker is a low-importance semantic block.
[0058] The purpose of the attribution markers is as follows: High-importance semantic blocks immediately enter the secondary memory slot writing process after settlement; medium-importance and low-importance semantic blocks are batched into the secondary memory slot writing process along with the semantic block set during primary memory consolidation. The three attribution markers are mutually exclusive, and only one attribution marker is assigned to the same semantic block at any given time. The attribution markers do not change the residency status of the semantic block in primary memory; all semantic blocks with all three attribution markers reside in primary memory until the selective clearing in step 6.
[0059] 3. Addressing and writing to slots in secondary memory. Secondary memory is Composed of slots, the first Each slot is determined by the key vector. Value vector With the quality of the slot constitute, This refers to the slot number. and All dimensions are , The length is 1. The value ranges from 0 to 1. (Slot quality) The physical meaning of is the importance level of the semantic content carried in the slot, and its value update method is given in Part 4. During initialization, the key vector and value vector of all slots are set to zero vectors, and the slot quality is set to 0, that is, all slots are in an unoccupied state.
[0060] The read operator for second-order memory is defined as follows: ; in, Readout operator for secondary memory; This is a query vector with a length of one unit. This is the retrieval temperature coefficient for secondary memory, with a value of 0.05. Indicates the slot number Perform normalized exponentiation on the indexed set.
[0061] Slot addressing is performed based on slot quality classification. Semantic block. The set of writable slots in secondary memory is denoted as . Determine using the following formula: ; in, For semantic blocks The set of writable slots refers to slots whose quality is no higher than the importance score of the semantic block. Unoccupied slots have a quality of 0, satisfying the condition... Therefore, all unoccupied slots are in the set of writable slots.
[0062] semantic blocks Write slot index set Determine by the following formula: ; in, This represents the number of slots for writing to secondary memory, with a value of 8. The number of elements is denoted as ; This represents the actual number of slots written to. For semantic blocks The aggregated bond vector. When When not empty, For the set of writable slots and The largest inner product Each slot has a computational complexity of O(n). .
[0063] when When the set is empty, the slot quality of all slots in the secondary memory is higher than the importance score of the semantic block. At this time, the slot with the lowest slot quality is released and used as the writing object. The slot release operation is the same as the slot release in the fifth part of step 6. After the release is completed, the slot quality of the slot is 0, and the key vector and value vector are zero vectors.
[0064] The addressing method based on slot quality classification serves the following purpose: low-importance semantic blocks can only be written to slots whose slot quality is no higher than their own importance score, thus avoiding overwriting slots created by higher-importance semantic content; high-importance semantic blocks can be written to all slots whose slot quality is no higher than their own importance score, including slots occupied by medium- and low-importance content, thereby achieving a slot replacement order from low to high importance.
[0065] The secondary memory slot key vector is formed as follows: the key vector of a slot is interpolated with the aggregated key vector of the semantic block during each write operation. Therefore, after semantic blocks with similar semantic content are repeatedly written to the same batch of slots, the key vectors of these slots converge to the central direction of that type of semantic content. For 512, Under the condition of 512, the expected absolute value of the inner product between random unit key vectors is approximately 0.044, while the inner product of aggregated key vectors within the same semantic category is no less than 0.60 under the ranking loss constraint of training phase one. Therefore, the addressing method of selecting the write slot based on the inner product can stably select slots of the same semantic category. (Retrieval temperature coefficient) Setting it to 0.05 concentrates the read weights on the slots with the largest inner product, while also considering the semantic blocks within the same semantic block. Each write slot is written with the same target representation, and the read weight is as described above. Changes in the distribution of slots do not alter the readout results.
[0066] 4. Perform secondary memory slot write and update slot quality. After selecting the write object, update the slot key vector and slot value vector using the following formula: ; ; in, For semantic blocks In the second level of memory Write intensity of the sleep cycle, This is the sequence number of the sleep cycle, and its calculation method is given in the fourth part of step 3; For semantic blocks The target representation in primary memory consolidation is calculated as given in Part 2 of Step 3. The slot bond vector, after normalization, still has a length of 1, consistent with the length constraint of the aggregate bond vector.
[0067] Slot quality is updated using the following formula: ; in, A coefficient of 0.5 is set for the quality of the slot. The positive part operation is performed. The asymptotic value of the updated expression is... That is, the slot's importance score after repeated reception is not higher than When the semantic block is defined, its slot quality converges to The positive part operation in the update formula ensures that slot quality is monotonically non-decreasing. That is, when the slot quality is already higher than the importance score of the currently written semantic block, the slot quality does not change, thus preventing low-importance writes from lowering the slot quality established by high-importance content. This is because the addressing rules are already limited. The positive part operation does not function under normal circumstances. Its function is to ensure the monotonicity of slot quality in abnormal situations such as using slot release as the writing object.
[0068] Taking a highly important semantic block as an example, its importance score is 0.90. If the importance score is 0.5, the slot quality is 0.405 after the first write, 0.628 after the second write, and 0.750 after the third write. After the third write, it exceeds the critical slot quality threshold of 0.65. Taking a semantic block with an importance score of 0.40 as an example, its asymptotic value for slot quality is 0.40, and it will not exceed 0.65 regardless of the number of writes. This property allows slot quality to simultaneously serve as both a memory strength measure and an importance filter.
[0069] 5. Determine the writing conditions and storage distribution of the three-level memory. Level 3 memory is Composed of slots, the first Each slot is determined by the key vector. Value vector With the quality of the slot constitute, This refers to the level 3 memory slot number. The level 3 memory read operator. Press and The same formula is defined, where the set of slot numbers is... Replace with The temperature coefficient is retrieved from Replace with , The value is 0.03.
[0070] The writing conditions for Level 3 memory are: the slot quality of the Level 2 memory slots is not lower than the critical slot quality threshold. , The value is 0.65. The writing of tertiary memory is not performed in the first-level memory consolidation. Instead, it is performed according to semantic category in the second-level memory consolidation in step 4. The calculation method for the writing object and the writing content is given in step 4.
[0071] Level 1 and Level 2 memory reside in the graphics processor's video memory. Level 2 memory is released at the end of the session; Level 3 memory is written to non-volatile storage at the end of the session and loaded at the start of the next session. Storage overhead is as follows: the key-value storage size for a single token in Level 1 memory is... Each 16-bit floating-point number, or 131,072 bytes, occupies 1,073,741,824 bytes of video memory (1 GiB) when there are 8,192 token locations. The storage size for the slot key vector, slot value vector, and slot quality in Level 2 memory is 2,099,200 bytes (2.0 MiB). The storage size for Level 3 memory is 525,312 bytes (0.50 MiB). Based on the number of bytes stored, the compression ratio from Level 1 to Level 2 memory is 512:1, and the compression ratio from Level 1 to Level 3 memory is 2048:1.
[0072] In step 3, when the occupancy rate of the primary memory reaches the primary consolidation trigger occupancy rate threshold, the target representation is read from the primary memory using the aggregated bond vector. The writing intensity and number of dormancy cycles are determined according to the importance score. The target representation, corrected by error feedback, is written to the secondary memory slot, and the relative reconstruction error is verified on the writing result. Specifically: The inputs to this step are the importance score and aggregated key vector of the settled semantic blocks output from step 1, the secondary memory slot status output from step 2, and the actual residence location of the semantic blocks in the primary memory. The output of this step is a consolidation completion flag for each semantic block. With relative reconstruction error The updated secondary memory slot key vector, slot value vector, and slot quality are also included. The consolidation completion flag and relative reconstruction error are passed down to the first part of step 6 to determine the semantic blocks that can be cleared. The updated secondary memory slots are passed down to the calculation in step 2 and the secondary memory consolidation in step 4. Step 3 includes the following six parts.
[0073] 1. Determine whether Level 1 memory consolidation has been triggered. The Level 1 consolidation unit continuously monitors the occupancy rate of Level 1 memory. The occupancy rate of Level 1 memory is defined as the number and capacity of currently resident token locations. The ratio. When the occupancy rate reaches the first-level consolidation trigger occupancy rate threshold. Triggering Level 1 memory consolidation at any time The value is 0.90, corresponding to the 7373rd token position. The execution target of Level 1 memory consolidation is all settled semantic blocks in Level 1 memory, i.e., the set. After execution, the content of primary memory remains unchanged; changes to the content of primary memory are accomplished by selective clearing in step 6.
[0074] 2. Read out the target representation of the semantic block in primary memory. semantic blocks The reinforcement strength control unit uses the aggregated key vector of this semantic block. For retrieval, read the target representation from primary memory. Calculate using the following formula: ; in, For semantic blocks The target representation in primary memory consolidation has the following dimensions: ; To read the dimension-reduced matrix, the shape is... OK List; and The token position in the 15th layer attention module The key vector and value vector obtained by concatenating all key headers; For semantic blocks The set of token locations. Since the set of token locations carried by the primary memory is consistent with the set of token locations of the semantic block at the time of settlement, reading out the set of token locations used is... The target representation is the high-precision content representation of the semantic block in primary memory, and it is the target for writing to the secondary memory slot.
[0075] 3. Execute a semantic block-level sleep loop. First-level memory consolidation performs a dormant loop on the semantic block-level representation, without recalculating the self-attention sublayers and feedforward sublayers of the backbone network, thus not changing the hidden states of the backbone network. For semantic blocks... The The sleep cycle executes the following four steps in sequence.
[0076] (1) Read the reconstructed vector from the secondary memory: ; in, For the first When the sleep cycle begins To query the reconstructed vector read from secondary memory, with dimension .
[0077] (2) Calculate the residual vector: ; in, For the first The difference between the target representation and the reconstructed vector in the sleep cycle has a dimension of . .
[0078] (3) Calculate the writing target after error feedback correction: ; in, For the first Write target after round sleep cycle correction, dimension is ; The error dimension reduction matrix for first-level memory consolidation has the following shape: OK List; This is the ascending error matrix for first-level memory consolidation, with the following shape: OK List; The upper limit of the error feedback amplitude is set to 0.5. The function of the above formula is as follows: the readout result from the secondary memory is given by a weighted sum of the normalized exponents of the 512 slots. The weights are distributed across multiple slots, and the norm of the readout result is lower than the norm of the target representation. If the target representation is directly used as the writing target, the writing process will gradually converge towards the already converged readout result, failing to improve accuracy. Increasing the writing target along the current residual direction allows the readout result to converge towards the target table. In the above formula... Its function is to restrict the norm of the correction quantity to the norm of the target representation. Within a factor of 1, when the residual is sufficiently small, the correction amount decreases proportionally, thereby ensuring that the error feedback correction maintains the compressibility mapping property throughout the entire iteration process.
[0079] (4) Write the corrected write target to the secondary memory slot. The execution method is the same as the slot key vector update formula, slot value vector update formula, and slot quality update formula described in Part 4 of Step 2. The slot to be written is... .
[0080] 4. Determine the write strength and the number of sleep cycles. No. semantic blocks in hierarchical memory In the The write intensity of the sleep cycle is calculated using the following formula: ; in, This is the memory level number, which can be 2 or 3; here, we choose 2. For semantic blocks In the Level 1 memory Write intensity during sleep cycles; For the first The lower bound of write strength for level memory; For the first Upper bound of write strength for Tier 1 memory; For the first The importance response index of primary memory; This is the round decay factor, with a value of 0.95; For semantic blocks Importance rating. Lower bound of secondary memory write strength. Value is 0.10, upper bound The value is 0.45. This represents the lower bound of the write strength for L3 memory. Value is 0.06, upper bound. The value is 0.35. The primary memory consolidation uses the secondary memory write strength parameter.
[0081] The design basis of the above formula is as follows: setting a lower bound for the write strength ensures that semantic blocks with lower importance scores still have sufficient write strength for convergence, thus enabling low-importance semantic blocks to pass the relative reconstruction error verification and enter the clearing process in step 6; setting an upper bound for the write strength, with the upper bound not exceeding 0.55, keeps the error compression factor between two adjacent sleep cycles positive, and the error compression factor... Calculate using the following formula: ; in, For the first Error compression factor of the sleep cycle. Not greater than 0.55 and Under the condition of 0.5, The value ranges from 0.17 to 0.78, so each sleep cycle will compress the relative reconstruction error to at least 0.78 times, and the sleep cycle process is a monotonically converging process.
[0082] Importance Response Index The value is 0.6. The value is 0.9. Because... Greater than The writing intensity of semantic blocks with lower importance scores in tertiary memory is lower relative to their lower bound, meaning that deep memory is more conservative in writing low-importance semantic blocks.
[0083] semantic blocks In the The number of dormant cycles in the primary memory is calculated using the following formula: ; in, For semantic blocks In the The number of dormant cycles in the hierarchical memory; and The first The lower and upper bounds of the number of dormant cycles in hierarchical memory; This is for floor function operations. The lower bound of the number of rounds for first-level memory consolidation. Value 4, upper bound The value is 8, therefore The value ranges from 4 to 8 rounds. Lower bound of the number of rounds for secondary memory consolidation. Value is 10, upper bound The value is 16. This is the lower bound for the number of rounds of third-level memory consolidation. Value is 20, upper bound The value is 32. The above value satisfies... and Therefore, the number of dormant cycles in deep memory for any semantic block is strictly greater than the number of dormant cycles in shallow memory.
[0084] 5. Perform consolidation verification and supplementary dormancy cycles. After each sleep cycle, the consolidation and validation unit calculates the relative reconstruction error using the following formula: ; in, For semantic blocks In the Level 1 memory The relative reconstruction error after the end of the sleep cycle; For semantic blocks In the Target representation in graded memory consolidation, When it is 2, take ; For the readout operator of the next level of memory, in When it is 2, take .
[0085] The decision threshold for consolidation verification is calculated using the following formula: ; in, For semantic blocks In the The threshold for determining memory consolidation at different levels; and The first The lower and upper bounds of the threshold for judging second-level memory consolidation. The lower bound of the threshold for judging second-level memory. Value is 0.08, upper bound. The value is 0.30. This is the lower bound of the threshold for determining tertiary memory. The value is 0.25, with the upper bound as follows: The value is 0.45. The judgment threshold decreases as the importance score increases, meaning that highly important semantic blocks need to achieve higher reconstruction accuracy to pass the verification.
[0086] when Not greater than When the consolidation verification unit terminates the dormant cycle of the semantic block, it will set the consolidation completion flag. Setting it to 1 indicates that the system has been consolidated, and the relative reconstruction error of this round is denoted as... .
[0087] when equal and Greater than When this happens, execute the supplementary sleep cycle as follows: ; in, To compensate for the increase in the number of sleep cycles, a value of 2 is used. This also serves as a cycle decay factor during the sleep cycle execution. A value of 1 means that the write strength in the supplementary sleep cycle no longer decays with each round. The supplementary sleep cycle can execute a maximum of [number] times. Second-rate, The value is 2, so the actual number of rounds for Level 1 memory consolidation does not exceed 12 rounds.
[0088] If the consolidation completion flag is displayed after the supplementary sleep cycle has been completed but the threshold has not been met. A value of 0 indicates no consolidation; this semantic block does not participate in the selective clearing in step 6 and is reserved for re-execution of the dormant loop during the next memory consolidation at this level. Verified semantic blocks are included in subsequent first-level memory consolidation iterations. Continuing to write constitutes repeated writing, which further improves the reconstruction accuracy on the basis of the original accuracy.
[0089] The feasibility of the above parameter values is illustrated using primary memory consolidation as an example. The lower bound of the secondary memory write strength is 0.10, and the upper bound is 0.45, with an importance response index of 0.6. For a semantic block with an importance score of 0.20, the write strength is 0.397, the error compression factor is 0.405, and the number of rotation wheels... The process involves 5 rounds. The relative reconstruction error is 0.405 after the first round, 0.176 after the second round, and 0.081 after the third round, which is lower than the judgment threshold of 0.256 for the semantic block. For a semantic block with an importance score of 0.90, the write strength is set to 0.538, the error compression factor to 0.194, and the number of rounds is [not specified]. The number of rounds is 8. The relative reconstruction error is 0.194 after the first round and 0.045 after the second round, which is lower than the judgment threshold of 0.102 for the semantic block. For the extreme case where the importance score is 0, the write strength is reduced to the lower bound of 0.15, the error compression factor is 0.775, and the number of rounds is... The process consists of 4 rounds; the relative reconstruction error is taken as 0.361 after 4 rounds. After two rounds of supplementary sleep cycles, the value is 0.217, which is lower than the decision threshold of 0.30 corresponding to this semantic block. When the slot has not yet been written, the read result of the secondary memory is close to the zero vector, and the relative reconstruction error does not exceed 1. The above convergence process shows that the decision threshold can be reached within the above rounds.
[0090] 6. Implement a safety net and mandatory consolidation. The consolidation strength control unit counts the number of clearable token positions in the first-level memory consolidation using the following formula: ; in, This represents the number of token positions that can be cleared. for Consolidate the set of semantic blocks marked with a 1. For semantic blocks The number of token locations.
[0091] When continuous After secondary memory consolidation All are less than the number of token positions that need to be released in one clear. At that time, the reinforcement strength control unit for For semantic blocks where the consolidation completion flag is 0, forced consolidation is performed: the write strength is determined by the following formula, and then executed. Supplementary dormant cycles with no round decay: ; in, The upper bound for the write strength of Level 1 memory consolidation is set at 0.55. The number of failure consolidation cycles is set to 3. The number of forced consolidation rounds is set to 4. The error compression factor of the forced consolidation is 0.175, so the relative reconstruction error drops to 0.175 after the first round and to 0.031 after the second round, which allows all semantic blocks to pass the decision threshold verification.
[0092] The determination parameter for the forced consolidation is the number of token positions that can be cleared, rather than a threshold. This is because relaxing the determination threshold based on importance scoring would reduce the reconstruction accuracy standard of all semantic content. However, performing forced consolidation with the upper bound of the write strength improves the convergence speed without changing the determination threshold, thereby keeping the verification standard constant throughout the entire operation.
[0093] In step 4, when the cumulative slot write count reaches the cumulative slot write count threshold, secondary memory slots with slot quality not lower than the critical slot quality threshold are classified into semantic categories. Class prototype value vectors are calculated according to the semantic categories, and these vectors are written to the tertiary memory slots. Specifically: The inputs to this step are the updated secondary memory slot key vector, slot value vector, and slot quality from steps 2 and 3, as well as the cumulative number of slot writes to the secondary memory. The outputs of this step are the updated tertiary memory slot key vector, value vector, and slot quality, as well as the semantic category carried by each tertiary memory slot. These tertiary memory slots are passed down to step 5 for tertiary memory consolidation and then down to the third part of step 6 for memory readout.
[0094] Step 4 includes the following five parts.
[0095] 1. Determine whether secondary memory consolidation has been triggered. The secondary consolidation unit maintains the cumulative slot write count. After each primary memory consolidation is completed, the cumulative slot write count is incremented using the following formula: ; in, This is the cumulative number of writes to the slot. This represents the number of write slots in secondary memory for a single semantic block, with a value of 8. This represents the number of settled semantic blocks in Level 1 memory, with a value of 128. Therefore, the increment of the cumulative slot write count after each Level 1 memory consolidation is 1024.
[0096] when Reaching the cumulative slot write count threshold Triggering secondary memory consolidation and Set to 0, The value is 2048. Under the above increment, secondary memory consolidation is triggered approximately once every two primary memory consolidations.
[0097] 2. Select secondary memory consolidation targets and categorize them into tertiary memory categories. After the triggering conditions described in Part 1 are met, the secondary consolidation unit performs the following two steps.
[0098] (1) Selecting consolidation targets. The targets for secondary memory consolidation are secondary memory slots whose slot quality satisfies the following formula: ; in, The critical slot quality threshold is set to 0.65. As can be seen from the slot quality update formula described in Part 4 of Step 2, the slot quality converges to the importance score of the semantic block repeatedly carried by the slot. Therefore, the importance score of the semantic block carried by the slot that satisfies the above formula is not lower than 0.65.
[0099] (2) Semantic category classification. For each secondary memory slot that satisfies the above formula, the slot management unit determines its tertiary memory slot according to the following formula: ; in, Secondary memory slot The corresponding tertiary memory slot number; For Level 3 memory The key vector of each slot.
[0100] when The similarity threshold for creating new slots is less than the threshold. Furthermore, when there are unoccupied slots in the Level 3 memory, the slot management unit creates a new slot in the Level 3 memory and sets the key vector of that slot to 0. , value vector set to The quality of the slot is set to ; The value is 0.60.
[0101] For each tertiary memory slot s, the set of tertiary memory slots it carries is denoted as . Determine using the following formula: ; Not satisfied Secondary memory slots do not participate in secondary memory consolidation. If they are continuously written to during subsequent primary memory consolidation, the quality of the slots will continue to improve, thus qualifying them for selection during subsequent secondary memory consolidation.
[0102] 3. Computational Prototype Target Representation For each tertiary memory slot s, the secondary consolidation unit calculates the class prototype key vector and class prototype value vector for that category using the following formula: ; ; in, The class prototype key vector for the semantic category carried by the third-level memory slot s; The class prototype value vector of the semantic category carried by the third-level memory slot s; Secondary memory slot The quality of the slot is used as a weighting coefficient in this formula. This is a weighted average of the slot quality of each member slot value vector in the secondary memory. It serves as the central representation of the semantic content of all members in this category, i.e., the skeleton of the semantic content of this category.
[0103] The target representation of secondary memory consolidation is taken from the prototype value vector of the class. The reason for not using the target representation of a single semantic block is as follows: The number of slots in the Level 3 memory is 128, and the slot dimension is 512. The set of Level 3 memory slot key vectors spans at most a 128-dimensional subspace in the 512-dimensional space. If the high-precision representation of a single semantic block is used as the consolidation target, its components outside this 128-dimensional subspace cannot be represented, and the decision threshold becomes unattainable. When the class prototype is used as the consolidation target, the class prototype itself is a weighted average of multiple member representations, located in the central direction of the space spanned by the member representations. This direction can be covered by the set of slots in the Level 3 memory. Therefore, the decision threshold becomes the convergence criterion of the writing process rather than the attainability criterion of representation capability. The upper limit of the number of categories in the Level 3 memory is 128. The Level 3 memory retains the semantic skeleton of each category rather than the details of each category member, corresponding to the high compression ratio setting of the Level 3 memory.
[0104] 4. Execute the hibernation cycle to tertiary memory. For each tertiary memory slot s, the secondary consolidation unit is... For query, with To represent the target, a sleep loop is executed on the tertiary memory slot s. The execution structure of the sleep loop is consistent with the third part of step 3, except that the read source is replaced by the tertiary memory instead of the secondary memory, and the read operator is changed from... Replace with The error feedback matrix is composed of and Replace with and The write slot is controlled by Replace with the set of three-level memory slots corresponding to this category.
[0105] The write strength of the hibernation cycle is calculated using the following formula: ; in, For the third-level memory slot s in the first Write intensity during sleep cycles; Three-level memory slots The importance parameter of the semantic category it carries is determined by the following formula: ; in, This represents the maximum quality of the member slots in this category, ranging from 0.65 to 1.
[0106] The number of dormancy cycles is calculated using the following formula: ; in, The number of sleep cycles for the tertiary memory slot s is 28 to 32. The lower bound of the write strength for tertiary memory consolidation is 0.06, the upper bound is 0.35, and the importance response index is 0.9.
[0107] To consolidate and verify the calculation of the relative reconstruction error, the following formula is used: ; in, For the third-level memory slot s in the first The relative reconstruction error after the end of the sleep cycle. The judgment threshold is determined by the following formula: ; in, The threshold for determining the level 3 memory slot s.
[0108] The feasibility of the above parameter values is Taking a lower bound of 0.65 as an example, the write strength is set to 0.257, the error compression factor to 0.615, and the number of sleep cycle rounds to 28. The relative reconstruction error is 0.615 after the first round, 0.390 after the second round, and 0.254 after the third round, which is lower than the corresponding judgment threshold of 0.320 for this category. The value vector of the level 3 memory slot s is set to... Therefore, the initial relative reconstruction error is small, and the above convergence process shows that the decision threshold can be reached within the above rounds budget.
[0109] 5. Perform three-level memory slot merging. When all of the third-level memories When all slots are occupied and a new slot needs to be created, the slot management unit determines the slot pair to be merged using the following formula: ; in, and The sequence numbers of the two most similar already occupied level 3 memory slots; and Let be the key vectors of the two; and The quality of the slots for both.
[0110] when Not less than the slot merging similarity threshold At that time, slot merging shall be performed according to the following three formulas. Value: 0.85 ; ; ; The slot key vector is re-normalized after being weighted by slot quality, while maintaining a length of 1. The slot value vector is weighted and averaged by slot quality, with the sum of the weighting coefficients being 1. Therefore, the value vector of the merged slot has the same common center representation as the two slot value vectors before the merge, and the readout scale remains unchanged. After the merge is completed, slot s_a becomes unoccupied and can accommodate newly created categories.
[0111] when Less than In this case, slot merging will not be performed during the second-level memory consolidation. The slot management unit will postpone the request to create a new slot to the next second-level memory consolidation. During this period, the category will be carried by the second-level memory.
[0112] In step 5, during periods when the computational resource utilization rate is continuously below the idle threshold or at the time of session switching, value vector re-anchoring, key vector orthogonalization, and slot quality decay are performed on the level 3 memory slots, and the level 3 memory slots are persistently stored to non-volatile storage media. Specifically: The inputs to this step are the updated L3 memory slot key vector, value vector, and slot quality from step 4, as well as the semantic category carried by each L3 memory slot output from step 4. The outputs of this step are the globally reorganized L3 memory slots and the persistent L3 memory records written to non-volatile storage media. These persistent records are loaded at the start of the session, becoming the initial L3 memory slots for that session.
[0113] Step 5 includes the following five parts.
[0114] 1. Determine whether level three memory consolidation has been triggered. The three-level consolidation unit continuously monitors the graphics processor's computing resource utilization, specifically the video memory bandwidth utilization. When the video memory bandwidth utilization continuously... Time below idle threshold Triggering Level 3 memory consolidation The value is 30 seconds. The value is 0.20. When a session switching event is detected, the Level 3 consolidation unit also triggers Level 3 memory consolidation. The target of Level 3 memory consolidation is all occupied Level 3 memory slots.
[0115] 2. Perform three-level memory slot value vector re-anchoring. For each occupied tertiary memory slot s, the tertiary consolidation unit uses the class prototype key vector of that slot. For querying, read the anchored representation from secondary memory. Calculate using the following formula: ; in, Anchoring representation of the third-level memory slot s, with dimension . .
[0116] Then update the value vector of the third-level memory slot using the following formula: ; in, For the third-level memory slot s in the first The write intensity of the deep sleep cycle is determined according to the write intensity formula described in Part 4 of Step 4. The value vector re-anchoring repeatedly aligns the value vector of the tertiary memory slot to the newer representation in the secondary memory during session idle periods or session switching times, thereby eliminating representation deviations generated during cross-session accumulation.
[0117] 3. Perform three-level memory slot key vector orthogonalization. In each deep sleep cycle, the key vectors of all occupied level 3 memory slots are orthogonally corrected once using the following formula: ; in, The orthogonalization step size is set to 0.05. For the first The slot quality of each slot is used as a weighting coefficient in this formula. The correction amount is proportional to the inner product of the key vectors of the slot and the other slots. The correction amount increases when the directions of the two slot key vectors are close. The correction direction is to remove the projected components of the slot in the directions of the other slots. Since the step size is less than 1 and the correction amount is proportional to the inner product, the correction process is a compression mapping. After multiple rounds of correction, the three-level memory slot key vectors converge to an approximately orthogonal configuration.
[0118] The purpose of orthogonalizing the slot key vectors of the third-level memory is as follows: The third-level memory has 128 slots with a slot dimension of 512. Orthogonalization ensures that the 128 slot key vectors are distributed as far as possible in unrelated directions, thereby maximizing the coverage of the main directions of the class prototypes of each category by the 128-dimensional subspace spanned by the 128 slots, thus reducing the relative reconstruction error described in the fourth part of step 4. The slot key vectors of the second-level memory are not orthogonalized; they freely form semantic clusters as described in the third part of step 2. Therefore, the second-level memory undertakes the function of self-consistency sinking, while the third-level memory undertakes the function of inter-class separability.
[0119] 4. Implement slot quality decay and hit rate improvement In each deep sleep cycle, a slot quality decay is performed once for any unhitted Level 3 memory slots using the following formula: ; in, This is the slot quality attenuation coefficient, with a value of 0.9. The level 3 memory slot participates in the value vector re-anchoring write during this sleep cycle, or its read weight in level 3 memory reads is greater than the slot hit weight threshold. When the time is right, it is considered a hit, and the quality of the slot is improved according to the following formula: ; in, The threshold value for slot hit weight is 0.2. This is the slot quality hit increment, with a value of 0.02.
[0120] Level 3 memory slots do not perform slot release. Slot release is only performed when the slot quality of a Level 3 memory slot decays to the slot release quality threshold. In the following cases, the slot retains its key vector and value vector and continues to participate in orthogonalization and slot merging. The value is 0.05. This differs from the processing method of secondary memory because tertiary memory is responsible for cross-session persistence, and releasing the slot will make cross-session knowledge unrecoverable.
[0121] 5. Perform persistent storage and session loading for three-level memory. After Level 3 memory consolidation is complete, the Level 3 storage unit writes the key vectors, value vectors, slot quality, class prototype key vectors, and class prototype value vectors of all Level 3 memory slots to the non-volatile storage medium. The storage size of a single persistent record is... and The product of these is multiplied by 4 bytes, resulting in a value of 1049600 bytes, or 1.0 MiB.
[0122] At the start of the session, the Level 3 storage unit reads the record and loads it into the Level 3 memory slot. After loading, the historical slot key vector and the newly generated slot key vector for this session are fused once using the following formula: ; in, This is the fused slot key vector; For slot key vectors loaded from persistent records; This is the newly generated slot key vector for this session. If no corresponding slot has been generated for this session, the vector is zero. The session payload weight is set to 0.3, giving historical persistent representation a major weight in the fusion process. Cross-session memory accumulation is achieved through the fusion of persistent writes and loads.
[0123] In step 6, when the occupancy rate of the first-level memory reaches the first-level consolidation trigger occupancy rate threshold, the validated semantic blocks are removed from the first-level memory in ascending order of importance score. During the prediction phase, result vectors are read from the first-level memory, second-level memory, and third-level memory respectively, and fused with confidence weights. The fused read vector is then injected into the memory injection layer, and the prediction distribution of the next token is output. Specifically: The inputs to this step are the consolidation completion flag and relative reconstruction error output from step 3, the updated third-level memory slot states from steps 2 and 4, and the hidden states for each prediction step. The outputs of this step are the updated first-level and second-level memory slot states and the predicted distribution of the next token. The updated first-level memory is passed down to step 2 in the next round as a receiving object, and the predicted distribution of the next token is directly output as the text generation result.
[0124] Step 6 includes the following five parts.
[0125] 1. Perform selective clear When the occupancy rate of primary memory reaches When this happens, the selectively cleared unit performs the following operations.
[0126] (1) Select semantic blocks from primary memory that satisfy the following formula: ; in, For semantic blocks The consolidation and completion of the mark; For semantic blocks The optimal relative reconstruction error recorded during primary memory consolidation; For semantic blocks The threshold for judgment. Semantic blocks that satisfy the above formula are denoted as verified semantic blocks, whose content has been carried by secondary and tertiary memory.
[0127] (2) Score the validated semantic blocks according to their importance. Sort in ascending order.
[0128] (3) Remove the token positions of the semantic block from the first-level memory in ascending order. The removal operation is to delete the key vector and value vector of the token position in each layer of the memory injection layer set. The cumulative number of removed token positions reaches Stop moving out when the time comes. The value is 1024. The removal operation does not change the slot content of the secondary and tertiary memories, so the information of the removed semantic block can still be obtained by reading from the secondary and tertiary memories.
[0129] (4) When the number of token locations provided by the semantic block obtained in (1) is insufficient In this round, the removal will be performed according to the actual number of blocks, and semantic blocks that have not passed verification will not be removed.
[0130] High-importance semantic blocks are located at the end of the ascending order due to their high importance scores, and they receive higher write strength and lower decision threshold during first-level memory consolidation, so they are given priority in preservation during the clearing process.
[0131] 2. Perform a gradual cleanup. The progressive clearing unit performs the removal of semantic blocks as described in Part 1, one by one at the semantic block granularity. After each semantic block is removed, the occupancy rate of the primary memory is immediately recalculated using the following formula: ; in, This represents the current occupancy rate of primary memory. This represents the number of token positions currently residing in the primary memory. When Down to The process of moving out will stop at the following time, thus forming a sliding window-style progressive clearing during continuous generation, instead of performing a one-time overall clear.
[0132] 3. Perform multi-level memory retrieval In each memory injection layer, the multi-level retrieval unit reads the result vector from the first-level memory, second-level memory, and third-level memory, respectively.
[0133] The first-level memory readout vector is calculated using the following formula: ; in, The first-level memory readout vector has a dimension of . ; For the first The output vector of the self-attention sublayer in each memory injection layer has a dimension of . ; and For the first Token position in layer attention module The key vector and value vector, obtained by concatenating all key headers, have dimensions of 1. .
[0134] The second-level memory readout vector and the third-level memory readout vector are calculated using the following formula: ; in, For the first Level memory readout vector, dimension is ; For the first The query vector for hierarchical memory is calculated using the following formula: ; in, For the first The query dimensionality reduction matrix of level memory has the following shape: OK List; For the first The query matrix of the first-order memory is of the following shape: OK The query vector has a length of 1, which is consistent with the length constraint of the aggregation key vector used as the query in steps 3 and 4. Therefore, the readout in the inference stage and the readout in the consolidation stage are consistent in terms of soft maximum calculation.
[0135] The readings of secondary and tertiary memory are performed in parallel within each memory injection layer, and their computational complexity is independent. The number of values taken was 327,680; the total number of read multiply-accumulate operations in the 8 memory injection layers was 2,621,440, accounting for 0.02% of the 1.3×10^10 forward multiply-accumulate operations per token in the backbone network.
[0136] 4. Perform confidence fusion and prediction output The confidence fusion unit calculates the confidence weights of each memory level using the following formula: ; in, For the first The confidence weights for the three levels of memory are equal to 1. To read the up-dimensional matrix, the shape is... OK List; The fusion temperature coefficient is set to 0.5. This is the quality weighting coefficient, with a value of 0.3; For the first Hierarchical quality indicators for memory levels; This is the lower limit of the hierarchical quality indicator, with a value of 10. -3 This is used to avoid logarithmic operations being undefined when the hierarchical quality index is 0.
[0137] The first term in the confidence formula is the cosine similarity between the query vector and the read vector after dimensionality increase, divided by the fusion temperature coefficient. Its function is to give higher weight to memory reads that are consistent with the current hidden state direction. The second term is the logarithm of the hierarchical quality index multiplied by the quality weight coefficient. Its function is to give higher weight to memory hierarchies with higher average quality.
[0138] Hierarchical quality indicators are defined by the following formula: ; in, The quality index for the first-level memory is the average importance score of the settled semantic blocks in the first-level memory. The quality index for secondary memory is the average quality of all slots in secondary memory. The quality index for the three-level memory is the average quality of all slots in the three-level memory. This represents the number of settled semantic blocks in primary memory. This is for calculating the larger value.
[0139] The fused readout vector is calculated using the following formula: ; in, To fuse the readout vector, the dimension is .
[0140] The memory injection sublayer will inject the fused readout vector into the hidden state using the following formula: ; in, For the first The output vector of the memory injection sub-layer in each memory injection layer; For the first The dimensionality reduction matrix of the memory injection layer has the shape of... OK List; The up-dimensional matrix shared by all memory injection layers has the following shape: OK List; For the first Injection gating coefficients for each memory injection layer; For layer normalization. The initial value is set to 0, so that the memory injection sublayer is an identity mapping at the beginning of training, thereby ensuring the stability of the training process when starting from the pre-trained weights of the backbone network.
[0141] No. The layer output of each memory injection layer and the predicted distribution of the next token are calculated using the following formula: ; ; in, For the first Layer output vector; For the first The feedforward sublayer of the layer; This is the layer output vector of the 32nd layer of the backbone network; The output projection matrix of the backbone network has the following shape. OK List; To in the generated sequence The next token under the condition The predicted distribution.
[0142] 5. Perform importance decay and secondary memory slot release. After each primary memory consolidation, the importance decay unit performs a slot quality decay on the secondary memory slots once using the following formula: ; In this round of primary memory consolidation, when a secondary memory slot is a write target, it is considered a hit and is not decayed. The slot quality is improved by updating the slot quality as described in Part 4 of Step 2.
[0143] When the secondary memory slots are satisfied When the importance decay unit is reached, the slot is released according to the following formula: ; After a slot is released, it becomes unoccupied and can be reselected by subsequent semantic blocks according to the addressing rules described in Part 3 of Step 2. Since the slot quality increases by 0.02 when hit and is multiplied by 0.9 when decaying, the slot quality of slots that have not been written to for a long time decreases by powers of 0.9. After about 30 decays, the quality drops to below 0.05 and is released, thus realizing the reclamation of storage space that has not been accessed for a long time.
[0144] This invention provides an end-to-end training method, wherein the end-to-end training module is executed sequentially in the following four training stages. The backbone network, memory injection layer, secondary memory slots, and tertiary memory slots constitute the same model, and the freeze and unfreeze ranges of the four training stages are different from each other.
[0145] Training Phase 1: Ranking Training of the Importance Scoring Network The objects frozen in this stage are the backbone network parameters, memory injection layer parameters, and all memory slots; the objects trained are the aggregate weight vectors. Four unnormalized weight parameters Dimensionality reduction matrix of aggregate bonds Reading the reduced-dimensional matrix Error dimensionality reduction matrix and Error ascending matrix and .
[0146] The supervision signal in this stage is the semantic block. Citations . The definition is: under the condition that attention is calculated on the training corpus in complete sequences, semantic blocks In the following The sum of attention weights for each token used as a key in a query, divided by the sum of attention weights for all tokens queried within that window, The value is 512.
[0147] This stage simultaneously determines the aforementioned two calibration values. and The determination method involves statistical analysis on calibration corpora independent of the training corpora. The mean negative log-likelihood of each semantic block and the mean attention value. The value is 32.
[0148] The sorting loss is calculated using the following formula: ; in, The ranking loss; To meet A set of semantic block pairs; The threshold for citation difference is set to 0.05. The sorting interval is 0.2. For set The number of elements. After this stage of training is completed, the distribution of importance scores matches the distribution of citations, and the normalized weights of the four sub-scorings are then determined.
[0149] Phase Two: Supervised Training of Memory Readout and Injection The parameters frozen in this phase are the backbone network parameters and all parameters determined in the first training phase. The training objects are the dimensionality reduction matrices of each memory injection layer. Shared up-dimensional matrix Reading the upgraded matrix Injection gating coefficient Query dimensionality reduction matrix and Query the upgraded matrix and .
[0150] The training objective in this stage is the sum of the language modeling loss and the auxiliary prediction loss. The auxiliary prediction loss is calculated using the following formula: ; in, To assist in predicting losses; The first factor matrix of the auxiliary prediction head has the following shape: OK List; The second factor matrix of the auxiliary prediction head has the following shape: OK List; To assist in the low-rank dimension of the prediction head, a value of 128 is used; Cross-entropy loss; This is the true successor token of the current token. The number of parameters in the auxiliary prediction header is... The value is 4,161,536. This prediction head is only used during the training phase and is removed during the inference phase.
[0151] The purpose of the auxiliary prediction loss is to enable the readout vectors of the second-level memory and the third-level memory to each have independent next token prediction capabilities, so as to avoid the deep memory readouts being completely masked by the first-level memory readouts in end-to-end training.
[0152] Training Phase 3: Multi-level BPTT global fine-tuning with segmented state replay In this phase, all parameters are unfrozen, and end-to-end training is performed on the complete consolidation and retrieval processes for the three memory levels.
[0153] (1) State transcription and replay. Since the consolidation process is executed in segments during offline periods, the computation graph of all dormant loops cannot reside in GPU memory simultaneously. Therefore, a state transcription and replay unit is set up. This unit defines a complete dormant loop round for a semantic block in one consolidation as a consolidation segment and saves the input of the consolidation segment, namely the aggregated key vector. Target representation Write to slot index set With each round of writing intensity The state transcription buffer capacity is A consolidation phase, The value is 64, and items are eliminated in a first-in, first-out (FIFO) order. The storage size of a single consolidation segment is... The buffer contains 1104 floating-point numbers, with a total storage capacity of no more than 282624 bytes, or 0.27 MiB.
[0154] (2) Backpropagation. During backpropagation, the state transcription and replay unit reads the input of the consolidation segment from the buffer, replays all dormant cycle rounds of the consolidation segment sequentially to reconstruct the computation graph, and backpropagates the gradients to adjacent shallower memory levels. All semantic blocks consolidated in one operation are processed sequentially. Each consolidation segment is divided into several backpropagation segments, with gradient truncation performed between segments. The number of semantic blocks for first-level memory consolidation is 128, therefore, one backpropagation of first-level memory consolidation is divided into 2 segments.
[0155] (3) Handling of non-differentiable operations. Slot addressing, slot merging, slot release and consolidation completion flag determination are all non-differentiable operations based on index or comparison. Their gradients are set to zero during backpropagation, and forward calculation is performed according to the actual selection results.
[0156] (4) Total Loss. The total loss of a multi-stage BPTT is calculated using the following formula: ; in, Total loss; Model cross-entropy loss for language; , , , , The values for each loss are 0.03, 0.5, 0.1, 0.001, and 1.0, respectively.
[0157] Reconstruction loss, orthogonalization regularization, and computational budget regularization are calculated using the following formulas: ; ; ; in, For reconstruction loss; To regularize the orthogonality; To calculate budget regularization; This refers to the set of semantic blocks involved in the current training batch. Its number of elements; For semantic blocks Relative reconstruction error in primary memory consolidation; , , The number of consecutive rounds is calculated using the following formula: ; in, For semantic blocks In the The number of consecutive rounds in the hierarchical memory. The calculation of budget regularization uses the number of consecutive rounds instead of the rounded number because the gradient of the rounding operation is 0 at almost all values. Using the number of consecutive rounds allows the gradient of the budget regularization calculation to be backpropagated to the four unnormalized weight parameters via importance scoring. This allows for the co-optimization of consolidation computation and the parameters of the importance scoring network.
[0158] The reconstruction loss directly uses the relative reconstruction error used in the consolidation and validation phase as the training target, thereby optimizing the determination of the writing intensity parameter, error feedback matrix, and number of sleep cycle rounds. The orthogonalization regularization and the orthogonalization correction in the third part of step 5 jointly constrain the pairwise inner product of the three-level memory slot key vectors. The computational budget regularization enables the model to compress the consolidation computation while maintaining reconstruction accuracy.
[0159] Training Phase Four: Policy Gradient Fine-tuning for Consolidation The objects frozen in this phase are the backbone network parameters and all memory slots, while the training objects are the consolidation policy learning units. Each consolidation policy learning unit consists of a policy network whose inputs are the current session's context length and first-level memory occupancy rate, and whose outputs are the lower bounds of the write intensity for the three memory levels. and Upper bound of write strength for the three memory levels and and three importance response indices , , The output layer uses Gaussian sampling.
[0160] The reward is calculated using the following formula: ; in, As a reward; For accuracy in long-context question answering tasks; This represents the average number of sleep cycles for all semantic blocks in the current evaluation round. This is the computational penalty coefficient, with a value of 0.002.
[0161] The policy gradient is calculated using the following formula: ; in, For policy network parameters; To optimize the strategy objectives; The action vector output by the policy network consists of three lower bounds of write strength, three upper bounds of write strength, and three importance response indices. The policy network outputs an action given the current state. The probability density; The moving average of the reward is used as a baseline to reduce gradient variance; The entropy of the policy distribution; is the entropy regularization coefficient, with a value of 0.01. The lower and upper bounds of the write intensity output by the policy network are fixed outside of the fourth training phase. Take 0.10, Take 0.45, Take 0.06, Take 0.35, Take 0.3, Take 0.6, We set it to 0.9. Policy gradients enable policy networks to balance long-context question-answering accuracy with consolidation computation, thereby autonomously learning graded consolidation policies.
[0162] This invention also provides a long context management system for large language models based on multi-level memory consolidation, applied to the aforementioned method for long context management of large language models based on multi-level memory consolidation. The system includes: a context receiving and preprocessing module, a three-level memory storage module, a hierarchical dormancy consolidation module, a memory retrieval and fusion module, an intelligent clearing strategy module, and an end-to-end training module. The data connection relationships between each module correspond one-to-one with the steps of the aforementioned method.
[0163] The context receiving and preprocessing module is used to perform step 1. This module includes the following units.
[0164] 1. Token Input Unit: This unit receives the input token sequence and outputs the input token sequence along with the hidden states of each layer of the backbone network. The output of this unit is then passed to the Semantic Blocking Unit, the Importance Evaluation Unit, and the First-Level Memory Unit.
[0165] 2. Semantic Blocking Unit: This unit divides semantic blocks according to the adjacent cosine similarity of the hidden states in the aggregation layer and the block length constraint, and outputs a set of semantic blocks. This is the set of token locations for each semantic block. The output of this unit is passed to the importance evaluation unit and the aggregation key generation unit.
[0166] 3. Importance Assessment Unit: This unit calculates four sub-scores, synthesizes the importance score, determines the settlement time, and outputs the importance score and pre-written tag for each semantic block. The output of this unit is passed to the three-level memory storage module, the hierarchical hibernation consolidation module, and the intelligent clearing strategy module, and is also passed to the calculation loop of the repetition sub-score.
[0167] 4. Aggregate Key Generation Unit: This unit calculates the aggregate key vector for each semantic block and outputs the aggregate key vector. The output of this unit is passed to the secondary memory unit, the primary consolidation unit, and the secondary consolidation unit.
[0168] The three-level memory storage module is used to perform step 2. This module includes the following units.
[0169] 1. The first-level memory unit is used to store the key vector and value vector of each layer of the memory injection layer set for the current resident token position, with a capacity of 8192 token positions; this unit provides the key vector and value vector required for reading the first-level memory to the memory retrieval and fusion module, and accepts the removal instruction from the intelligent clearing strategy module.
[0170] 2. Secondary memory unit, used to store the key vector, value vector and slot quality of 512 secondary memory slots; this unit provides read objects to the primary consolidation unit and memory retrieval and fusion module, accepts write instructions from the primary consolidation unit, and provides key vector, value vector and slot quality to the secondary consolidation unit.
[0171] 3. The third-level memory unit is used to store the key vector, value vector, slot quality, class prototype key vector and class prototype value vector of 128 third-level memory slots, and writes persistent records to the non-volatile storage medium at the end of the session and loads the records at the beginning of the session; this unit accepts write instructions from the second-level consolidation unit and the third-level consolidation unit, and provides read objects to the memory retrieval and fusion module.
[0172] 4. The slot management unit is used to perform slot addressing, slot quality grading and filtering, slot creation, slot merging, and slot release. The output of this unit is written to the slot index set and passed to the first-level and second-level consolidation units. The addressing, merging, releasing, and creation operations of this unit are all index-based operations, and the gradients are zeroed during end-to-end training.
[0173] The graded hibernation consolidation module is used to execute steps 3, 4, and 5. This module includes the following units.
[0174] 1. Primary consolidation unit, used when the occupancy rate of primary memory reaches a certain level. The target representation of the semantic block in the primary memory is read out, a semantic block-level sleep loop is executed, and the data is written to the secondary memory unit. The input of this unit is the key vector and value vector of the primary memory unit, the position of the semantic block in the primary memory, the aggregated key vector output by the aggregated key generation unit, and the slot state of the secondary memory unit; the output of this unit is the updated secondary memory slot and the consolidation completion flag of each semantic block.
[0175] 2. Secondary consolidation unit, used to consolidate the number of writes to the cumulative slot until a certain number of writes are achieved. The prototype of the class is calculated for secondary memory slots that meet the slot quality conditions, and a sleep loop is executed for the tertiary memory unit. The input of this unit is the slot state of the secondary memory unit; the output of this unit is the updated tertiary memory slot.
[0176] 3. Three-level consolidation unit, used when the memory bandwidth utilization is continuously below [a certain value]. During the time period or session switching time, value vector re-anchoring, key vector orthogonalization, and slot quality degradation are performed. The input of this unit is the tertiary memory unit and the secondary memory unit; the output of this unit is the globally reorganized tertiary memory slot.
[0177] 4. Consolidation Strength Control Unit: This unit determines the write strength, number of sleep cycles, decision threshold, and number of supplementary sleep cycles based on the importance score, and performs forced consolidation when there are insufficient token positions to clear. The inputs to this unit are the importance score output by the importance assessment unit and the decision result output by the consolidation verification unit; the output of this unit is passed to the first-level consolidation unit and the second-level consolidation unit.
[0178] 5. Consolidation Verification Unit: This unit calculates the relative reconstruction error, determines whether the judgment threshold has been reached, outputs the consolidation completion flag and the relative reconstruction error, and outputs the judgment result to the consolidation intensity control unit. The consolidation completion flag and relative reconstruction error output by this unit are transmitted to the intelligent clearing strategy module.
[0179] The memory retrieval and fusion module is used to perform steps 6, parts three and four. This module includes the following units.
[0180] 1. A multi-level retrieval unit is used to read result vectors from the first-level, second-level, and third-level memory units, respectively, using the output of the self-attention sublayer of each memory injection layer as the query, and to calculate the query dimensionality reduction matrix and query dimensionality increase matrix of the corresponding query vectors. The output of this unit is the read vectors of the three memory levels.
[0181] 2. The confidence fusion unit is used to calculate the hierarchical quality index and confidence weights of the three memory levels, and to obtain the fused readout vector by weighted summation of the upscaled readout vectors. The input of this unit is the readout vector of the multi-level retrieval unit and the slot quality of the second-level and third-level memory units; the output of this unit is the fused readout vector.
[0182] 3. The hidden state injection unit is used to superimpose the fused readout vector, after layer normalization, dimensionality reduction matrix transformation, and shared dimensionality increase matrix transformation, onto the output of the self-attention sub-layer according to the injection gating coefficients, and output the output vector of the memory injection sub-layer. The output of this unit is passed to the feedforward sub-layer of the backbone network.
[0183] 4. Importance attenuation unit, used to perform slot quality attenuation and slot release for secondary memory slots. The output of this unit is transmitted to the secondary memory unit and the slot management unit.
[0184] The intelligent clearing strategy module is used to execute the first and second parts of step 6. This module includes the following units.
[0185] 1. Selective clearing of units, used when the occupancy rate reaches a certain level. Semantic blocks with a consolidation completion flag of 1 and a relative reconstruction error not exceeding the judgment threshold are selected and removed in ascending order of importance score until the cumulative number of removed token positions reaches a certain threshold. The inputs to this unit are the outputs of the consolidation and verification unit and the importance assessment unit; the output of this unit is a shift instruction, which is passed to the first-level memory unit.
[0186] 2. Progressive clearing unit: This unit is used to remove elements one by one at the semantic block level and recalculate the occupancy rate after each removal. The clearing process continues until the occupancy rate drops to a certain level. The following will terminate the removal process. This unit, in conjunction with the selective clear unit, controls the resident content of the primary memory unit.
[0187] The end-to-end training module is used to perform the four training phases mentioned above. This module includes the following units.
[0188] 1. Multi-level BPTT training unit, used to perform training according to the parameters of the importance evaluation unit, the parameters of the memory injection layer, the error feedback matrix and the memory slots in training phase one, training phase two and training phase three.
[0189] 2. Consolidate the policy learning unit, which is used to update the strength parameter and importance response index according to the policy gradient in training phase four.
[0190] 3. State transcription and replay unit, used to save the input of the consolidation segment during the training phase, replay the forward computation of the consolidation segment sequentially to reconstruct the computation graph, and replay the input of each consolidation segment in sequence. Each consolidation segment is divided into backpropagation segments.
[0191] Embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0192] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0193] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0194] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0195] Contents not described in detail in this specification are prior art known to those skilled in the art. It is hereby indicated that the above description is intended to help those skilled in the art understand this invention, but does not limit the scope of protection of this invention. Any equivalent substitutions, modifications, improvements, or simplifications of the above descriptions that do not depart from the essential content of this invention fall within the scope of protection of this invention.
Claims
1. A method for long context management in large language models based on multi-level memory consolidation, characterized in that, The large language model includes multiple memory injection layers, and the method includes: Step 1: Obtain the input token sequence, divide the input token sequence into semantic blocks according to the neighbor similarity of the hidden state of the aggregation layer, calculate the importance score and aggregation key vector of each semantic block, and settle the importance score when the settlement condition is met; Step 2: Write the key vector and value vector of the input token sequence into the primary memory, determine the ownership tag of the semantic block according to the importance score, and write the semantic blocks that meet the ownership conditions into the secondary memory slots and update the slot quality according to the slot quality level. Step 3: When the occupancy rate of the primary memory reaches the primary consolidation trigger occupancy rate threshold, the target representation is read from the primary memory using the aggregated bond vector. The writing intensity and the number of dormancy cycles are determined according to the importance score. The target representation after error feedback correction is written into the secondary memory slot, and the relative reconstruction error is verified on the writing result. Step 4: When the cumulative number of slot writes reaches the cumulative number of slot writes threshold, divide the secondary memory slots with slot quality not lower than the critical slot quality threshold into semantic categories, calculate the class prototype value vector according to the semantic category, and write the class prototype value vector into the tertiary memory slot. Step 5: During periods when the computing resource utilization rate is continuously lower than the idle threshold or during session switching, perform value vector re-anchoring, key vector orthogonalization, and slot quality decay on the level 3 memory slots, and persistently store the level 3 memory slots to non-volatile storage media. Step 6: When the occupancy rate of the first-level memory reaches the first-level consolidation trigger occupancy rate threshold, the semantic blocks that have passed the verification are removed from the first-level memory in ascending order of importance score. During the prediction phase, the result vectors are read from the first-level memory, second-level memory and third-level memory respectively and fused with confidence weight. The fused read vector is injected into the memory injection layer and the prediction distribution of the next token is output.
2. The method of claim 1, wherein, In step 1, the input token sequence is divided into semantic blocks based on the neighbor similarity of the hidden states in the aggregation layer. The importance score and aggregation key vector for each semantic block are calculated, and the importance score is calculated when the settlement conditions are met. Specifically: The cosine similarity between two adjacent token positions in the hidden state of the aggregation layer is calculated sequentially. When the cosine similarity is less than the semantic boundary judgment threshold, a semantic boundary is set between the two adjacent token positions. At the same time as setting the semantic boundary, upper and lower bound constraints on block length are applied. The aggregation layer is a layer in the large language model that does not belong to the memory injection layer. The set of token positions obtained by the semantic boundary segmentation constitutes a semantic block. The importance score is obtained by multiplying the task relevance sub-score, novelty sub-score, inference chain criticality sub-score, and repetition sub-score by their respective normalized weights and then summing them. The task relevance sub-score represents the degree of relevance between the semantic block and the user instruction. The novelty sub-score represents the degree of unexpectedness of the semantic block relative to the existing information in the session. The inference chain criticality sub-score represents the intensity of attention the semantic block receives in subsequent generation. The repetition sub-score represents the degree of overlap between the semantic block and the existing slot key vector in secondary memory. All four sub-scores are in the range of 0 to 1. The key vectors of each token in the semantic block at the aggregation layer are weighted and summed according to the aggregation weight. The weighted sum is then reduced in dimension and normalized to obtain the aggregated key vector. The aggregation weight is obtained by normalizing the hidden state of each token in the semantic block at the aggregation layer. The length of the normalized aggregated key vector is 1. Set the settlement conditions as follows: the number of subsequent tokens of the semantic block reaches the length of the attention settlement window; the number of subsequent tokens of the semantic block reaches the minimum settlement window and the occupancy rate of the first-level memory reaches the threshold of the first-level consolidation trigger occupancy rate; a session switching event is detected; wherein the minimum settlement window is less than the length of the attention settlement window.
3. The method of claim 2, wherein, In step 2, the key vector and value vector of the input token sequence are written into the primary memory. The ownership tag of the semantic block is determined according to the importance score. Then, the semantic blocks that meet the ownership criteria are written into the secondary memory slots and the slot quality is updated according to the slot quality classification. Specifically: Write the key vectors and value vectors of all token positions in the input token sequence into the first-level memory, where the writing is not affected by the importance score; The attribution tags are determined as high-importance semantic blocks, medium-importance semantic blocks, or low-importance semantic blocks according to the value range of the importance score. The high-importance semantic blocks enter the writing process of the secondary memory slot after the importance score is calculated. The medium-importance semantic blocks and low-importance semantic blocks enter the writing process of the secondary memory slot in batches during step 3. The three attribution tags are mutually exclusive. The set of writable slots for a semantic block is determined based on the condition that the slot quality is not higher than the importance score of the semantic block. The writable slots are selected from the set of writable slots in descending order of inner product. The inner product is the inner product of the aggregation key vector of the semantic block and the key vector of the secondary memory slot. When the set of writable slots is empty, the secondary memory slot with the smallest slot quality is released and used as the writing object. The key vector written to the slot is interpolated to the aggregate key vector of the semantic block and then normalized. The value vector written to the slot is interpolated to the target representation of the semantic block obtained in step 3. The slot quality of the slot is updated according to the importance score. The updated slot quality is monotonically non-decreasing and uses the importance score of the semantic block as the asymptotic value. The condition for writing a secondary memory slot to a tertiary memory slot in step 4 is that the slot quality of the secondary memory slot is not lower than the critical slot quality threshold.
4. The method of claim 3, wherein, In step 3, when the occupancy rate of the primary memory reaches the primary consolidation trigger occupancy rate threshold, the target representation is read from the primary memory using the aggregated bond vector. The writing intensity and number of dormancy cycles are determined according to the importance score. The target representation, corrected by error feedback, is written to the secondary memory slot, and the relative reconstruction error is verified on the writing result. Specifically: Using the aggregated key vector of the semantic block as the query, the normalized exponential operation is performed on the key vector of each token in the aggregate layer within the semantic block to obtain the readout weight. The value vectors of each token in the aggregate layer within the semantic block are weighted and summed according to the readout weight, and then the dimension is reduced to obtain the target representation. The hibernation cycle is executed round by round according to the number of hibernation cycles. In each hibernation cycle, a reconstruction read, a residual calculation, an error feedback correction, and a slot write are executed sequentially. The result of the reconstruction read is the result vector read from the secondary memory with the aggregation key vector of the semantic block as the query. The result of the residual calculation is the difference between the target representation and the result vector read from the reconstruction. The error feedback correction is to reduce the dimension of the residual vector and then increase the dimension to obtain the correction direction vector and add it to the target representation according to the correction amount. The slot write is to write the corrected target representation into the secondary memory slot according to the write intensity. The correction ratio is the smaller of the ratio of the norm of the target representation to the norm of the correction direction vector and the upper limit of the error feedback amplitude. The upper limit of the error feedback amplitude is the maximum multiple of the single-round correction amount relative to the norm of the target representation. The correction ratio is no greater than 1 so that the dormant loop is a compressed mapping. The write intensity is taken as the interpolation between the lower bound of the write intensity of the memory level and the upper bound of the write intensity of the memory level. The interpolation ratio is determined by the power of the importance response index of the importance score and the round decay factor. The write intensity remains monotonically constant with the importance score and decreases with the number of sleep cycle rounds. The upper bound of the write intensity is set to make the error compression factor of the sleep cycle positive. The importance response index of the memory level with the larger memory level number is greater than the importance response index of the memory level with the smaller memory level number. The number of dormant cycle rounds is taken as the interpolation between the lower bound of the number of dormant cycle rounds for that level of memory and the upper bound of the number of dormant cycle rounds for that level of memory, and then rounded up. The interpolation ratio is the importance score. The lower bound and upper bound of the number of dormant cycle rounds for each of the first-level memory consolidation, second-level memory consolidation and third-level memory consolidation increase sequentially and do not overlap. After each dormancy cycle, the relative reconstruction error is determined by the ratio of the norm of the difference between the target representation and the next level memory readout result vector to the norm of the target representation. The judgment threshold of the semantic block in the memory consolidation of that level is determined by the interpolation between the upper bound and the lower bound of the memory consolidation judgment threshold of that level. The interpolation ratio is the importance score, and the judgment threshold decreases as the importance score increases. When the relative reconstruction error is not greater than the determination threshold, the dormancy loop of the semantic block is terminated and the consolidation completion flag is set to consolidated; When the number of sleep cycles reaches the number of sleep cycle rounds and the relative reconstruction error is greater than the judgment threshold, the number of sleep cycle rounds is added by the increment of the number of supplementary sleep cycles. In the sleep cycle that is added, the round decay factor is set to 1. The number of additional executions does not exceed the upper limit of the number of supplementary sleep cycles. If the judgment threshold is still not met after the additional execution is completed, the consolidation completion flag remains as unconsolidated. The number of token positions provided by semantic blocks marked as consolidated in the primary memory is counted to obtain the number of token positions that can be cleared. When the number of token positions that can be cleared after a number of consecutive failure consolidation cycles is less than the number of token positions that need to be released in one clearing, the write strength of semantic blocks marked as not consolidated is taken as the upper bound of the write strength of that level of memory, and a sleep cycle with a forced consolidation round is executed with the round decay factor set to 1.
5. The method of claim 4, wherein, In step 4, when the cumulative slot write count reaches the cumulative slot write count threshold, secondary memory slots with slot quality not lower than the critical slot quality threshold are classified into semantic categories. Class prototype value vectors are calculated according to the semantic categories, and these vectors are written to the tertiary memory slots. Specifically: After each primary memory consolidation is completed, the number of slots actually written to the secondary memory slots during this primary memory consolidation is added to the cumulative slot write count. When the cumulative slot write count reaches the cumulative slot write count threshold, secondary memory consolidation is triggered, and the cumulative slot write count is set to zero. For each secondary memory slot whose quality is not lower than the key slot quality threshold, calculate the inner product of the key vector of the secondary memory slot and the key vector of each tertiary memory slot. Take the tertiary memory slot with the largest inner product as the semantic category to which the secondary memory slot belongs. When the largest inner product is less than the slot creation similarity threshold and there are unoccupied slots in the tertiary memory, create a new slot in the tertiary memory and take the key vector, value vector and slot quality of the new slot as the key vector, value vector and slot quality of the secondary memory slot, respectively. Secondary memory slots whose quality is lower than the key slot quality threshold do not participate in step 4. The class prototype value vector is obtained by weighting the value vectors of all second-level memory slots under the same semantic category according to the slot quality of the second-level memory slots. The class prototype key vector is obtained by weighting and combining the key vectors of all second-level memory slots under the same semantic category according to the slot quality of the second-level memory slots and then normalizing them. Using class prototype key vectors as queries and class prototype value vectors as target representations, the hibernation cycle is executed on the tertiary memory slots according to the write intensity of the secondary memory consolidation and the number of hibernation cycle rounds, and verification is performed according to the relative reconstruction error. The judgment threshold of the secondary memory consolidation is greater than the judgment threshold of the primary memory consolidation, and the total number of slots of the tertiary memory slots is less than the dimension of the slot vector, so that the tertiary memory slots retain the semantic category skeleton. When all three-level memory slots are occupied and a new slot needs to be created, the two slots with the largest inner product among the occupied three-level memory slots are identified as slot pairs to be merged. When the inner product of the slot pairs to be merged is not less than the slot merging similarity threshold, the key vector of one slot in the pair is taken as the result of the normalized combination of the slot quality of the two slot key vectors, the value vector of the slot is taken as the slot quality weighted average of the two slot value vectors, the slot quality of the slot is taken as the smaller value between the sum of the two slot qualities and 1, and the key vector, value vector and slot quality of the other slot in the pair are set to zero. When the inner product of the slot pairs to be merged is less than the slot merging similarity threshold, the current slot creation request is postponed to the next second-level memory consolidation execution.
6. The method of claim 5, wherein, In step 5, during periods when the computational resource utilization rate is continuously below the idle threshold or at the time of session switching, value vector re-anchoring, key vector orthogonalization, and slot quality decay are performed on the level 3 memory slots, and the level 3 memory slots are persistently stored to non-volatile storage media. Specifically: The computing resource utilization rate is taken as the graphics processor memory bandwidth utilization rate. When the continuous idle determination time of the memory bandwidth utilization rate is lower than the idle determination threshold, or when a session switching event is detected, the third-level memory consolidation is triggered. The execution object of the third-level memory consolidation is all occupied third-level memory slots. Using the class prototype key vector as a query, the anchor representation is read from the secondary memory, and the value vector of the tertiary memory slot is interpolated and updated to the anchor representation according to the writing intensity of the tertiary memory consolidation. The anchor representation is the current central representation of the semantic category in the secondary memory. The correction amount is determined by the inner product of the tertiary memory slot key vector and the key vectors of the other occupied tertiary memory slots, the slot quality of the other tertiary memory slots, and the orthogonalization step size. The correction amount is subtracted from the tertiary memory slot key vector and then normalized. The orthogonalization step size is less than 1 to orthogonalize the key vectors into a compressed mapping. The orthogonalization of the key vectors causes the key vectors of the tertiary memory slots to converge to an approximately orthogonal configuration. Multiply the slot quality of the unhit tertiary memory slots in this round by the slot quality decay coefficient. Increase the slot quality of the hit tertiary memory slots in this round by the slot quality hit increment, not exceeding 1. The hit determination condition is that the tertiary memory slot participates in the value vector re-anchoring write in this round of dormancy, or the read weight of the tertiary memory slot in tertiary memory read is greater than the slot hit weight threshold. When the slot quality of the tertiary memory slot decays to below the slot release quality threshold, the key vector, value vector and slot quality of the tertiary memory slot are not released and continue to participate in the key vector orthogonalization and slot merging. After the Level 3 memory consolidation is completed, the key vectors, value vectors, slot quality, class prototype key vectors, and class prototype value vectors of all Level 3 memory slots are written to the non-volatile storage medium. At the start of the session, the slot contents stored in the non-volatile storage medium are read and loaded into the Level 3 memory slots. The loaded slot key vectors and the newly generated slot key vectors in this session are weighted and summed according to the session load weight, and then normalized. The session load weight is the weight of the newly generated slot key vectors in this session in the weighted sum. The session load weight is less than 0.5, so that the historical persistence representation has a major weight in the weighted sum.
7. The method of claim 6, wherein, In step 6, when the occupancy rate of the first-level memory reaches the first-level consolidation trigger occupancy rate threshold, the validated semantic blocks are removed from the first-level memory in ascending order of importance score. During the prediction phase, result vectors are read from the first-level memory, second-level memory, and third-level memory respectively, and fused with confidence weights. The fused read vector is then injected into the memory injection layer, and the prediction distribution of the next token is output. Specifically: Semantic blocks that have passed verification are selected from the primary memory, marked as having been consolidated and whose relative reconstruction error is no greater than the judgment threshold. They are then sorted by importance score from smallest to largest and removed sequentially. Removal involves deleting the key vector and value vector of the token position in the memory injection layer and aggregation layer of the semantic block. Removal stops when the cumulative number of removed token positions reaches the number of token positions required to be released in one clearing. If the number of token positions provided by the verified semantic blocks is less than the number of token positions required to be released in one clearing, the actual number of token positions are removed without removing unverified semantic blocks. Removal does not change the content of the secondary and tertiary memory slots, so that the information of the removed semantic blocks can still be read from the secondary and tertiary memories. After each semantic block is removed, the occupancy rate of the first-level memory is recalculated based on the ratio of the number of token positions currently residing in the first-level memory to the capacity of the first-level memory. The removal is terminated when the recalculated occupancy rate drops below the first-level consolidation trigger occupancy rate threshold. In each memory injection layer, the output of the self-attention sublayer of that memory injection layer is used as the basis for the query vector. The slot value vectors are read from the second-level memory and the third-level memory respectively according to the slot weights and weighted summation to obtain the result vector. The query vector is read from the first-level memory according to the normalized exponential weight of the key vector of each token in the aggregation layer in the semantic block and weighted summation of the value vector of each token in the aggregation layer in the semantic block, and then dimension reduction is performed to obtain the result vector. The query vector is obtained by dimension reduction and then dimension increase and normalization of the output of the self-attention sublayer. The retrieval temperature coefficient of the third-level memory is smaller than that of the second-level memory. The readout vectors of the three memory levels are weighted and summed after dimensionality increase according to the confidence weights to obtain the fused readout vector. The sum of the three confidence weights is 1. The confidence weights are jointly determined by the cosine similarity between the output of the self-attention sublayer and the dimensionality increase readout vector, as well as the hierarchical quality index of the memory level. The hierarchical quality index of the first-level memory is the average importance score of the settled semantic blocks in the first-level memory, and the hierarchical quality index of the second-level and third-level memories is the average quality of all slots in each memory level. The fused readout vector is normalized and transformed by layer, and then superimposed on the output of the self-attention sub-layer of the memory injection layer according to the injection gating coefficient. The result is input into the feedforward sub-layer of the memory injection layer. The output of the last layer of the large language model is projected by the output projection matrix and then the prediction distribution of the next token is obtained by normalization exponent operation. The initial value of the injection gating coefficient is 0, so that the memory injection layer is an identity mapping at the beginning of training. After each primary memory consolidation is completed, the slot quality of the secondary memory slots that were not hit in this round is multiplied by the slot quality decay coefficient. When the slot quality of the secondary memory slot is less than the slot release quality threshold, the key vector, value vector and slot quality of the secondary memory slot are set to zero and the slot is converted to an unoccupied state. Among them, the secondary memory slots that participated in the primary memory consolidation writing in this round are judged as hits and no slot quality decay is performed.
8. A long context management system for large language models based on multi-level memory consolidation, applied to any one of the long context management methods for large language models based on multi-level memory consolidation as described in claims 1-7, characterized in that, include: The module includes a context receiving and preprocessing module, a three-level memory storage module, a hierarchical hibernation consolidation module, a memory retrieval and fusion module, an intelligent clearing strategy module, and an end-to-end training module. The context receiving and preprocessing module is used to obtain the input token sequence, divide the input token sequence into semantic blocks according to the neighbor similarity of the hidden state of the aggregation layer, calculate the importance score and aggregation key vector of each semantic block, and settle the importance score when the settlement conditions are met. The three-level memory storage module is used to write the key vector and value vector of the input token sequence into the first-level memory, determine the ownership tag of the semantic block according to the importance score, and write the semantic block that meets the ownership condition into the second-level memory slot and update the slot quality according to the slot quality level addressing. It is also used to store the second-level memory slot and the third-level memory slot and persistently store the third-level memory slot to a non-volatile storage medium. The hierarchical dormancy consolidation module is used to read the target representation from the primary memory using aggregated key vectors when the occupancy rate of the primary memory reaches the primary consolidation trigger occupancy rate threshold. It then determines the write intensity and number of dormancy cycles based on importance scores, writes the target representation corrected by error feedback into the secondary memory slot, and performs relative reconstruction error verification on the write results. Furthermore, when the cumulative number of slot writes reaches the cumulative number of slot writes threshold, it classifies secondary memory slots with slot quality not lower than the critical slot quality threshold into semantic categories, calculates class prototype value vectors according to semantic categories, and writes the class prototype value vectors into the tertiary memory slots. Finally, it performs value vector re-anchoring, key vector orthogonalization, and slot quality decay on the tertiary memory slots during periods when the computing resource occupancy rate is continuously lower than the idle determination threshold or during session switching. The intelligent clearing strategy module is used to remove verified semantic blocks from the first-level memory in ascending order of importance score when the occupancy rate of the first-level memory reaches the first-level consolidation trigger occupancy rate threshold. The memory retrieval and fusion module is used to read the result vectors from the first-level memory, second-level memory and third-level memory respectively in the prediction stage, and fuse them with confidence weight. After injecting the fused read vector into the memory injection layer, the prediction distribution of the next token is output. The end-to-end training module is connected to the context receiving and preprocessing module, the three-level memory storage module, the hierarchical dormancy consolidation module, the memory retrieval and fusion module, and the intelligent clearing strategy module, respectively, and is used to sequentially execute four training phases on the large language model, all memory slots, and the parameters on which the importance score is based.
