Layered dynamic memory throttling processing method and apparatus, and readable storage medium

Abstract

This invention discloses a hierarchical dynamic memory throttling method, apparatus, and readable storage medium. The method includes: data acquisition and preprocessing, three-level memory hierarchical processing, semantic dynamic scoring, topic drift detection, adaptive throttling compression, and incremental summary updating and context reconstruction steps. This application's hierarchical dynamic memory throttling method relies on three-level memory hierarchical differentiated processing, dynamic low-relevance content filtering, and an incremental summary iteration mechanism. Based on three-dimensional semantic weights, it accurately filters cross-turn key logic, preconditions, and core user demands. It compresses tokens in ordinary multi-turn dialogue scenarios, fully preserves the latest dialogue details in real-time hot memory, maintains no loss of question-and-answer accuracy in long dialogue scenarios, preserves core question-and-answer logic in short-term warm memory, and no longer stores the original complete text in long-term cold memory, only storing iteratively generated structured incremental summaries. This achieves reduced token compression rate and effective throttling.

Description

Technical Field

[0001] This invention relates to the field of large language model artificial intelligence interaction technology, and in particular to a hierarchical dynamic memory throttling processing method, apparatus and readable storage medium. Background Technology

[0002] Multi-turn dialogue systems driven by large language models have become a core application scenario for artificial intelligence. The current industry-standard processing logic is a full historical context concatenation mechanism. This means that each time a user initiates a new question, the system concatenates all user questions and model responses since the start of the dialogue into the current prompt word, converts them into tokens, and sends them to the model for reasoning. This basic mechanism suffers from four unavoidable technical flaws: First, the cumulative effect of token redundancy is significant. As the number of dialogue rounds increases, a large amount of repetitive semantics, meaningless greetings, outdated and invalid topic content, and repetitive explanatory statements are generated in the context, and the total number of tokens increases linearly with the number of dialogue rounds. In long dialogue scenarios, the number of tokens can easily exceed the upper limit of the model's context window, while significantly increasing server memory usage, GPU computing power consumption, and commercial API call costs, thereby increasing enterprise operating costs.

[0003] Second, traditional static compression strategies suffer from severe semantic loss. Existing token optimization methods are mostly static processing schemes such as fixed-round summaries, fixed-length header / tail truncation, and global sparse attention, where compression rules are not dynamically adjusted according to dialogue semantics. When dialogues involve pre-existing logical dependencies, cross-round related conclusions, and multi-branch questions, static truncation and fixed summaries are prone to losing key constraints, numerical values, and reasoning premises, causing biases in model intent recognition, misalignment of question-and-answer logic, and breaks in contextual understanding, significantly reducing the quality of dialogue interaction.

[0004] Third, existing unified storage and compression methods cannot simultaneously meet the requirements of fidelity and cost-saving. Current technologies employ a unified storage and compression strategy for all historical dialogue content, failing to differentiate between recent, highly relevant dialogues and distant, less relevant historical dialogues. For distant dialogues that only serve a temporary auxiliary purpose, a large amount of token resources are still consumed, resulting in low compression efficiency; while for newly generated key dialogue details, excessive compression leads to information loss, failing to balance the requirements of fidelity and cost-saving.

[0005] Fourth, the ability to adapt to topic-shifting scenarios is poor. Real-world human-computer dialogues often involve complex interactions such as topic switching, branching questions, and historical topic revisiting. Existing compression algorithms cannot automatically identify topic switching boundaries or accurately invalidate historical memories corresponding to expired topics. When the dialogue jumps to a completely new area, irrelevant content from the old topic continues to occupy context space; when users revisit historical topics, they cannot quickly retrieve the corresponding prior information, easily leading to contradictions in questions and answers and logical disconnects.

[0006] In summary, existing multi-turn dialogue token optimization technologies suffer from industry pain points such as rigid compression rules, incomplete redundancy filtering, easy loss of key semantics, and weak adaptability to multiple topic scenarios. There is an urgent need for a dynamic, layered, adaptive, and low-loss token throttling optimization scheme that can achieve efficient token saving while ensuring the integrity of dialogue logic and accurate intent recognition. Summary of the Invention

[0007] In order to overcome the shortcomings of the prior art, the present invention aims to provide a hierarchical dynamic memory throttling processing method, device and readable storage medium to solve the problems of rigid compression rules, incomplete redundancy filtering, easy loss of key semantics and weak adaptability to multi-topic scenarios in the existing multi-turn dialogue token optimization technology.

[0008] This invention proposes a hierarchical dynamic memory throttling method, the method comprising: Data acquisition and preprocessing steps: Real-time acquisition of raw data from multi-turn dialogues and shallow noise reduction preprocessing to remove explicit redundant text and output standardized dialogue units; The three-level memory stratification process involves dynamically dividing historical dialogues into three levels—real-time hot memory, short-term warm memory, and long-term cold memory—based on dialogue rounds, time weight, and semantic heat. Differentiated information processing strategies are then applied to the memory content at each level. Semantic dynamic scoring steps: Combine the current round of user input text, calculate the semantic weight score of each historical dialogue unit from three dimensions: semantic similarity, topic relevance, and logical dependency. Filter valid historical information based on preset filtering thresholds and remove low-relevance redundant content. Topic drift detection steps: Determine whether topic drift has occurred by the semantic vector offset between the previous and subsequent dialogues. If topic drift is determined, the low-association long-term memory corresponding to the previous topic is automatically marked as expired memory and will no longer be included in the current context splicing, while retaining the cross-topic common core information. Adaptive throttling compression steps: Real-time monitoring of the total amount of tokens currently occupied in the context, dynamic adjustment of compression intensity based on the current amount of tokens occupied, and differential compression of memory content at each level; Incremental summary update and context reconstruction steps: Based on the existing long-term memory summary, incremental updates are performed to add key information and correct expired logic, generating iterative new summaries. The real-time hot memory complete content, the short-term warm memory simplified content, and the long-term cold memory incremental summaries are reconstructed into the model input context.

[0009] The three-level memory stratification step includes: The real-time hot memory is the complete original content of the most recent N rounds of dialogue, where N is a preset positive integer, and no compression processing is performed on the real-time hot memory; The short-term warm memory is a lightweight and simplified version of the dialogue from round N+1 to round M, where M is a preset positive integer greater than N. The core question-and-answer logic is retained in the short-term warm memory, while decorative and repetitive statements are removed. The long-term cold memory refers to the incremental structured summary of historical dialogues prior to round M, without retaining the original dialogue text.

[0010] In the semantic dynamic scoring step, the semantic weight score S_i of the i-th historical dialogue unit is calculated using the following formula: S_i = α × Sim_i + β × Rel_i + γ × Dep_i Where Sim_i is the semantic similarity between the i-th historical dialogue unit and the current round of user input, Rel_i is the topic relevance between the i-th historical dialogue unit and the current round of user input, and Dep_i is the logical dependency between the i-th historical dialogue unit and the current round of user input; α, β, and γ are preset weight coefficients, α∈[0,1], β∈[0,1], and γ∈[0,1], and satisfy α+β+γ=1.

[0011] The semantic similarity Sim_i is calculated in the following way: Input text from the current user in the current round. Convert to semantic vector Convert the i-th historical dialogue unit H_i into a semantic vector. Calculate the cosine similarity between the two. The calculation results are then corrected for their range to ensure that Sim_i ∈ [0,1]. The formula is as follows:

[0012] In the above formula, Mole: The text entered by the user in the current round Corresponding semantic vector The i-th historical dialogue unit H_i is converted into a semantic vector. The dot product of two vectors; Denominator: User input text in the current round Corresponding semantic vector The i-th historical dialogue unit H_i is converted into a semantic vector. The product of the L2 norms of two vectors (vector magnitudes); The calculation results are automatically normalized to [-1, 1], and the range is corrected.

[0013] After correction, Sim∈[0,1], which conforms to the unified value specification; The meanings of the letters in the above formulas are as follows: The text entered by the user in the current round; : The i-th historical dialogue unit (single-turn Q&A: user asks + model answers); : The corresponding semantic vector (Embedding); Historical Dialogue The corresponding semantic vector.

[0014] The logical dependency degree Dep_i is calculated in the following way: Extract the logical features between the i-th historical dialogue unit and the current round of user input, the logical features including referential dependency, conditional dependency and continuation dependency; Calculate the logical dependency degree based on the hit logical features and their corresponding preset scores:

[0015] in, The text entered by the user in the current round; : The i-th historical dialogue unit (single-turn Q&A: user asks + model answers); m is the number of logical features hit, which is a positive integer; Score_j is the score corresponding to the j-th hit logical feature, and Dep_i∈[0,1]; When Dep=0: There is no logical dependency, and the historical content is irrelevant to the current question and answer; When Dep=1: The current question completely depends on this history as the preceding logic.

[0016] The topic drift detection step includes: Calculate the topic vector offset between adjacent rounds of dialogue. When the offset exceeds a preset threshold, it is determined to be topic drift. Automatically mark low-relevance long-term memories of the previous topic as expired memories, and no longer include them in the current round of context splicing; at the same time, retain common background information, core settings, key user needs and other common content across topics to avoid information loss; The topic vector offset is calculated as follows: Get the previous dialogue vector and the user input vector of the current round Calculate the cosine similarity between the two:

[0017] Calculate the topic vector offset D: The value range is [0,2]. The larger the value, the greater the difference in the topics. A topic shift is determined to occur when D > D_threshold, where D_threshold is a preset shift determination threshold.

[0018] The adaptive throttling compression step includes: The system presets a global token limit threshold and monitors the token usage in the current context in real time. When the token usage is below the low threshold L_threshold: only shallow noise reduction is performed, fully preserving dialogue details with zero compression loss; When the token holding is in the middle range: perform regular layered compression, simplify redundant statements, and retain the core logic; When the token usage exceeds the high threshold H_threshold: an extreme throttling strategy is activated to strengthen the long-term memory summary, significantly remove low-relevance historical content, and prioritize the computing power for current dialogue reasoning. The low threshold L_threshold = 0.6 × Token_max, and the high threshold H_threshold = 0.9 × Token_max, where Token_max is the preset maximum number of tokens; When the token usage exceeds the high threshold H_threshold: the extreme throttling strategy is activated, which includes: preserving the complete content of real-time hot memory, strongly simplifying short-term warm memory, heavily summarizing long-term cold memory, and batch eliminating low-value historical dialogues with a comprehensive score lower than the screening threshold.

[0019] The incremental summary update and context reconstruction steps include: The underlying summary of the long-term cold memory storage; When the short-term warm memory accumulates to a preset number of rounds, new key information and change information are extracted from the short-term warm memory. The new key information is added to the basic summary, and the expiration logic in the basic summary is corrected based on the change information to generate a new version summary. The model input context reconstruction is performed by splicing together the complete content of real-time hot memory, the simplified content of short-term warm memory, and the incremental summary of long-term cold memory.

[0020] This invention also proposes a communication tunnel dynamic processing device, comprising: The data acquisition and preprocessing module is used to acquire multi-turn dialogue data in real time and perform shallow noise reduction preprocessing to remove explicit redundant text and output standardized dialogue units. The three-level memory layering module is used to dynamically divide historical dialogues into three levels: real-time hot memory, short-term warm memory, and long-term cold memory based on dialogue rounds, time weights, and semantic heat. The semantic dynamic scoring module is used to calculate the semantic weight score of each historical dialogue unit from three dimensions: semantic similarity, topic relevance, and logical dependency. The topic drift detection module is used to calculate the topic vector offset between adjacent rounds of dialogue and determine topic drift. An adaptive throttling compression module is used to dynamically adjust the compression intensity based on the real-time token usage. The incremental summary update and context reconstruction module is used to perform incremental updates based on existing long-term memory summaries and reconstruct the model input context.

[0021] The present invention also proposes a computer-readable storage medium storing a communication tunnel dynamic processing program, wherein when the communication tunnel dynamic processing program is executed by a processor, the steps of the hierarchical dynamic memory throttling processing method described in any of the preceding claims are implemented.

[0022] The hierarchical dynamic memory-based throttling method described in this application relies on three-level memory hierarchical differentiated processing, dynamic low-association content filtering, and incremental summary iteration mechanism. Based on three-dimensional semantic weights, it accurately filters key logic, preconditions, and core user demands across rounds. It compresses tokens in ordinary multi-turn dialogue scenarios, fully preserves the latest dialogue details in real-time hot memory without losing referential, follow-up, and temporary constraint information, and maintains no loss of question-and-answer accuracy in long dialogue scenarios. Short-term warm memory retains core question-and-answer logic and eliminates modifying and repetitive statements. Long-term cold memory no longer stores the original complete text, but only stores the iteratively generated structured incremental summary, thereby saving token compression rate and achieving effective throttling. Attached Figure Description

[0023] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a flowchart of the first embodiment of the hierarchical dynamic memory throttling method of the present invention; Figure 2 This is a flowchart of the third embodiment of the hierarchical dynamic memory throttling method of the present invention; Figure 3 This is a flowchart of the fourth embodiment of the hierarchical dynamic memory throttling method of the present invention; Figure 4 This is a schematic diagram of the hierarchical dynamic memory throttling device of the present invention. Detailed Implementation

[0024] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0025] In the following description, the use of suffixes such as "module," "part," or "unit" to denote elements is solely for the purpose of illustrative purposes and has no specific meaning in itself. Therefore, "module," "part," or "unit" may be used interchangeably.

[0026] This invention relates to a hierarchical dynamic memory throttling processing method, specifically a token dynamic throttling compression algorithm with low semantic loss and high compression efficiency for multi-turn continuous human-computer dialogue scenarios. It can be widely applied to all systems and terminal devices that rely on large language models (LLM) to achieve continuous interaction, such as general AI intelligent assistants, enterprise intelligent customer service, long text human-computer interaction, retrieval augmented generation (RAG) question answering, and multi-turn dialogue robots.

[0027] Tokens are the smallest processing units for parsing and operating text in large language models. Each round of dialogue inference requires converting the historical dialogue text into token vectors to input into the model. The total number of tokens directly determines the model's memory usage, inference latency, and API call billing costs. This invention proposes a hierarchical dynamic memory throttling scheme to address the context redundancy problem in multi-turn dialogues, significantly reducing the number of input tokens while fully preserving the core semantics and logical connections of the dialogue. The implementation method is illustrated in detail below through examples.

[0028] Example 1 To achieve the above-mentioned objectives, this invention provides a hierarchical dynamic memory throttling method. Please refer to [link to relevant documentation]. Figures 1 to 3 The method includes the following steps: S10. Real-time data acquisition and preprocessing steps: Real-time acquisition of raw data from multi-turn dialogues and completion of shallow noise reduction preprocessing, removal of explicit redundant text, and output of standardized dialogue units. Specifically, data such as user questions, model responses, dialogue timestamps, and round numbers are collected in real time during multi-turn dialogues. The collected dialogue data undergoes shallow noise reduction preprocessing to remove explicit redundant text, such as filtering pure punctuation, empty content, and greetings without actual semantics (e.g., "hello," "uh-huh," "okay," etc.), and explicit token redundancy is removed, thus completing the initial lightweight noise reduction.

[0029] S20, Three-level memory stratification steps: Based on dialogue rounds, time weights, and semantic heat, historical dialogues are dynamically divided into three levels: real-time hot memory, short-term warm memory, and long-term cold memory, and differentiated information processing strategies are implemented for the memory content at each level. Specifically, memory levels are divided based on the number of dialogue rounds, and the processing strategies for each level are dynamically adjusted according to semantic popularity. All historical dialogue units are divided into three types of storage spaces: real-time hot memory, short-term warm memory, and long-term cold memory, and information retention strategies are executed differently. The real-time hot memory is the complete original content of the most recent N rounds of dialogue, where N is a preset positive integer, preferably 3 rounds, and no compression processing is performed on the real-time hot memory; Specifically, the content of the most recent three rounds of questions and answers in the dialogue is stored in its original text without any deletion, summarization or compression, ensuring that the temporary details, immediate references and short-term logical connections of the current dialogue are completely intact; The short-term warm memory is a lightweight and simplified version of the dialogue from round N+1 to round M, where M is a preset positive integer greater than N, preferably M=10. The core question-and-answer logic is retained in the short-term warm memory, while embellishment and repetitive statements are removed. Specifically, the core subjects and verbs, key values, constraints, and core conclusions of the dialogues from rounds 4 to 10 will be retained, while adjectives, repetitive small talk, and redundant modifiers will be removed, resulting in lightweight text simplification. The long-term cold memory refers to the incremental structured summary of historical dialogues prior to round M, without retaining the original dialogue text.

[0030] Specifically, all historical dialogues from 10 rounds ago will no longer be stored as original, complete texts, but only as iteratively generated structured incremental summaries, achieving extreme token throttling.

[0031] This application achieves a token compression rate of 40%-80% in multi-turn dialogue scenarios through three-level hierarchical storage, dynamic redundancy filtering, and incremental summary iterative updates. The more dialogue rounds, the more significant the cost-saving effect, greatly reducing model inference costs, API call costs, and GPU memory usage.

[0032] S30, Semantic Dynamic Scoring Step: Combining the current round of user input text, calculate the semantic weight score of each historical dialogue unit from three dimensions: semantic similarity, topic relevance, and logical dependency. Filter valid historical information based on preset filtering thresholds and remove low-relevance redundant content. Specifically, for historical dialogue units corresponding to short-term warm memory and long-term cold memory, the effective weight score S of a single historical dialogue is quantitatively calculated from three dimensions—semantic similarity, topic relevance, and logical dependency—based on the current round of user input text. The semantic weight score S_i of the i-th historical dialogue unit is calculated using the following formula: S_i = α × Sim_i + β × Rel_i + γ × Dep_i Where Sim_i is the semantic similarity between the i-th historical dialogue unit and the current round of user input, Rel_i is the topic relevance between the i-th historical dialogue unit and the current round of user input, and Dep_i is the logical dependency between the i-th historical dialogue unit and the current round of user input; α, β, and γ are preset weight coefficients, α∈[0,1], β∈[0,1], and γ∈[0,1], and satisfy α+β+γ=1.

[0033] In this embodiment, the closer the value of S is to 1, the stronger the correlation.

[0034] The semantic similarity represents the overall similarity between historical text and current input at both the literal and semantic levels. It is used to measure the overlap of text content itself and is the most basic association indicator.

[0035] The semantic similarity Sim_i is calculated in the following way: Input text from the current user in the current round. Convert to semantic vector Convert the i-th historical dialogue unit H_i into a semantic vector. Calculate the cosine similarity between the two. The calculation results are then corrected for their range to ensure that Sim_i ∈ [0,1]. The formula is as follows:

[0036] In the above formula, Mole: The text entered by the user in the current round Corresponding semantic vector The i-th historical dialogue unit H_i is converted into a semantic vector. The dot product of two vectors; Denominator: User input text in the current round Corresponding semantic vector The i-th historical dialogue unit H_i is converted into a semantic vector. The product of the L2 norms of two vectors (vector magnitudes); The calculation results are automatically normalized to [-1, 1], and the range is corrected.

[0037] After correction, Sim∈[0,1], which conforms to the unified value specification; The meanings of the letters in the above formulas are as follows: The text entered by the user in the current round; : The i-th historical dialogue unit (single-turn Q&A: user asks + model answers); : The corresponding semantic vector (Embedding); Historical Dialogue The corresponding semantic vector.

[0038] The aforementioned semantic vectors can be generated into dense vectors of fixed dimensions using general text embedding models (such as BERT, Sentence-BERT, LLM built-in vectorization interfaces, etc.). Semantic similarity is suitable for capturing scenarios where keywords, sentences, and surface semantic overlap overlap, and it performs best in recognizing pure paraphrasing, synonymous questions, and repetitive requests. However, it should be noted that this metric lacks contextual logic awareness and only represents the similarity of the text itself; it needs to be combined with the latter two metrics for comprehensive judgment.

[0039] The topic relevance characterizes the degree of matching between the theme, domain, and discussion scope of the historical dialogue and the current topic. It is used to distinguish different scenarios such as "continuation of the same topic", "entry into a new topic", and "branching of the topic". It is one of the core features of this invention and effectively solves the problem of identifying dialogue topic drift.

[0040] In this embodiment, the topic relevance calculation adopts a dual-path determination method combining topic clustering and topic matching, specifically including the following steps: S301 Step: Topic Tag Extraction.

[0041] Extract topics from all dialogue texts to obtain the current input topic tag set. (Keywords / Domain Tags) and H_i's Collection of Topic Tags in Historical Conversations .

[0042] Step S302: Calculate topic relevance (Jaccard similarity) based on the intersection of sets.

[0043]

[0044] The meanings of each expression in the formula are as follows: The number of elements in the intersection of two sets (shared topic tags) The number of elements in the union of two sets (all unique hashtags). Natural constraint on range: Rel∈[0,1] Rel=1 indicates that the themes are completely identical; Rel=0 indicates that it is a completely new topic.

[0045] It's important to clarify that the fundamental difference between semantic similarity and topic relevance lies in the fact that semantic similarity measures whether sentences are "similar," while topic relevance measures whether the conversation is about the same thing or the same area. A typical scenario is that different questions on the same topic may have low semantic similarity but high topic relevance; this metric can effectively preserve this type of valuable historical information.

[0046] The logical dependency degree represents whether the current question depends on the preconditions, contextual logic, established conclusions, and constraint rules of the historical dialogue, and determines whether the historical content is a necessary logical premise for the current question and answer.

[0047] The calculation method adopts a quantization approach that combines rule features with semantic logic reasoning, and the final output is Dep∈[0,1].

[0048] Step A: First define logical dependency features. The logical dependency degree Dep_i is calculated in the following way: Extract the logical features between the i-th historical dialogue unit and the current round of user input. The logical features include referential dependency, conditional dependency, and continuation dependency. Each logical feature corresponds to a fixed score. The total score is 1, as shown in the table below: 0.4 + 0.3 + 0.3 = 1.

[0049]

[0050] Step B: Calculate the logical dependency degree based on the hit logical features and their corresponding preset scores:

[0051] in, The text entered by the user in the current round; : The i-th historical dialogue unit (single-turn Q&A: user asks + model answers); m is the number of logical features hit, which is a positive integer; Score_j is the score corresponding to the j-th hit logical feature, and Dep_i∈[0,1]; When Dep=0: There is no logical dependency, and the historical content is irrelevant to the current question and answer; When Dep=1: The current question completely depends on this history as the preceding logic.

[0052] For example: Historical conversation: Help me calculate what 20% of 100 is. Current conversation: Then calculate 30%. If the dependency is a directive and a dependency is a continuation, and the Dep value is close to 1, it is considered a strong dependency and must be retained. Historical dialogue: Introducing artificial intelligence; Current topic: How's the weather today? If there are no logical characteristics and Dep=0, it is determined to be undependent and can be compressed / removed.

[0053] Further illustrative examples using the above formulas: Current input: Then multiply this result by 2. Historical dialogue: 100 × 20% = 20 calculate: Sim=0.75 (semantic similarity) Rel=0.9 (also a topic of mathematical calculation) Dep=1.0 (Strong reference + logical dependency) Let α = 0.3, β = 0.4, and γ = 0.3. S=0.3×0.75+0.4×0.9+0.3×1.0=0.885 If the filtering threshold T=0.5, and 0.885>0.5, it is considered valid content and will be retained.

[0054] This application's embodiments abandon the traditional one-size-fits-all fixed compression mode, dynamically filtering key information based on semantic relevance, fully preserving real-time dialogue details and cross-turn core logic, effectively solving the problem that traditional compression algorithms easily lose key information and cause question-answer misalignment. Tests have shown that, at the same compression rate, the dialogue accuracy of this invention is significantly better than traditional schemes such as fixed truncation and fixed summarization.

[0055] S40. Topic drift detection steps: Determine whether topic drift has occurred by the semantic vector offset of the previous and subsequent dialogues. If topic drift is determined, the low-association long-term memory corresponding to the previous topic is automatically marked as expired memory and will no longer be included in the current context splicing, while retaining the cross-topic common core information. The topic drift detection steps specifically include: S401. Calculate the topic vector offset between adjacent rounds of dialogue. When the offset exceeds a preset threshold, it is determined to be topic drift. S402: Automatically mark low-association long-term memories of the previous topic as expired memories and no longer include them in the current round of context splicing; at the same time, retain common content such as cross-topic background information, core settings, and key user needs to avoid information loss. The topic vector offset is calculated as follows: Get the previous dialogue vector and the user input vector of the current round Calculate the cosine similarity between the two:

[0056] Calculate the topic vector offset D: The value range is [0,2]. The larger the value, the greater the difference in the topics. A topic shift is determined to occur when D > D_threshold, where D_threshold is a preset shift determination threshold.

[0057] The method described in this application can perfectly adapt to various multi-turn interaction scenarios, such as single-topic continuous dialogue, multi-topic alternating dialogue, topic backtracking, and branching questions. Through the topic drift detection mechanism, it can accurately identify topic switching and automatically manage the memory boundaries of different topics, and its adaptive capability is far superior to traditional static compression algorithms.

[0058] S50, Adaptive throttling compression steps: Real-time monitoring of the total amount of tokens currently occupied in the context, dynamic adjustment of compression intensity based on the current amount of tokens occupied, and differential compression of memory content at each level; The system presets a global token limit threshold and monitors the token usage in the current context in real time. When the token usage is below the low threshold L_threshold: only shallow noise reduction is performed, fully preserving dialogue details with zero compression loss; When the token holding is in the middle range: perform regular layered compression, simplify redundant statements, and retain the core logic; When the token usage exceeds the high threshold H_threshold: an extreme throttling strategy is activated to strengthen the long-term memory summary, significantly remove low-relevance historical content, and prioritize the computing power for current dialogue reasoning. The low threshold L_threshold = 0.6 × Token_max, and the high threshold H_threshold = 0.9 × Token_max, where Token_max is the preset maximum number of tokens; When token usage exceeds the high threshold H_threshold: An extreme throttling strategy is activated, which includes: Full real-time hot memory retains the complete content, preserving the original text of the last 3 rounds of dialogue without deletion, ensuring the details of the current dialogue; Short-term memory is heavily simplified by removing adjectives, small talk, and repetitive expressions, retaining only the core subject and predicate of the question and answer, key values, and constraints. Heavy summarization of long-term cold memories, completely discarding the original text of early dialogues, and retaining only a minimalist structured summary of key points; Low-value historical conversations with a comprehensive score below the screening threshold will be eliminated in batches. Historical conversations with a comprehensive score below the screening threshold and those that have been determined to be outdated will be deleted and will no longer be included in the context.

[0059] The embodiments of this application can be perfectly adapted to various multi-turn interaction scenarios such as single-topic continuous dialogue, multi-topic alternating dialogue, topic backtracking, and branching questions. Through the topic drift detection mechanism, it can accurately identify topic switching and automatically manage the memory boundaries of different topics, and its adaptive capability is far superior to traditional static compression algorithms.

[0060] S60, Incremental Summary Update and Context Reconstruction Steps: Based on the existing long-term memory summary, incremental updates are performed to add key information and correct expired logic, generating iterative new summaries. The real-time hot memory complete content, the short-term warm memory simplified content, and the long-term cold memory incremental summary are reconstructed into the model input context.

[0061] Specifically, abandoning the high-loss traditional approach of rewriting the entire summary in each round, an incremental update mechanism is adopted for long-term cold memory. That is, based on the stored historical basic summary, only the newly added key information in the short-term warm memory is supplemented and the invalid and outdated logic in the summary is corrected to generate an iterative new version summary. Finally, the context sent into the large model is composed of three parts: complete real-time hot memory text + simplified short-term warm memory text + incrementally updated long-term cold memory structured summary.

[0062] This application's embodiments employ an incremental digest update mechanism instead of a full digest rewrite, eliminating the need for additional large-scale model computational overhead. Utilizing a lightweight computational approach with minimal operational costs, it achieves a virtuous cycle where the cost savings far outweigh the losses.

[0063] S70. Repeat steps S10-S60 in multiple iterations.

[0064] After each round of dialogue reasoning is completed, the above steps S10-S60 are repeated to dynamically update the three-level memory hierarchy, semantic weights, and summary content, thereby realizing a token throttling optimization method for continuous, dynamic, and low-loss multi-round dialogue.

[0065] The embodiments of this application do not rely on the structural design of a specific large model and can be adapted to all mainstream LLM models (including GPT series, Claude series, Llama series, Wenxin Yiyan, Tongyi Qianwen, etc.). It can be seamlessly embedded into various application scenarios such as intelligent customer service, human-computer dialogue, RAG system, and AI assistant, with low implementation cost and extremely strong compatibility and scalability.

[0066] The hierarchical dynamic memory-based throttling method described in this application relies on three-level memory hierarchical differentiated processing, dynamic low-relevance content filtering, and incremental summary iteration mechanism. In ordinary multi-turn dialogue scenarios, the token compression rate can reach 40%–80%. The more dialogue rounds, the more significant the throttling optimization effect, which can greatly reduce GPU computing power, server memory, and commercial API call costs. This application accurately filters key logic, preconditions, and core user demands across rounds based on three-dimensional semantic weights; it completely retains the latest dialogue details in real-time hot memory without losing referential, follow-up, and temporary constraint information; the question-and-answer accuracy remains unaffected in long dialogue scenarios; short-term warm memory retains core question-and-answer logic and eliminates modifying and repetitive statements; and long-term cold memory no longer stores the original complete text, but only stores iteratively generated structured incremental summaries, thereby achieving token compression rate savings and effective throttling.

[0067] Example 2 Based on the above embodiments, this application also provides a hierarchical dynamic memory throttling processing device. The device is applied to the method described in the first embodiment for throttling processing. The device includes a data acquisition and preprocessing unit 10, a memory hierarchical unit 20, a semantic dynamic scoring unit 30, a topic drift detection unit 40, an adaptive throttling compression unit 50, and a summary update and reconstruction unit 60 that are interconnected and interact with each other.

[0068] The data acquisition and preprocessing unit 10 is used to acquire raw data from multi-turn dialogues in real time, perform shallow noise reduction preprocessing, remove explicit redundant text, and output standardized dialogue units.

[0069] Specifically, data such as user questions, model responses, dialogue timestamps, and round numbers are collected in real time during multi-turn dialogues. The collected dialogue data undergoes shallow noise reduction preprocessing to remove explicit redundant text, such as filtering out pure punctuation, empty statements, and casual greetings without actual semantic meaning, thus completing initial lightweight noise reduction.

[0070] The memory layering unit 20 is used to dynamically divide historical dialogues into three levels—real-time hot memory, short-term warm memory, and long-term cold memory—based on dialogue rounds, time weights, and semantic heat, and to implement differentiated information processing strategies for the memory content at each level.

[0071] Specifically, memory levels are divided based on dialogue rounds, and the processing strategies for each level are dynamically adjusted according to semantic popularity. All historical dialogue units are divided into three storage spaces: real-time hot memory, short-term warm memory, and long-term cold memory, with differentiated information retention strategies. The retention strategy for each type of storage space is as follows: The real-time hot memory is the complete original content of the most recent N rounds of dialogue, where N is a preset positive integer, preferably 3 rounds, and no compression processing is performed on the real-time hot memory; The short-term warm memory is a lightweight and simplified version of the dialogue from round N+1 to round M, where M is a preset positive integer greater than N, preferably M=10. The core question-and-answer logic is retained in the short-term warm memory, while embellishment and repetitive statements are removed. The long-term cold memory consists of incremental structured summaries of historical dialogues up to round M, without retaining the original dialogue text. All historical dialogues up to round 10 are no longer stored in their original, complete text; only iteratively generated structured incremental summaries are stored, achieving extreme token throttling.

[0072] The semantic dynamic scoring unit 30 is used to combine the current round of user input text and calculate the semantic weight score of each historical dialogue unit from three dimensions: semantic similarity, topic relevance, and logical dependency. It filters effective historical information based on a preset filtering threshold and removes low-relevance redundant content.

[0073] Specifically, for historical dialogue units corresponding to short-term warm memory and long-term cold memory, the effective weight score S of a single historical dialogue is quantitatively calculated from three dimensions: semantic similarity, topic relevance, and logical dependency, based on the current user input text. The closer the S value is to 1, the stronger the relevance.

[0074] The semantic weight score S_i of the i-th historical dialogue unit is calculated using the following formula: S_i = α × Sim_i + β × Rel_i + γ × Dep_i Where Sim_i is the semantic similarity between the i-th historical dialogue unit and the current round of user input, Rel_i is the topic relevance between the i-th historical dialogue unit and the current round of user input, and Dep_i is the logical dependency between the i-th historical dialogue unit and the current round of user input; α, β, and γ are preset weight coefficients, α∈[0,1], β∈[0,1], and γ∈[0,1], and satisfy α+β+γ=1.

[0075] The semantic similarity Sim_i is calculated in the following way: Input text from the current user in the current round. Convert to semantic vector Convert the i-th historical dialogue unit H_i into a semantic vector. Calculate the cosine similarity between the two. The calculation results are then corrected for their range to ensure that Sim_i ∈ [0,1]. The formula is as follows:

[0076] The calculation results are automatically normalized to [-1, 1], and the range is corrected.

[0077] After correction, Sim∈[0,1], which conforms to the unified value specification.

[0078] The topic relevance calculation employs a dual-path decision method combining topic clustering and theme matching; the semantic dynamic scoring unit includes a topic extraction unit 301 and a topic relevance calculation unit 302. The topic extraction unit 301 is used to extract topic tags to obtain the currently input topic tag set. (Keywords / Domain Tags) and H_i's Collection of Topic Tags in Historical Conversations .

[0079] The topic relevance calculation unit 302 is used to calculate the topic relevance based on the intersection of sets.

[0080] The logical dependency degree represents whether the current question depends on the preconditions, contextual logic, established conclusions, and constraint rules of the historical dialogue, and determines whether the historical content is a necessary logical premise for the current question and answer.

[0081] The calculation method adopts a quantization approach that combines rule features with semantic logic reasoning, and the final output is Dep∈[0,1].

[0082] The topic drift detection unit 40 is used to determine whether topic drift has occurred by the semantic vector offset of the previous and subsequent dialogues. If topic drift is determined, the low-association long-term memory corresponding to the previous topic is automatically marked as expired memory and is no longer included in the current context splicing, while retaining the cross-topic common core information.

[0083] The topic vector offset is calculated as follows: Get the previous dialogue vector and the user input vector of the current round Calculate the cosine similarity between the two:

[0084] Calculate the topic vector offset D: The value range is [0,2]. The larger the value, the greater the difference in the topics. A topic shift is determined to occur when D > D_threshold, where D_threshold is a preset shift determination threshold.

[0085] The adaptive throttling compression unit 50 is used to monitor the total amount of tokens currently occupied in the context in real time, dynamically adjust the compression intensity based on the current amount of tokens occupied, and perform differentiated compression on the memory content at each level.

[0086] Specifically, the system presets a global token upper limit threshold and monitors the token usage of the current context in real time: When the token usage is below the low threshold L_threshold: only shallow noise reduction is performed, fully preserving dialogue details with zero compression loss; When the token holding is in the middle range: perform regular layered compression, simplify redundant statements, and retain the core logic; When the token usage exceeds the high threshold H_threshold: an extreme throttling strategy is activated to strengthen the long-term memory summary, significantly remove low-relevance historical content, and prioritize the computing power for current dialogue reasoning. The low threshold L_threshold = 0.6 × Token_max, and the high threshold H_threshold = 0.9 × Token_max, where Token_max is the preset maximum number of tokens; When token usage exceeds the high threshold H_threshold: An extreme throttling strategy is activated, which includes: Full real-time hot memory retains the complete content, preserving the original text of the last 3 rounds of dialogue without deletion, ensuring the details of the current dialogue; Short-term memory is heavily simplified by removing adjectives, small talk, and repetitive expressions, retaining only the core subject and predicate of the question and answer, key values, and constraints. Heavy summarization of long-term cold memories, completely discarding the original text of early dialogues, and retaining only a minimalist structured summary of key points; Low-value historical conversations with a comprehensive score below the screening threshold will be eliminated in batches. Historical conversations with a comprehensive score below the screening threshold and those that have been determined to be outdated will be deleted and will no longer be included in the context.

[0087] The summary update and reconstruction unit 60, based on the existing long-term memory summary, incrementally updates and adds key information and corrects expired logic to generate an iterative new summary, reconstructing the real-time hot memory complete content, the short-term warm memory simplified content, and the long-term cold memory incremental summary into the model input context.

[0088] This application abandons the high-loss scheme of traditional full-scale summary rewriting in each round. Instead, it adopts an incremental update mechanism for long-term cold memory through the summary update and reconstruction unit 60. That is, based on the stored historical basic summary, it only supplements the newly added key information in short-term warm memory and corrects the invalid and outdated logic in the summary to generate an iterative new version summary. Finally, the context sent to the large model is composed of three parts: complete real-time hot memory text + simplified short-term warm memory text + incrementally updated long-term cold memory structured summary.

[0089] The hierarchical dynamic memory throttling device described in this application uses a memory hierarchical unit 20 to perform three-level memory hierarchical differentiation processing on dialogue rounds, time weights, and semantic heat. The semantic dynamic scoring unit 30 calculates the semantic weight score of each historical dialogue unit based on three dimensions: semantic similarity, topic relevance, and logical dependency of the current user input text. Valid historical information is filtered based on a preset screening threshold, and low-relevance redundant content is removed. With dynamic low-relevance content filtering and incremental summary iteration mechanisms, the token compression rate in ordinary multi-turn dialogue scenarios can reach 40% to 80%. The more dialogue rounds there are, the more significant the throttling optimization effect, which can greatly reduce GPU computing power, server video memory, and commercial API call costs. Through the topic drift detection unit 40, the adaptive throttling compression unit 50, and the summary update and reconstruction unit 60, key logic, preconditions, and core user demands across rounds are accurately selected. The latest dialogue details are fully preserved in real-time hot memory without losing referential, follow-up, or temporary constraint information. The accuracy of question and answer is not lost in long dialogue scenarios. The core question and answer logic is preserved in short-term warm memory, while decorative and repetitive statements are removed. The original complete text is no longer stored in long-term cold memory; only the iteratively generated structured incremental summary is stored. This can save token compression rate and achieve effective throttling.

[0090] Example 3 Based on the above embodiments, the present invention also proposes a computer-readable storage medium, including a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the computer program is executed by the processor, it implements the steps of the hierarchical dynamic memory throttling processing method as described in any of the above embodiments.

[0091] It should be noted that the above-described medium embodiments and method embodiments belong to the same concept. The specific implementation process can be found in the method embodiments, and the technical features in the method embodiments are also applicable to the medium embodiments, which will not be repeated here.

[0092] The hierarchical dynamic memory-based throttling method, apparatus, and computer-readable storage medium of this invention achieve a token compression rate of 40%–80% in ordinary multi-turn dialogue scenarios through three-level memory hierarchical differentiated processing, dynamic low-relevance content filtering, and incremental summary iteration mechanism. The more dialogue rounds, the more significant the throttling optimization effect, which can greatly reduce GPU computing power, server video memory, and commercial API call costs. This application accurately filters key logic, preconditions, and core user demands across rounds based on three-dimensional semantic weights; it completely retains the latest dialogue details in real-time hot memory without losing referential, follow-up, and temporary constraint information, and maintains no loss of question-and-answer accuracy in long dialogue scenarios; short-term warm memory retains core question-and-answer logic and eliminates modifying and repetitive statements; and long-term cold memory no longer stores the original complete text, but only stores the iteratively generated structured incremental summaries, thereby achieving token compression rate savings and effective throttling processing.

[0093] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0094] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0095] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0096] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A hierarchical dynamic memory-based throttling method, characterized in that, The method includes: Data acquisition and preprocessing steps: Real-time acquisition of raw data from multi-turn dialogues and shallow noise reduction preprocessing to remove explicit redundant text and output standardized dialogue units; The three-level memory stratification process involves dynamically dividing historical dialogues into three levels—real-time hot memory, short-term warm memory, and long-term cold memory—based on dialogue rounds, time weight, and semantic heat. Differentiated information processing strategies are then applied to the memory content at each level. Semantic dynamic scoring steps: Combine the current round of user input text, calculate the semantic weight score of each historical dialogue unit from three dimensions: semantic similarity, topic relevance, and logical dependency. Filter valid historical information based on preset filtering thresholds and remove low-relevance redundant content. Topic drift detection steps: Determine whether topic drift has occurred by the semantic vector offset between the previous and subsequent dialogues. If topic drift is determined, the low-association long-term memory corresponding to the previous topic is automatically marked as expired memory and will no longer be included in the current context splicing, while retaining the cross-topic common core information. Adaptive throttling compression steps: Real-time monitoring of the total amount of tokens currently occupied in the context, dynamic adjustment of compression intensity based on the current amount of tokens occupied, and differential compression of memory content at each level; Incremental summary update and context reconstruction steps: Based on the existing long-term memory summary, incremental updates are performed to add key information and correct expired logic, generating iterative new summaries. The real-time hot memory complete content, the short-term warm memory simplified content, and the long-term cold memory incremental summaries are reconstructed into the model input context.

2. The hierarchical dynamic memory-based throttling method according to claim 1, characterized in that, The three-level memory stratification steps include: The real-time hot memory is the complete original content of the most recent N rounds of dialogue, where N is a preset positive integer, and no compression processing is performed on the real-time hot memory; The short-term warm memory is a lightweight and simplified version of the dialogue from round N+1 to round M, where M is a preset positive integer greater than N. The core question-and-answer logic is retained in the short-term warm memory, while decorative and repetitive statements are removed. The long-term cold memory refers to the incremental structured summary of historical dialogues prior to round M, without retaining the original dialogue text.

3. The hierarchical dynamic memory-based throttling method according to claim 2, characterized in that, In the semantic dynamic scoring step, the semantic weight score S_i of the i-th historical dialogue unit is calculated using the following formula: S_i = α × Sim_i + β × Rel_i + γ × Dep_i Where Sim_i is the semantic similarity between the i-th historical dialogue unit and the current round of user input, Rel_i is the topic relevance between the i-th historical dialogue unit and the current round of user input, and Dep_i is the logical dependency between the i-th historical dialogue unit and the current round of user input; α, β, and γ are preset weight coefficients, α∈[0,1], β∈[0,1], and γ∈[0,1], and satisfy α+β+γ=1.

4. The hierarchical dynamic memory-based throttling method according to claim 3, characterized in that, The semantic similarity Sim_i is calculated in the following way: Input text from the current user in the current round. Convert to semantic vector Convert the i-th historical dialogue unit H_i into a semantic vector. Calculate the cosine similarity between the two. The calculation results are then corrected for their range to ensure that Sim_i ∈ [0,1]. The formula is as follows: ； In the above formula, Mole: The text entered by the user in the current round Corresponding semantic vector The i-th historical dialogue unit H_i is converted into a semantic vector. The dot product of two vectors; Denominator: User input text in the current round Corresponding semantic vector The i-th historical dialogue unit H_i is converted into a semantic vector. The product of the L2 norms of two vectors (vector magnitudes); The calculation results are automatically normalized to [-1, 1], and the range is corrected. ; After correction, Sim∈[0,1], which conforms to the unified value specification; The meanings of the letters in the above formulas are as follows: The text entered by the user in the current round; : The i-th historical dialogue unit (single-turn Q&A: user asks + model answers); : The corresponding semantic vector (Embedding); Historical Dialogue The corresponding semantic vector.

5. The hierarchical dynamic memory throttling method according to claim 3, characterized in that, The logical dependency degree Dep_i is calculated in the following way: Extract the logical features between the i-th historical dialogue unit and the current round of user input, the logical features including referential dependency, conditional dependency and continuation dependency; Calculate the logical dependency degree based on the hit logical features and their corresponding preset scores: ; in, The text entered by the user in the current round; : The i-th historical dialogue unit (single-turn Q&A: user asks + model answers); m is the number of logical features hit, which is a positive integer; Score_j is the score corresponding to the j-th hit logical feature, and Dep_i∈[0,1]; When Dep=0: There is no logical dependency, and the historical content is irrelevant to the current question and answer; When Dep=1: The current question completely depends on this history as the preceding logic.

6. The hierarchical dynamic memory-based throttling method according to claim 3, characterized in that, The topic drift detection steps include: Calculate the topic vector offset between adjacent rounds of dialogue. When the offset exceeds a preset threshold, it is determined to be topic drift. Automatically mark low-relevance long-term memories of the previous topic as expired memories, and no longer include them in the current round of context splicing; at the same time, retain common background information, core settings, key user needs and other common content across topics to avoid information loss; The topic vector offset is calculated as follows: Get the previous dialogue vector and the user input vector of the current round Calculate the cosine similarity between the two: ; Calculate the topic vector offset D: The value range is [0,2]. The larger the value, the greater the difference in the topics. A topic shift is determined to occur when D > D_threshold, where D_threshold is a preset shift determination threshold.

7. The hierarchical dynamic memory-based throttling method according to claim 1, characterized in that, The adaptive throttling compression step includes: The system presets a global token limit threshold and monitors the token usage in the current context in real time. When the token usage is below the low threshold L_threshold: only shallow noise reduction is performed, fully preserving dialogue details with zero compression loss; When the token holding is in the middle range: perform regular layered compression, simplify redundant statements, and retain the core logic; When the token usage exceeds the high threshold H_threshold: an extreme throttling strategy is activated to strengthen the long-term memory summary, significantly remove low-relevance historical content, and prioritize the computing power for current dialogue reasoning. The low threshold L_threshold = 0.6 × Token_max, and the high threshold H_threshold = 0.9 × Token_max, where Token_max is the preset maximum number of tokens; When the token usage exceeds the high threshold H_threshold: the extreme throttling strategy is activated, which includes: preserving the complete content of real-time hot memory, strongly simplifying short-term warm memory, heavily summarizing long-term cold memory, and batch eliminating low-value historical dialogues with a comprehensive score lower than the screening threshold.

8. The method according to claim 1, characterized in that, The incremental summary update and context reconstruction steps include: The underlying summary of the long-term cold memory storage; When the short-term warm memory accumulates to a preset number of rounds, new key information and change information are extracted from the short-term warm memory. The new key information is added to the basic summary, and the expiration logic in the basic summary is corrected based on the change information to generate a new version summary. The model input context reconstruction is performed by splicing together the complete content of real-time hot memory, the simplified content of short-term warm memory, and the incremental summary of long-term cold memory.

9. A hierarchical dynamic memory throttling device, characterized in that, include: The data acquisition and preprocessing module is used to acquire multi-turn dialogue data in real time and perform shallow noise reduction preprocessing to remove explicit redundant text and output standardized dialogue units. The three-level memory layering module is used to dynamically divide historical dialogues into three levels: real-time hot memory, short-term warm memory, and long-term cold memory based on dialogue rounds, time weights, and semantic heat. The semantic dynamic scoring module is used to calculate the semantic weight score of each historical dialogue unit from three dimensions: semantic similarity, topic relevance, and logical dependency. The topic drift detection module is used to calculate the topic vector offset between adjacent rounds of dialogue and determine topic drift. An adaptive throttling compression module is used to dynamically adjust the compression intensity based on the real-time token usage. The incremental summary update and context reconstruction module is used to perform incremental updates based on existing long-term memory summaries and reconstruct the model input context.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a hierarchical dynamic memory throttling program, which, when executed by a processor, implements the steps of the hierarchical dynamic memory throttling method as described in any one of claims 1 to 8.

Images