Conversation method and device based on KV cache and large language model, equipment and medium

Abstract

The invention discloses a conversation method, device and equipment based on a KV cache and a large language model and a medium, and relates to the technical field of large models, and the method comprises the steps: obtaining and storing the KV cache in real time, and generating a semantic abstract vector according to a KV cache screening result; determining a target score according to the occurrence record of the semantic unit block, and determining whether to store the semantic unit block and the semantic abstract vector to a first database based on the target score; storing the first KV cache in a second database, converting the new problem of the user into a query vector, retrieving a target semantic abstract vector of which the semantic similarity with the query vector meets a target threshold in the first database, and determining a second KV cache based on the target semantic abstract vector; and splicing the second KV cache to obtain spliced data, adjusting the spliced data by using the target lightweight neural network, and reasoning based on the adjusted data and the large language model to obtain a reply corresponding to the new question of the user. And efficient and long-term reuse of the dialogue history is realized.

Description

Technical Field

[0001] This invention relates to the field of large model technology, and in particular to dialogue methods, apparatus, devices and media based on KV caching and large language models. Background Technology

[0002] In the autoregressive inference process of Large Language Models (LLMs) (such as the GPT (Generative Pre-trained Transformer) series and LLaMA series), the model generates output token by token through the Transformer decoder. To improve efficiency and avoid redundant computation of historical tokens, the model caches the key and value vectors of already generated tokens in memory, forming a "KV cache". As LLM applications expand to long contexts (such as 100,000+ tokens), the memory footprint (tens of gigabytes per dialogue) and computational redundancy of the KV cache become bottlenecks. Existing well-known optimization techniques are mainly divided into three categories: Memory compression: Reduce cache size through quantization (e.g., FP16->INT8), pruning (removing key-value pairs with low attention weights), and sharing (multi-head attention key-value pair dimensionality reduction sharing).

[0003] Cache reuse: Cache key-value pairs for recurring text prefixes (such as system prompts and fixed instructions) to avoid redundant calculations (i.e., "prefix caching").

[0004] State management: Inactive caches are temporarily stored in external storage and loaded on demand (i.e., "store-based computation").

[0005] However, current solutions only address the basics of optimizing key-value (KV) caching for text symbol associations, failing to handle issues such as semantically similar but textually different states, redundant long-term context states, and cross-session state preservation. Therefore, resolving the shortcomings of existing technologies—namely, manipulating text symbols rather than computational states, ignoring attention value in static summarization, lacking semantic long-term storage, and relying on full caching for reuse—and ultimately achieving intelligent, long-term reuse of computational states is a pressing issue. Summary of the Invention

[0006] In view of this, the purpose of this invention is to provide a dialogue method, apparatus, device, and medium based on KV caching and a large language model. This invention addresses the shortcomings of existing technologies caused by "operating on text symbols rather than computational states," "static summaries ignoring attention value," "storage not being semantically long-term," and "reuse relying on the complete cache," by reconstructing the management objects, storage strategies, and reuse methods of the KV cache. Ultimately, it achieves intelligent, long-term reuse of computational states. The specific solution is as follows: Firstly, this application discloses a dialogue method based on KV caching and a large language model, including: The key-value cache generated in each round of dialogue during the reasoning process of the large language model is acquired and stored in real time. The key-value cache is filtered according to the attention weight of the large language model, and a semantic summary vector is generated according to the corresponding filtering results. The target score is determined based on the occurrence record of the semantic unit block in the historical multi-turn dialogue. Based on the target score, it is determined whether to store the semantic unit block and the semantic summary vector corresponding to the semantic unit block in the first database. The semantic unit block is used to represent a group of consecutive words or characters whose vector cosine similarity satisfies a preset threshold. The first KV cache that meets the preset conditions in the KV cache is stored in the second database. The user's new question is converted into a query vector. The target semantic summary vector in the first database that has a semantic similarity to the query vector that meets the target threshold is retrieved. The second KV cache is determined based on the target semantic summary vector. The query vector is a vector with the same dimension as the semantic summary vector. The second KV cache is concatenated to obtain corresponding concatenated data. The concatenated data is adjusted using a target lightweight neural network. Reasoning is performed based on the adjusted data and the large language model to obtain the answer to the user's new question.

[0007] Optionally, the real-time acquisition and storage of the key-value cache generated in each round of dialogue during the large language model inference process includes: The key-value caches of each output layer of the large language model are obtained in real time through hook functions; The KV cache is serialized into a binary file using the dialogue round ID and output layer number as keys, and the binary file is stored in a memory buffer.

[0008] Optionally, the step of filtering the KV cache according to the attention weights of the large language model and generating a semantic summary vector based on the corresponding filtering results includes: The attention weights are determined based on the attention matrix of the last layer of the decoder in the large language model. Each word in the KV cache is valued according to the attention weight to obtain the corresponding score result; The KV cache is filtered based on the relationship between the scoring results and the first threshold to obtain the corresponding filtering results; The filtering results are flattened into a one-dimensional vector, and the one-dimensional vector is compressed into a low-dimensional vector by an autoencoder. The low-dimensional vector is then determined as the semantic summary vector.

[0009] Optionally, determining the target score based on the occurrence record of semantic unit blocks in historical multi-turn dialogues includes: The frequency of occurrence, mean weight, and variance of a semantic unit block are determined based on its occurrence records in historical multi-turn dialogues; the mean weight is the average attention weight of the semantic unit block in each occurrence; and the variance of the weight is the variance of the weight of the semantic unit block in each occurrence. Determine the target sum of 1 and the weighted variance; Determine the target ratio between the weighted mean and the target sum; The product of the frequency of occurrence and the target ratio is determined as the target score; Accordingly, determining whether to store the semantic unit block and the corresponding semantic summary vector in the first database based on the target score includes: If the target score is greater than the second threshold, the semantic unit block and the semantic summary vector corresponding to the semantic unit block are stored in the first database.

[0010] Optionally, storing the first KV cache that meets the preset conditions in the KV cache to the second database includes: According to the circular queue structure, the first KV cache in the KV cache of the target round dialogue whose score result is greater than the first threshold is stored in the second database.

[0011] Optionally, retrieving target semantic summary vectors from the first database whose semantic similarity to the query vector meets a target threshold includes: The target semantic summary vector in the first database that has a semantic similarity to the query vector that meets the target threshold is retrieved using an approximate nearest neighbor search algorithm.

[0012] Optionally, determining the second KV cache based on the target semantic summary vector includes: Based on the ID of the target semantic summary vector, query the associated filter result ID in the metadata table, and determine the second KV cache from the first database or the second database based on the filter result ID.

[0013] Optionally, concatenating the second KV cache to obtain corresponding concatenated data includes: The second KV cache is concatenated according to the word order of the second KV cache to obtain the corresponding concatenated data; Accordingly, the step of adjusting the concatenated data using a target lightweight neural network, and performing inference based on the adjusted data and the large language model to obtain the answer to the user's new question, includes: The concatenated data is adjusted using a target lightweight neural network, and the adjusted data is stored in the decoder KV cache buffer of the large language model so that the large language model can generate a response to the user's new question based on the adjusted data and the text segmentation of the user's new question.

[0014] Secondly, this application discloses a dialogue device based on KV caching and a large language model, comprising: The cache filtering module is used to acquire and store the key-value cache generated in each round of dialogue during the reasoning process of the large language model in real time, filter the key-value cache according to the attention weight of the large language model, and generate semantic summary vectors according to the corresponding filtering results. The vector storage module is used to determine the target score based on the occurrence record of the semantic unit block in the historical multi-turn dialogue, and to determine whether to store the semantic unit block and the semantic summary vector corresponding to the semantic unit block into the first database based on the target score; the semantic unit block is used to represent a group of consecutive words or characters whose vector cosine similarity satisfies a preset threshold. The second KV cache determination module is used to store the first KV cache that meets the preset conditions in the KV cache to the second database, convert the user's new question into a query vector, retrieve the target semantic summary vector in the first database whose semantic similarity to the query vector meets the target threshold, and determine the second KV cache based on the target semantic summary vector; the query vector is a vector with the same dimension as the semantic summary vector. The inference module is used to concatenate the second KV cache to obtain corresponding concatenated data, adjust the concatenated data using a target lightweight neural network, and perform inference based on the adjusted data and the large language model to obtain the answer corresponding to the user's new question.

[0015] Thirdly, this application discloses an electronic device, including: Memory, used to store computer programs; A processor is used to execute computer programs to implement dialogue methods based on key-value caching and large language models, as described above.

[0016] Fourthly, this application discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the aforementioned dialogue method based on KV caching and a large language model.

[0017] This application first acquires and stores the key-value (KV) cache generated in each round of dialogue during the reasoning process of a large language model in real time. The KV cache is then filtered according to the attention weights of the large language model, and a semantic summary vector is generated based on the filtering results. A target score is determined based on the occurrence records of semantic unit blocks in historical multi-round dialogues. Based on the target score, it is determined whether to store the semantic unit block and its corresponding semantic summary vector in a first database. The semantic unit block represents a group of consecutive words or characters whose vector cosine similarity satisfies a preset threshold. The first KV cache that meets the preset conditions is stored in a second database. The user's new question is converted into a query vector. Target semantic summary vectors in the first database whose semantic similarity to the query vector satisfies the target threshold are retrieved. A second KV cache is determined based on the target semantic summary vector. The query vector is a vector with the same dimension as the semantic summary vector. Finally, the second KV cache is concatenated to obtain concatenated data. The concatenated data is adjusted using a target lightweight neural network. Reasoning is performed based on the adjusted data and the large language model to obtain the answer to the user's new question. As can be seen, this application directly manages the intermediate computational states (KV cache tensors) generated during the reasoning process of a large language model, thereby manipulating the model's "understanding state" rather than merely processing surface text. It identifies high-semantic-value tokens by analyzing the model's internal attention weights and filters redundant caches; combined with autoencoder compression to generate low-dimensional semantic summary vectors, it achieves a lightweight representation of the computational state. A two-layer storage structure is constructed, and finally, through three steps—query vector generation, semantic retrieval, and concatenation of core cache blocks—a closed-loop reuse of "retrieval-location-injection" is achieved, skipping the recomputation of the original text. This enables cross-text reuse, reducing redundant computation by more than 70%. It systematically solves the state management problem in long-context interactions. This not only achieves efficient reuse of semantically similar states across texts and sessions but also ensures the high fidelity and high security of the reused states. The system can automatically identify and retain universal knowledge states for a long time, while adapting them flexibly and dynamically to new contexts when reused. This truly endows the large language model with stable, reliable and accurate "persistent memory" capabilities, fundamentally overcoming the shortcomings of existing technologies such as rigid cached content, high reuse risk and limited long-term value. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0019] Figure 1 This is a flowchart of a dialogue method based on KV caching and a large language model disclosed in this application; Figure 2 This is a schematic diagram of a KV cache acquisition and storage method disclosed in this application; Figure 3 This is a schematic diagram of a two-layer semantic caching library disclosed in this application; Figure 4 This is a schematic diagram of a second KV cache determination method disclosed in this application; Figure 5 This is a schematic diagram of a cache injection and inference process disclosed in this application; Figure 6 This is a schematic diagram of a dialogue system architecture based on KV caching and a large language model disclosed in this application; Figure 7 This is a flowchart of intelligent management and cross-session reuse of KV cache disclosed in this application; Figure 8 This is a schematic diagram of a dialogue device structure based on KV caching and a large language model disclosed in this application; Figure 9 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] In some existing technologies, the inability to reuse computational states that are semantically similar but have different texts is due to the limitation of "operation objects to text symbols". The static / coarse-grained nature of the summarization mechanism (without incorporating model attention weights) prevents the dynamic preservation of "high-value computational states". The failure to separate "computational states" from their attached "text symbols" and the lack of a long-term indexable "semantic summary" prevents computational states from existing independently of the original text, maintaining their freshness over time, and enabling efficient retrieval. Furthermore, the reliance on loading a complete cache for reuse makes it impossible to "accurately reconstruct lightweight computational states". To address these technical problems, this application discloses a dialogue method, apparatus, device, and medium based on KV caching and a large language model. By reconstructing the management objects, storage strategies, and reuse methods of the KV cache, this invention solves the defects of existing technologies caused by "operating on text symbols instead of computational states", "static summarization ignoring attention value", "storage not being semantically long-term", and "reuse relying on a complete cache", ultimately achieving intelligent long-term reuse of computational states.

[0022] See Figure 1 As shown, this embodiment of the invention discloses a dialogue method based on KV caching and a large language model, including: Step S11: Real-time acquisition and storage of KV cache generated in each round of dialogue during the reasoning process of the large language model; filtering of the KV cache according to the attention weight of the large language model; and generating semantic summary vectors according to the corresponding filtering results.

[0023] In this embodiment, the complete key-value (KV) cache generated in each round of dialogue is first captured and stored in real time during model inference, providing raw data for subsequent summarization. The KV cache of each output layer of the large language model is obtained in real time through hook functions; the KV cache is serialized into a binary file based on the dialogue round ID and output layer number as the key, and the binary file is stored in a memory buffer. The process is as follows: Figure 2 As shown, the interception timing is as follows: After each layer's output from the LLM decoder (such as TransformerDecoder), the Key (K) and Value (V) tensors of all tokens in the current layer are captured via a hook function, and stacked layer by layer to form a complete KV cache (dimension: L×H×d, where L = number of tokens, H = number of attention heads, and d = vector dimension). Storage format: Using "dialogue round ID + layer number" as the key, the KV cache is serialized into a binary file (such as Protobuf format) and temporarily stored in a memory buffer (to avoid blocking the main inference thread). Data flow: User asks a question → Model generates a response → After inference, a "record event" is triggered, packaging the complete KV cache (including all layers) for this round and sending it to the "dynamic semantic digester".

[0024] Then, attention weights are determined based on the attention matrix of the last layer of the large language model decoder; each word in the KV cache is value-scored according to the attention weights to obtain the corresponding score result; the KV cache is filtered according to the relationship between the score result and a first threshold to obtain the corresponding filtering result; the filtering result is flattened into a one-dimensional vector, and the one-dimensional vector is compressed into a low-dimensional vector through an autoencoder, and the low-dimensional vector is determined as the semantic summary vector. Specifically, weights (dimension: L×L, where L is the total number of tokens in the current round) are extracted from the attention matrix of the last layer of the LLM decoder (which best reflects global semantic attention). Value scoring: For each token i, its average attention weight on all subsequent tokens is calculated. : ; Among them, A j,i Let represent the attention weight of Token j to Token i. The higher the weight, the more critical the Token is to subsequent reasoning.

[0025] Threshold setting: Based on historical data statistics, a dynamic threshold θ = μ + σ (μ is the average weight, σ is the standard deviation) is set, retaining... The key-value block corresponding to the token with value >θ is defined as the core cache block. For example, a fixed threshold (such as the top 20% of attention weight) is used for the first N rounds (N=10). After completing M rounds of dialogue or accumulating a certain number of tokens, the global μ and σ are recalculated.

[0026] Redundancy filtering: For multi-Token KV blocks of the same semantic unit (such as the continuous noun phrase "market risk of Company A"), merge them into a single "semantic unit block" (determined by the vector cosine similarity of adjacent Tokens > 0.9).

[0027] Semantic summary vector generation: Vector compression: The selected core cache blocks (containing K / V tensors) are flattened into one-dimensional vectors (dimension N×d, where N is the number of core tokens), and compressed into low-dimensional vectors (such as 128-dimensional vectors) by an autoencoder, which serve as "semantic summary vectors".

[0028] Association storage: Associate the "semantic summary vector" with the "core cache block", "round ID" and "time stamp" to form a summary record.

[0029] Step S12: Determine the target score based on the occurrence record of the semantic unit block in the historical multi-turn dialogue, and determine whether to store the semantic unit block and the semantic summary vector corresponding to the semantic unit block in the first database based on the target score; the semantic unit block is used to represent a group of continuous words or characters whose vector cosine similarity satisfies a preset threshold.

[0030] In this embodiment, the frequency of occurrence, mean weight, and variance of a semantic unit block are determined based on its occurrence records in historical multi-turn dialogues; the mean weight is the average attention weight of the semantic unit block in each occurrence; the variance of the weight is the variance of the weight of the semantic unit block in each occurrence; a target sum of 1 and the variance of the weight is determined; a target ratio between the mean weight and the target sum is determined; the product of the frequency of occurrence and the target ratio is determined as the target score; correspondingly, determining whether to store the semantic unit block and the semantic summary vector corresponding to the semantic unit block in the first database based on the target score includes: if the target score is greater than a second threshold, then storing the semantic unit block and the semantic summary vector corresponding to the semantic unit block in the first database.

[0031] Specifically, the core cache blocks and semantic summary vectors are stored in a hierarchical manner to achieve synergy between "short-term precise memory" and "long-term semantic indexing." The process is as follows: Figure 3 As shown. To ensure that the "computational state summary" stored in the long-term memory layer has universality and reusability across dialogue contexts, and to avoid weight bias in a single context, the system designs a dual guarantee mechanism: Multiple rounds of stability checks (quality control during the storage phase): Before deciding to store a "semantic unit block" and its summary vector into the long-term memory layer (first database), the system checks the occurrence records of that semantic unit in historical multi-turn related dialogues. The system maintains a "semantic unit weight stability score" S, which is calculated by taking into account the following factors: Frequency of occurrence: The number of times it appears in dialogues on similar topics.

[0032] Mean weight: The average attention weight α for each occurrence.

[0033] Weight variance: the degree of dispersion of the weights over time, σ ω .

[0034] The scoring formula simplifies to S = frequency × α / (1 + σ) ω Only when the score S is higher than a set threshold is the semantic unit deemed to have stable high value and eligible to be transferred from working memory to long-term memory. This mechanism ensures that long-term memory stores universally valid "core knowledge states" that have been validated over time and in various contexts, rather than temporary contextual focuses.

[0035] Long-term memory layer (long-term semantic index)

[0036] Stored content: "semantic summary vectors" of all historical dialogues, building a vector index library (such as the IVF-PQ (Indexed Vector Fields-Probabilistic Quantization) index of the FAISS (FacebookAI Similarity Search, an open-source similarity search tool) library).

[0037] Data Structures: Vector database: Stores "semantic summary vector (128 dimensions) + associated core cache block ID + timestamp"; Metadata table: Records the mapping relationship between digest vectors and core cache blocks (core cache blocks are actually stored on disk, and only pointers are stored in the long memory layer).

[0038] Search optimization: Cosine similarity is used as the metric, and a similarity threshold τ=0.8 is set (if it is higher than the threshold, it is considered relevant). Top-N search is supported (N=3, returning the 3 most relevant historical topics).

[0039] Eviction strategies: To avoid data being added without being deleted, algorithms such as timestamp-based (LRU (Least Recently Used)) and usage frequency-based (LFU (Least Frequently Used)) can be used to evict data.

[0040] The long-term memory layer essentially constructs a semantic subspace. This space is based on low-dimensional semantic summary vectors and organized using vector indexing technology, enabling the compression, structured storage, and efficient retrieval of the core semantics of massive historical computational states.

[0041] In this way, the present invention uses decoder attention weights to dynamically score and filter core tokens (w i =avg(A j,i By combining autoencoder compression to generate low-dimensional summary vectors, the reused state always contains "semantic anchors" that are crucial for subsequent reasoning.

[0042] Step S13: Store the first KV cache that meets the preset conditions in the KV cache to the second database, convert the user's new question into a query vector, retrieve the target semantic summary vector in the first database that has a semantic similarity to the query vector that meets the target threshold, and determine the second KV cache based on the target semantic summary vector; the query vector is a vector with the same dimension as the semantic summary vector.

[0043] In this embodiment, according to the circular queue structure, the first KV cache whose rating result in the KV cache of the target round dialogue is greater than the first threshold is stored in the second database. Specifically, the working memory layer (second database, short-term precise memory): Storage content: The core cache block of the most recent K rounds of dialogue (K is configurable, such as K=5), which adopts a circular queue structure (first-in, first-out, the earliest round is evicted when the queue is full).

[0044] Data structure: Hash table (key = round ID, value = {core cache block, timestamp, associated digest vector ID}), supporting block retrieval with O(1) time complexity.

[0045] Function: To ensure the continuity of short conversations (e.g., when a user asks follow-up questions, the most recent high-fidelity cache block can be reused directly).

[0046] In this way, the working memory layer and the long-term memory layer together constitute the hierarchical reuse architecture based on semantic subspace of this invention. To address the limitation of "storage not being semantically long-term", a two-layer library of "working memory layer (short-term high-fidelity) + long-term memory layer" is constructed, using summary vectors as indexes to achieve cross-session state freshness and second-level retrieval.

[0047] In this embodiment, the user's new question is converted into a query vector. A near nearest neighbor search algorithm is used to retrieve target semantic summary vectors from the first database whose semantic similarity to the query vector meets a target threshold. The user's new question is then converted into a query vector, which is used to retrieve semantically similar historical state summaries from the computational state summary library (long-term memory layer), thereby locating reusable core cache blocks. The process is as follows: Figure 4 As shown. First, query vector generation (generating keys for retrieving the state summary): Encoding method: Using the same encoding system as that used to generate the "semantic summary vector" (i.e., first through an LLM encoder, then through the same linear projection layer), the new question text is compressed into a query vector of the same dimension as the "semantic summary vector" in the library. The purpose of this step is to generate a "key" that can be matched with the state summary library, rather than semantically caching the text itself.

[0048] Secondly, semantic retrieval and ranking: Retrieval Algorithm: An Approximate Nearest Neighbor Search (ANN) is performed on the FAISS index, which stores all historical "semantic summary vectors." The retrieval objective is to find the historical state summary most similar to the current query vector, rather than similar historical text. Relevance Verification: Candidate summary vectors are filtered using a combination of timestamps (prioritizing memories from the last 3 months) and user IDs (multi-user isolation), ultimately selecting the optimal historical topic.

[0049] Finally, core cache block localization: According to the retrieved abstract vector ID, query the associated core cache block ID in the metadata table, and then load the core cache block from the working memory layer (if the round is within the last 5 rounds) or the disk (long-term storage). That is, query the relevant filtered result ID in the metadata table based on the ID of the target semantic abstract vector, and determine the second KV cache from the first database or the second database based on the filtered result ID.

[0050] Step S14: Concatenate the second KV cache to obtain corresponding concatenated data, use a target lightweight neural network to adjust the concatenated data, and perform inference based on the corresponding adjusted data and the large language model to obtain a reply corresponding to the user's new question.

[0051] In this embodiment, the second KV cache is concatenated according to the word order of the second KV cache to obtain corresponding concatenated data; then, a target lightweight neural network is used to adjust the concatenated data, and the corresponding adjusted data is stored in the decoder KV cache buffer of the large language model, so that the large language model generates a reply corresponding to the user's new question based on the adjusted data and the text segmentation of the user's new question. Specifically, the retrieved core cache block is reconstructed into a "historical calculation status packet", injected into the model, and the inference process is controlled to achieve efficient generation. The process is as Figure 5 shown.

[0052] In this process, first, status packet reconstruction: Concatenation rule: Concatenate according to the Token order of the core cache block (from early to late) to form a "historical calculation status packet" (dimension: L′×H×d, L′ is the total number of core Tokens, usually L′<<L original).

[0053] Layer alignment: Ensure that the KV caches of each layer of the status packet correspond one-to-one with the LLM decoder layers (for example, the status packet of the i-th layer corresponds to the i-th layer of the decoder).

[0054] In addition, it should be noted that when the historical core cache blocks retrieved from the long-term memory layer are located and ready to be injected into the current session, they will first pass through a "context adapter". This adapter is a lightweight neural network module (such as a single-layer feedforward network), which takes the current query vector and the original abstract vector of the cache block to be injected as joint inputs, and outputs a set of lightweight scaling factors a and bias vectors β.

[0055] Adaptation operation: These factors are used to perform an element-wise affine transformation on the Key / Value vector of the injected cache block, i.e., KVadapted = a⊙KVretrieved + β; where ⊙ represents element-wise multiplication; KVadapted is the adapted KV cache vector; and KVretrieved is the retrieved original KV cache vector.

[0056] Functional Purpose: This fine-tuning process enables reused historical states to undergo subtle semantic shifts and weight recalibrations based on the specific context of the current new problem, thereby better integrating them into the new dialogue flow and avoiding interference or conflicts caused by differences in the original context. This process has extremely low computational overhead but significantly improves the accuracy and security of state reuse.

[0057] Then, cache injection: Injection location: Before LLM inference begins, the state packet is directly loaded into the model's decoder KV cache buffer (overwriting the initial empty cache), skipping the re-segmentation, embedding, and forward computation processes of the original historical text.

[0058] Finally, incremental inference control: Input processing: Only the user's new question text is segmented and input into the model (no need to include historical text). The model calculates attention based on "historical calculation state packet + new question token" and generates a response.

[0059] State update: After the inference is completed, the "record event" is triggered, and the newly generated KV cache in this round is sent to the "dynamic semantic digester" to complete the closed loop.

[0060] In this way, the efficiency bottleneck of "reusing the full cache" is addressed by reconstructing a lightweight state package (1%-10% of the original cache size) through "retrieval-concatenation-injection", skipping text re-segmentation and forward computation, and achieving efficient reasoning with "instant recall".

[0061] As can be seen, this application directly manages the intermediate computational states (KV cache tensors) generated during the reasoning process of a large language model, thereby manipulating the model's "understanding state" rather than merely processing surface text. It identifies high-semantic-value tokens by analyzing the model's internal attention weights and filters redundant caches; combined with autoencoder compression to generate low-dimensional semantic summary vectors, it achieves a lightweight representation of the computational state. A two-layer storage structure is constructed, and finally, through three steps—query vector generation, semantic retrieval, and concatenation of core cache blocks—a closed-loop reuse of "retrieval-location-injection" is achieved, skipping the recomputation of the original text. This enables cross-text reuse, reducing redundant computation by more than 70%. It systematically solves the state management problem in long-context interactions. This not only achieves efficient reuse of semantically similar states across texts and sessions but also ensures the high fidelity and high security of the reused states. The system can automatically identify and retain universal knowledge states for a long time, while adapting them flexibly and dynamically to new contexts when reused. This truly endows the large language model with stable, reliable and accurate "persistent memory" capabilities, fundamentally overcoming the shortcomings of existing technologies such as rigid cached content, high reuse risk and limited long-term value.

[0062] Based on the previous embodiment, this application achieves intelligent management and cross-session reuse of KV cache (intermediate computation state for model inference) through a closed loop of "dynamic summarization - hierarchical storage - semantic retrieval - state reuse". The system consists of five core modules: KV cache interception and recording module, dynamic semantic summarizer, two-layer semantic cache library (including working memory layer and long-term memory layer), semantic retrieval engine, and cache injection and inference control module. The overall system principle block diagram is as follows. Figure 6 As shown, the workflow follows a five-stage closed loop of "recording-summarizing-storing-retrieval-reusing," with the core objective of upgrading the model from a stateless mode of "rereading the full text each time" to a highly efficient state mode of "persistent memory + instant recall." The system's core innovation lies in constructing a "semantic subspace" (i.e., a computational state summary library): it does not store text, but rather stores state summary vectors generated from historical dialogues, representing their core semantics, and uses these vectors to index the corresponding original computational state blocks. The entire system achieves a closed loop from "new text input" to "relevant historical state matching and injection," managing the essence of the state rather than the textual appearance that triggers it. The flowchart for the system's intelligent management and cross-session reuse of KV caching is shown below. Figure 7 As shown.

[0063] In summary, this invention systematically solves the state management problem in long-context interactions by replacing "indirect optimization of text symbols" with "direct management of computational state," supplemented by the dual guarantees of "multi-round stability verification" and "dynamic context adaptation." This not only enables efficient reuse of semantically similar states across texts and sessions but also ensures the high fidelity and security of the reused states. The system can automatically identify and retain universal knowledge states for a long time, while flexibly and dynamically adapting them to new contexts at the moment of reuse. This truly endows large language models with stable, reliable, and accurate "persistent memory" capabilities, fundamentally overcoming the shortcomings of existing technologies such as rigid cached content, high reuse risk, and limited long-term value.

[0064] See Figure 8 As shown, this embodiment of the invention discloses a dialogue device based on KV caching and a large language model, comprising: The cache filtering module 11 is used to acquire and store the KV cache generated in each round of dialogue during the reasoning process of the large language model in real time, filter the KV cache according to the attention weight of the large language model, and generate a semantic summary vector according to the corresponding filtering results. The vector storage module 12 is used to determine the target score based on the occurrence record of the semantic unit block in the historical multi-turn dialogue, and to determine whether to store the semantic unit block and the semantic summary vector corresponding to the semantic unit block into the first database based on the target score; the semantic unit block is used to represent a group of consecutive words or characters whose vector cosine similarity satisfies a preset threshold. The second KV cache determination module 13 is used to store the first KV cache that meets the preset conditions in the KV cache to the second database, convert the user's new question into a query vector, retrieve the target semantic summary vector in the first database whose semantic similarity to the query vector meets the target threshold, and determine the second KV cache based on the target semantic summary vector; the query vector is a vector with the same dimension as the semantic summary vector. The inference module 14 is used to concatenate the second KV cache to obtain corresponding concatenated data, adjust the concatenated data using a target lightweight neural network, and perform inference based on the adjusted data and the large language model to obtain the answer corresponding to the user's new question.

[0065] Since the embodiments of the device part correspond to the embodiments described above, please refer to the embodiments described in the method part for the embodiments of the device part, and will not be repeated here.

[0066] As can be seen, this application directly manages the intermediate computational states (KV cache tensors) generated during the reasoning process of a large language model, thereby manipulating the model's "understanding state" rather than merely processing surface text. It identifies high-semantic-value tokens by analyzing the model's internal attention weights and filters redundant caches; combined with autoencoder compression to generate low-dimensional semantic summary vectors, it achieves a lightweight representation of the computational state. A two-layer storage structure is constructed, and finally, through three steps—query vector generation, semantic retrieval, and concatenation of core cache blocks—a closed-loop reuse of "retrieval-location-injection" is achieved, skipping the recomputation of the original text. This enables cross-text reuse, reducing redundant computation by more than 70%. It systematically solves the state management problem in long-context interactions. This not only achieves efficient reuse of semantically similar states across texts and sessions but also ensures the high fidelity and high security of the reused states. The system can automatically identify and retain universal knowledge states for a long time, while adapting them flexibly and dynamically to new contexts when reused. This truly endows the large language model with stable, reliable and accurate "persistent memory" capabilities, fundamentally overcoming the shortcomings of existing technologies such as rigid cached content, high reuse risk and limited long-term value.

[0067] Furthermore, embodiments of this application also disclose an electronic device, Figure 9 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0068] Figure 9 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the dialogue method based on KV caching and a large language model disclosed in any of the foregoing embodiments. Alternatively, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0069] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0070] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0071] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the KV-based dialogue method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.

[0072] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned disclosed dialogue method based on KV caching and a large language model. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0073] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0074] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0075] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0076] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0077] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A dialogue method based on KV caching and a large language model, characterized in that, include: The key-value cache generated in each round of dialogue during the reasoning process of the large language model is acquired and stored in real time. The key-value cache is filtered according to the attention weight of the large language model, and a semantic summary vector is generated according to the corresponding filtering results. The target score is determined based on the occurrence record of the semantic unit block in the historical multi-turn dialogue. Based on the target score, it is determined whether to store the semantic unit block and the semantic summary vector corresponding to the semantic unit block in the first database. The semantic unit block is used to represent a group of consecutive words or characters whose vector cosine similarity satisfies a preset threshold. The first KV cache that meets the preset conditions in the KV cache is stored in the second database. The user's new question is converted into a query vector. The target semantic summary vector in the first database that has a semantic similarity to the query vector that meets the target threshold is retrieved. The second KV cache is determined based on the target semantic summary vector. The query vector is a vector with the same dimension as the semantic summary vector; The second KV cache is concatenated to obtain corresponding concatenated data. The concatenated data is adjusted using a target lightweight neural network. Reasoning is performed based on the adjusted data and the large language model to obtain the answer to the user's new question.

2. The dialogue method based on KV caching and a large language model according to claim 1, characterized in that, The real-time acquisition and storage of key-value caches generated during each round of dialogue in the large language model inference process includes: The key-value caches of each output layer of the large language model are obtained in real time through hook functions; The KV cache is serialized into a binary file using the dialogue round ID and output layer number as keys, and the binary file is stored in a memory buffer.

3. The dialogue method based on KV caching and a large language model according to claim 1, characterized in that, The step of filtering the KV cache according to the attention weights of the large language model and generating a semantic summary vector based on the filtering results includes: The attention weights are determined based on the attention matrix of the last layer of the decoder in the large language model. Each word in the KV cache is valued according to the attention weight to obtain the corresponding score result; The KV cache is filtered based on the relationship between the scoring results and the first threshold to obtain the corresponding filtering results; The filtering results are flattened into a one-dimensional vector, and the one-dimensional vector is compressed into a low-dimensional vector by an autoencoder. The low-dimensional vector is then determined as the semantic summary vector.

4. The dialogue method based on KV caching and a large language model according to claim 1, characterized in that, The step of determining the target score based on the occurrence record of semantic unit blocks in historical multi-turn dialogues includes: The frequency of occurrence, mean weight, and variance of a semantic unit block are determined based on its occurrence records in historical multi-turn dialogues; the mean weight is the average attention weight of the semantic unit block in each occurrence; and the variance of the weight is the variance of the weight of the semantic unit block in each occurrence. Determine the target sum of 1 and the weighted variance; Determine the target ratio between the weighted mean and the target sum; The product of the frequency of occurrence and the target ratio is determined as the target score; Accordingly, determining whether to store the semantic unit block and the corresponding semantic summary vector in the first database based on the target score includes: If the target score is greater than the second threshold, the semantic unit block and the semantic summary vector corresponding to the semantic unit block are stored in the first database.

5. The dialogue method based on KV caching and a large language model according to claim 3, characterized in that, The step of storing the first KV cache that meets the preset conditions in the KV cache to the second database includes: According to the circular queue structure, the first KV cache in the KV cache of the target round dialogue whose score result is greater than the first threshold is stored in the second database.

6. The dialogue method based on KV caching and a large language model according to claim 1, characterized in that, The step of retrieving target semantic summary vectors from the first database whose semantic similarity to the query vector meets a target threshold includes: The target semantic summary vector in the first database that has a semantic similarity to the query vector that meets the target threshold is retrieved using an approximate nearest neighbor search algorithm.

7. The dialogue method based on KV caching and a large language model according to claim 1, characterized in that, The step of determining the second KV cache based on the target semantic summary vector includes: Based on the ID of the target semantic summary vector, query the associated filter result ID in the metadata table, and determine the second KV cache from the first database or the second database based on the filter result ID.

8. The dialogue method based on KV caching and a large language model according to any one of claims 1 to 7, characterized in that, The step of concatenating the second KV cache to obtain the corresponding concatenated data includes: The second KV cache is concatenated according to the word order of the second KV cache to obtain the corresponding concatenated data; Accordingly, the step of adjusting the concatenated data using a target lightweight neural network, and performing inference based on the adjusted data and the large language model to obtain the answer to the user's new question, includes: The concatenated data is adjusted using a target lightweight neural network, and the adjusted data is stored in the decoder KV cache buffer of the large language model so that the large language model can generate a response to the user's new question based on the adjusted data and the text segmentation of the user's new question.

9. A dialogue device based on KV caching and a large language model, characterized in that, include: The cache filtering module is used to acquire and store the key-value cache generated in each round of dialogue during the reasoning process of the large language model in real time, filter the key-value cache according to the attention weight of the large language model, and generate semantic summary vectors according to the corresponding filtering results. The vector storage module is used to determine the target score based on the occurrence record of the semantic unit block in the historical multi-turn dialogue, and to determine whether to store the semantic unit block and the semantic summary vector corresponding to the semantic unit block into the first database based on the target score; the semantic unit block is used to represent a group of consecutive words or characters whose vector cosine similarity satisfies a preset threshold. The second KV cache determination module is used to store the first KV cache that meets the preset conditions in the KV cache to the second database, convert the user's new question into a query vector, retrieve the target semantic summary vector in the first database whose semantic similarity to the query vector meets the target threshold, and determine the second KV cache based on the target semantic summary vector; The query vector is a vector with the same dimension as the semantic summary vector; The inference module is used to concatenate the second KV cache to obtain corresponding concatenated data, adjust the concatenated data using a target lightweight neural network, and perform inference based on the adjusted data and the large language model to obtain the answer corresponding to the user's new question.

10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing a computer program to implement the steps of the dialogue method based on KV caching and a large language model as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, A computer program is stored on a computer-readable storage medium, and when executed by a processor, the computer program implements the steps of the dialogue method based on KV caching and a large language model as described in any one of claims 1 to 8.

Images