Emotional chat scene cost reduction method and system based on kv sparse

Abstract

The invention relates to the technical field of chat scene cost reduction, and provides a kv sparse-based emotion chat scene cost reduction method, which comprises the following steps: S1, defining a Page Atten tion optimization-based H2O sparse algorithm to optimize KV storage and calculation efficiency, comprising the steps of defining an expelling strategy generation process under the limitation of a limited KV cache, defining a dynamic sub-module, designing a dynamic KV cache strategy and optimizing KV storage based on Page Attention. S2, performing partition management on the Token tube based on a Keyformer scoring mechanism, including Logs regularization, selection of regularization distribution, key Token score identification based on Keyformer scoring and implementation of partition management on the Token; and S3, carrying out sparsification on the KV matrix through Average Pooling so as to reduce consumption of storage and computing resources, wherein the method comprises the steps of carrying out sparsification processing on the KV matrix by adopting a constant KV strategy maintained by SnapKV and the Average Pooling. According to the method, the sparse algorithm is fused into the paged attention, and the management mechanism of the block table is optimized, so that the performance and the efficiency of the emotional chat system are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of chat scene cost reduction, and in particular to a method and system for reducing the cost of an emotional chat scene based on kv sparseness. Background Art

[0002] Emotional chat system is an important application field of artificial intelligence technology, covering various forms such as chatbots, dialogue systems and human-machine customer service. With the continuous advancement of natural language processing (NLP) and deep learning technology, the conversation ability and intelligence level of these systems have been significantly improved, and they can provide efficient and natural interactive experience in a variety of emotional chat scenarios. According to data from market research institutions, the global chatbot market size is growing rapidly and is expected to reach billions of dollars in the next few years. This growth is mainly due to the need for companies to improve customer satisfaction and reduce operating costs. However, in the face of increasingly complex dialogue scenarios and a large number of user requests, existing technologies and methods still have many problems and challenges:

[0003] (1) Context management: When processing long conversations, the system needs to remember and process a large amount of context information, which results in excessive token length, seriously affecting the inference speed and computing cost.

[0004] In emotional chat systems, in order to achieve natural and smooth conversations, the system needs to remember and process a large amount of contextual information. As the conversation progresses, the length of tokens that need to be processed continues to increase, which is particularly prominent in long conversations and complex emotional exchanges. Excessively long token lengths can lead to the following problems:

[0005] a) Decreased inference speed: Each additional token requires additional computing resources, which slows down the inference process. This not only affects the system’s response speed, but may also lead to a decline in user experience.

[0006] b) Increased computing cost: Processing long contexts requires more computing resources and storage space, which directly leads to increased computing costs. In a cloud service environment, this means higher operating expenses and resource consumption.

[0007] (2) Concurrent processing capability: Due to the limitation of graphics card capacity (such as NVIDIA 4090 graphics card), when processing high concurrent requests, the existing methods encounter bottlenecks in graphics memory I / O and capacity, resulting in low service concurrency.

[0008] Graphics card capacity and I / O performance are key factors affecting the concurrent processing capability of the emotional chat system. For example, although the NVIDIA 4090 graphics card has powerful computing power, in actual applications, the graphics memory I / O and capacity limitations will significantly affect the system's concurrency.

[0009] likeFigure 1 As shown in the figure, as the number of concurrent requests increases, the inference time of the NVIDIA 4090 graphics card increases significantly. When the concurrency is 8, the inference time is short, with an average inference time of 0.17 seconds and a 90th percentile inference time of 0.30 seconds, showing good concurrent processing capabilities. However, when the concurrency increases to 16, the inference time almost doubles, to 0.27 seconds and 0.60 seconds, respectively. When it further increases to 32 concurrent requests, the inference time rises sharply, with an average inference time of 11.63 seconds and a 90th percentile inference time of 21.84 seconds. This performance degradation is mainly due to the competition for graphics card memory and I / O resources. Each concurrent request needs to be allocated a certain amount of video memory and I / O bandwidth, and graphics card resources are limited. When the number of concurrent requests increases, the resources allocated to each request decrease, resulting in an increase in processing time. This situation is particularly evident on the NVIDIA 4090 graphics card. Despite its large computing power, the bottlenecks of video memory and I / O bandwidth still exist.

[0010] (3) Resource consumption and performance requirements: The emotional chat system needs to process a large number of requests, which places high demands on server resources. Existing methods have bottlenecks in resource consumption and performance, resulting in service delays and quality issues.

[0011] In the emotional chat system, each service request consumes a lot of computing and storage resources. This resource consumption not only limits the concurrent processing capability of the system, but also increases operating costs and affects service quality. Specifically, this problem is mainly reflected in the following aspects:

[0012] a) High computing resource consumption:

[0013] In order to generate high-quality conversation content, emotional chat systems usually use large-scale language models. These models contain hundreds of millions of parameters and require a lot of computing resources for reasoning and generation. In each request, the system needs to perform complex calculations and processing on the input context information, consuming a lot of GPU computing power. This high computing resource consumption can lead to problems such as extended response time and resource competition.

[0014] b) High storage resource consumption:

[0015] Emotional chat systems need to store a large amount of contextual information and intermediate results in order to generate coherent and emotionally logical conversations. This information requires a large amount of video memory and memory resources. When processing long conversations or complex emotional exchanges, storage requirements increase further, resulting in insufficient video memory or excessive memory usage. Summary of the invention

[0016] In view of the above problems, the object of the present invention is to provide a cost reduction method and system for emotional chat scenes based on KV sparseness, by integrating the sparse algorithm into paged attention and optimizing the management mechanism of block table, significantly improving the performance and efficiency of the emotional chat system. By introducing partition management and sparse processing, the storage and computing costs of KV cache can be effectively reduced, the concurrent processing capability and response speed of the system can be improved, thereby improving user experience and service quality. By solving the key problems in the prior art, the present invention will provide an efficient and scalable solution for the design and implementation of emotional chat system.

[0017] The above-mentioned object of the present invention is achieved through the following technical solutions:

[0018] A cost reduction method for emotional chat scenes based on kv sparseness includes the following steps:

[0019] S1: Define the H2O sparse algorithm based on Paged Attention optimization to optimize KV storage and computing efficiency, including defining the eviction strategy generation process under limited KV cache constraints, defining dynamic submodules, designing dynamic KV cache strategies, and optimizing KV storage based on Paged Attention;

[0020] S2: Partition management of tokens based on the Keyformer scoring mechanism, including Logits regularization, selection of regularized distribution, identification of key tokens based on Keyformer scoring, and partition management of tokens;

[0021] S3: The KV matrix is ​​sparsely processed by Average Pooling to reduce the consumption of storage and computing resources, including using SnapKV to maintain a constant KV strategy and Average Pooling to sparsely process the KV matrix.

[0022] Furthermore, in step S1, the eviction strategy generation process under the limited KV cache limit is defined, including:

[0023] Define the eviction strategy generation process under the limited KV cache limit, and the attention query matrix is ​​expressed as The attention value key matrix is n represents the number of rows, d represents the number of columns, Q i,* represents the i-th row of Q, K ≤i,* represents the first i rows of K, let k represent the space budget, and k <n, represents a submatrix of K, from which S is selected i Rows, for unselected rows [i]\S i, all set to 0, the eviction strategy is defined as:

[0024] |S i |=k, which means the size of the KV cache does not change over time;

[0025] |S i \S i-1 |≤1 or equivalently |S i ∩S i-1 |≥k-1, indicating that at most 1KV key-value pair is removed from the KV cache;

[0026] The generation process of the eviction strategy is defined as:

[0027] make Represents the Token in the KV cache when predicting the i-th Token;

[0028] The information obtained is a vector of length i

[0029] Among them, the scalar The evicted KVs are set to 0 and need to be subtracted when calculating the normalization. Replace S with [i] in the definition. i and D i Leading to the standard generation process, the eviction strategy is based on S i-1 and corresponding information update S i .

[0030] Furthermore, in step S1, the dynamic submodule is defined and the dynamic KV cache strategy is designed, specifically:

[0031] Define the dynamic submodule: Define the dynamic attention score calculation with space constraints as a dynamic submodule type problem, define the function For any collection Assumptions is a submodule function wrt Z;

[0032] Design the dynamic KV cache strategy: Definition Represents a score function, let S i-1 represents the source set, S i represents the target set, then the eviction strategy g:S i-1 →S i st represents the following:

[0033] |S i |=k, which means the KV cache size does not change over time;

[0034] |S i \S i-1|≤1 or equivalently |S i ∩S i-1 |≥k-1, indicating that at most 1KV key-value pair is removed from the KV cache;

[0035] structure To dynamic KV cache strategy, where S i is the token set in the KV cache when predicting the i-th token, which is updated in the current step, S i-1 is the set of tokens in the KV cache when predicting the i-1th token, which is the result of the previous step. {i} is the currently predicted i-th token. This part is to find the score function F score Get the maximum value element v and remove it from (S i-1 ∪{i}) is removed and the result is recorded as \{u}.

[0036] Furthermore, in step S1, the KV storage is optimized based on Paged Attention, specifically:

[0037] Treat blocks as pages, tokens as bytes, and sequences as processes;

[0038] The continuous logical blocks of the sequence are mapped to non-continuous physical blocks through the block table. The physical blocks are allocated on demand when generating new tokens. When sampling in parallel, the same prompt generates multiple output sequences. These sequences share the prompt's attention calculation and video memory when they are generated.

[0039] Similar to the way processes share physical pages in the OS, different sequences share blocks by mapping their logical blocks to the same physical block. Paged Attention tracks the reference count of the physical block and implements a copy-on-write mechanism.

[0040] Furthermore, in step S2, Logits regularization and selection regularization distribution are performed, specifically:

[0041] Logits Regularization:

[0042] In the prompt processing stage, nk tokens are removed from the context, after which the Keyformer uses the logits regularization technique to regularize the reduced logits by introducing an additional distribution EAD. The Keyformer adds noise to the unnormalized logits derived from QKT, i.e.:

[0043]

[0044] Here, yi is the adjusted logits, x i are the unnormalized logits, ζ i is the regularized additional distribution, Q[i,:] is the i-th row of the attention query matrix Q, and K[:,i] T is the i-th column of the attention value key matrix K, To adjust the calculation result scale parameters for preset;

[0045] Choosing a regularized distribution: The Gumbel distribution is chosen for the key token recognition task. The standard Gumbel probability density function applied to the unnormalized logit is as follows:

[0046]

[0047] The probability density function of logits adjusted with Gumbel addition is as follows:

[0048]

[0049] In order to quantify the spread of the probability distribution after normalization, entropy is defined as H(p) = -∑p i log(p i ), p i Represents a probability value in the probability distribution. Gumbel-based logits adjustment promotes a more uniform distribution, indicating its effectiveness as a regularization technique for key token identification. The proof is as follows:

[0050] z=softmax(y)

[0051]

[0052] Among them, y is the processed vector, which is converted into probability distribution z through the softmax function. represents the expectation of the probability distribution obtained by using the logits adjusted by the Gumbel distribution, represents the expectation of the original probability distribution. The inequality shows that the probability distribution after Gumbel adjustment is more uniform and has a larger entropy, which shows that Gumbel distribution is effective as a regularization technique for key token identification.

[0053] Furthermore, in step S2, key tokens are identified based on the Keyformer score and partition management is performed on the tokens, specifically:

[0054] Identify key tokens based on Keyformer scoring: Use the scoring function Keyformer, represented by f θ , the cumulative attention-based scoring function fθ (acc attn), and introduce a temperature parameter, denoted as τ, as follows:

[0055]

[0056] Among them, higher τ (τ→∞) values ​​produce uniform probabilities, assigning equal scores to all tokens. Conversely, lower τ (τ→0) values ​​produce cleaner distributions, prioritizing specific tokens based on unnormalized logits. This parameter determines the degree of randomness of the probability;

[0057] When a Token is deleted from its KV cache, the dynamic properties of τ at each decoding iteration t are as follows:

[0058]

[0059] Among them, τ init is the initial value of the temperature parameter, t is the current number of decoding iterations, Δτ is the change in the temperature parameter in each iteration, τ end is the final value of the temperature parameter, T is the total number of decoding iterations;

[0060] Partition management of tokens: Use continuous batch processing to optimize the block table management mechanism of large language models in emotional scenarios, obtain high-scoring tokens based on the scoring results of key tokens identified based on Keyformer scoring, and store these tokens separately from tokens that are closer in time series.

[0061] Furthermore, in step S3, SnapKV is used to maintain a constant KV strategy and Average Pooling is used to perform sparse processing on the KV matrix, specifically:

[0062] SnapKV is used to maintain a constant KV strategy: the KV matrix is ​​divided into several windows of fixed size, each window contains a fixed number of key-value pairs, and each window is set to contain N key-value pairs;

[0063] In each fixed window, all key-value pairs are aggregated. Specifically, all key-value pairs in the window are averaged to generate a representative key-value pair. The aggregated key-value pair KV snap (i) is expressed as:

[0064]

[0065] Where KV(i,j) represents the jth key-value pair in the i-th window, and N is the window size;

[0066] Average Pooling performs sparse processing on the KV matrix:

[0067] During the generation process, it is necessary to identify the key-value pairs that are most important to the conversation. The scoring mechanism is used to determine the most important key-value pairs in the current conversation. The Keyformer scoring mechanism identifies the key key-value pairs by calculating the importance score of each key-value pair:

[0068] Importance(KV i )=Keyformer Score(KV i )

[0069] Next, in each fixed-size window, calculate the average value of all key-value pairs in the window. Assume that each window contains N key-value pairs, and the key-value pairs in the window are {KV1, KV2, …, KV N}, then the average value KV of the key-value pairs in the window avg It is expressed by the following formula:

[0070]

[0071] Replace the key-value pairs in each window with the average value KV of the window avg , the original KV matrix will become more sparse, but still retain the main information. The size of the replaced sparse KV matrix will be smaller than the original matrix, and the storage requirements and computational costs will also be reduced;

[0072] In the early stages of a conversation, a larger window is used for faster sparsification, while in the later stages of a conversation, a smaller window is used to retain more context information. The window size is adjusted dynamically to achieve the best sparsification effect according to the complexity and length of the conversation. The formula is as follows:

[0073] Window_Size=f(Dialogue_Complexity,Dialogue_Length)

[0074] Among them, Window_Size is the size of the window, Dialogue_Complexity is the complexity of the dialogue, and Dialogue_Length is the complexity of the dialogue;

[0075] The sparse KV matrix is ​​optimized for calculation and attention calculation is performed on the sparse KV matrix.

[0076] A cost reduction system for emotional chat scenes based on kv sparseness, which is used to adopt the cost reduction method for emotional chat scenes based on kv sparseness as described above, comprises:

[0077] The sparse algorithm definition module is used to define the H2O sparse algorithm based on Paged Attention optimization to optimize KV storage and computing efficiency, including defining the eviction strategy generation process under limited KV cache restrictions, defining dynamic submodules, designing dynamic KV cache strategies, and optimizing KV storage based on Paged Attention.

[0078] Token partition management module, which is used to perform partition management on the token based on the Keyformer scoring mechanism, including Logits regularization, selecting regularized distribution, identifying key tokens based on Keyformer scoring, and implementing partition management on tokens;

[0079] The matrix sparse optimization module is used to sparse the KV matrix through Average Pooling to reduce the consumption of storage and computing resources, including using SnapKV to maintain a constant KV strategy and Average Pooling to sparse the KV matrix.

[0080] A computer device comprises a memory and one or more processors, wherein the memory stores computer codes, and when the computer codes are executed by the one or more processors, the one or more processors execute the above method.

[0081] A computer-readable storage medium stores computer codes. When the computer codes are executed, the above method is executed.

[0082] Compared with the conventional large language model in emotional chat scenarios, the present invention has the following advantages:

[0083] (1) Use the virtual memory mechanism to manage the intermediate results of the model and achieve mapping.

[0084] In order to solve the problem of increased inference speed and computational cost due to long context in existing methods, this patent introduces a virtual memory mechanism to manage the intermediate results of the model. By storing the intermediate results of the model in external memory and mapping them when needed, the direct dependence on physical video memory and memory is reduced.

[0085] (2) Optimize the token management mechanism of large language models and store tokens with different functions separately.

[0086] In order to solve the problem of low service concurrency caused by graphics card capacity limitation in existing methods, this patent optimizes the token management mechanism of large language models. The specific method includes storing tokens with different functions separately to achieve partition management. By more effectively managing video memory and I / O resources, the system can handle more concurrent requests at the same time. Partition management reduces resource competition, improves the processing speed of each request, and reduces the bottleneck problems of video memory and I / O bandwidth.

[0087] (3) The average pooling method reduces the size of the intermediate result kv matrix.

[0088] In order to deal with the problem of service requests consuming a lot of resources, this patent introduces an average pooling method to reduce the size of the KV matrix of the intermediate results. By reducing the size of the intermediate results, the computational burden is reduced and the reasoning and generation speed is improved.

[0089] The solution proposed by this patent has achieved remarkable results in the following aspects:

[0090] (1) Improve response speed: Through the virtual memory mechanism and average pooling method, the computing and storage resource usage is reduced, and the system response speed is accelerated.

[0091] (2) Enhanced concurrent processing capabilities: Optimized token management mechanism and partition management, enabling the system to handle more concurrent requests and improving service quality.

[0092] (3) Reduce operating costs: Effective resource management and optimization methods reduce the overall resource requirements of the system and reduce operating costs.

[0093] This method can be widely used in various emotional chat scenarios, such as:

[0094] (1) Customer Service: Applying this method in a customer service system can improve the system response speed and concurrent processing capabilities, and provide more efficient customer support.

[0095] (2) Online consultation: In online consultation systems such as medical and legal services, this method can handle complex conversations and provide more accurate and timely advice.

[0096] (3) Social chatbots: In social media platforms, this method can improve the interaction quality and response speed of chatbots and enhance user experience.

[0097] By combining the above-mentioned optimization methods, this patent not only solves the key problems in the existing emotional chat system, but also significantly improves the overall performance and user experience of the system, providing solid technical support for the widespread application of emotional chat scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0098] Figure 1 A schematic diagram of the relationship between the inference time of a graphics card and the number of concurrent requests in the prior art;

[0099] Figure 2 The overall flow chart of the cost reduction method for emotional chat scenes based on kv sparseness of the present invention;

[0100] Figure 3 A schematic diagram of a block copy when modified for the present invention;

[0101] Figure 4 It is a schematic diagram of sparse optimization of the SnapKV matrix of the present invention;

[0102] Figure 5 This is the overall structural diagram of the cost reduction system for emotional chat scenarios based on kv sparseness of the present invention. DETAILED DESCRIPTION

[0103] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0104] Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a", "an", "said" and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0105] Explanation of the professional terms of the present invention:

[0106] (1)Token

[0107] The basic unit of language model processing, usually corresponding to a word, a subword (such as a subword in WordPiece or Byte Pair Encoding), or a character in the text. Token plays an important role in language models, and they constitute the basic elements of model input.

[0108] (2) KV Matrix

[0109] That is, a matrix used to store the key and value corresponding to each Token in the input sequence to help the model calculate attention weights and perform context encoding.

[0110] (3)Paged Attention

[0111] That is, a method to optimize the attention mechanism, which aims to solve the problem of high computational and storage complexity when processing long sequences.

[0112] (4) H2O sparse algorithm

[0113] That is, a novel KV cache management strategy that significantly reduces the memory footprint of large language models and improves reasoning efficiency by dynamically retaining a small number of tokens that are critical to generating long texts.

[0114] (5)Keyformer

[0115] That is, an innovative reasoning time method that improves the reasoning efficiency of the generated language model by selecting key tokens to reduce the size of the key-value cache and memory bandwidth usage while maintaining the accuracy of the model.

[0116] (6) Average Pooling

[0117] It is a commonly used pooling operation that is used to reduce the spatial size of an image or feature map while retaining its main features.

[0118] The present invention mainly has the following technical breakthroughs:

[0119] (1) Use Paged Attention to optimize the H2O sparse algorithm and store and calculate the intermediate results of H2O in a block table structure, so that they can be mapped to the block table position of KV and reduce the impact on memory limitations.

[0120] (2) Use Keyfprmer's scoring function to score tokens, and use continuous batch processing and token partition storage to optimize inference speed and performance pressure. We divide the storage of tokens into two parts, one for storing tokens with higher scores and the other for storing tokens with closer scores, so that it can better adapt to KV sparse algorithms such as h2o.

[0121] (3) The present invention uses the Average Pooling sparse algorithm to optimize SnapKV. For the KV matrix generated by model reasoning, we use Average Pooling to sparse it, reducing the size of the matrix while ensuring the reasoning accuracy and alleviating memory pressure.

[0122] First embodiment

[0123] like Figure 2 As shown, this embodiment provides a cost reduction method for emotional chat scenarios based on kv sparseness, which is mainly used to reduce the cost of large language models deployed in emotional chat scenarios, accelerate the reasoning speed and improve the concurrency of services. It is mainly divided into: (1) H2O sparse algorithm based on Paged Attention optimization; (2) Partition Token management mechanism based on Keyformer scoring; (3) SnapKV matrix sparse optimization based on Average Pooling.

[0124] The following is a detailed description of each step in this embodiment:

[0125] S1: Define the H2O sparse algorithm based on Paged Attention optimization to optimize KV storage and computing efficiency, including defining the eviction strategy generation process under limited KV cache constraints, defining dynamic sub-modules, designing dynamic KV cache strategies, and optimizing KV storage based on Paged Attention.

[0126] The first step of this embodiment is to define the eviction strategy generation process under the limited KV cache limit, including:

[0127] Define the eviction strategy generation process under the limited KV cache limit, and the attention query matrix is ​​expressed as The attention value key matrix is n represents the number of rows, d represents the number of columns, Q i,* represents the i-th row of Q, K ≤i,* denotes the first i rows of K, let k denote the space budget, and represents a submatrix of K, from which S is selected i Rows, for unselected rows [i]\S i , all set to 0, the eviction strategy is defined as:

[0128] |S i |=k, which means the size of the KV cache does not change over time;

[0129] |S i \S i-1 |≤1 or equivalently |S i ∩S i-1 |≥k-1, indicating that at most 1KV key-value pair is removed from the KV cache;

[0130] The generation process of the eviction strategy is defined as:

[0131] make Represents the Token in the KV cache when predicting the i-th Token;

[0132] The information obtained is a vector of length i

[0133] Among them, the scalar The evicted KVs are set to 0 and need to be subtracted when calculating the normalization. Replace S with [i] in the definition. i and D i Leading to the standard generation process, the eviction strategy is based on S i-1 and corresponding information update S i .

[0134] Our goal is to find a KV cache eviction policy that makes the output of the generated process similar or comparable to the output of the original process without limiting the cache size.

[0135] Step S11: define the dynamic submodule:

[0136] Define the dynamic attention score calculation with space constraints as a dynamic submodule type problem and define the function For any collection Assumptions is a submodule function wrt Z; for example:

[0137] For all collections satisfy

[0138] For all elements x∈[n], x∈[n]\Y.

[0139] Because f(X∪{x})-f(X)≥f(Y∪{x})-f(Y), where f(·):=F(Z,·). X represents an existing word in the KV cache. Y is any superset of X. x can be regarded as a "word" newly added to or deleted from the KV cache. Example f can be an attention score. If the sequence S1, S2, ···, Sn is loaded (guaranteed |S i |≤k and ), that is, for any i∈[n], choose Z=S i , then it becomes a special case of the dynamic submodule problem.

[0140] Step S12: Design the dynamic KV cache strategy:

[0141] definition Represents a score function, let S i-1 represents the source set, S irepresents the target set, then the eviction strategy g:S i-1 →S i st represents the following:

[0142] |S i |=k, which means the KV cache size does not change over time;

[0143] |S i \S i-1 |≤1 or equivalently |S i ∩S i-1 |≥k-1, indicating that at most 1KV key-value pair is removed from the KV cache;

[0144] structure To dynamic KV cache strategy, where S i is the token set in the KV cache when predicting the i-th token, which is updated in the current step, S i-1 is the set of tokens in the KV cache when predicting the i-1th token, which is the result of the previous step. {i} is the currently predicted i-th token.

[0145] This part is to find the score function F score Get the maximum value element v and remove it from (S i-1 ∪{i}) is removed and the result is recorded as \{u}.

[0146] Step S13: Optimizing the KV storage based on Paged Attention:

[0147] The KV cache used for autoregressive generation occupies a large amount of video memory. Inspired by the idea of ​​virtual memory and paging in the OS, this attention optimization algorithm is proposed to store continuous keys and values ​​in discontinuous video memory space. It is used to divide the KV cache of each sequence into blocks, each of which contains a fixed number of token key and value tensors.

[0148] Because blocks do not have to be contiguous in video memory, keys and values ​​can be managed in a more flexible way, just like OS virtual memory paging:

[0149] Treat blocks as pages, tokens as bytes, and sequences as processes;

[0150] The continuous logical blocks of the sequence are mapped to non-continuous physical blocks through the block table. The physical blocks are allocated on demand when generating new tokens, so only the last block will waste video memory, which is less than 4%. When sampling in parallel, the same prompt generates multiple output sequences, and these sequences share the prompt's attention calculation and video memory when they are generated;

[0151] Similar to the way processes share physical pages in OS, different sequences share blocks by mapping their logical blocks to the same physical block. To ensure sharing safety, Paged Attention tracks the reference count of physical blocks and implements a copy-on-write mechanism, that is, a block copy is copied only when it needs to be modified, such as Figure 3 shown.

[0152] S2: Partition management of tokens based on the Keyformer scoring mechanism, including Logits regularization, selection of regularized distribution, identification of key tokens based on Keyformer scoring, and partition management of tokens.

[0153] Keyformer exploits the inherent sparsity within the decoder layer by using a mixture of recent tokens to identify key tokens. It adjusts for changes in the score function caused by discarded tokens by applying regularization to the unnormalized logits to identify key tokens. For the token score calculation in step 1, this method uses the Keyformer function for scoring.

[0154] Step S21: Logits Regularization:

[0155] In the prompt processing stage, the present invention removes nk tokens from the context, which helps us keep the KV cache size of k tokens unchanged during the generation process and prevent unnecessary memory expansion. Thereafter, Keyformer uses logits regularization technique to regularize the reduced logits by introducing additional distribution EAD (added distribution, abbreviated as EAD), so that the model can remain robust and adaptive. It helps to identify key tokens even in the presence of unknown context during reasoning time. In addition, the added distribution type significantly affects the generated probability distribution. Keyformer adds noise to the unnormalized logarithm derived from QKT, that is:

[0156]

[0157] Here, y i is the adjusted logits, x i are the unnormalized logits, ζ iis the regularized additional distribution, Q[i,:] is the i-th row of the attention query matrix Q, and K[:,i] T is the i-th column of the attention value key matrix K, To adjust the calculation result scale parameters for preset;

[0158] Step S22: Select the regularization distribution:

[0159] The regularized distribution added to the unnormalized logits affects key token identification and model quality. Therefore, the goal of this approach is to draw intuition using the semantics of LLM.

[0160] The Gumbel distribution (Cooray, 2010) chosen in this method is particularly suitable for the key mark identification task because it characterizes the distribution of the maximum value in a set of samples and is biased towards the initial mark.

[0161] The Gumbel distribution is chosen for the key token recognition task, and the standard Gumbel probability density function applied to the unnormalized logit is as follows:

[0162]

[0163] The probability density function of logits adjusted with Gumbel addition is as follows:

[0164]

[0165] It is worth noting that the Gumbel distribution is of great significance in statistical theory. It captures the essence of the Gumbel limit theorem, which asserts that common probability distributions (such as normal, exponential, uniform, etc.) converge to the Gumbel distribution. This emphasizes its appropriateness for modeling key token recognition.

[0166] Theoretically, choosing a regularized distribution that promotes uniformity after normalization helps identify key tokens. This is crucial during inference because information about discarded tokens is not available. To quantify the spread of the probability distribution after normalization, entropy is defined as H(p) = -

[0167] ∑p i log(p i ), p i Represents a probability value in the probability distribution. Gumbel-based logits adjustment promotes a more uniform distribution, indicating its effectiveness as a regularization technique for key token identification. The proof is as follows:

[0168] z=softmax(y)

[0169]

[0170] Among them, y is the processed vector, which is converted into probability distribution z through the softmax function. represents the expectation of the probability distribution obtained by using the logits adjusted by the Gumbel distribution, represents the expectation of the original probability distribution. The inequality shows that the probability distribution after Gumbel adjustment is more uniform and has a larger entropy, which shows that Gumbel distribution is effective as a regularization technique for key token identification.

[0171] Step S23: Identify key tokens based on Keyformer scoring:

[0172] The scoring function Keyformer is used, expressed as f θ , the cumulative attention-based scoring function f θ (accattn), this new scoring function integrates the Gumbel noise distribution into the unnormalized logits. However, it fails to take into account the dropped tokens when forming the underlying probability distribution. To correct this, Keyformer introduces a temperature parameter, denoted as τ, as follows:

[0173]

[0174] Among them, the "temperature" parameter (τ) is key in regulating the smoothness of the probability distribution. Higher τ (τ→∞) values ​​produce uniform probabilities, assigning equal scores to all tokens. Conversely, lower τ (τ→0) values ​​produce a clearer distribution, prioritizing specific tokens based on unnormalized logits. This parameter determines the degree of randomness of the probability;

[0175] This is crucial when tokens are removed from their KV cache, as they cannot be reintroduced without recalculating their keys and values. The dynamic nature of τ at each decoding iteration t is as follows:

[0176]

[0177] Among them, τ init is the initial value of the temperature parameter, t is the current number of decoding iterations, Δτ is the change in the temperature parameter in each iteration, τ end is the final value of the temperature parameter, T is the total number of decoding iterations;

[0178] Step S24: Implement partition management for Token:

[0179] Unlike traditional deep learning models, LLM's batch processing approach is complicated because its reasoning is iterative. Intuitively, this is because requests may complete early, but it is tricky to release its resources and add new requests that may be in different completion states. This means that the GPU is underutilized as the generation length of different sequences in the batch is different from the maximum generation length.

[0180] Continuous batching uses iteration-level scheduling, where the batch size is determined on a per-iteration basis. As a result, once a sequence in a batch has finished being generated, a new sequence can be inserted in its place, achieving higher GPU utilization than static batching.

[0181] The present invention uses continuous batch processing to optimize the block table management mechanism of the large language model in the emotional scene, obtains high-scoring tokens based on the scoring results of identifying key tokens based on Keyformer scoring, and stores these tokens separately from tokens that are closer in time series. In this way, the model can manage and access different types of tokens more effectively, thereby improving computing efficiency and storage utilization. This adjustment helps to optimize the performance of the model when processing long sequences and reduce video memory usage and computing burden.

[0182] S3: The KV matrix is ​​sparsely processed by Average Pooling to reduce the consumption of storage and computing resources, including using SnapKV to maintain a constant KV strategy and Average Pooling to sparsely process the KV matrix.

[0183] In the previous steps, Paged Attention was used to optimize KV storage and computing efficiency, and the Keyformer scoring mechanism was used to partition tokens. Based on these optimizations, this step further sparses the KV matrix through AveragePooling to further reduce the consumption of storage and computing resources. This process includes SnapKV's constant KV strategy and Average Pooling sparse processing.

[0184] Average Pooling is a technique commonly used in convolutional neural networks, such as Figure 4 As shown in Figure 1, by averaging the local area of ​​the feature map, the size of the feature map is reduced while retaining the main information. In the sparse optimization of the KV matrix, we draw on this idea and perform sparse processing on the KV matrix to reduce its storage and computational costs.

[0185] Step S31: Use SnapKV to maintain a constant KV strategy:

[0186] The SnapKV mechanism is designed to solve the problem of infinite expansion of the KV matrix in emotional chat systems as the length and complexity of conversations increase. By maintaining a constant size of the KV matrix, SnapKV not only reduces the demand for video memory and computing resources, but also ensures the system's responsiveness and concurrent processing capabilities.

[0187] The KV matrix is ​​divided into several windows of fixed size. Each window contains a fixed number of key-value pairs. Each window is set to contain N key-value pairs. This fixed window approach prevents the KV matrix from expanding infinitely as the conversation length increases, ensuring that the consumption of storage and computing resources is within a controllable range.

[0188] In each fixed window, all key-value pairs are aggregated. Specifically, all key-value pairs in the window are averaged to generate a representative key-value pair. The aggregated key-value pair KV snap (i) is expressed as:

[0189]

[0190] Among them, KV(i,j) represents the jth key-value pair in the i-th window, and N is the window size. By averaging the key-value pairs in the window, the generated representative key-value pairs retain the main information in the window, while significantly reducing the storage requirements and computational overhead of the KV matrix.

[0191] The SnapKV mechanism ensures that the total size of the KV matrix remains constant by fixing the window size and in-block aggregation. For each window, a representative key-value pair is generated by averaging, and these representative key-value pairs are combined to form a sparse KV matrix. We denote the average value of all key-value pairs in the i-th window as:

[0192]

[0193] This representative key-value pair retains the main information within the window while significantly reducing the storage requirements and computational overhead of the KV matrix.

[0194] Step S32: Average Pooling performs sparse processing on the KV matrix:

[0195] In order to optimize the performance of the emotional chat system, we introduced the Average Pooling method to sparse the KV matrix. This method aims to reduce storage requirements and computing burden. The specific implementation steps are as follows:

[0196] During the generation process, it is necessary to identify the key-value pairs that are most important to the conversation. A scoring mechanism (such as Keyformer scoring) is used to determine the most important key-value pairs in the current conversation. The Keyformer scoring mechanism identifies key key-value pairs by calculating the importance score of each key-value pair. These scores determine which key-value pairs will be retained in further processing and which will be merged or discarded:

[0197] Importance(KV i )=Keyformer Score(KV i )

[0198] Next, in each fixed-size window, calculate the average value of all key-value pairs in the window. Assume that each window contains N key-value pairs, and the key-value pairs in the window are {KV1, KV2, …, KV N}, then the average value KV of the key-value pairs in the window avg It is expressed by the following formula:

[0199]

[0200] In this way, multiple key-value pairs within the window are merged into a representative key-value pair, thereby reducing the number of key-value pairs that need to be processed. This average pooling operation not only reduces the size of the KV matrix, but also retains the main information, making subsequent calculations more efficient.

[0201] Replace the key-value pairs in each window with the average value KV of the window avg , the original KV matrix will become more sparse, but still retain the main information. The size of the replaced sparse KV matrix will be smaller than the original matrix, and the storage requirements and computational costs will also be reduced;

[0202] In the early stages of a conversation, a larger window is used for faster sparsification, while in the later stages of a conversation, a smaller window is used to retain more context information. The window size is adjusted dynamically to achieve the best sparsification effect according to the complexity and length of the conversation. The formula is as follows:

[0203] Window_Size=f(Dialogue_Complexity,Dialogue_Length)

[0204] Among them, Window_Size is the size of the window, Dialogue_Complexity is the complexity of the dialogue, and Dialogue_Length is the complexity of the dialogue;

[0205] Finally, the calculation of the sparse KV matrix is ​​optimized. Since the matrix has been sparse, many unimportant key-value pairs can be skipped during the calculation process, thereby reducing calculation time and resource consumption. Attention calculations using the sparse KV matrix can significantly improve reasoning speed and system performance.

[0206] Second embodiment

[0207] like Figure 5 As shown, this embodiment provides a KV-sparse based emotional chat scene cost reduction system for adopting the KV-sparse based emotional chat scene cost reduction method as in the first embodiment, including:

[0208] Sparse algorithm definition module 1 is used to define the H2O sparse algorithm based on Paged Attention optimization to optimize KV storage and computing efficiency, including defining the eviction strategy generation process under limited KV cache restrictions, defining dynamic submodules, designing dynamic KV cache strategies, and optimizing KV storage based on Paged Attention;

[0209] Token partition management module 2 is used to perform partition management on the token based on the Keyformer scoring mechanism, including Logits regularization, selecting regularized distribution, identifying key tokens based on Keyformer scoring, and implementing partition management on tokens;

[0210] The matrix sparse optimization module 3 is used to sparse the KV matrix through Average Pooling to reduce the consumption of storage and computing resources, including using SnapKV to maintain a constant KV strategy and Average Pooling to sparse the KV matrix.

[0211] A computer-readable storage medium stores a computer code. When the computer code is executed, the above method is executed. A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program. The program can be stored in a computer-readable storage medium. The storage medium can include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0212] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

[0213] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0214] It should be noted that the above embodiments can be freely combined as needed. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered as the protection scope of the present invention.

Claims

1. A cost reduction method for emotional chat scenes based on kv sparseness, characterized in that: The following steps are involved: S1: Define the H2O sparse algorithm based on Paged Attention optimization to optimize KV storage and computing efficiency, including defining the eviction strategy generation process under limited KV cache constraints, defining dynamic submodules, designing dynamic KV cache strategies, and optimizing KV storage based on Paged Attention; S2: Partition management of tokens based on the Keyformer scoring mechanism, including Logits regularization, selection of regularized distribution, identification of key tokens based on Keyformer scoring, and partition management of tokens; S3: The KV matrix is ​​sparsely processed by Average Pooling to reduce the consumption of storage and computing resources, including using SnapKV to maintain a constant KV strategy and Average Pooling to sparsely process the KV matrix.

2. The cost reduction method for emotional chat scenes based on kv sparseness according to claim 1 is characterized in that: In step S1, the eviction strategy generation process under the limited KV cache limit is defined, including: Define the eviction strategy generation process under the limited KV cache limit, and the attention query matrix is ​​expressed as The attention value key matrix is n represents the number of rows, d represents the number of columns, Q i,* represents the i-th row of Q, K ≤i,* represents the first i rows of K, let k represent the space budget, and k <n, represents a submatrix of K, from which S is selected i Rows, for unselected rows [i]\S i , all set to 0, the eviction strategy is defined as: |S i |=k, which means the size of the KV cache does not change over time; |S i \S i-1 |≤1 or equivalently |S i ∩S i-1 |≥k-1, indicating that at most 1KV key-value pair is removed from the KV cache; The generation process of the eviction strategy is defined as: make Represents the Token in the KV cache when predicting the i-th Token; The information obtained is a vector of length i Among them, the scalar The evicted KVs are set to 0 and need to be subtracted when calculating the normalization. Replace S with [i] in the definition. i and D i Leading to the standard generation process, the eviction strategy is based on S i-1 and corresponding information update S i .

3. The cost reduction method for emotional chat scenes based on kv sparseness according to claim 2 is characterized in that: In step S1, the dynamic submodule is defined and the dynamic KV cache strategy is designed, specifically: Define the dynamic submodule: Define the dynamic attention score calculation with space constraints as a dynamic submodule type problem, and define the function F: For any collection Assume F(Z,·): is a submodule function wrt Z; Design the dynamic KV cache strategy: Define F score : Represents a score function, let S i-1 represents the source set, S i represents the target set, then the eviction strategy g:S i-1 →S i st represents the following: |S i |=k, which means the KV cache size does not change over time; |S i \S i-1 |≤1 or equivalently |S i ∩S i-1 |≥k-1, indicating that at most 1KV key-value pair is removed from the KV cache; structure To dynamic KV cache strategy, where S i is the Token set in the KV cache when predicting the i-th Token, which is updated in the current step, S i-1 is the set of tokens in the KV cache when predicting the i-1th token, which is the result of the previous step. {i} is the currently predicted i-th token. This part is to find the score function F score Get the element v with the maximum value and remove it from (S i-1 ∪{i}) is removed and the result is recorded as \{u}.

4. The cost reduction method for emotional chat scenes based on kv sparseness according to claim 3 is characterized in that: In step S1, the KV storage is optimized based on Paged Attention, specifically: Treat blocks as pages, tokens as bytes, and sequences as processes; The continuous logical blocks of the sequence are mapped to non-continuous physical blocks through the block table. The physical blocks are allocated on demand when generating new tokens. When sampling in parallel, the same prompt generates multiple output sequences. These sequences share the prompt's attention calculation and video memory when they are generated. Similar to the way processes share physical pages in the OS, different sequences share blocks by mapping their logical blocks to the same physical block. Paged Attention tracks the reference count of the physical block and implements a copy-on-write mechanism.

5. The cost reduction method for emotional chat scenes based on kv sparseness according to claim 2 is characterized in that: In step S2, Logits regularization and selection regularization distribution are performed as follows: Logits Regularization: In the prompt processing stage, nk tokens are removed from the context, after which the Keyformer uses the logits regularization technique to regularize the reduced logits by introducing an additional distribution EAD. The Keyformer adds noise to the unnormalized logits derived from QKT, i.e.: Here, y i is the adjusted logits, x i are the unnormalized logits, ζ i is the regularized additional distribution, Q[i,:] is the i-th row of the attention query matrix Q, and K[:,i] T is the i-th column of the attention value key matrix K, To adjust the calculation result scale parameters for preset; Choosing a regularized distribution: The Gumbel distribution is chosen for the key token recognition task. The standard Gumbel probability density function applied to the unnormalized logit is as follows: The probability density function of logits adjusted with Gumbel addition is as follows: In order to quantify the spread of the probability distribution after normalization, entropy is defined as H(p) = -∑p i log(p i ), p i Represents a probability value in the probability distribution. Gumbel-based logits adjustment promotes a more uniform distribution, indicating its effectiveness as a regularization technique for key token identification. The proof is as follows: z=softmax(y) Among them, y is the processed vector, which is converted into probability distribution z through the softmax function. represents the expectation of the probability distribution obtained by using the logits adjusted by the Gumbel distribution, represents the expectation of the original probability distribution. The inequality shows that the probability distribution after Gumbel adjustment is more uniform and has a larger entropy, which shows that Gumbel distribution is effective as a regularization technique for key token identification.

6. The cost reduction method for emotional chat scenes based on kv sparseness according to claim 1 is characterized in that: In step S2, key tokens are identified based on the Keyformer score and the tokens are partitioned. Specifically: Identify key tokens based on Keyformer scoring: Use the scoring function Keyformer, represented by f θ , the cumulative attention-based scoring function f θ (acc attn), and introduce a temperature parameter, denoted as τ, as follows: Among them, higher τ (τ→∞) values ​​produce uniform probabilities, assigning equal scores to all tokens. Conversely, lower τ (τ→0) values ​​produce a cleaner distribution, prioritizing specific tokens based on unnormalized logits. This parameter determines the degree of randomness of the probability; When a Token is deleted from its KV cache, the dynamic properties of τ at each decoding iteration t are as follows: Among them, τ init is the initial value of the temperature parameter, t is the current number of decoding iterations, Δτ is the change in the temperature parameter in each iteration, τ end is the final value of the temperature parameter, T is the total number of decoding iterations; Partition management of tokens: Use continuous batch processing to optimize the block table management mechanism of large language models in emotional scenarios, obtain high-scoring tokens based on the scoring results of key tokens identified based on Keyformer scoring, and store these tokens separately from tokens that are closer in time series.

7. The cost reduction method for emotional chat scenes based on kv sparseness according to claim 1 is characterized in that: In step S3, SnapKV is used to maintain a constant KV strategy and Average Pooling is used to perform sparse processing on the KV matrix, specifically: SnapKV is used to maintain a constant KV strategy: the KV matrix is ​​divided into several windows of fixed size, each window contains a fixed number of key-value pairs, and each window is set to contain N key-value pairs; In each fixed window, all key-value pairs are aggregated. Specifically, all key-value pairs in the window are averaged to generate a representative key-value pair. The aggregated key-value pair KV snap (i) is expressed as: Where KV(i,j) represents the jth key-value pair in the i-th window, and N is the window size; Average Pooling performs sparse processing on the KV matrix: During the generation process, it is necessary to identify the key-value pairs that are most important to the conversation. The scoring mechanism is used to determine the most important key-value pairs in the current conversation. The Keyformer scoring mechanism identifies the key key-value pairs by calculating the importance score of each key-value pair: Importance(KV i )=Keyformer Score(KV i ) Next, in each fixed-size window, calculate the average value of all key-value pairs in the window. Assume that each window contains N key-value pairs, and the key-value pairs in the window are {KV1, KV2, …, KV N }, then the average value KV of the key-value pairs in the window avg It is expressed by the following formula: Replace the key-value pairs in each window with the average value KV of the window avg , the original KV matrix will become more sparse, but still retain the main information. The size of the replaced sparse KV matrix will be smaller than the original matrix, and the storage requirements and computational costs will also be reduced; In the early stages of a conversation, a larger window is used for faster sparsification; in the later stages of a conversation, a smaller window is used to retain more contextual information. The window size is adjusted dynamically according to the complexity and length of the conversation to achieve the best sparsification effect. The formula is as follows: Window_Size=f(Dialogue_Complexity,Dialogue_Length) Among them, Window_Size is the size of the window, Dialogue_Complexity is the complexity of the dialogue, and Dialogue_Length is the complexity of the dialogue; The sparse KV matrix is ​​optimized for calculation and attention calculation is performed on the sparse KV matrix.

8. A cost reduction system for emotional chat scenes based on kv sparseness, which is used to adopt the cost reduction method for emotional chat scenes based on kv sparseness as described in any one of claims 1 to 7, characterized in that: include: The sparse algorithm definition module is used to define the H2O sparse algorithm based on Paged Attention optimization to optimize KV storage and computing efficiency, including defining the eviction strategy generation process under limited KV cache restrictions, defining dynamic submodules, designing dynamic KV cache strategies, and optimizing KV storage based on Paged Attention. The Token partition management module is used to perform partition management on the Token based on the Keyformer scoring mechanism, including Logits regularization, selecting regularized distribution, identifying key Tokens based on Keyformer scoring, and implementing partition management on Tokens. The matrix sparse optimization module is used to sparse the KV matrix through Average Pooling to reduce the consumption of storage and computing resources, including using SnapKV to maintain a constant KV strategy and Average Pooling to sparse the KV matrix.

9. A computer device comprising a memory and one or more processors, wherein the memory stores computer codes, and when the computer codes are executed by the one or more processors, the one or more processors execute the method according to any one of claims 1 to 7. 10 . A computer-readable storage medium storing a computer code. When the computer code is executed, the method according to claim 1 is executed.

Images