A data processing method, device and electronic equipment

Abstract

The application provides a data processing method and device and electronic equipment; the method comprises: obtaining a key-value cache generated by an inference model in an inference process; dividing the key-value cache into a plurality of key-value data blocks, each key-value data block containing a key tensor and a value tensor corresponding to a word piece generated in a plurality of inference steps; in response to a state parameter of the key-value cache meeting a set screening condition, determining a first data block and a second data block from the plurality of key-value data blocks. The second data block is compressed to obtain a third data block, the data amount of the third data block being less than that of the second data block; the third data block at least includes context information of the second data block; in a current inference step of the inference model, the first data block and the third data block are taken as key-value inputs for attention calculation, and a query vector of the current inference step is subjected to attention calculation to obtain an attention calculation result and determine an output of the current inference step based on the attention calculation result.

Description

Technical Field

[0001] This application relates to computer technology, and more particularly to a data processing method, apparatus, and electronic device. Background Technology

[0002] During model inference, for each new lexical unit generated, the model needs to calculate the attention weights between the current query vector and all historical key-value caches, and write the key tensor and value tensor of that lexical unit into the key-value cache. As the number of dialogue rounds or the length of the context continues to increase, the storage size of the key-value cache and the computational complexity of attention increase in tandem. Current data processing methods typically discard low-importance key-value data directly, or prune early context information in chronological order when the cache overflows. The information that is directly discarded or pruned is no longer involved in subsequent inference in any form. Therefore, current data processing methods suffer from the problems of uncontrollable loss of historical context information and the lack of long-term context state of the model. Summary of the Invention

[0003] This application provides a data processing method, apparatus, and electronic device.

[0004] This application provides a data processing method, comprising: acquiring a key-value cache generated by an inference model during the inference process; dividing the key-value cache into multiple key-value data blocks, each key-value data block containing key tensors and value tensors corresponding to lexical units generated in multiple inference steps; determining a first data block and a second data block from the multiple key-value data blocks in response to the key-value cache's state parameters satisfying set filtering conditions; the first data block having a higher correlation with the current inference step than the second data block; compressing the second data block to obtain a third data block, the third data block having a smaller data size than the second data block; the third data block including at least the context information of the second data block; in the current inference step of the inference model, using the first data block and the third data block as key-value inputs for attention calculation, performing attention calculation with the query vector of the current inference step to obtain an attention calculation result, and determining the output of the current inference step based on the attention calculation result.

[0005] According to one embodiment of this application, determining the first data block and the second data block from the plurality of key-value data blocks includes: determining an importance score for each key-value data block; the importance score is determined based on at least one of the average attention weight of the key-value data block in historical attention calculation, the access frequency of the key-value data block within a set time window, and the time decay factor of the inference step corresponding to the key-value data block; sorting the key-value data blocks in descending order of the importance scores, and determining the first number of key-value data blocks at the top of the sorted list as the first data block; and determining the key-value data blocks other than the first data block from the plurality of key-value data blocks as the second data block.

[0006] According to one embodiment of this application, the step of compressing the second data block to obtain a third data block includes: obtaining a currently existing historical third data block; the historical third data block is a compressed representation of the historical context information corresponding to the key-value cache determined to be the second data block; performing intra-block aggregation and inter-block aggregation on each key-value data block in the second data block to obtain a candidate update vector; and performing recursive fusion based on the candidate update vector and the currently existing historical third data block to obtain the third data block.

[0007] According to one embodiment of this application, the step of recursively fusing the candidate update vector and the currently existing historical third data block to obtain the third data block includes: determining a gating factor based on the number of data blocks in the second data block and the cache occupancy data corresponding to the key-value cache; the gating factor characterizing the degree of influence of the key-value cache determined as the second data block on the historical third data block; and recursively fusing the candidate update vector with the currently existing historical third data block based on the gating factor to obtain the third data block.

[0008] According to one embodiment of this application, the method further includes: determining the candidate update vector as the third data block in response to the absence of the historical third data block.

[0009] According to one embodiment of this application, determining the gating factor based on the number of data blocks in the second data block and the cache occupancy data corresponding to the key-value cache includes: obtaining the number of data blocks and a pre-set number of reference evicted blocks; obtaining the current cache occupancy ratio and a pre-set cache pressure threshold included in the cache occupancy data; calculating the ratio of the number of data blocks to the number of reference evicted blocks, and the difference between the current cache occupancy ratio and the cache pressure threshold; and inputting the ratio and the difference into a pre-set gating function to obtain the gating factor.

[0010] According to one embodiment of this application, the gating function further includes a weight parameter; the method further includes: after the third data block participates in attention calculation, determining a first average attention weight of the third data block and a second average attention weight of the first data block; and adjusting the weight parameter based on the difference between the second average attention weight and the first average attention weight.

[0011] According to one embodiment of this application, the step of using the first data block and the third data block as key-value inputs for attention calculation, and performing attention calculation with the query vector of the current inference step, includes: concatenating the key tensor corresponding to the first data block and the key tensor corresponding to the third data block according to the sequence dimension to obtain a final key tensor; concatenating the value tensor corresponding to the first data block and the value tensor corresponding to the third data block according to the sequence dimension to obtain a final value tensor; and performing attention calculation based on the query vector of the current inference step, the final key tensor, and the final value tensor to obtain the attention calculation result.

[0012] This application also provides a data processing apparatus, comprising: an acquisition module for acquiring a key-value cache generated by an inference model during inference; a partitioning module for partitioning the key-value cache into multiple key-value data blocks, each key-value data block containing key tensors and value tensors corresponding to lexical units generated in multiple inference steps; a determination module for determining a first data block and a second data block from the multiple key-value data blocks in response to a set filtering condition satisfying a state parameter of the key-value cache; the first data block having a higher correlation with the current inference step than the second data block; a compression module for compressing the second data block to obtain a third data block, the third data block having a smaller data size than the second data block; the third data block including at least the context information of the second data block; and a calculation module for using the first data block and the third data block as key-value inputs for attention calculation in the current inference step of the inference model, performing attention calculation with the query vector of the current inference step, obtaining an attention calculation result, and determining the output of the current inference step based on the attention calculation result.

[0013] This application also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform: acquiring a key-value cache generated by an inference model during inference; dividing the key-value cache into multiple key-value data blocks, each key-value data block containing key tensors and value tensors corresponding to tokens generated in multiple inference steps; in response to the key-value cache's state parameters satisfying set filtering conditions, determining a first data block and a second data block from the multiple key-value data blocks; the first data block having a higher correlation with the current inference step than the second data block; compressing the second data block to obtain a third data block, the third data block having a smaller data size than the second data block; the third data block including at least the context information of the second data block; in the current inference step of the inference model, using the first data block and the third data block as key-value inputs for attention calculation, performing attention calculation with the query vector of the current inference step to obtain an attention calculation result, and determining the output of the current inference step based on the attention calculation result. Attached Figure Description

[0014] The above and other objects, features, and advantages of exemplary embodiments of this application will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. Several embodiments of this application are illustrated in the drawings by way of example and not limitation, in which: In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts.

[0015] Figure 1 This application illustrates a schematic diagram of the data processing method provided in an embodiment. Figure 1 ; Figure 2 This application illustrates a schematic diagram of the data processing method provided in an embodiment. Figure 2 ; Figure 3 This illustration shows an application scenario diagram of the data processing method provided in an embodiment of this application; Figure 4 This illustration shows another application scenario of the data processing method provided in the embodiments of this application; Figure 5 An optional schematic diagram of the data processing apparatus provided in an embodiment of this application is shown; Figure 6 A schematic diagram of the composition structure of the electronic device provided in the embodiments of this application is shown. Detailed Implementation

[0016] To make the objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0017] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0018] In the following description, the terms "first" and "second" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0020] The processing flow of the data processing method provided in the embodiments of this application will be described. See [link to relevant documentation]. Figure 1 , Figure 1 This is a schematic diagram of the data processing method provided in the embodiments of this application. Figure 1 , will combine Figure 1 Steps S101-S105 are explained below.

[0021] Step S101: Obtain the key-value cache generated by the inference model during the inference process.

[0022] In some embodiments, the inference model may include a large language model based on the Transformer inference architecture. The inference model may also include other types of models, which are not limited in this application embodiment. The key-value cache (KV cache) may include a set of key tensors and value tensors corresponding to a consecutive plurality of tokens stored during the inference process.

[0023] In the inference process of a standard autoregressive Transformer model, each time a new token is generated, the model performs the following steps: Calculate the Query (query vector) for the current token. Read the Key / Value pairs (KVCache) corresponding to all tokens in the historical context and perform Attention(Q,K,V) calculation, where Q is the query vector, K is the key tensor, and V is the value tensor. Generate the output for the current token and append the K / V pairs of that token to the KVCache.

[0024] Step S102: Divide the key-value cache into multiple key-value data blocks. Each key-value data block contains key tensors and value tensors corresponding to the lexical units generated in multiple inference steps.

[0025] In some embodiments, the key-value data block may include: multiple key-value tensors corresponding to multiple words obtained by splitting the key-value cache into a preset size. Specifically, the key-value data block may be a group of multiple consecutive key-value tensors corresponding to multiple words obtained by splitting the key-value cache into a preset size. Words can serve as the basic components of the model's input and output, and each word can generate a set of key tensors and value tensors through a linear transformation.

[0026] The process of dividing a key-value cache into multiple key-value data blocks can be represented by the following formula:

[0027] in, This represents the i-th key-value data block, where B is the preset key-value data block size. Let the key tensor of the t-th inference step be represented. Let i represent the value tensor of the t-th inference step. i can be the position number of the key-value data block in the cache sequence, and t can be the position number of the term in the inference sequence.

[0028] In step S103, in response to the key-value cache state parameters satisfying the set filtering conditions, a first data block and a second data block are determined from multiple key-value data blocks; the first data block has a higher correlation with the current inference step than the second data block.

[0029] In some embodiments, the state parameters may include: the total number of key-value data blocks, the video memory usage ratio of the key-value cache, or the number of consecutive inference steps. The state parameters may also include other parameters, which are not limited in this embodiment. Filtering conditions may include: the total number of key-value data blocks exceeding a preset threshold, the video memory usage ratio exceeding a preset ratio, or the number of consecutive inference steps reaching a preset interval. Filtering conditions may also include other filtering conditions corresponding to other parameters, which are not limited in this embodiment. The first data block may be a key-value data block with a high degree of correlation to the current inference step. The first data block can be directly used as key-value input to participate in the attention calculation of the current inference step. The second data block may be a key-value data block with a low degree of correlation to the current inference step. The second data block can participate in attention calculation after compression.

[0030] Step S104: Compress the second data block to obtain a third data block. The data size of the third data block is smaller than that of the second data block. The third data block includes at least the context information of the second data block.

[0031] In some embodiments, the third data block may include a summary key-value data block after the second data block has been compressed. The third data block can retain the contextual information of the second data block with a smaller data volume. The third data block can serve as a low-dimensional state representation of historical contextual information and continuously participate in subsequent attention calculations, thereby avoiding the uncontrollable loss of historical information.

[0032] Step S105: In the current inference step of the inference model, the first data block and the third data block are used as key-value inputs for attention calculation, and attention calculation is performed with the query vector of the current inference step to obtain the attention calculation result and determine the output of the current inference step based on the attention calculation result.

[0033] In some embodiments, the query vector may include a vector representation generated by linear transformation of the input terms in the current inference step. The attention calculation result may include a vector obtained by weighted aggregation of the query vector and the key-value input through attention calculation. Based on the attention calculation result, the probability distribution ultimately generated in the current inference step can be determined.

[0034] As an example, when performing long text continuation inference using a large language model, the key-value cache generated in previous inference steps is first retrieved and divided into multiple key-value data blocks. Each key-value data block contains key tensors and value tensors corresponding to 32 lexical units. The state parameters of the key-value cache are monitored. When the total number of key-value data blocks exceeds a preset threshold of 200, based on the correlation between each key-value data block and the current inference step, the top 50 key-value data blocks with the highest correlation to the current inference step are determined as the first data block, and the remaining 150 key-value data blocks are determined as the second data block.

[0035] The second data block is compressed to obtain the third data block. The third data block has a smaller data size than the second data block, but retains the context information of the second data block. In the current inference step, the key-value tensors of the first and third data blocks are used as key-value inputs and attention is calculated with the query vector generated from the current term to obtain the attention calculation result. Based on the attention calculation result, the output of the current term in the current inference step is determined, and the key-value tensor corresponding to the current term is written to the key-value cache to participate in the next round of inference.

[0036] The method in this embodiment divides the key-value cache into multiple key-value data blocks. Based on the degree of association with the current inference step, the first data block is selected to directly participate in attention calculation, while the second data block with a lower degree of association is compressed into a smaller third data block and then participates in attention calculation together. This achieves that, under fixed storage overhead, the second data block, after compression, still retains its contextual statistical features in the form of a third data block and continues to participate in attention calculation, avoiding the uncontrollable loss of historical information. Furthermore, the compression update of the third data block is triggered only when the state parameters of the key-value cache meet the set selection conditions, and the update frequency is related to the cache management event and is significantly lower than the update frequency at the inference step level. At the same time, the first and third data blocks directly participate in standard attention calculation as key-value inputs, and the state update process does not introduce additional model calculation paths beyond the original attention calculation, thus not affecting the parallelism of the original attention calculation of the model.

[0037] In some embodiments, step S103, which involves determining the first data block and the second data block from a plurality of key-value data blocks, includes: determining the importance score of each key-value data block; sorting the key-value data blocks in descending order of importance score; determining the first number of key-value data blocks that are ranked first as the first data block; and determining the key-value data blocks other than the first data block from the plurality of key-value data blocks as the second data block.

[0038] In this embodiment, the importance score is determined based on at least one of the following: the average attention weight of the key-value data block in historical attention calculations, the access frequency of the key-value data block within a set time window, and the time decay factor of the inference step corresponding to the key-value data block. The first quantity may include a pre-set quantity threshold. The first quantity is the number of key-value data blocks directly involved in attention calculations.

[0039] As an example, importance score The process of determining can be represented by the following formula:

[0040] in It can be determined by any or a combination of the following: 1. Determining the average attention weight based on key-value data blocks in historical attention calculation

[0041] Where q represents the query vector corresponding to each word in the key-value data block, k represents the key vector corresponding to each word in the key-value data block, and Q represents the set of query vectors corresponding to the key-value data block. This represents the key tensor of the i-th key-value data block. 2. Determining based on the access frequency of key-value data blocks within a set time window.

[0042] in, This represents the historical attention counting function, used to count the number of times key-value data blocks are read by attention calculation within a preset time window.

[0043] 3. Time decay factor of the inference step corresponding to the key-value data block

[0044] Where α∈(0,1), Indicates the current reasoning step. This represents the steps involved in generating the i-th key-value data block.

[0045] As an example, firstly, historical attention weight statistics and access frequency determination are performed on all cached key-value data blocks. Specifically, for each key-value data block, the corresponding average attention weight score is determined; simultaneously, the corresponding access frequency score is determined. These two scores are then weighted and fused according to preset weight coefficients (e.g., average attention weight accounts for 70%, access frequency accounts for 30%) to generate an importance score for the key-value data block. Based on the importance score, all key-value data blocks are sorted from high to low, and the first number of key-value data blocks in the sorted list are determined as the first data block, with the remaining key-value data blocks determined as the second data block.

[0046] The method in this application embodiment determines the importance score based on at least one of the average attention weight in historical attention calculation, the access frequency within a set time window, and the time decay factor of the corresponding inference step, and filters the first data block and the second data block according to the score. This can more accurately identify the key context of the current inference step, so that the historical context information can exist in a controllable and continuous state for a long time, avoiding the uncontrollable loss of historical semantic information and the long-term lack of the model's context state.

[0047] In some embodiments, the processing flow of the data processing method is illustrated. Figure 2 ,like Figure 2As shown, the compression process of the second data block in step S104 to obtain the third data block may include: Step S201: Obtain the currently existing historical third data block.

[0048] In this embodiment, the historical third data block is a compressed representation of the historical context information corresponding to the key-value cache that was previously identified as the second data block. The historical third data block can continuously store historical context information from distant periods. The compressed representation can represent historical context information with a data volume smaller than that of the original key-value cache.

[0049] Step S202: Perform intra-block aggregation and inter-block aggregation on each key-value data block in the second data block to obtain candidate update vectors.

[0050] Intra-block aggregation can be expressed by the following formula:

[0051]

[0052] in, This represents the key tensor of the i-th key-value data block. Let represent the value tensor of the i-th key-value data block, k represent the key vector corresponding to each word in the key-value data block, and v represent the value vector corresponding to each word in the key-value data block. Represents the aggregated bond vector. This represents an aggregated value vector.

[0053] Inter-block aggregation can be expressed by the following formula:

[0054]

[0055] in, This indicates the number of data blocks in the second data block. This represents the key vector in the candidate update vector. This represents the value vector within the candidate update vector. The candidate update vector can include a combination of key and value vectors generated through intra-block and inter-block aggregation. The candidate update vector can represent the context information of all second data blocks in this iteration.

[0056] Step S203: Based on the candidate update vector and the currently existing historical third data block, perform recursive fusion to obtain the third data block.

[0057] In some embodiments, step S203 may include: determining a gating factor based on the number of data blocks in the second data block and the cache occupancy data corresponding to the key-value cache. Based on the gating factor, recursively fusing the candidate update vector with the currently existing historical third data block to obtain the third data block. The gating factor characterizes the degree of influence of the key-value cache determined as the second data block on the historical third data block. The number of data blocks in the second data block represents the eviction scale to be aggregated in the current inference step.

[0058] As an example, determining the gating factor It can be expressed by the following formula:

[0059] Where P represents the cache usage data corresponding to the key-value cache, i.e., the current cache usage ratio. Gating factor The g(·) represents the degree of influence of all second data blocks on the historical third data blocks.

[0060] Based on the gating factor, the recursive fusion of the candidate update vector with the currently existing historical third data block can be expressed by the following formula:

[0061]

[0062] in, This represents the key tensor of the currently existing historical third data block. The tensor representing the value of the currently existing historical third data block. This represents the key tensor corresponding to the third data block. This represents the value tensor corresponding to the third data block.

[0063] As an example, the currently existing third data block is read. Intra-block aggregation is performed on all the second data blocks identified in this analysis; that is, the arithmetic mean of the key tensors and value tensors corresponding to all terms within each key-value data block is calculated to obtain the aggregated key vector and aggregated value vector for each key-value data block. Inter-block aggregation is then performed on the aggregated vectors of all second data blocks, calculating the arithmetic mean of each aggregated key vector and aggregated value vector to obtain the candidate update vector corresponding to the second data block. Based on the number of second data blocks and the current cache occupancy ratio, a gate control factor is calculated using a gating function. The candidate update vector is then recursively fused with the currently existing historical third data blocks according to the gate control factor to generate the third data block. The third data block can participate in attention calculation together with the first data block.

[0064] The method in this application embodiment obtains candidate update vectors by first performing intra-block aggregation and then inter-block aggregation on the second data block, and then recursively merges them with the historical third data block. This allows the third data block to have a much smaller data volume than the original second data block. Even when the KV Cache is pruned or compressed, historical information still participates in reasoning in the form of state. This effectively controls the size of the KV Cache, reduces the computational complexity of attention, and continuously retains historical context information under fixed storage overhead, avoiding uncontrollable loss of historical context information and long-term loss of model context state.

[0065] In some embodiments, the method further includes: determining a candidate update vector as a third data block in response to the absence of a current historical third data block.

[0066] In this embodiment, when the state parameters of the key-value cache meet the filtering conditions for the first time during the model inference process, since the state parameters of the key-value cache did not meet the filtering conditions before, it is determined that there is no historical third data block. Intra-block aggregation and inter-block aggregation are performed on each key-value data block in the second data block to obtain a candidate update vector, and the candidate update vector is determined as the third data block.

[0067] As an example, determining the candidate update vector as the third data block can be expressed by the following formula:

[0068]

[0069] in, This represents the key tensor corresponding to the third data block. This represents the value tensor corresponding to the third data block.

[0070] The method in this application embodiment directly determines the candidate update vector as the third data block when there is no historical third data block; when the filtering conditions are met for the first time or the cache reaches the pressure threshold for the first time, the initial third data block can be quickly established and continuously participate in subsequent attention calculations, so that the historical context information can still participate in attention calculations as the overall semantic state after being compressed, thereby avoiding the uncontrollable loss of historical information.

[0071] In some embodiments, determining a gating factor based on the number of data blocks in the second data block and the cache occupancy data corresponding to the key-value cache includes: obtaining the number of data blocks and a pre-set number of reference evicted blocks; obtaining the current cache occupancy ratio and a pre-set cache pressure threshold included in the cache occupancy data; calculating the ratio of the number of data blocks to the number of reference evicted blocks, and the difference between the current cache occupancy ratio and the cache pressure threshold; and inputting the ratio and the difference into a pre-set gating function to obtain the gating factor.

[0072] In this embodiment, the current cache occupancy ratio may include the percentage of key-value cache used storage capacity out of the total cache capacity. The cache pressure threshold may include a pre-set cache occupancy ratio threshold. This embodiment does not limit the specific current cache occupancy ratio and cache pressure threshold. The gating function may include a state update intensity calculation model based on eviction scale and cache pressure. The gating factor may be an adjustment parameter controlling the recursive fusion ratio of candidate update vectors and historical third data blocks.

[0073] As an example, during the long text continuation inference process of the model, when the state parameters of the key-value cache meet the set filtering conditions, the number of data blocks identified as the second data block in the current inference step is determined, and the pre-set number of reference eviction blocks is read. The current cache occupancy ratio of the key-value cache is obtained, and the pre-set cache pressure threshold is read. The ratio of the number of data blocks to the number of reference eviction blocks is calculated, and the difference between the current cache occupancy ratio and the cache pressure threshold is also calculated. Finally, the calculated ratio and difference are input into a pre-set gating function and summed to obtain the gating factor.

[0074] The method in this application determines the gating factor based on the number of data blocks in the second data block and the cache usage data, and uses the gating factor to control the recursive fusion ratio of the candidate update vector and the historical third data block. This achieves the matching of the summary state update intensity with the current elimination scale and cache pressure, avoiding the uncontrollable loss of historical information and the long-term lack of model context state.

[0075] In some embodiments, the gating function further includes a weight parameter, and the method further includes: after the third data block participates in attention calculation, determining a first average attention weight of the third data block and a second average attention weight of the first data block; and adjusting the weight parameter based on the difference between the second average attention weight and the first average attention weight.

[0076] As an example, determining the gating factor It can be expressed by the following formula:

[0077] in, The Sigmoid activation function is used. The set reference number of discarded blocks, Here, is the cache pressure threshold, and 'a' and 'b' are adjustable weight parameters. To control the weighting parameters of the impact of the elimination scale on the state update, The weighted parameter P, representing the current cache occupancy percentage, controls the impact of cache pressure on state updates. (Gating factor) This characterizes the extent to which the entire second data block affects the historical third data block.

[0078] Determine the first average attention weight for the third data block. It can be expressed by the following formula:

[0079] Where H represents the number of attention heads. This represents the attention weight function of the h-th attention head. h represents the index of a single attention channel in a multi-head attention mechanism.

[0080] Determine the second average attention weight for the first data block. It can be expressed by the following formula:

[0081] in, Indicates the first data block. This represents the i-th first data block.

[0082] The weight parameters are adjusted based on the difference between the second average attention weight and the first average attention weight, which can be expressed by the following formula:

[0083]

[0084] Where clip(·) represents the clipping function. Let 'a' represent the adjustable weight parameter 'a' for the (t+1)th inference step. Let b represent the adjustable weight parameter for the (t+1)th inference step. Let a represent the adjustable weight parameter 'a' for the t-th inference step. Let b represent the adjustable weight parameter for the t-th inference step, and η represent the learning rate. This represents the lower bound of the weight parameter a. This represents the upper limit of the weight parameter 'a'. This represents the lower bound of the weight parameter b. This represents the upper limit of the weight parameter b.

[0085] In this embodiment, after the third data block participates in the attention calculation of the current inference step, the attention weight distribution corresponding to the third data block, i.e., the first average attention weight, is statistically analyzed and compared with the second average attention weight corresponding to the first data block. When the first average attention weight is lower than the second average attention weight, it indicates that the historical information expression of the third data block is insufficient, and the recursive fusion strength of subsequent inference steps is correspondingly enhanced. Conversely, when the first average attention weight is higher than the second average attention weight, the recursive fusion strength of subsequent inference steps is correspondingly reduced to avoid excessive interference of historical states on the new context.

[0086] The method in this application adjusts the weight parameters of the gating function based on the difference in average attention weight between the third data block and the first data block after the third data block participates in the attention calculation. When the attention of the third data block is lower than that of the first data block, the subsequent fusion intensity is enhanced; otherwise, the fusion intensity is reduced to avoid excessive interference, uncontrollable loss of historical information, and loss of long-term contextual state of the model.

[0087] In some embodiments, step S105, using the first data block and the third data block as key-value inputs for attention calculation and performing attention calculation with the query vector of the current inference step, may include: concatenating the key tensor corresponding to the first data block and the key tensor corresponding to the third data block according to the sequence dimension to obtain the final key tensor; concatenating the value tensor corresponding to the first data block and the value tensor corresponding to the third data block according to the sequence dimension to obtain the final value tensor; and performing attention calculation based on the query vector, the final key tensor, and the final value tensor of the current inference step to obtain the attention calculation result.

[0088] As an example, the final key tensor and final value tensor The process of determining can be represented by the following formula:

[0089]

[0090] in, This represents the key tensor corresponding to the first data block. This represents the key tensor corresponding to the third data block. This represents the value tensor corresponding to the third data block. This represents the value tensor corresponding to the first data block.

[0091] Based on the query vector, final key tensor, and final value tensor of the current inference step, attention computation can be represented by the following formula:

[0092] in, This indicates attention calculation.

[0093] As an example, the first and third data blocks are read. The key tensors corresponding to the first and third data blocks are concatenated end-to-end along the same sequence dimension to obtain the final key tensor. Similarly, the value tensors corresponding to the first and third data blocks are concatenated end-to-end along the same sequence dimension to obtain the final value tensor. The query vector in the current inference step is obtained. Attention is calculated based on the query vector, the final key tensor, and the final value tensor to obtain the attention calculation result.

[0094] The method in this application embodiment performs attention calculation by concatenating the key tensors and value tensors of the first data block and the third data block along the sequence dimension. This allows the current query vector to retrieve recent context information and long-term compressed context information simultaneously in a single attention operation. The state update process does not introduce additional model computation paths and does not affect the original attention parallelism of the Transformer. It obtains long-term memory capabilities without increasing computational path complexity, and avoids the uncontrollable loss of historical information and the lack of long-term context state of the model.

[0095] refer to Figure 3 This application provides an example of an application scenario diagram of a data processing method, which is applied to a KV Cache data processing flow based on recursive state compression.

[0096] In the local inference scenario of the AI ​​(Artificial Intelligence) model platform, the user first inputs a lengthy project background into the model, followed by multiple rounds of question-and-answer sessions surrounding the project background. As the dialogue progresses, the model generates a new token and simultaneously produces a corresponding key-value pair, which is then written into a key-value block set. The importance score of each block is calculated in real time, and blocks that reach the preset criteria are marked as reserved blocks. After Top-K filtering, these blocks can be directly used as key-value inputs for attention calculation.

[0097] Blocks that do not reach the threshold are marked as unselected blocks. When the cache occupancy ratio P meets the eviction criteria, they enter the recursive compression module to generate compressed key-value blocks. Finally, the retained blocks and the compressed key-value blocks are assembled together as key-value inputs to participate in attention calculations.

[0098] Even after the dialogue has gone through ten or so rounds, the original KV cache corresponding to the early context is no longer fully preserved. However, key background information continues to participate in subsequent reasoning through the Summary state, thereby avoiding semantic breaks, ignoring early key information, or losing long-term memory in subsequent responses.

[0099] refer to Figure 4 Another application scenario diagram of the data processing method provided in this application embodiment is applied to the KV Block processing flow in the recursive compression module.

[0100] In the recursive compression module, the first inputs are the set of unselected KV Blocks, the cache occupancy ratio P, and the Block eviction status. Block-level aggregation is then performed on the unselected KV Block set, compressing multiple key-value data blocks into candidate update vectors through intra-block and inter-block aggregation.

[0101] The system determines whether a Summary Block (summary key-value data block) exists. If it does not exist, it initializes the Summary Block and uses the candidate update vector directly as the updated Summary Block.

[0102] If it exists, load the historical Summary Block from the storage medium, perform a recursive state update between the candidate update vector and the loaded Summary Block, and generate the updated Summary Block.

[0103] The updated Summary Block is written to the cache for storage, so that it can be reloaded and continue to participate in recursive compression when a subsequent Block eviction occurs.

[0104] The exemplary structure of the data processing apparatus 90 provided in the embodiments of this application will be further described below. In some embodiments, such as... Figure 5 As shown, the data processing device 90 may include: an acquisition module 901, which can be used to acquire the key-value cache generated by the inference model during the inference process; The partitioning module 902 can be used to divide the key-value cache into multiple key-value data blocks, each key-value data block containing key tensors and value tensors corresponding to the lexical units generated in multiple inference steps; The determination module 903 can be used to determine a first data block and a second data block from multiple key-value data blocks in response to the state parameters of the key-value cache meeting the set filtering conditions; the first data block has a higher correlation with the current inference step than the second data block. Compression module 904 can be used to compress the second data block to obtain a third data block, the data size of the third data block being smaller than that of the second data block; the third data block includes at least the context information of the second data block; The calculation module 905 can be used to take the first data block and the third data block as key-value inputs for attention calculation in the current inference step of the inference model, perform attention calculation with the query vector of the current inference step, obtain the attention calculation result, and determine the output of the current inference step based on the attention calculation result.

[0105] In some embodiments, the determining module 903 may be used to: determine the importance score of each key-value data block; the importance score is determined based on at least one of the following: the average attention weight of the key-value data block in historical attention calculation, the access frequency of the key-value data block within a set time window, and the time decay factor of the inference step corresponding to the key-value data block; sort the key-value data blocks in descending order of importance score, and determine the first number of key-value data blocks at the top of the sort as the first data block; and determine the key-value data blocks other than the first data block among the multiple key-value data blocks as the second data block.

[0106] In some embodiments, the compression module 904 can be used to: obtain the currently existing historical third data block; the historical third data block is a compressed representation of the historical context information corresponding to the key-value cache determined to be the second data block; perform intra-block aggregation and inter-block aggregation on each key-value data block in the second data block to obtain a candidate update vector; and perform recursive fusion based on the candidate update vector and the currently existing historical third data block to obtain the third data block.

[0107] In some embodiments, the compression module 904 can be used to: determine a gating factor based on the number of data blocks of the second data block and the cache occupancy data corresponding to the key-value cache; the gating factor characterizes the degree of influence of the key-value cache determined as the second data block on the historical third data block; and based on the gating factor, recursively merge the candidate update vector with the currently existing historical third data block to obtain the third data block.

[0108] In some embodiments, the data processing apparatus 90 is further configured to: determine the candidate update vector as the third data block in response to the absence of a current historical third data block.

[0109] In some embodiments, the compression module 904 can be used to: obtain the number of data blocks and a preset number of reference evicted blocks; obtain cache occupancy data including the current cache occupancy ratio and a preset cache pressure threshold; calculate the ratio of the number of data blocks to the number of reference evicted blocks, and the difference between the current cache occupancy ratio and the cache pressure threshold; input the ratio and the difference into a preset gating function to obtain a gating factor.

[0110] In some embodiments, the gating function further includes a weight parameter; the data processing device 90 is further configured to: determine a first average attention weight of the third data block and a second average attention weight of the first data block after the third data block participates in attention calculation; and adjust the weight parameter based on the difference between the second average attention weight and the first average attention weight.

[0111] In some embodiments, the calculation module 905 may be used to: concatenate the key tensor corresponding to the first data block and the key tensor corresponding to the third data block according to the sequence dimension to obtain the final key tensor; concatenate the value tensor corresponding to the first data block and the value tensor corresponding to the third data block according to the sequence dimension to obtain the final value tensor; and perform attention calculation based on the query vector, the final key tensor and the final value tensor of the current inference step to obtain the attention calculation result.

[0112] It should be noted that the description of the apparatus in this application embodiment is similar to the description of the method embodiment above, and has similar beneficial effects as the method embodiment; therefore, it will not be repeated. For any technical details not covered in the data processing apparatus provided in this application embodiment, please refer to... Figures 1 to 5 The meaning is understood in accordance with the description of any of the accompanying drawings.

[0113] According to embodiments of this application, this application also provides an electronic device and a non-transitory computer-readable storage medium.

[0114] Figure 6 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of this application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0115] like Figure 6 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. The RAM 803 may also store various programs and data required for the operation of the electronic device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0116] Multiple components in electronic device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of displays, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows electronic device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0117] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as data processing methods. For example, in some embodiments, the data processing method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform data processing methods by any other suitable means (e.g., by means of firmware).

[0118] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0119] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0120] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0121] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0122] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0123] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0124] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0125] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0126] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data processing method, comprising: Retrieve the key-value cache generated by the inference model during the inference process; The key-value cache is divided into multiple key-value data blocks, and each key-value data block contains key tensors and value tensors corresponding to the lexical units generated in multiple inference steps; In response to the key-value cache's state parameters satisfying the set filtering conditions, a first data block and a second data block are determined from the plurality of key-value data blocks; the first data block has a higher correlation with the current inference step than the second data block; The second data block is compressed to obtain a third data block, the data size of the third data block being smaller than that of the second data block; the third data block includes at least the context information of the second data block. In the current inference step of the inference model, the first data block and the third data block are used as key-value inputs for attention calculation, and attention calculation is performed with the query vector of the current inference step to obtain the attention calculation result. The output of the current inference step is determined based on the attention calculation result.

2. The method according to claim 1, wherein determining the first data block and the second data block from the plurality of key-value data blocks comprises: Determine the importance score for each key-value data block; The importance score is determined based on at least one of the following: the average attention weight of the key-value data block in historical attention calculations, the access frequency of the key-value data block within a set time window, and the time decay factor of the inference step corresponding to the key-value data block. The key-value data blocks are sorted in descending order of importance score, and the first number of key-value data blocks at the top of the sort is determined as the first data block. The key-value data block other than the first data block among the plurality of key-value data blocks is determined as the second data block.

3. The method according to claim 1, wherein compressing the second data block to obtain the third data block comprises: Retrieve the currently existing historical third data block; The historical third data block is a compressed representation of the historical context information corresponding to the key-value cache that was determined to be the second data block; Perform intra-block aggregation and inter-block aggregation on each key-value data block in the second data block to obtain a candidate update vector; The third data block is obtained by recursively fusing the candidate update vector and the currently existing historical third data block.

4. The method according to claim 3, wherein the recursive fusion based on the candidate update vector and the currently existing historical third data block to obtain the third data block includes: The gating factor is determined based on the number of data blocks in the second data block and the cache usage data corresponding to the key-value cache. The gating factor characterizes the degree of influence of the key-value cache of the second data block on the historical third data block; Based on the gating factor, the candidate update vector is recursively fused with the currently existing historical third data block to obtain the third data block.

5. The method according to claim 3, further comprising: In response to the absence of the historical third data block, the candidate update vector is determined as the third data block.

6. The method according to claim 4, wherein determining the gating factor based on the number of data blocks in the second data block and the cache occupancy data corresponding to the key-value cache includes: Get the number of data blocks and the pre-set reference number of discarded blocks; The cache usage data obtained includes the current cache usage ratio and the preset cache pressure threshold; Calculate the ratio of the number of data blocks to the number of reference evicted blocks, and the difference between the current cache occupancy ratio and the cache pressure threshold; The ratio and the difference are input into a pre-set gating function to obtain the gating factor.

7. The method according to claim 6, wherein the gating function further includes weight parameters; the method further includes: After the third data block participates in the attention calculation, the first average attention weight of the third data block and the second average attention weight of the first data block are determined. The weight parameters are adjusted based on the difference between the second average attention weight and the first average attention weight.

8. The method according to claim 1, wherein the step of using the first data block and the third data block as key-value inputs for attention calculation, and performing attention calculation with the query vector of the current inference step, comprises: The key tensor corresponding to the first data block and the key tensor corresponding to the third data block are concatenated according to the sequence dimension to obtain the final key tensor; The value tensor corresponding to the first data block and the value tensor corresponding to the third data block are concatenated according to the sequence dimension to obtain the final value tensor; Based on the query vector of the current inference step, the final key tensor, and the final value tensor, attention calculation is performed to obtain the attention calculation result.

9. A data processing apparatus, comprising: The acquisition module is used to retrieve the key-value cache generated by the inference model during the inference process. The partitioning module is used to divide the key-value cache into multiple key-value data blocks, each key-value data block containing key tensors and value tensors corresponding to the lexical units generated in multiple inference steps; The determination module is used to determine a first data block and a second data block from the plurality of key-value data blocks in response to the state parameters of the key-value cache meeting the set filtering conditions; the first data block has a higher correlation with the current inference step than the second data block. A compression module is used to compress the second data block to obtain a third data block, wherein the data size of the third data block is smaller than that of the second data block; The third data block includes at least the context information of the second data block; The calculation module is used to take the first data block and the third data block as key-value inputs for attention calculation in the current inference step of the inference model, perform attention calculation with the query vector of the current inference step, obtain the attention calculation result, and determine the output of the current inference step based on the attention calculation result.

10. An electronic device, comprising: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform: Retrieve the key-value cache generated by the inference model during the inference process; The key-value cache is divided into multiple key-value data blocks, and each key-value data block contains key tensors and value tensors corresponding to the lexical units generated in multiple inference steps; In response to the key-value cache's state parameters satisfying the set filtering conditions, a first data block and a second data block are determined from the plurality of key-value data blocks; the first data block has a higher correlation with the current inference step than the second data block; The second data block is compressed to obtain a third data block, the data size of the third data block being smaller than that of the second data block; the third data block includes at least the context information of the second data block. In the current inference step of the inference model, the first data block and the third data block are used as key-value inputs for attention calculation, and attention calculation is performed with the query vector of the current inference step to obtain the attention calculation result. The output of the current inference step is determined based on the attention calculation result.

Images