A method and system for generating context adaptive compression

Abstract

The application relates to the technical field of large language model reasoning, and discloses a context-adaptive compression generation method and system. A cache response generated by a target historical cache unit to a constructed query state is obtained, and a post-attention representation formed after the target historical cache unit participates in attention calculation is obtained; the post-attention representation is mapped to a candidate key-value state, a key-value state is determined after position recoding, and a residual key-value state is formed according to a cache response that is not adopted; a verification query state is used to verify joint replacement and form a replacement condition; when the joint cache occupancy is less than the cache occupancy of the target historical cache unit, the target historical cache unit is transferred to a backup storage and an active key-value cache is released; according to whether the online query state meets the replacement condition, the key-value state and the residual key-value state are called or the target historical cache unit is restored, so that the release of the active key-value cache corresponds to the actual attention calculation demand in the future.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of large language model inference and key-value cache management, and particularly relates to a context adaptive compression generation method and system. BACKGROUND

[0002] Large language models usually generate text in an autoregressive manner. In the autoregressive generation process, each Transformer layer performs attention calculation on the query state corresponding to the current generation position and the key state and value state corresponding to the previous generation position. To avoid repeated calculation of the key state and value state that has been formed during continuous generation, the key state and value state corresponding to the historical word are usually saved in the key-value cache and continue to be called in the subsequent generation process.

[0003] As the input context length and the generated content length increase, the historical key state and value state saved in the key-value cache continue to increase, and the corresponding storage resources occupied also increase. In long context generation, long round dialogue, and long text generation scenarios, the active storage space occupied by the key-value cache may become a factor limiting the continuous generation length and inference concurrency of the model. Therefore, reducing the key-value cache occupation while maintaining the effect of the historical context on subsequent generation is an important processing requirement in the inference process of large language models.

[0004] The existing key-value cache processing method can reduce cache occupation by selectively retaining part of the historical key-value state, removing part of the historical key-value state, reducing the data representation precision of the key-value state, or compressing the representation of the key-value state. The corresponding processing usually determines the cache content to be retained or compressed according to the access situation of the historical key-value state, the attention relationship, or the data characteristics of the key-value state itself, in order to reduce the number of historical states or the amount of historical state data that need to be continuously saved for subsequent attention calculation.

[0005] However, when a large language model continuously performs autoregressive generation, the actual calculation effect of the historical key-value state on different generation stages will change with the query state. Part of the calculation effect of the historical key-value state produced in the previous attention calculation may have been reflected in the subsequent formed internal state of the model, while part of the calculation effect still needs to be maintained in the subsequent query state. If the cache retention range is still determined according to the access characteristics of the historical key-value state or the fixed compression relationship, it is easy to occur that the historical key-value state that has formed repeated calculation effect still continuously occupies the active cache, or the historical calculation effect that still needs to participate in the subsequent attention calculation is lost with the cache compression, thereby making it difficult to maintain a corresponding relationship between the key-value cache resource release and the actual subsequent attention calculation demand. SUMMARY

[0006] The application aims to provide a context adaptive compression method and system to solve the problem that the release basis of the history key value state cache during the continuous execution of the large language model self-regressive generation process cannot correspond to the actual attention calculation demand, resulting in continuous increase of the active key value cache or loss of the history calculation effect due to cache compression.

[0007] To solve the above technical problems, the application adopts the following technical solutions.

[0008] In one aspect, the application provides a context adaptive compression method, comprising: obtaining cache responses of a target history cache unit to a plurality of constructed query states in a large language model self-regressive generation process; obtaining a post-attention representation formed after the target history cache unit completes attention calculation according to a corresponding generation step call, mapping the post-attention representation to a candidate key value state, re-encoding the position of the candidate key value state according to the history position distribution of the target history cache unit, and determining the key value state according to the cache response recurrence degree; generating a residual key value state according to the cache response difference of the target history cache unit and the key value state to the same constructed query state; verifying the joint replacement of the verification query state not participating in the key value state and the residual key value state, and forming a replacement condition according to the verification result; when the joint cache occupancy of the key value state and the residual key value state is less than the cache occupancy of the target history cache unit, transferring the target history cache unit to a backup storage and releasing the active key value cache resource occupied by the target history cache unit; for the online query state, calling the key value state and the residual key value state to participate in attention calculation when the replacement condition is met, and restoring the target history cache unit when the replacement condition is not met.

[0009] Further, when determining the key value state, the post-attention representation formed by each constructed query state according to the corresponding generation step is obtained, each post-attention representation is mapped to the key state space and value state space corresponding to the target Transformer layer to form a candidate key value state, the cache response difference reduction amount before and after each candidate key value state is added to the current key value state group is determined, and the candidate key value state is determined according to the updated cache response difference to form the key value state.

[0010] Further, when re-encoding the position of the candidate key value state, at least one source position anchor point is determined according to the attention contribution of the constructed query state to different history positions in the target history cache unit; the position encoding state of the candidate key value state is obtained, or the inverse position transformation is performed on the candidate key value state to which the position encoding has been applied; and the position encoding of the obtained candidate key value state is re-applied according to the source position anchor point.

[0011] Furthermore, when generating residual key-value states, at least one residual key-value slot is established, and the joint cache response generated by the accepting state and the residual key-value slot is calculated using the constructed query state. The residual key-value slot is adjusted according to the difference between the joint cache response and the remaining cache response of the target historical cache unit. When the difference in the remaining cache response exceeds the allowable deviation, and the joint cache occupancy of the accepting state and the residual key-value slot is less than the cache occupancy of the target historical cache unit, the number of residual key-value slots is increased.

[0012] Furthermore, when forming alternative conditions, the baseline cache response of the target historical cache unit and the joint cache response generated by the acceptance state and the residual key value state are determined by verifying the query state respectively; the verification query state in which the difference in attention normalization contribution and the difference in value weighted response do not exceed their respective allowable deviations are determined as the pass state; the verification query anchor point and the corresponding coverage are determined according to the state relationship between the pass states, and the alternative conditions corresponding to the online query state are formed by using the coverage.

[0013] Furthermore, for online query states that do not meet the alternative conditions, the target historical cache unit is restored from the backup storage; the cache response corresponding to the online query state is calculated using the restored target historical cache unit, the inherited state, and the residual key-value state, respectively; when the difference in attention normalization contribution and the difference in value-weighted response between the two cache responses do not exceed their respective allowable deviations, the corresponding online query state is added to the pass state, and the coverage corresponding to the verification query anchor point is updated.

[0014] Furthermore, when releasing the active key-value cache resources occupied by the target historical cache unit, the combined cache occupancy of the inherited state and the residual key-value state, as well as the occupancy status of the physical cache block where the target historical cache unit is located, are determined. If the combined cache occupancy is less than the cache occupancy of the target historical cache unit, and there are no other key-value states in the corresponding physical cache block that need to remain active, the target historical cache unit is transferred to the backup storage, and the corresponding physical cache block is released.

[0015] Furthermore, when the accepting state participates in the attention calculation in the subsequent autoregressive generation process and forms a new post-attention representation, the new post-attention representation is mapped to the second candidate accepting key value state, and the second accepting state and the direct residual key value state are redefined for the original target historical cache unit; when the second accepting state and the direct residual key value state are jointly verified by substitution for the original target historical cache unit, the direct substitution relationship corresponding to the original target historical cache unit is established, and the intermediate substitution relationship formed through the original accepting state is canceled.

[0016] Furthermore, the cache usage of the active key-value cache is monitored. When the cache usage reaches the cache budget, historical cache units that have formed a construction query state and corresponding cache response, and have formed a verification query state that has not participated in the generation of the acceptance state and residual key-value state, are identified as candidate historical cache units. The processing order is determined according to the formation state of the verification query state corresponding to each candidate historical cache unit and the cache usage, and the target historical cache unit is determined according to the processing order.

[0017] On the other hand, the present invention provides a context-adaptive compression system, including an inference processor, an active key-value cache, a backup memory, and a program memory, wherein the active key-value cache, the backup memory, and the program memory establish data access relationships with the inference processor.

[0018] The active key-value cache is used to store the historical key-value states that participated in the attention calculation of the large language model; the backup memory is used to store the historical key-value states that exited the active key-value cache and are retained for recovery purposes; the program memory stores the computer program executed by the inference processor.

[0019] The inference processor is used to obtain the cache response generated by the target historical cache unit for multiple constructed query states, obtain the post-attention representation formed after the target historical cache unit participates in attention calculation, and form the acceptance state based on the post-attention representation; form the residual key-value state based on the difference in cache response between the target historical cache unit and the acceptance state; form the substitution condition by verifying the query state; after satisfying the joint substitution requirement and the cache resource benefit requirement, transfer the target historical cache unit to the backup storage and release the corresponding active key-value cache resources; and, depending on whether the online query state satisfies the substitution condition, call the acceptance state and the residual key-value state to participate in attention calculation, or restore the target historical cache unit from the backup storage.

[0020] Compared with the prior art, the beneficial effects achieved by the present invention are as follows: 1. This invention obtains the post-attention representation formed after the target historical cache unit has participated in attention calculation, and constructs the receiving state from the post-attention representation, so that the historical calculation effect that has been reflected in the internal state of the subsequent model can be re-represented as a key-value state that can participate in attention calculation, thereby reducing the situation where the same historical calculation effect is repeatedly occupied by the complete historical key-value state in the active cache.

[0021] 2. For historical computational actions that have not yet been maintained in the receiving state, this invention forms a residual key-value state based on the difference between the cache response generated by the target historical cache unit and the receiving state for the same constructed query state. This residual key-value state supplements the attention normalization action and value state output action that have not yet been received, reducing the repeated representation of computational actions that have already been maintained by the receiving state in the residual state, thereby controlling the active cache resources occupied by the alternative state.

[0022] 3. This invention uses a verification query state that was not involved in the generation of the succession state and residual key-value state to verify the joint substitution relationship, and forms substitution conditions based on the verification results. This ensures that whether a historical cache unit exits the active key-value cache is simultaneously constrained by the actual query state, which is separate from the construction process. When the online query state exceeds the already verified substitution range, the complete target historical cache unit can be restored, thereby reducing the possibility that the fixed deletion of historical key-value states will not be able to adapt to subsequent query state changes.

[0023] 4. Based on the establishment of joint substitution, this invention uses the condition that the joint cache occupancy of the accepting state and the residual key-value state is less than the cache occupancy of the target historical cache unit as the cache release condition, and releases the corresponding active cache resources when the physical cache block release condition is met, so that the substitution of historical computing role is further transformed into reallocatable active storage space, thereby reducing the pressure of active key-value cache growth during the continuous generation of long context.

[0024] 5. For new post-attention representations formed during the continuous generation process, the present invention can reconstruct new acceptance states and establish direct substitution relationships for the original target historical cache units. After verification, the intermediate substitution relationships are canceled, so that the acceptance states, residual key-value states, and substitution association records will not form uncontrolled multi-level relationships during the generation process, which is beneficial to controlling the cache management overhead during the long-term generation process. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the overall process of the context-adaptive compression method of the present invention; Figure 2 This is a schematic diagram of the generation context adaptive compression system of the present invention; Figure 3 This is a schematic diagram illustrating the processing relationships of cache response acceptance, residual compensation, and joint substitution in this invention; Figure 4 This is a schematic diagram of the method for alternative calculation and historical cache recovery in the online query state of the present invention; Figure 5 This is a schematic diagram of the method for generating and iteratively adjusting residual key value states according to the present invention; Figure 6 A schematic diagram of the method for determining the target historical cache unit of the present invention; Figure 7 This is a schematic diagram of the method flow for joint substitution verification and substitution condition formation of the present invention; Figure 8 This is a schematic diagram of the method for merging multi-generational inheritance relationships according to the present invention. Detailed Implementation

[0026] This invention can be modified in many ways and has many embodiments, with specific embodiments shown in the accompanying drawings for detailed description. However, this does not mean that the invention is limited to a specific implementation; it should be understood that all modifications, equivalents, and even substitutions falling within the concept and technical scope of this invention are included in this invention. Similar reference numerals are used for similar constituent elements in the description of the drawings.

[0027] The terms “first,” “second,” “A,” “B,” etc., are used to describe a wide variety of constituent elements, but these constituent elements are not limited by these terms. These terms are used to distinguish one constituent element from others. For example, without departing from the scope of the invention, a first constituent element may be named a second constituent element, and similarly, a second constituent element may be named a first constituent element. The term “and / or” includes a combination of multiple associated descriptions, or one of multiple associated descriptions.

[0028] When it is mentioned that a certain constituent element is "connected" or "linked" to other constituent elements, it can mean not only that the element is directly connected or linked to the other constituent element, but also that there are other constituent elements in between. Conversely, when it is mentioned that a certain constituent element is "directly connected" or "directly linked" to other constituent elements, there are no other constituent elements in between.

[0029] The terminology used in this application is for illustrative purposes only and is not intended to limit the scope of the invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. Terms such as "comprising" or "having" as used herein do not preclude the possibility of the presence or addition of features, numbers, stages, actions, constituent elements, components, or combinations thereof described in the specification.

[0030] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0031] Terms defined in common dictionaries should be interpreted as having the same meaning as in the context of the relevant technology, and should not be interpreted as having an ideal or overly formal meaning unless explicitly defined in this application.

[0032] Example 1 refer to Figures 1 to 8 As shown in this embodiment, a generation context adaptive compression method is provided for dynamically processing the historical key-value cache that increases with the generation length during the autoregressive generation process of a large language model.

[0033] After the large language model completes the pre-filling of the input context, each Transformer layer generates key and value states corresponding to historical lexical units and stores these states in the active key-value cache. As autoregressive decoding continues, the key and value states corresponding to newly generated lexical units are continuously written into the active key-value cache, causing its resource consumption to continuously increase.

[0034] Existing key-value cache compression methods, based on attention intensity, access frequency, or a fixed retention ratio, primarily determine the retention targets based on the historical key-value states themselves or the current access results. When the subsequent model's internal state already reflects the computational effects generated by some historical key-value states, completely retaining the corresponding historical key-value states will result in duplicate activity cache usage; directly deleting the corresponding historical key-value states may cause the loss of computational effects that have not yet been retained by the subsequent model's internal state.

[0035] Based on this, this embodiment utilizes the query states generated during the actual autoregressive generation process of the large language model to obtain the attention calculation effects already generated by the historical key-value states, and constructs the continuation state using the post-attention representation formed after the target historical cache unit participates in the attention calculation. For the calculation effects that have not yet been maintained by the continuation state, residual key-value states are formed. After the continuation state and residual key-value states are verified by the query state to jointly maintain the calculation effects of the historical cache unit, the corresponding historical key-value states are removed from the active key-value cache.

[0036] The method provided in this embodiment includes: S101, monitor the resource usage status of the active key-value cache, determine the target historical cache unit, and obtain the cache response generated by the target historical cache unit for multiple construction query states; S102, Generate candidate acceptance key value states based on the post-attention representation formed after the target historical cache unit participates in the attention calculation, determine the acceptance state, and recode the acceptance state according to the historical position distribution of the target historical cache unit. S103, Generate residual key-value state based on the difference in cache response generated by the target historical cache unit and the position re-encoded acceptance state for the same construction query state; S104, use the verification query state that did not participate in the generation of the acceptance state and residual key value state to verify the joint substitution relationship, and form substitution conditions based on the verification results; S105: Transfer the target historical cache unit that meets the joint substitution requirements to the backup storage and release the active key-value cache. Perform substitution calculation, historical cache recovery and substitution relationship update according to the relationship between online query status and substitution conditions.

[0037] S101, monitor the resource usage status of the active key-value cache, determine the target historical cache unit, and obtain the cache response generated by the target historical cache unit for multiple construction query states.

[0038] Specifically, after the large language model completes the pre-filling of the input context, each Transformer layer forms a key state and value state corresponding to the historical lexical units, and writes the key state and value state into the active key-value cache. As autoregressive decoding continues, the key state and value state corresponding to newly generated lexical units are continued to be written into the active key-value cache, and the cache management program synchronously obtains the storage resources currently occupied by the active key-value cache.

[0039] The active key-value cache is configured with a corresponding cache budget. The cache budget is determined based on the key-value cache storage space allocated by the inference device for the current large language model inference task, and is maintained by the cache management program during model execution or updated based on inference resource scheduling results. By using a cache budget corresponding to the actual inference resources, the context compression triggering condition directly reflects the active key-value cache's resource capacity to continue accommodating historical key-value states, rather than being triggered by a fixed generation length or a fixed number of lexical units.

[0040] Furthermore, the cache management program organizes the key and value states in the active key-value cache into multiple historical cache units, based on the data storage granularity that allows for independent reading, migration, or release. Each historical cache unit is associated with a corresponding historical term position range, Transformer layer, key-value header, and active cache position, enabling the cache management program to determine the active cache resources occupied by different historical cache units.

[0041] In one implementation, when using a paginated key-value cache, the historical cache unit is formed according to the key-value state corresponding to the logical cache page; when using a contiguous cache structure, the historical cache unit is formed according to a contiguous key-value state region that can independently perform cache migration. By using historical cache units with the same granularity as the original cache management of the inference framework, the existing cache mapping relationship can be directly used to complete the transfer and release after the replacement conditions are met, reducing the additional data movement caused by reorganizing historical key-value data.

[0042] While the active key-value cache has not yet reached its cache budget, the large language model continues to write newly generated key-value states according to the normal autoregressive generation path. Simultaneously, it saves query response records generated during the model's normal attention calculation process for historical cache units. These query response records include the constructed query state, the corresponding generation location, and the cached responses generated by historical cache units in response to the constructed query state.

[0043] Furthermore, for the same historical cache unit, the query response records used for constructing subsequent successor states and residual key-value states are distinguished from the verification query states and their baseline cache responses that did not participate in the above construction process, according to their generation sequence. By using two types of records that are separated in time, the construction data and verification data of candidate alternative states both originate from the actual generation process of the current inference task, while avoiding the use of the same set of query states to simultaneously complete the formation of residual states and the verification of alternative effects.

[0044] When the cache usage of the active key-value cache reaches the cache budget, the cache management program determines candidate historical cache units from the current historical cache units. Only historical cache units that have formed construction query states and corresponding cache responses used for constructing the accepting state and residual key-value states, and that have formed verification query states that did not participate in the above construction process, are included in the candidate scope.

[0045] Furthermore, for multiple historical cache units that simultaneously meet the candidate conditions, the processing order is determined based on the formation status of the verification query status and the cache occupancy of the historical cache units. Historical cache units that have formed verification query statuses that can be used for joint alternative verification are processed first; when the verification query status formation statuses are the same, historical cache units with larger cache occupancy are processed first, so as to prioritize the release of historical cache units with larger potential active cache release space.

[0046] When the active key-value cache reaches the cache budget and there are no historical cache units that meet the candidate conditions, the joint replacement process described in this embodiment is not executed, and the inference system continues to process the current active key-value cache according to the existing cache resource management method.

[0047] After determining the target historical cache unit according to the processing order, the construction query status and corresponding cache response associated with the target historical cache unit are obtained, and the calculation benchmark of the target historical cache unit is established.

[0048] Specifically, when performing autoregressive decoding in a large language model, the current Transformer layer constructs a query state based on the hidden state at the current generation position. When the target historical cache unit participates in the attention calculation corresponding to the constructed query state, the attention normalization contribution and value-weighted response of the target historical cache unit are obtained, and the attention normalization contribution and value-weighted response are used together as the cache response generated by the target historical cache unit for the constructed query state.

[0049] Among them, the attention normalization contribution is used to characterize the degree to which the key state in the target history cache unit participates in the attention normalization calculation of the current constructed query state; the value-weighted response is used to characterize the state contribution of the value state in the target history cache unit to the current attention output after being weighted by the corresponding attention weights.

[0050] Furthermore, the attention normalization contribution and value-weighted response are combined as the cache response, so that the computational effect of the target historical cache unit simultaneously includes its influence on the attention normalization relation and the attention output content. Therefore, the subsequent selection of successor states and the formation of residual key-value states are no longer based solely on the access intensity of historical key states, but rather on the cache response formed by the actual participation of historical cache units in the model attention computation as the processing benchmark.

[0051] In one implementation, the large language model employs block-based attention computation. When the attention computation kernel processes the key-value data block corresponding to the target history cache unit, it reads the locally normalized intermediate states and accumulated local values ​​already formed in the key-value data block, and forms a cache response based on these local normalized intermediate states and accumulated local values. This approach utilizes the intermediate states already generated during normal attention computation to complete cache response acquisition, eliminating the need to separately generate and save the complete lexical-level attention matrix, thereby controlling the additional memory usage and redundant computation introduced by compression processing.

[0052] S102, generate candidate acceptance key value states based on the post-attention representation formed after the target historical cache unit participates in the attention calculation, determine the acceptance state, and recode the position of the acceptance state according to the historical position distribution of the target historical cache unit.

[0053] Specifically, based on the construction query response record corresponding to the target historical cache unit, the generation steps for forming the corresponding construction query state are determined, and the post-attention representation formed after the corresponding generation steps call the target historical cache unit to complete the attention calculation is obtained.

[0054] The post-attention representation originates from the internal model state formed after the construction of the query state and the completion of attention calculations by the active key-value cache, including the target history cache unit. The value states in the target history cache unit have already participated in the formation of the current attention output according to the corresponding attention relationships. Therefore, the post-attention representation includes the attention calculation effect actually generated by the target history cache unit in the corresponding generation step.

[0055] Furthermore, this embodiment does not directly use the key and value states formed before the attention computation is performed in the current Transformer layer as candidate acceptance states. The key and value states of the current Transformer layer are formed by projecting the hidden states before entering the attention sublayer, and at the time of their formation, the target history cache unit has not yet been acted upon by the corresponding generation step through the current attention computation. By using the post-attention representation formed after the attention computation is completed as the data source for the candidate acceptance states, the formation basis of the candidate acceptance states can include the actual attention computation that has already occurred in the target history cache unit.

[0056] In one implementation, the post-attention representation is a state representation formed by projecting the attention weighted result of the current Transformer layer onto the output. In another implementation, the post-attention representation is a state representation formed by fusing the attention output and the attention sub-layer input with residuals. For the same target history cache unit, a consistent post-attention representation is used to obtain the position, so as to maintain a consistent data source for the candidate succession states formed by different generation steps.

[0057] Specifically, after obtaining the post-attention representation, the post-attention representation is mapped to the key state space and value state space corresponding to the target Transformer layer, respectively, to form candidate accepting key-value states.

[0058] In one implementation, when the post-attention representation has a data dimension that matches the input state corresponding to the existing key projection and value projection of the target Transformer layer, the post-attention representation is projected using the existing key projection matrix and value projection matrix of the target Transformer layer to form candidate accepting key states and candidate accepting value states. For models using grouped query attention, candidate accepting key states and candidate accepting value states are formed according to the state dimension of the key-value header group corresponding to the target historical cache unit.

[0059] Furthermore, when the state form of the post-attention representation is inconsistent with the input form of the existing key and value projections of the target Transformer layer, a state mapping corresponding to the hidden state dimension of the target Transformer layer can be performed on the post-attention representation first, and then the key and value state projections can be performed. The state mapping is only used to enable the post-attention representation to be transformed into the key-value state space of the target Transformer layer, without changing the original model parameters of the target large language model.

[0060] The resulting candidate key-value states do not correspond to newly generated text lexical units, nor are they the lexical output of the language model. Instead, they serve as virtual key-value states derived from the post-attention representation within the model, used to evaluate whether the historical computational role contained in the post-attention representation can be re-represented as key and value states that can participate in attention computation.

[0061] Furthermore, the above transformation is performed on the post-attention representations formed in the generation steps corresponding to different constructed query states, forming a set of candidate accepting key-value states. The candidate accepting key-value states are associated with the corresponding constructed query states, generation positions, and target historical cache units, so that the subsequent cache response reproduction results can be traced back to the data source of the corresponding post-attention representation.

[0062] Since the candidate accepting key-value state is formed in a subsequent generation step, while the target historical cache unit corresponds to a previous historical position, the candidate accepting key-value state and the original key-value state of the target historical cache unit have different positional encoding relationships. If the cache response generated by the target historical cache unit is directly reproduced using the candidate accepting key-value state, the cache response difference will be affected by both the difference in state content and the difference in positional encoding.

[0063] Therefore, before evaluating the reproducibility of the cache response of the candidate accepting key state, the candidate accepting key state is recoded based on the historical location distribution of the target historical cache unit.

[0064] Specifically, based on the attention contributions generated when the constructed query state acts on different historical positions within the target historical cache unit, historical positions with continuous historical position relationships and similar attention contribution trends are merged to form historical position regions; the source position anchor point is determined based on the cumulative attention contribution of each position within the corresponding historical position region.

[0065] The above method is used to form source location anchors, so that the location recoding corresponds to the historical location action distribution when the target historical cache unit actually generates a cache response, rather than manually setting the location representative value according to the page header, page footer or fixed position of the target historical cache unit.

[0066] In one implementation, the large language model employs rotational positional encoding. For a candidate accepting key state, the key state before applying rotational positional encoding is obtained; when the active cache stores candidate accepting key states that have already been rotated and positionally encoded, an inverse positional transformation is performed based on the corresponding generated position of the candidate accepting key state to eliminate the positional encoding effect of the original generated position, and then positional encoding is reapplied based on the corresponding source position anchor point.

[0067] After positional recoding, the state content represented by the candidate accepting key-value state originates from the post-attention representation formed after the target historical cache unit participates in attention calculation. When the candidate accepting key-value state participates in subsequent attention matching, it adopts the positional encoding relationship corresponding to the historical calculation effect of the target historical cache unit. By adopting the above processing, the source of the state content of the candidate accepting key-value state and the attention positional relationship can be controlled separately, so that the subsequent cache response comparison mainly reflects the degree to which the candidate accepting key-value state preserves the calculation effect of the target historical cache unit.

[0068] Specifically, the cache response generated by the candidate accepting key value state after position recoding is calculated using the construction query state corresponding to the target historical cache unit, and compared with the cache response generated by the target historical cache unit for the same construction query state.

[0069] For each candidate accepting key-value state, determine the change in cache response difference before and after the candidate accepting key-value state is added to the current accepting state group, and determine the degree of preservation of the historical computation effect of the corresponding candidate accepting key-value state based on the degree of reduction in cache response difference.

[0070] Furthermore, candidate accepting key-value states that maximize the reduction in cache response difference are added to the current accepting state group, and the remaining cache response difference is recalculated based on the updated current accepting state group. For other candidate accepting key-value states that have not yet been added to the current accepting state group, the corresponding reduction in cache response difference is re-determined based on the updated remaining cache response difference.

[0071] By adopting a process of updating the remaining cache response difference in successive updates, the candidate accepting key-value states added later will prioritize maintaining the historical calculation role that has not yet been reproduced in the current accepting state group, thereby reducing the active cache resource occupation caused by multiple candidate accepting key-value states repeatedly representing the same historical cache response.

[0072] In one implementation, when continuing to increase the number of candidate accepting key-value states can no longer further reduce the remaining cache response difference, the increase of candidate accepting key-value states is stopped.

[0073] Furthermore, the active cache occupancy of the current accepting state group is simultaneously determined during the process of increasing the candidate accepting key value state. When the active cache occupancy of the current accepting state group reaches the active cache occupancy of the target historical cache unit, the accepting state construction is stopped, and the target historical cache unit remains in an active state.

[0074] By using both cache response reproducibility and active cache resource benefits to constrain the formation of the receiving state, we can avoid creating too many virtual key-value states in order to reproduce the computational effect of the target historical cache unit, thus causing the receiving state itself to lose its cache compression value.

[0075] The final determined candidate accepting key-value states together form the accepting state. The accepting state is used to represent the historical computation that has been re-formed and maintained by the post-attention representation after the target historical cache unit has acted; the cache response difference that still exists between the target historical cache unit and the accepting state corresponds to the historical computation that has not yet been maintained by the accepting state.

[0076] S103, based on the difference in cache response generated by the target historical cache unit and the position re-encoded acceptance state for the same construction query state, generate residual key value state.

[0077] Specifically, after determining the acceptance state and completing the location recoding, the construction query state corresponding to the target historical cache unit and the baseline cache response corresponding to each construction query state are obtained. The acceptance cache response generated by the acceptance state is calculated using the same construction query state, and the cache response difference is determined based on the difference between the baseline cache response and the acceptance cache response.

[0078] The cache response difference includes the attention normalization contribution difference and the value-weighted response difference. The attention normalization contribution difference is used to characterize the attention normalization effect of the received state relative to the target historical cache unit that has not yet been maintained; the value-weighted response difference is used to characterize the value state output effect of the received state relative to the target historical cache unit that has not yet been maintained.

[0079] Furthermore, when comparing the remaining cache response difference corresponding to different constructed query states, the attention normalization contribution difference and the value-weighted response difference are scaled uniformly according to their respective allowable deviations. The remaining cache response level of the corresponding constructed query state is determined based on the two types of response differences after scale uniformity, so that the cache response difference corresponding to different constructed query states can be compared under the unified response retention requirement.

[0080] Furthermore, the cached response difference is not directly added as an additional value to the attention output of the Transformer layer, but is used to generate residual key-value states that can participate in subsequent query processing according to the model's existing attention calculation method. The residual key-value states include at least one residual key state and a corresponding residual value state. The residual key state is used to establish an attention matching relationship between the subsequent query state and the unreceived computational action, while the residual value state is used to supplement the output of value states that have not yet been maintained under the corresponding matching relationship.

[0081] Using the above method, both the receiving state and the residual key-value state can maintain the data form of key-value states and participate together in the original attention calculation process of the model, without the need to add output compensation processing independent of the model inference path after the attention calculation is completed.

[0082] Specifically, the construct query state with the largest remaining cache response difference is determined based on the cache response difference corresponding to each construct query state, and a residual key-value slot is established. The residual key-value slot is associated with the Transformer layer where the target historical cache unit is located and the corresponding key-value header. It does not correspond to newly generated text lexical units of the language model, nor does it participate in the lexical output process.

[0083] In one implementation, based on the attention relationships formed when the constructed query state acts on each historical position within the target historical cache unit, the historical key states in the target historical cache unit are weighted to form an initial state of the residual key states. This ensures that the initial residual key states correspond to query access directions that are not yet fully preserved, reducing the additional computational load caused by re-searching for effective attention matching directions during subsequent adjustments.

[0084] Furthermore, based on the difference between the value-weighted response generated by the target historical cache unit for the constructed query state and the value-weighted response generated by the accepting state for the same constructed query state, an initial state of residual value state is formed, so that the residual value state preferentially represents the value state output content that the accepting state has not yet maintained.

[0085] When the target history cache unit covers multiple historical locations, the source location anchor is determined for the residual key state based on the attention contribution distribution corresponding to the uncomputed computational effects at different historical locations. Position encoding is then applied to the residual key state based on the source location anchor. By preserving the historical positional relationships corresponding to the uncomputed computational effects, the residual key state can be queried in subsequent attention calculations according to its positional relationship with the corresponding historical context.

[0086] Furthermore, after establishing the residual key-value slots, the original model parameters of the large language model, the target historical cache unit, and the already determined acceptance state are kept unchanged, and only the residual key state and residual value state are locally adjusted.

[0087] Specifically, the baseline cache response corresponding to the target historical cache unit is obtained by constructing the query state, and the joint cache response formed by the acceptance state and the current residual key value slot participating in the attention calculation. The residual state is adjusted based on the difference in attention normalization contribution and the difference in value-weighted response between the two.

[0088] In one implementation, the local attention calculation process performs error backpropagation based on the cached response difference, and only updates the residual key state and the residual value state. For the attention normalization contribution difference, the matching relationship between the constructed query state and the residual key state, as well as the degree to which the residual key value slot participates in the attention normalization calculation, are changed by adjusting the residual key state; for the value-weighted response difference, the state content provided by the residual key value slot to the attention output is changed by adjusting the residual value state.

[0089] The existing model parameters of the large language model remain frozen during the above adjustment process, so that the generation of residual key-value states is manifested as an adjustment of the runtime cache state formed for the current inference task, without changing the existing parameters of the large language model or the model structure used for subsequent normal generation.

[0090] Furthermore, after each round of residual key-value state adjustment, the joint cache response formed by the accepting state and the residual key-value state for all constructed query states is recalculated, and the remaining cache response difference between the joint cache response and the baseline cache response of the target historical cache unit is re-determined.

[0091] When the remaining cache response difference still exceeds the allowable deviation, the construct query state with the largest remaining cache response difference is re-determined from the constructed query state, and a new residual key-value slot is established based on the updated remaining cache response. The newly added residual key-value slot is used to supplement the calculations of existing inherited states and existing residual key-value states that have not yet been maintained.

[0092] By adopting the method of successively forming residual key-value slots based on the difference of the remaining cache response after the update, the subsequent residual key-value slots can supplement the calculation function of the previous residual key-value slots that have not yet been maintained, thereby reducing the resource consumption caused by multiple residual key-value slots repeatedly representing the same cache response.

[0093] In one implementation, the allowable bias is determined based on the inference accuracy configuration of the target large language model. Specifically, allowable biases corresponding to attention normalization contribution and value-weighted response can be set according to the data representation accuracy used in the inference process, the quantization configuration, and the attention output error allowed by the target deployment task. Only when the difference between the two types of cached responses is within the corresponding allowable range is the current joint cached response determined to meet the residual representation requirements.

[0094] Furthermore, during the formation of residual key-value states, the combined cache occupancy corresponding to the accepting state and all residual key-value states is determined simultaneously. When the combined cache occupancy reaches the cache occupancy of the target historical cache unit, the addition of residual key-value slots is stopped.

[0095] If adding residual key-value slots still fails to reduce the remaining cache response difference to the allowable deviation, or if adding new residual key-value slots no longer reduces the remaining cache response difference, the target historical cache unit remains active and the corresponding cache replacement is not performed.

[0096] By setting cache response retention requirements and cache resource benefit requirements, the target historical cache unit will only continue to enter the substitution verification process when the limited acceptance state and residual key value state can jointly maintain the corresponding historical calculation function and form actual cache resource compression space.

[0097] S104 uses the verification query state that did not participate in the generation of the acceptance state and residual key value state to verify the joint substitution relationship, and forms substitution conditions based on the verification results.

[0098] Specifically, after the acceptance status and residual key value status are formed, the verification query status that was pre-reserved in S101 and did not participate in the acceptance status filtering, position recoding, and residual key value status adjustment is obtained.

[0099] The verification query state originates from the actual autoregressive generation process of the large language model when the target historical cache unit is still in a fully active state, and is separated from the construction query state according to the generation sequence. Each verification query state is also associated with the baseline cache response actually generated by the target historical cache unit in the corresponding generation step.

[0100] By using pre-reserved verification query states for joint substitution verification, joint substitution verification does not need to wait for new generation steps to form verification data after the active key-value cache reaches its cache budget. At the same time, it avoids the construction query states used to generate candidate substitution states from also undertaking substitution verification functions.

[0101] Specifically, for each verification query state, the baseline cache response already saved in the target historical cache unit is read, and the corresponding candidate alternative attention calculation is performed together with the position-recoded successor state and the residual key-value state to form a joint cache response.

[0102] Furthermore, the joint cache response is characterized using the same attention computation object as the baseline cache response, including the corresponding attention normalization contribution and value-weighted response. This allows for comparison of the attention computation effects generated by combinations of complete target historical cache units and candidate alternative states under the same verification query state.

[0103] In one implementation, while the target historical cache unit has not yet been released, the target historical cache path and the candidate alternative path can be executed simultaneously using the verification query state. The candidate alternative path runs as a verification bypass, and its attention output is not written into the current normal generation path of the large language model; it is only used to obtain the joint cache response and perform alternative verification.

[0104] When using bypass verification, the normal generation of the large language model continues to use the complete target history cache unit, thereby avoiding the influence of unverified succession state and residual key value state on the subsequent generation state, and further changing the data conditions on which the verification process is based.

[0105] Specifically, the joint cache response is compared with the corresponding baseline cache response to determine the attention normalization contribution difference and the value-weighted response difference. Only when both types of cache response differences are within their respective allowable deviation ranges is the corresponding verification query status determined as passed.

[0106] For verification query states where the cache response difference exceeds the allowable deviation, the corresponding verification result is recorded, and the corresponding verification query state is used as constraint data for subsequently determining alternative condition boundaries.

[0107] Furthermore, based on the distribution relationship of the query states corresponding to each pass state in the query state space of the target Transformer layer, at least one verification query anchor point and the coverage area corresponding to the verification query anchor point are formed.

[0108] In one implementation, a verification query anchor is selected based on the state distance between passing states. Passing states that can be covered by existing verification query anchors are associated with the corresponding verification query anchors; passing states that cannot be covered by existing verification query anchors form new verification query anchors.

[0109] For each verification query anchor point, the initial coverage area is determined based on the pass status associated with the verification query anchor point, and the initial coverage area is limited based on the adjacent verification failure query status, so that the obtained coverage area does not extend to the query status area that has been verified as not meeting the joint substitution requirements.

[0110] Furthermore, the state distance can be determined using the normalized vector distance of the query states, the directional difference, or the distance of the query states in a preset low-dimensional projection space. The state distance calculation uses query states corresponding to the same Transformer layer and the same attention head to maintain a consistent state space for the data participating in the distance comparison.

[0111] If the online query status is within the coverage area of ​​any verification query anchor point, it is considered that the alternative condition is met; if the online query status is not within any coverage area, it is determined that the online query status does not meet the alternative condition.

[0112] By simultaneously utilizing both validated and unvalidated query states to determine coverage, alternative conditions correspond to query state regions validated by actual cached responses, rather than directly inferring subsequent applicability based on the constructed results of accepting states and residual key-value states.

[0113] S105: Transfer the target historical cache unit that meets the joint substitution requirements to the backup storage and release the active key-value cache. Perform substitution calculation, historical cache recovery and substitution relationship update according to the relationship between online query status and substitution conditions.

[0114] Specifically, after the alternative conditions are formed, the active cache resources jointly occupied by the accepting state and the residual key-value state are determined and compared with the active cache resources occupied by the target historical cache unit.

[0115] When the combined cache occupancy of the accepting state and the residual key-value state is less than the cache occupancy of the target historical cache unit, it is determined that the current combined replacement has active cache resource benefits; when the combined cache occupancy is not less than the cache occupancy of the target historical cache unit, the target historical cache unit is kept in an active state and no corresponding cache release is performed.

[0116] Furthermore, after meeting the requirements for the benefits of active cache resources, the complete key and value states corresponding to the target historical cache unit are transferred to the backup storage, and the logical cache identifier, historical location range, Transformer layer, key-value header, and backup storage address corresponding to the target historical cache unit are saved.

[0117] The backup storage is used to save the complete historical key-value states that have exited the active key-value cache but still require recovery. In one embodiment, the backup storage is located in host memory, low-priority device storage space, or other storage space that has data transfer relationships with the inference processor.

[0118] Furthermore, after the transfer is completed, a substitution association record is established between the target historical cache unit and the acceptance status, residual key value status, and substitution conditions.

[0119] In the implementation using paginated key-value caching, the current occupancy status of the physical cache block where the target historical cache unit is located is further obtained. When there are no other key-value states that still need to be kept active in the corresponding physical cache block, the active mapping between the logical cache identifier of the target historical cache unit and the corresponding physical cache block is released, and the physical cache block is released.

[0120] If the corresponding physical cache block still contains other key-value states that need to be kept active, retain the physical cache block until the corresponding physical cache block meets the overall release condition.

[0121] By employing the above processing, the logical replacement of the target historical cache unit can further form physical cache resources that can be reallocated and used, rather than simply marking the target historical cache unit as no longer participating in computation. The freed-up active cache space can continue to be used to store new key-value states generated subsequently by autoregression.

[0122] Furthermore, when the large language model continues to perform autoregressive generation, it identifies newly formed query states that have not yet participated in the verification of substitution relationships as online query states, and determines whether the online query states meet the substitution conditions based on the state relationship between the online query states and the verification query anchors and corresponding coverage areas.

[0123] When the online query status meets the substitution conditions, the complete target history cache unit in the backup storage is not read. Instead, the accepting status and residual key-value status are called to participate in the current attention calculation, so that the accepting status and residual key-value status jointly provide the historical calculation function corresponding to the original target history cache unit.

[0124] When the online query status does not meet the alternative conditions, the complete key status and value status are obtained according to the backup storage address corresponding to the target historical cache unit. The complete key status and value status are written into the currently available active cache space, and the mapping relationship between the logical cache identifier of the target historical cache unit and the current active cache address is restored.

[0125] The restored target historical cache unit re-participates in the attention calculation corresponding to the current online query state according to the saved historical location range, Transformer layer, and key-value header relationship.

[0126] Furthermore, after the target historical cache unit is restored, the baseline cache response of the target historical cache unit is obtained using the current online query state, and the joint cache response generated by the successor state and the residual key-value state for the same online query state is calculated.

[0127] When the difference in attention normalization contribution and the difference in value-weighted response between the baseline cache response and the joint cache response are both within their respective allowable deviation ranges, the current online query status is added to the pass status set, and the coverage of the existing verification query anchor is adjusted according to the updated pass status, or a new verification query anchor is formed.

[0128] If the current online query status verification fails, the corresponding alternative conditions are not expanded, and the target historical cache unit is kept involved in the attention calculation of the current generation stage.

[0129] Furthermore, when the verification results of subsequent online query status cause the alternative conditions to cover the query status range of the current generation stage again, and the acceptance status and residual key-value status still meet the requirements of active cache resource benefits, the target historical cache unit can be transferred to the backup storage again and the corresponding active key-value cache can be released.

[0130] Through the above online verification and recovery process, the replacement state of the target historical cache unit can change according to subsequent actual query needs, rather than forming an irreversible permanent deletion state after the first compression.

[0131] Furthermore, in the long-term autoregressive generation process, after the original receiving state participates in subsequent attention calculations, the corresponding generation steps form a new post-attention representation. A second candidate receiving key value state is then formed based on this new post-attention representation.

[0132] For the second candidate accepting key value state, the position recoding is performed on the second candidate accepting key value state according to the historical position distribution corresponding to the original target historical cache unit. The degree of reproduction of the calculation effect of the second candidate accepting key value state on the original target historical cache unit is re-evaluated by using the construction query state and cache response saved in the original target historical cache unit, so as to determine the second accepting state.

[0133] Based on the difference in cache response generated by the original target historical cache unit and the second accepting state for the same constructed query state, a direct residual key-value state is formed, and the joint substitution verification is re-executed using the verification query state corresponding to the original target historical cache unit.

[0134] When the second acceptance state and the direct residual key-value state pass the joint substitution verification, a direct substitution association is established between the original target historical cache unit and the second acceptance state and the direct residual key-value state, and the intermediate substitution association formed through the original acceptance state is cancelled.

[0135] Furthermore, before canceling intermediate substitution associations, the active cache usage and substitution association record usage of the direct substitution state and the original multi-level substitution state are compared. Intermediate substitution relationship merging is only performed if the direct substitution relationship can maintain the computational function of the original target historical cache unit and does not increase the corresponding cache management resource usage.

[0136] By adopting the above processing, we can avoid the formation of multi-level succession relationships and the continuous accumulation of intermediate succession states and alternative associated records as the generation process continues, so that the context compression structure can maintain a controlled cache resource scale during the long-term generation process.

[0137] Example 2 This embodiment provides a context-adaptive compression system for compressing and restoring continuously increasing historical key-value states during the autoregressive generation process of a large language model.

[0138] The generative context adaptive compression system includes an inference processor, an active key-value cache, a backup memory, and a program memory. The active key-value cache, backup memory, and program memory each establish data access relationships with the inference processor. The program memory stores computer programs used to perform large language model inference and generative context adaptive compression processing. The active key-value cache saves the key and value states currently involved in the large language model attention calculation. The backup memory saves the complete historical key-value states that have exited the active key-value cache but still require retrieval.

[0139] Specifically, the inference processor reads the computer program from the program memory, performs pre-filling processing on the input context, and forms key states and value states corresponding to historical lexical units according to the Transformer hierarchical structure of the large language model. The formed key states and value states are written into the active key-value cache and continuously participate in the attention calculation corresponding to the query state during subsequent autoregressive generation. As new generated lexical units are continuously formed, the corresponding key states and value states continue to be written into the active key-value cache.

[0140] Furthermore, the inference processor forms historical cache units for the historical key-value states in the active key-value cache according to the data storage granularity that can be independently read, migrated, or released, and establishes a correspondence between historical cache units and the Transformer layer, key-value header, historical location range, and active cache address. Therefore, after the historical computation role is replaced, cache migration and resource release can be performed on the corresponding historical cache units without having to copy or reorganize the entire historical key-value cache.

[0141] In one implementation, the active key-value cache employs a paging cache management structure, where historical cache units correspond to key-value data blocks that can be accessed via logical cache identifiers. A mapping is maintained between logical cache identifiers and actual physical cache blocks, enabling the inference processor to unmap the original physical cache after a historical cache unit exits its active state and reuse the released physical cache space for storing subsequently generated key-value states.

[0142] Specifically, when the inference processor participates in the attention calculation corresponding to the constructed query state in the target historical cache unit in the active key-value cache, it obtains the cache response formed by the target historical cache unit. The cache response includes the attention normalization contribution and value-weighted response formed by the target historical cache unit for the constructed query state, and establishes an association with the corresponding constructed query state.

[0143] The formation of the cached response utilizes the intermediate computational state already generated during the normal attention computation process performed by the inference processor, without requiring the additional generation of a complete word-level attention matrix. Therefore, the computational effects already generated by historical cached units can be obtained while maintaining the original large language model's inference path, and the redundant computations and intermediate data storage required for cache compression can be reduced.

[0144] Furthermore, after the target history cache unit participates in the attention calculation corresponding to the query state, the inference processor obtains the post-attention representation formed in the corresponding generation step. The post-attention representation is formed after the target history cache unit has completed its state contribution to the current attention calculation, and therefore includes the computational role already performed by the target history cache unit in the current generation step.

[0145] The inference processor maps the post-attention representation to the key state space and value state space corresponding to the target Transformer layer to form candidate accepting key-value states.

[0146] In one implementation, when the post-attention representation has a data dimension that matches the input state corresponding to the existing key projection and value projection of the target Transformer layer, candidate accepting key states and candidate accepting value states are formed using the key projection relationship and value projection relationship corresponding to the target Transformer layer, respectively. The candidate accepting key-value states do not correspond to new text terms, but are stored in the active key-value cache as virtual key-value states used to re-represent the historical computation.

[0147] Furthermore, the inference processor determines the source location anchor point based on the attention contribution of different historical positions within the target historical cache unit to the formation of the query state, and adjusts the position encoding relationship of the candidate accepting key state according to the source location anchor point, so that the candidate accepting key value state can participate in subsequent attention calculation according to the position relationship corresponding to the historical calculation role of the target historical cache unit.

[0148] In the implementation using rotational positional encoding, the positional encoding effect corresponding to the original generation position is removed from the candidate acceptor key state, and then positional encoding is reapplied according to the source position anchor point. By processing the source of the candidate acceptor key value state's state content and its historical positional relationship separately, the interference caused by the difference between the original generation position and the historical position on the cached response reproduction result can be reduced.

[0149] Specifically, the inference processor uses the construction query state corresponding to the target history cache unit to calculate the cache response formed by the candidate accepting key value state for the same construction query state, and compares the obtained cache response with the cache response pre-saved by the target history cache unit.

[0150] When multiple candidate accept key-value states exist, the inference processor determines the accept state sequentially based on the degree of reduction in cache response difference after adding the candidate accept key-value states to the current accept state. After determining each new accept state, the cache response that has not yet been maintained is recalculated, and other candidate accept key-value states are evaluated again for the updated remaining cache response.

[0151] Through the above processing, the subsequent acceptance states are given priority to maintain the historical calculation role that has not yet been reproduced by the previous acceptance states, thereby reducing the active cache occupation caused by multiple acceptance states repeatedly representing the same historical cache response.

[0152] Furthermore, for cached responses whose inherited states have not yet been maintained, the inference processor establishes residual key-value states in the active key-value cache. The residual key-value states include residual key states and residual value states. The residual key states are used to establish an attention matching relationship between the query state and the unaccomplished historical computation actions, while the residual value states are used to supplement the output actions of value states whose inherited states have not yet been maintained under the corresponding attention relationship.

[0153] Specifically, the inference processor maintains the original model parameters of the large language model and the already determined acceptance state unchanged, and adjusts the residual key state and residual value state according to the difference in the remaining cache response corresponding to the constructed query state. During the adjustment process, the acceptance state and the residual key value state jointly participate in the local attention calculation and form a joint cache response; the inference processor updates the residual key value state according to the difference between the joint cache response and the original cache response of the target historical cache unit.

[0154] Furthermore, when the existing residual key-value state is still insufficient to meet the allowable deviation for the combined cache response, a new residual key-value state is formed based on the updated remaining cache response. When the combined cache occupancy of the inheriting state and the residual key-value state reaches the cache occupancy of the target historical cache unit, the formation of new residual key-value states stops, and the target historical cache unit remains active.

[0155] This ensures that the system continues to perform cache replacement only when the target historical cache unit can maintain its computational function in the acceptance state and residual key-value state, and when it can generate actual active cache resource benefits.

[0156] Specifically, after the acceptance state and residual key-value state are formed, the inference processor obtains the verification query state that did not participate in the above state formation process, and uses the saved baseline cache response corresponding to the verification query state to verify the joint cache response formed by the acceptance state and residual key-value state.

[0157] When both the attention normalization contribution difference and the value-weighted response difference between the joint cache response and the baseline cache response are within their respective allowable deviation ranges, the corresponding verification query state is determined as a verification passed state. The inference processor forms verification query anchor points and corresponding coverage areas based on the distribution relationship of multiple verification passed states in the query state space, and restricts the coverage areas based on verification failed states, thereby forming alternative conditions for determining whether subsequent online query states can continue to use the successor state and residual key-value state.

[0158] Furthermore, before the joint substitution verification is completed, the target history cache unit remains in the active key-value cache. The successor state and the residual key-value state participate in the verification calculation as candidate substitution states, and the corresponding verification calculation results are not written into the normal generation path of the large language model, thereby avoiding unverified candidate substitution states from changing the subsequent generation state of the model.

[0159] After successful verification and the formation of alternative conditions, the inference processor compares the combined cache occupancy of the inherited state and the residual key-value state with the cache occupancy of the target history cache unit. If the combined cache occupancy is less than the cache occupancy of the target history cache unit, the complete key-value state corresponding to the target history cache unit is transferred to the backup memory.

[0160] During the transfer process, the inference processor saves the logical cache identifier, historical location range, Transformer layer, key value header, and backup storage location corresponding to the target historical cache unit, so that the complete historical key value state can still be restored according to the corresponding association after exiting the active key value cache.

[0161] Furthermore, after the target historical cache unit completes its backup transfer, the inference processor determines the occupancy status of the physical cache block corresponding to the target historical cache unit. If there are no other key-value states in the physical cache block where the target historical cache unit is located that need to remain active, the active mapping between the corresponding logical cache identifier and the physical cache block is released, and the corresponding physical cache resources are released.

[0162] By transferring the complete historical key-value states from the active key-value cache to the backup storage and reusing the freed physical cache space for subsequent state generation, the aforementioned computational function can be replaced by the actual release of active storage resources of the inference device, rather than simply reducing the number of historical states logically involved in attention computation.

[0163] Specifically, in the subsequent autoregressive generation process, the inference processor obtains the new online query status and determines whether the substitution conditions are met based on the relationship between the online query status and the verification query anchor and coverage.

[0164] When the online query status meets the substitution conditions, the inference processor directly reads the acceptance status and residual key-value status from the active key-value cache, and the acceptance status and residual key-value status jointly participate in the current attention calculation, without reading the complete target history cache unit in the backup memory.

[0165] When the online query status does not meet the alternative conditions, the inference processor obtains the complete key status and value status according to the backup storage location corresponding to the target historical cache unit, writes the complete key-value status into the currently available active cache space, and restores the mapping relationship between the logical cache identifier of the target historical cache unit and the current active cache address, so that the target historical cache unit can re-participate in the current attention calculation according to the original historical position relationship.

[0166] Furthermore, after the target historical cache unit is restored, the inference processor uses the current online query state to re-compare the complete target historical cache unit with the cache responses generated by the receiving state and the residual key-value state. When both cache responses still meet their respective allowable deviation requirements, the online query state is included in the verification passed state, and the corresponding verification query anchor point and coverage are updated; when the allowable deviation requirements are not met, the active state of the target historical cache unit is maintained.

[0167] Therefore, the target history cache unit in the system is not permanently deleted after compression, but can be migrated between the active key-value cache and the backup storage according to the subsequent actual query status, so that the active cache status changes with the actual attention calculation needs during the generation process.

[0168] Furthermore, as autoregressive generation continues, when the receiving state participates in subsequent attention calculations to form a new post-attention representation, the inference processor forms a new candidate receiving key-value state based on the new post-attention representation, and re-executes position recoding, cache response reproduction, residual key-value state formation, and joint substitution verification for the original target historical cache unit.

[0169] After the new accepting state and the corresponding direct residual key-value state are jointly verified for the original target history cache unit, the inference processor establishes a direct substitution relationship between the original target history cache unit and the new accepting state and the direct residual key-value state, and cancels the intermediate substitution relationship formed through the original accepting state.

[0170] Furthermore, before canceling the intermediate substitution relationship, the inference processor compares the active cache usage and substitution record usage of the direct substitution relationship with the original multi-level substitution relationship. If the direct substitution relationship can maintain the computational function of the original target historical cache unit without increasing cache management resource usage, then the intermediate substitution relationship is canceled.

[0171] The above processing can suppress the continuous increase of the successor state and substitution relationship hierarchy during the long-term autoregressive generation process, so that the system can maintain controlled activity key value cache usage and substitution relationship maintenance overhead.

[0172] Working principle and effects This invention leverages the characteristic that historical key-value states continuously participate in subsequent attention calculations during the autoregressive generation process of large language models, and re-represents the computational effects already generated by historical key-value states. When a target historical cache unit participates in attention calculation, it obtains the corresponding cache response and the post-attention representation formed after the attention calculation, and uses the post-attention representation to form candidate accepting key-value states. The candidate accepting key-value states are re-encoded according to the historical positional relationship corresponding to the target historical cache unit, and the accepting state is determined based on the degree of reproduction of the historical cache response by the candidate accepting key-value states. This allows the historical computational effects already reflected in the post-attention representation to participate in subsequent attention calculations again in the form of key-value states.

[0173] For computational actions where the receiving state has not yet been maintained, a residual key-value state is formed based on the difference in cache responses generated by the target historical cache unit and the receiving state for the same constructed query state. This allows the receiving state and the residual key-value state to jointly maintain the attention normalization effect and value state output effect of the target historical cache unit. Furthermore, the joint substitution relationship is verified using the verification query state that did not participate in the formation of the receiving state and the residual key-value state, and substitution conditions are formed based on the verification results, thereby limiting the scope of the receiving state and the residual key-value state applicable to subsequent query states.

[0174] When a joint substitution is established and the combined cache size of the accepting state and the residual key-value state is less than the cache size of the target historical cache unit, the complete target historical cache unit is transferred to the backup storage and the corresponding active key-value cache resources are released. If the online query state meets the substitution conditions, the accepting state and the residual key-value state are used in attention calculation; if the substitution conditions are not met, the complete target historical cache unit is restored. Through this process, the continuous occupation of the active cache by the complete historical key-value state can be reduced while maintaining the historical role required for subsequent attention calculations. Furthermore, the cache compression state can be adjusted according to actual query needs, thereby reducing the pressure on active key-value cache resources during the continuous generation of long contexts.

[0175] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0176] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0177] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0178] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0179] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A generative context adaptive compression method, characterized in that, include: Obtain the cached responses generated by the target historical cache unit for multiple constructed query states during the autoregressive generation process of a large language model; Obtain the post-attention representation formed after the target historical cache unit completes the attention calculation in the corresponding generation step, map the post-attention representation to the candidate acceptance key value state, re-encode the position of the candidate acceptance key state according to the historical position distribution of the target historical cache unit, and determine the acceptance state according to the degree of cache response reproduction. Generate residual key-value states based on the difference between the target historical cache unit and the cache response of the same constructed query state for the receiving state; The verification query state verification is jointly replaced by the verification query state verification that did not participate in the generation of the acceptance state and residual key value state, and the replacement conditions are formed based on the verification results; When the combined cache size of the accepting state and the residual key-value state is less than the cache size of the target historical cache unit, the target historical cache unit is transferred to the backup storage and its occupied active key-value cache resources are released. For online query states, if the substitution conditions are met, the accepting state and the residual key-value state are invoked to participate in attention calculation; otherwise, the target historical cache unit is restored.

2. The context-adaptive compression method according to claim 1, characterized in that, The determination of the acceptance status includes: Obtain the post-attention representation formed by the generation steps corresponding to each constructed query state; The post-attention representation is mapped to the key state space and value state space corresponding to the target Transformer layer, respectively, to form candidate accepting key-value states, and the candidate accepting key states are re-encoded in position. Determine the reduction in cache response difference relative to the target historical cache unit before and after each candidate accepting key value state is added to the current accepting state group; The candidate accepting key-value state with the largest reduction in cache response difference is added to the current accepting state group, and selection continues based on the updated cache response difference until adding candidate accepting key-value states no longer reduces the cache response difference. The resulting current accepting state group is then determined as the accepting state.

3. The context-adaptive compression method according to claim 2, characterized in that, The location recoding includes: Based on the attention contribution of the constructed query state to different historical positions in the target historical cache unit, at least one source position anchor point is determined. Obtain the state of the candidate accepting key before position encoding, or perform an inverse position transformation on the candidate accepting key state that has already been position encoded; The resulting candidate acceptor key states are re-encoded based on the source location anchor point.

4. The context-adaptive compression method according to claim 1, characterized in that, The state for generating residual key values ​​includes: Establish at least one residual key-value slot, and use the constructed query state to calculate the joint cache response of the acceptance state and the residual key-value slot; Determine the remaining cache response difference of the joint cache response relative to the target historical cache unit, construct the query state with the largest remaining cache response difference and the corresponding cache response difference as the fitting sample, and adjust the residual key value slot; When the remaining cache response difference exceeds the allowable deviation, and the combined cache occupancy of the receiving state and the residual key value slot is less than the cache occupancy of the target historical cache unit, the residual key value slot is increased and the adjustment is repeated.

5. The generation context adaptive compression method according to claim 1, characterized in that, The conditions for forming alternatives include: The baseline cache response of the target historical cache unit and the joint cache response generated by the acceptance state and the residual key value state are calculated using the verification query state respectively. The verification query status where the difference in attention normalization contribution and the difference in value-weighted response between the joint cache response and the baseline cache response both do not exceed their respective allowable deviations is determined as a pass status; At least one verification query anchor point and its corresponding coverage area are determined based on the state distance between the states, and the online query state being within any of the coverage areas is taken as satisfying the alternative condition.

6. The context-adaptive compression method according to claim 5, characterized in that: For online query states that do not meet the alternative conditions, the target historical cache unit is restored from the backup storage; The cache response for the online query status is calculated using the restored target history cache unit, the acceptance status, and the residual key value status, respectively. When the difference in attention normalization contribution and the difference in value-weighted response between the two cached responses do not exceed their respective allowable deviations, the online query status is added to the pass status, and the coverage of the corresponding verification query anchor point is updated.

7. The generation context adaptive compression method according to claim 1, characterized in that, The release of the activity key-value cache resources includes: Determine the combined cache occupancy of the receiving state and the residual key-value state, as well as the occupancy status of the physical cache block where the target historical cache unit is located; When the combined cache usage is less than the cache usage of the target historical cache unit, and the physical cache block does not have any other key-value states that need to remain active, the target historical cache unit is transferred to the backup storage, and the physical cache block is released.

8. The generation context adaptive compression method according to claim 1, characterized in that: When the acceptance state participates in the attention calculation in the subsequent autoregressive generation process, the second post-attention representation formed after the attention calculation is completed in the corresponding generation step is obtained, and the second post-attention representation is mapped to the second candidate acceptance key value state. The position of the second candidate accepting key is recoded according to the historical position distribution of the original target historical cache unit, and the second accepting state is determined according to the construction query state and cache response corresponding to the original target historical cache unit. A direct residual key-value state is generated based on the difference in cache response between the original target historical cache unit and the second accepting state for the same constructed query state. When the second acceptance state and the direct residual key-value state pass the joint substitution verification for the original target historical cache unit, a direct substitution relationship is established and the intermediate substitution relationship formed through the original acceptance state is cancelled.

9. The generation context adaptive compression method according to claim 1, characterized in that, The determination of the target history cache unit includes: Monitor the cache usage of the active key-value cache; When the cache usage reaches the cache budget, the historical cache units that have formed the construction query state and the corresponding cache response, and have formed the verification query state that did not participate in the generation of the acceptance state and the residual key value state, are identified as candidate historical cache units. The processing order is determined based on the formation status of the verification query status and the cache usage of each candidate historical cache unit, and the target historical cache unit is determined according to the processing order.

10. A generative context adaptive compression system, characterized in that, It includes an inference processor, an active key-value cache, a backup memory, and a program memory, wherein the active key-value cache, the backup memory, and the program memory are respectively connected to the inference processor; The active key-value cache is used to store the historical key-value states that participated in the attention calculation of the large language model; the backup memory is used to store the historical key-value states that exited the active key-value cache and retained the recovery requirements; and the program memory stores the computer program executed by the inference processor. The inference processor is configured to: acquire cache responses generated by the target history cache unit for multiple constructed query states; The process involves obtaining the post-attention representation formed after the target historical cache unit completes attention calculation in the corresponding generation step, mapping the post-attention representation to candidate acceptance key-value states, re-encoding the candidate acceptance key-value states according to the historical position distribution of the target historical cache unit, and determining the acceptance state based on the degree of cache response reproducibility; generating residual key-value states based on the difference between the cache response of the target historical cache unit and the acceptance state; and using the verification query state verification that did not participate in the generation of acceptance state and residual key-value states to jointly replace and form replacement conditions. The inference processor is further configured to: when the combined cache occupancy of the acceptance state and the residual key-value state is less than the cache occupancy of the target historical cache unit, transfer the verified target historical cache unit to the backup storage and release the corresponding active key-value cache resources; and, depending on whether the online query state meets the alternative condition, call the acceptance state and the residual key-value state to participate in attention calculation, or restore the target historical cache unit from the backup storage.

Images