Method and apparatus for selective re-computation of kv cache generation facing retrieval enhancement

Abstract

The application discloses a KV cache selective recalculation method and device for retrieval enhancement generation, the method obtains a multiplexing context KV cache corresponding to a user query, takes a query token of the user query as an attention query end, takes a context object corresponding to the multiplexing context KV cache as an evaluated object, calculates attention weights in multiple network layers of a Transform model and performs cross-layer fusion to obtain context object importance information, then determines a to-be-recomputed object set according to the importance information and a recalculation constraint, performs cache recalculation based on a current request context and updates the multiplexing context KV cache to obtain an updated cache, and finally performs decoding generation based on the updated cache and the user query. The method can concentrate the recalculation budget on key evidence positions related to the user query, and reduce long context reasoning overhead.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically, to a method and apparatus for selective recalculation of KV cache for retrieval enhancement generation. Background Technology

[0002] With the application of large language models in scenarios such as enterprise knowledge question answering, intelligent operation and maintenance, code analysis, and industry knowledge bases, retrieval enhancement generation technology is widely used to improve the factuality and domain adaptability of model answers. Existing prefix cache reuse technology requires different requests to have the same prompt prefix, which is difficult to adapt to the dynamic changes, changes in the arrangement order, and changes in the combination of document fragments in retrieval enhancement generation. Although position-independent key-value cache reuse can independently pre-compute key-value caches for document blocks and combine and reuse them when a request arrives, the independently pre-computed document block caches lack cross-document attention interaction, and direct combination can easily affect the generation quality. Summary of the Invention

[0003] The purpose of this invention is to provide a method and apparatus for selective recalculation of KV cache for retrieval enhancement generation.

[0004] In a first aspect, embodiments of the present invention provide a method for selective recalculation of a key-value cache for retrieval enhancement generation, comprising:

[0005] Retrieve the reused context key-value cache corresponding to the user query;

[0006] The query token corresponding to the user query is used as the attention query endpoint, and the context object corresponding to the reused context KV cache is used as the evaluation object. The attention weight from the query token to the context object is calculated in multiple network layers of the Transformer model, and cross-layer fusion is performed based on the attention weights of multiple network layers to obtain the importance information of the context object. The context object includes a context token, or includes a continuous token span obtained by merging the importance information of adjacent context tokens.

[0007] The set of objects to be recomputed is determined based on the context object importance information and recomputed constraints;

[0008] Based on the current request context, cache recalculation is performed on the set of objects to be recalculated, and the cache representation obtained from the recalculation is used to update the reuse context key-value cache to obtain the updated cache; the current request context includes multiple retrieval context fragments and the user query;

[0009] Decoding is performed based on the updated cache and the user query.

[0010] In one possible implementation, obtaining the reused context key-value cache corresponding to the user query includes:

[0011] Divide the knowledge base document into multiple document blocks;

[0012] For each document block, prefill is performed independently to obtain the document block KV cache representation corresponding to each document block. The document block KV cache representation includes key representations and value representations on multiple network layers.

[0013] In response to the user query, multiple target document blocks are retrieved, and the document block key-value cache representations corresponding to the multiple target document blocks are read.

[0014] The read document block KV cache representations are combined according to the order in which the multiple target document blocks are arranged in the current request context to obtain the reuse context KV cache.

[0015] In one possible implementation, the step of combining the read document block KV cache representations according to the order of the plurality of target document blocks in the current request context to obtain the reused context KV cache includes:

[0016] Read document block key-value cached representations that support position-independent multiplexing or relocatable multiplexing;

[0017] Based on the actual combined position of the multiple target document blocks in the context of this request, supplement the position information of the document block KV cache representation or perform position correction.

[0018] The reused context KV cache is obtained by combining the document block KV cache representation based on supplementary location information or execution location correction.

[0019] In one possible implementation, the step involves using the query token corresponding to the user query as the attention query endpoint, and the context object corresponding to the reused context key-value cache as the evaluation object. Attention weights from the query token to the context object are calculated across multiple network layers of the Transformer model. Cross-layer fusion is then performed based on these attention weights to obtain context object importance information, including:

[0020] Without performing a full prefill precomputation on the multiple retrieval context fragments, the query token accesses the context key cache in the reused context KV cache;

[0021] In multiple network layers of the Transformer model, the attention weights from the query token to the context object are calculated;

[0022] The context object importance score for each network layer is obtained based on the attention weights.

[0023] The importance scores of the context objects from multiple network layers are fused across layers to obtain the importance information of the context objects.

[0024] In one possible implementation, obtaining the context object importance score for each network layer based on the attention weights, and then fusing the context object importance scores across multiple network layers to obtain the context object importance information, includes:

[0025] For multiple query tokens in the same network layer, the attention weights about the same context object are aggregated by average, column sum, maximum value, Top-m average, or weighted average to obtain the context object importance score for each network layer.

[0026] When using the weighted average aggregation, at least one strong constraint query token is identified from the entity name, time, indicator name, relational term, and constraint condition in the user query, and the strong constraint query token is assigned a weight higher than that of the ordinary query token.

[0027] The context object importance score is obtained from all network layers or a representative network layer of the Transformer model, wherein the representative network layer includes at least two layers from shallow, medium and deep layers;

[0028] The importance scores of the context objects are fused across layers using mean fusion, weighted average fusion, maximum value fusion, or group fusion to obtain the importance information of the context objects.

[0029] In one possible implementation, determining the set of objects to be recomputed based on the context object importance information and recomputed constraints includes:

[0030] The recalculation constraint is determined, and the recalculation constraint includes a recalculation ratio or a fixed number of recalculations;

[0031] When the recalculation constraint includes a recalculation ratio, the total amount of context tokens in the current request context is proportionally converted according to the recalculation ratio, and the target recalculation quantity is determined according to a preset rounding rule.

[0032] When the recomputation constraint includes a fixed number of recomputations, the fixed number of recomputations is determined as the target number of recomputations;

[0033] The context objects are sorted according to their importance information.

[0034] Based on the target number of recalculations, select the corresponding number of context objects from the sorting results to obtain the set of objects to be recalculated.

[0035] In one possible implementation, determining the recomputation constraint includes:

[0036] Obtain the request information and / or system operating status, which includes at least one of the following: context length, number of retrieved document blocks, user query length, query complexity, search result confidence, search result concentration, graphics processor memory remaining, batch size, service load, request accuracy level, and historical quality feedback.

[0037] Based on the current request information and / or system operating status, determine the recalculation ratio or the fixed number of recalculations to obtain the recalculation constraint.

[0038] In one possible implementation, after determining the set of objects to be recalculated based on the context object importance information and recalculation constraints, and before performing cache recalculation on the set of objects to be recalculated based on the current request context, the method further includes:

[0039] Based on the distribution of the set of objects to be recalculated in the multiple retrieval context segments, the hit information of each retrieval context segment is determined;

[0040] Based on the hit information, the multiple search context fragments are rearranged, and the search context fragments whose hit information meets the preset conditions are arranged in a position close to the user query.

[0041] Based on the rearranged multiple retrieval context fragments and the user query, the rearranged current request context is formed;

[0042] When performing the cache recalculation, a joint recalculation is performed on the set of objects to be recalculated and the query token corresponding to the user query, so that the cache representation of the query token is recalculated based on the rearranged current request context.

[0043] In one possible implementation, the step of performing cache recalculation on the set of objects to be recalculated based on the current request context, and updating the reuse context KV cache using the recalculated cache representation to obtain the updated cache, includes:

[0044] When the set of objects to be recalculated includes tokens that are adjacent in position, the adjacent tokens are merged to form a continuous token span;

[0045] The continuous token span is modified based on at least one of the minimum span length, maximum span length, maximum quota per document block, and fragmentation constraint to obtain the modified set of objects to be recalculated.

[0046] Based on the number of the corrected set of objects to be recalculated in each retrieval context segment, the importance score determined by the importance information of the context objects, and / or the span length, the document block-level hit score of each retrieval context segment is determined;

[0047] The multiple search context fragments are rearranged according to the document block-level hit score, so that the search context fragments with higher document block-level hit scores are located closer to the user query, and the rearranged multiple search context fragments and the user query are used to form the rearranged current request context;

[0048] In each network layer of the Transformer model, based on the rearranged current request context, a complete cross-layer recalculation is performed on the corrected set of objects to be recalculated and the query token corresponding to the user query. This allows the context objects in the corrected set of objects to be recalculated to absorb cross-document information from the current combined context, obtaining the key and value representations corresponding to the context objects. The key and value representations of the query token are then recalculated based on the rearranged current request context.

[0049] Context objects that have not yet entered the revised set of objects to be recomputed continue to use the cached representation in the reused context KV cache;

[0050] The recalculated Key and Value representations of the context object, as well as the recalculated Key and Value representations of the query token, are written into the reused context KV cache at the positions corresponding to the rearranged current request context, thus obtaining the updated cache.

[0051] Secondly, embodiments of the present invention provide a KV cache selective recalculation apparatus for retrieval enhancement generation, comprising:

[0052] The retrieval module is used to retrieve the reused context key-value cache corresponding to the user query;

[0053] The calculation module is used to take the query token corresponding to the user query as the attention query endpoint, and the context object corresponding to the reused context KV cache as the evaluation object. It calculates the attention weights from the query token to the context object in multiple network layers of the Transformer model, and performs cross-layer fusion based on these attention weights to obtain context object importance information. The context object includes a context token, or a continuous token span obtained by merging the importance information of adjacent context tokens. It determines a set of objects to be recalculated based on the context object importance information and recalculation constraints. Based on the current request context, it performs cache recalculation on the set of objects to be recalculated, and updates the reused context KV cache using the recalculated cache representation to obtain the updated cache. The current request context includes multiple retrieval context fragments and the user query.

[0054] The calculation module is also used to rearrange multiple retrieval context fragments according to the distribution of the set of objects to be recalculated in multiple retrieval context fragments, and to include the query token corresponding to the user query in the recalculation scope when performing cache recalculation.

[0055] An execution module is used to perform decoding and generation based on the updated cache and the user query.

[0056] Compared to existing technologies, the beneficial effects of this invention include: This invention calculates the attention weight from the query token to the context object based on the user query, and determines the importance information of the context object through cross-layer fusion of multiple network layers. This enables more accurate identification of key evidence tokens or consecutive token spans related to the current user query, avoiding the recalculation budget being occupied by statically positioned, globally significant but query-irrelevant tokens. By performing cache recalculation only on the set of objects to be recalculated, and writing the recalculated cache representation to the response using the context KV cache, this invention can repair the cross-document attention interaction that is missing in independent document block caching while retaining most of the benefits of document block cache reuse. Therefore, this invention can reduce the prefill computation overhead and first token latency in long context retrieval enhancement generation scenarios, and improve the ability of generated answers to utilize key evidence under limited recalculation budgets. Attached Figure Description

[0057] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be considered as limiting the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 This is a flowchart illustrating the steps of the KV cache selective recalculation method for retrieval enhancement provided in an embodiment of the present invention.

[0059] Figure 2 This is a schematic diagram of cross-layer attention importance fusion and Top-K selection provided in an embodiment of the present invention;

[0060] Figure 3 This is a schematic diagram of the two-stage selective recalculation process provided in an embodiment of the present invention;

[0061] Figure 4 A schematic block diagram of the structure of the KV cache selective recalculation device for retrieval enhancement generation provided in the embodiments of the present invention;

[0062] Figure 5 A schematic block diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0064] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0065] In order to solve the technical problems mentioned in the background art Figure 1 This is a flowchart illustrating the selective recalculation method for KV cache generated for retrieval enhancement provided in this embodiment of the disclosure. The following is a detailed description of this selective recalculation method for KV cache generated for retrieval enhancement.

[0066] Step S201: Obtain the reused context KV cache corresponding to the user query;

[0067] Step S202: Using the query token corresponding to the user query as the attention query endpoint and the context object corresponding to the reused context KV cache as the evaluation object, the attention weight from the query token to the context object is calculated in multiple network layers of the Transformer model, and cross-layer fusion is performed based on the attention weights of multiple network layers to obtain the importance information of the context object; the context object includes a context token, or includes a continuous token span obtained by merging the importance information of adjacent context tokens;

[0068] Step S203: Determine the set of objects to be recalculated based on the context object importance information and recalculation constraints;

[0069] Step S204: Perform cache recalculation on the set of objects to be recalculated based on the current request context, and update the reuse context KV cache using the cache representation obtained from the recalculation to obtain the updated cache; the current request context includes multiple retrieval context fragments and the user query;

[0070] Step S205: Decoding and generation are performed based on the updated cache and the user query.

[0071] In this embodiment of the invention, for example, the server is applied to an enterprise knowledge base question-and-answer system. The enterprise knowledge base pre-stores product manuals, upgrade instructions, error code explanations, network configuration documents, historical work orders, and maintenance logs. During the offline phase or when a document block is first hit, the server segments the aforementioned knowledge base documents into multiple retrieval context fragments. Each retrieval context fragment is a document block of fixed length, or a document block segmented according to semantic boundaries such as paragraphs or sentence windows. The server independently performs a Transformer model prefill calculation on each document block to obtain the Key and Value representations of the document block in multiple network layers. The server then saves the Key and Value representations, along with the document block identifier, token range, model version, and cache expiration date, as a document block KV cache. This document block KV cache uses a position-independent or relocatable cache representation. During the request phase, the server supplements the position information according to the actual combination position in the search results, or performs position correction on the read cache representation, thereby enabling the same document block to be reused in different search orders.

[0072] After receiving a user query, "How should I handle connection timeouts after upgrading device A to V3.2?", the server retrieves multiple search context fragments, including an upgrade instruction document block, a connection timeout error code document block, a network port configuration document block, and a historical work order document block. The server reads the document block key-value cache corresponding to each of these search context fragments from the cache library and combines them according to the order of the search results in the prompts, resulting in a reused context key-value cache corresponding to the user query. Each context token in this reused context key-value cache already has a local key-value representation within its respective document block. The server does not perform a full prefill on the complete input sequence at this stage; instead, it uses this combined cache as the basis for subsequent query-driven evaluation and selective recalculation. Since each document block is pre-calculated independently, the aforementioned reused context key-value cache lacks cross-document attention interactions between the upgrade instructions, error code explanations, port configurations, and historical work orders. Therefore, the server continues with subsequent steps to repair the cache representation most relevant to the current user query within a limited budget.

[0073] The server uses the query tokens corresponding to user queries as attention endpoints, including query tokens such as "Model A", "Upgrade to V3.2", "Connection Timeout", and "How to Handle". The server uses the context objects corresponding to the reused context key-value cache as the evaluation objects, where the context objects include context tokens from each retrieval context fragment. In multiple network layers of the Transformer model, the server causes the aforementioned query tokens to access the combined context key cache layer by layer, and calculates the attention weight Attention_l(q,t) from each query token to each context token. At this stage, the server only calculates the attention distribution from query tokens to context tokens, and does not perform full self-attention calculations of s×s scale between complete context tokens, thus making the evaluation cost primarily related to the product of the number of query tokens and the context length. The server performs averaging, column summation, maximum value, Top-m averaging, or weighted averaging on the attention weights of multiple query tokens with respect to the same context token within the same network layer, obtaining the context object importance score α_l(t) for each network layer. In one specific implementation, the server calculates the importance score of the context token t in the l-th network layer according to α_l(t) = Average_{q∈Q_s}(Attention_l(q,t)), where Q_s represents the set of query tokens corresponding to the user query, and Attention_l(q,t) represents the attention weight from query token q to context token t in the l-th network layer. This importance score is used to characterize the degree to which the context token t is given overall attention by the current user query. Specifically, the server assigns higher weights to strongly constrained query tokens such as entity names, version numbers, and fault phenomena (e.g., "Model A", "V3.2", "Connection Timeout") than to ordinary query tokens, thus giving higher importance scores to context tokens corresponding to precise evidence locations.

[0074] The server further performs cross-layer fusion on the importance scores obtained from multiple network layers to obtain context object importance information. Attention weights in shallow network layers reflect more local lexical associations and short-range matching, allowing the server to identify context tokens related to literal meanings such as "connection timeout," "port," and "upgrade package." Attention weights in mid- and deep network layers reflect more entity relationships, fault cause chains, and processing steps, allowing the server to identify context tokens such as "remote management port closed by default after upgrade" in historical work orders, "open ports 443 and 8443" in network configuration documents, and "timeout triggered by TLS handshake failure" in error code documents. The server uses mean fusion, weighted fusion, maximum value fusion, or group fusion methods to fuse the importance scores from each network layer into a unified importance score α_bar(t). Figure 2 As shown, the server merges the importance scores of context objects obtained from different network layers, and then selects the Top-K context tokens or token spans for subsequent full cross-layer recalculation based on the merged importance scores. The server also merges consecutively occurring high-importance context tokens into consecutive token spans based on the importance information of adjacent context tokens. For example, it merges multiple adjacent tokens corresponding to "re-enable remote management port after upgrade" into one token span, and merges multiple adjacent tokens corresponding to "error code E504 indicates connection handshake timeout" into another token span, thus forming context objects that can be recalculated in units of spans.

[0075] The server determines the set of objects to be recalculated based on the importance information of context objects and recalculation constraints. Recalculation constraints include recalculation ratio, fixed recalculation quantity, maximum quota per document block, minimum span length, maximum span length, and fragmentation constraints. The server reads the context length of this request, the number of retrieved context fragments, the user query length, the current remaining GPU memory, batch size, service load, and historical quality feedback to determine the recalculation ratio for this request. For the aforementioned fault handling queries, the server identifies that the user query contains multiple strong constraints such as device model, software version, fault symptoms, and processing purpose, and therefore allocates a higher recalculation budget to this request. The server sorts the context objects according to the merged importance score and selects the top-ranked context tokens or consecutive token spans to form the set of objects to be recalculated. When forming the set, the server performs neighbor merging on adjacent high-scoring tokens, expands excessively short spans, truncates or splits spans exceeding the maximum span length, and selects results according to the maximum quota limit per document block to avoid a single document block consuming the entire recalculation budget, while reducing the gather and scatter overhead caused by discrete indexes.

[0076] The server performs cached recalculation on the set of objects to be recalculated based on the context of this request. This request context includes upgrade documentation blocks, error code explanation documentation blocks, network configuration documentation blocks, historical ticket documentation blocks, and user queries. The server fixes the set of objects to be recalculated and performs full cross-layer recalculation on the context tokens or consecutive token spans within the set in each network layer of the Transformer model, such as... Figure 3 As shown, the server identifies context objects worthy of recalculation through Stage I and performs a full cross-level recalculation on the selected context objects through Stage II. Together, these two stages achieve query-driven two-stage selective recalculation, enabling these selected context objects to recalculate their Key and Value representations according to the actual document composition relationships in the current request. Through this full cross-level recalculation, the "connection handshake timeout" token in the error code explanation document block can absorb the "default policy change after upgrade" information from the upgrade instructions document block, and the "port opening rule" token in the network configuration document block can absorb the "remote management port needs to be re-enabled after upgrade" information from the historical ticket document block, thereby repairing the cross-document connections missing in the original independent document block cache. The server writes the recalculated new Key and Value representations to the corresponding positions in the context KV cache used for replying to the set of objects to be recalculated. Context objects not included in the set of objects to be recalculated continue to use their original pre-calculated cache representations, thus obtaining the updated cache.

[0077] The server performs decoding and generation based on the updated cache and user query. During the decoding phase, the server uses the updated KV cache that has already been written back to the key evidence location, allowing the model to directly access the key evidence tokens repaired through cross-document information when generating the answer. Simultaneously, it continues to reuse a large number of pre-computed cached tokens that have not been recomputed. The server-generated answer revolves around the handling process of "connection timeout after upgrading device A to V3.2," and the output includes steps such as confirming the error code, checking the upgraded network policy, re-enabling the remote management port, opening the specified port, restarting related services, and verifying the connection status. During this process, the server does not perform a full prefill on all retrieval context fragments; instead, it concentrates the recomputation budget on context tokens or token spans directly related to the user query, reducing the first token latency in long-context RAG scenarios while maintaining the key information required for the answer.

[0078] In this embodiment of the invention, the step of obtaining the reused context KV cache corresponding to the user query can be implemented through the following example.

[0079] Divide the knowledge base document into multiple document blocks;

[0080] For each document block, prefill is performed independently to obtain the document block KV cache representation corresponding to each document block. The document block KV cache representation includes key representations and value representations on multiple network layers.

[0081] In response to the user query, multiple target document blocks are retrieved, and the document block key-value cache representations corresponding to the multiple target document blocks are read.

[0082] The read document block KV cache representations are combined according to the order in which the multiple target document blocks are arranged in the current request context to obtain the reuse context KV cache.

[0083] In this embodiment of the invention, for example, the server is deployed on an enterprise knowledge base question-and-answer platform. The knowledge base stores knowledge base documents such as product manuals, interface specifications, upgrade records, fault tickets, and operation and maintenance logs. The server first parses and segments the knowledge base documents, dividing each document into multiple document blocks according to a fixed token length, paragraph boundaries, or sentence windows. Metadata such as a document block identifier, source document identifier, token range, model version, and cache validity period are established for each document block. The server performs prefill precomputation on each document block, enabling the document block to independently form a multi-layer KV cache representation without depending on other document blocks. For each network layer in the model, the server saves the key and value representations corresponding to each context token within the document block, thereby obtaining a document block KV cache representation that can be reused in subsequent requests.

[0084] After receiving a user query for "How to handle connection timeouts after upgrading device A", the server sends the query to the retrieval module. The retrieval module returns multiple target document blocks from the knowledge base based on the query semantics, including a model A upgrade instructions document block, a connection timeout error code document block, a network port configuration document block, and historical fault ticket document blocks. The server accesses the key-value (KV) cache library based on the document block identifier of each target document block and reads the pre-saved KV cache representation of each target document block. While reading the cache, the server also obtains the token range, layer number information, and location relocation information for each document block, enabling the KV caches of document blocks from different sources to be combined according to the actual context order of this request.

[0085] The server concatenates and assembles the key-value cache representations of the read document blocks according to their order of appearance within the context of the current request. For example, the server arranges the upgrade instruction document block first, followed by the error code explanation document block, network configuration document block, and historical ticket document block, and then combines the key and value representations of each network layer according to this arrangement to obtain a reused context key-value cache. This reused context key-value cache retains the local context representations obtained by independently prefilling each document block, reducing the computational overhead of the server performing full prefilling for duplicate document blocks, and providing an input basis for subsequent attention evaluation based on user queries, cross-layer fusion selection, and selective cache recalculation.

[0086] In this embodiment of the invention, the step of combining the read document block KV cache representations according to the order of the plurality of target document blocks in the current request context to obtain the reused context KV cache can be implemented through the following example.

[0087] Read document block key-value cached representations that support position-independent multiplexing or relocatable multiplexing;

[0088] Based on the actual combined position of the multiple target document blocks in the context of this request, supplement the position information of the document block KV cache representation or perform position correction.

[0089] The reused context KV cache is obtained by combining the document block KV cache representation based on supplementary location information or execution location correction.

[0090] In this embodiment of the invention, for example, after obtaining the document block KV cache representations corresponding to multiple target document blocks, the server further performs position adaptation processing on each document block KV cache representation. Each document block in the enterprise knowledge base is independently prefilled during the pre-computation stage. When the server saves the Key and Value representations of the document block at multiple network layers, it also saves the original token range, document block identifier, pre-computation position identifier, position encoding type, and relocatable parameters of the document block. The document block KV cache representation supports position-independent reuse or relocatable reuse, enabling the server to read and recombine the same upgrade instruction document block, error code explanation document block, or historical work order document block even when they appear in different orders in different user queries. Specifically, the server does not fix the absolute position encoding during the document block pre-computation stage, or only saves relative position parameters that can be restored during the request stage, allowing the document block KV cache representation to supplement or correct position information based on the actual combined position in the current request context.

[0091] After receiving a user query for "How to handle connection timeouts after upgrading device A to V3.2", the server retrieves an upgrade instruction document block, an error code explanation document block, a port configuration document block, and a historical work order document block. The server reads the key-value cache representations of the target document blocks and determines the actual combined position of these target document blocks in the current request context. For example, the server places the upgrade instruction document block at the beginning of the context, follows the error code explanation document block, and places the port configuration document block and historical work order document block in subsequent areas. Based on the start and end token positions of each target document block in the current combined context, the server supplements the read key and value representations with position information for the current request. For cache representations that already carry pre-calculated position information, the server performs position correction based on the position offset between the current position and the pre-calculated position, ensuring that the key-value cache representation of the document block matches its actual position in the current request context.

[0092] After the server completes location supplementation or correction, it combines the Key and Value representations in each network layer according to the order of the multiple target document blocks in the current request context. Specifically, from layer 1 to the last layer, the server writes the corresponding Key and Value tensors of the layer into the corresponding positions of the combined cache in the order of upgrade instruction document block, error code explanation document block, port configuration document block, and historical ticket document block. The resulting reused context KV cache retains the local representations generated by the independent prefilling of each document block, and can be accessed by the subsequent query attention evaluation module in the actual document arrangement order of the current request. This process avoids cache location mismatch caused by changes in the document block arrangement order and provides a stable cache foundation for subsequent selective recalculation based on user queries to select key context tokens or consecutive token spans.

[0093] In this embodiment of the invention, the step of using the query token corresponding to the user query as the attention query endpoint, using the context object corresponding to the reused context KV cache as the evaluation object, calculating the attention weight from the query token to the context object in multiple network layers of the Transformer model, and performing cross-layer fusion based on the attention weights of multiple network layers to obtain the importance information of the context object can be implemented through the following example.

[0094] Without performing a full prefill precomputation on the multiple retrieval context fragments, the query token accesses the context key cache in the reused context KV cache;

[0095] In multiple network layers of the Transformer model, the attention weights from the query token to the context object are calculated;

[0096] The context object importance score for each network layer is obtained based on the attention weights.

[0097] The importance scores of the context objects from multiple network layers are fused across layers to obtain the importance information of the context objects.

[0098] In this embodiment of the invention, for example, after obtaining the reused context key-value cache corresponding to the user query, the server enters the query-driven importance evaluation phase. The user query received by the server is "How to handle connection timeouts after upgrading device A to V3.2", and the server segments the user query into multiple query tokens, including query tokens such as "model A", "upgrade to V3.2", "connection timeout", and "handle", which represent device entities, software versions, fault phenomena, and handling intentions. The server uses these query tokens as attention query ends and uses the context tokens in the reused context key-value cache corresponding to the upgrade instruction document block, error code explanation document block, port configuration document block, and historical work order document block as the evaluation objects. In this phase, the server does not re-perform a complete prefill pre-computation for all context tokens in the above-mentioned document blocks. Instead, it allows the user query token to access the already combined context key cache in multiple network layers of the Transformer model to obtain the attention distribution from query token to context token with lower computational cost.

[0099] The server performs attention weight calculations in layers 1 through L of the Transformer model. Within each network layer, the server calculates the attention weight from each query token to each context token based on the query representation corresponding to the query token and the key representation in the context key cache. For example, the query token "Connection Timeout" has a high attention weight in the error code explanation document block for the context token corresponding to "E504 indicates connection handshake timeout"; the query token "Upgrade to V3.2" has a high attention weight in the upgrade instructions document block for the context token corresponding to "Default security policy changed after upgrade"; and the query token "Process" has a high attention weight in the historical ticket document block for the context token corresponding to "Re-enable remote management port and restart service". The server performs averaging, weighted averaging, or Top-m averaging on the attention weights of multiple query tokens pointing to the same context token within the same network layer to obtain the context object importance score for that network layer.

[0100] The server further performs cross-layer fusion of the context object importance scores obtained from multiple network layers. Scores output from shallow network layers reflect local literal matching relationships such as "connection timeout," "port," and "upgrade." Scores output from mid-level network layers reflect semantic associations between error codes, configuration items, and fault phenomena. Scores output from deep network layers reflect the inference chain between upgrade policy changes, port closures, and connection timeouts. The server employs mean fusion, weighted fusion, maximum value fusion, or group fusion methods to merge the importance scores from different network layers into unified context object importance information. From this, the server obtains a context token or context object value signal for the current user's query and provides this value signal to the subsequent budget selection module to determine the key evidence token or continuous tokenspan that requires selective recalculation.

[0101] In this embodiment of the invention, obtaining the context object importance score for each network layer based on the attention weights, and fusing the context object importance scores of multiple network layers across layers to obtain the context object importance information, includes:

[0102] For multiple query tokens in the same network layer, the attention weights about the same context object are aggregated by average, column sum, maximum value, Top-m average, or weighted average to obtain the context object importance score for each network layer.

[0103] When using the weighted average aggregation, at least one strong constraint query token is identified from the entity name, time, indicator name, relational term, and constraint condition in the user query, and the strong constraint query token is assigned a weight higher than that of the ordinary query token.

[0104] The context object importance score is obtained from all network layers or a representative network layer of the Transformer model, wherein the representative network layer includes at least two layers from shallow, medium and deep layers;

[0105] The importance scores of the context objects are fused across layers using mean fusion, weighted average fusion, maximum value fusion, or group fusion to obtain the importance information of the context objects.

[0106] In this embodiment of the invention, for example, after obtaining the attention weights from the query token to the context object, the server obtains the context object importance score for each network layer based on the attention weights. The user query processed by the server is "A device of model A experienced a connection timeout after upgrading to V3.2 in December 2024, error code E504. How should this be handled?" This query includes information such as device model, time, software version, fault symptoms, error code, and processing purpose. Within the same network layer, the server aggregates the attention weights of multiple query tokens regarding the same context object to obtain the context object importance score for that network layer. The server performs average aggregation on the attention weights of query tokens such as "Model A", "December 2024", "V3.2", "Connection Timeout", and "E504" that point to the same context token, to reflect the degree of attention the context token receives from the overall query. The server also performs column sum aggregation on the attention weights of each query token on the same context object to identify evidence locations where multiple query tokens jointly pay attention. The server further uses maximum aggregation to identify context objects that are strongly paid attention to by only a few key query tokens, and uses Top-m average aggregation to reduce the impact of noisy query tokens on importance scores.

[0107] When using weighted average aggregation, the server identifies strongly constrained query tokens in user queries and assigns them higher weights than ordinary query tokens. The server identifies "Model A" as an entity name, "December 2024" as a time period, "E504" as an indicator name or error code, and "Connection timeout" and "Upgrade to V3.2" as fault symptoms and constraints, thereby increasing the weight of these query tokens in attention aggregation. Contextual objects such as "The new security policy is enabled by default after upgrading to V3.2" in the upgrade documentation block, "E504 indicates a connection handshake timeout" in the error code documentation block, and "The remote management port needs to be re-enabled after the upgrade" in the historical ticket documentation block, therefore receive higher importance scores.

[0108] The server obtains context object importance scores from all or representative network layers of the Transformer model. It selects at least two layer groups from shallow, mid, and deep layers as representative network layers. Shallow layer groups capture word literal matching and local associations; mid-layer groups capture semantic associations between error codes, port configurations, and fault phenomena; and deep layer groups capture inference links where connection timeouts occur due to upgrade strategy changes. The server performs cross-layer fusion of importance scores from multiple network layers, using methods such as mean fusion, weighted average fusion, maximum value fusion, or group fusion. Mean fusion provides stable overall importance information; weighted average fusion strengthens deep inference signals; maximum value fusion retains key evidence prominently appearing in a few network layers; and group fusion synthesizes the scores from shallow, mid, and deep layers respectively, ultimately yielding context object importance information for subsequent Top-K selection and selective recalculation.

[0109] In this embodiment of the invention, the step of determining the set of objects to be recalculated based on the context object importance information and recalculation constraints can be implemented through the following example.

[0110] The recalculation constraint is determined, and the recalculation constraint includes a recalculation ratio or a fixed number of recalculations;

[0111] When the recalculation constraint includes a recalculation ratio, the total amount of context tokens in the current request context is proportionally converted according to the recalculation ratio, and the target recalculation quantity is determined according to a preset rounding rule.

[0112] When the recomputation constraint includes a fixed number of recomputations, the fixed number of recomputations is determined as the target number of recomputations;

[0113] The context objects are sorted according to their importance information.

[0114] Based on the target number of recalculations, select the corresponding number of context objects from the sorting results to obtain the set of objects to be recalculated.

[0115] In this embodiment of the invention, for example, after obtaining the importance information of the context objects, the server continues to determine the set of objects to be recalculated based on the recalculation constraints. The request processed by the server is an enterprise knowledge base Q&A request, with the user query being "How should I handle the connection timeout and error code E504 after upgrading device A to V3.2?" The context of this request consists of an upgrade instruction document block, an error code explanation document block, a port configuration document block, and a historical work order document block. The server counts the total number of context tokens in this request context as 8000 and reads the recalculation constraints from the system configuration. These recalculation constraints include two forms: a recalculation ratio and a fixed recalculation quantity. The server selects the currently effective constraint based on the request type, service load, and quality requirements.

[0116] In a configuration where the recompile constraint uses a recompile ratio, the server reads the recompile ratio p as 0.2 and performs a ratio conversion based on the total number of context tokens in the current request context. The server calculates the target recompile quantity according to a preset rounding rule, resulting in a target recompile quantity k of 1600 context objects. This preset rounding rule uses an up-rounding method to ensure that the number of recomputed objects is not less than the budget scale corresponding to the recompile ratio. In a configuration where the recompile constraint uses a fixed recompile quantity, the server directly reads the fixed recompile quantity, for example, 1200, and determines this fixed recompile quantity as the target recompile quantity. Therefore, the server can uniformly obtain the target recompile quantity for selecting objects to be recomputed across different online service states.

[0117] The server sorts all context objects corresponding to the reused context key-value cache based on the importance information obtained in the previous steps. Context objects with higher importance scores include the token "E504 indicates connection handshake timeout" in the error code document block, the token "default security policy changed after V3.2 upgrade" in the upgrade instructions document block, the token "remote management port needs to be reopened" in the port configuration document block, and the token "re-enable remote management port and restart service" in the historical ticket document block. The server generates a sorting result from high to low importance scores and selects a corresponding number of context objects from the sorting result based on the target recalculation quantity. The selected context objects constitute the set of objects to be recalculated, while the unselected context objects retain their original reused key-value cache representation. Through this process, the server concentrates its limited recalculation budget on key evidence locations directly related to the current user query, rather than distributing it evenly across all retrieval context fragments.

[0118] In this embodiment of the invention, after determining the set of objects to be recalculated based on the importance information of the context object and the recalculation constraints, and before performing cache recalculation on the set of objects to be recalculated based on the current request context, the following example can be used for implementation.

[0119] Based on the distribution of the set of objects to be recalculated in the multiple retrieval context segments, the hit information of each retrieval context segment is determined;

[0120] Based on the hit information, the multiple search context fragments are rearranged, and the search context fragments whose hit information meets the preset conditions are arranged in a position close to the user query.

[0121] Based on the rearranged multiple retrieval context fragments and the user query, the rearranged current request context is formed;

[0122] When performing the cache recalculation, a joint recalculation is performed on the set of objects to be recalculated and the query token corresponding to the user query, so that the cache representation of the query token is recalculated based on the rearranged current request context.

[0123] In an embodiment of the invention, for example, after determining the set of objects to be recalculated, the server further statistically analyzes the distribution of the set of objects to be recalculated in each retrieval context segment to determine the hit information for each retrieval context segment. The hit information is used to characterize the concentration of context objects in the corresponding retrieval context segment that are relevant to the current user query and require cache repair. The hit information may include at least one of the following: the number of context tokens selected into the set of objects to be recalculated in the retrieval context segment, the number of consecutive token spans, the sum of the importance scores of the selected context objects, the average importance score, the maximum importance score, the coverage length of the selected token or token span in the retrieval context segment, and the coverage ratio. The server can determine the hit information for each retrieval context segment based on one or more of the above information, or it can perform weighted fusion of the above information to obtain the hit score for the corresponding retrieval context segment.

[0124] For example, if the server processes a user query such as "After upgrading device A to V3.2, a connection timeout occurred, error code E504. How should this be handled?", the request context includes an upgrade instruction document block, an error code explanation document block, a port configuration document block, and a historical ticket document block. Based on the set of objects to be recalculated determined in the preceding steps, the server includes the token corresponding to "Default security policy changed after V3.2 upgrade" in the upgrade instruction document block, the token corresponding to "E504 indicates connection handshake timeout" in the error code explanation document block, the token corresponding to "Remote management port needs to be reopened" in the port configuration document block, and the token or consecutive token span corresponding to "Re-enable remote management port and restart service" in the historical ticket document block. The server counts the number and importance score of the above-mentioned set of objects to be recalculated in each document block and determines the hit information for each retrieval context segment accordingly.

[0125] In one implementation, the server uses the number of context objects hit in each search context segment as the hit information. In another implementation, the server uses the sum of the importance scores of the context objects hit in each search context segment as the hit information. In yet another implementation, the server weights and fuses the hit count, importance score, and span length to obtain the hit score for the corresponding search context segment. For example, although the number of hit tokens in a historical work order document block is small, its hit token span is highly relevant to the query intent of "how to handle" and has a high importance score, so this historical work order document block can obtain high hit information. Similarly, the "E504 indicates connection handshake timeout" hit in the error code explanation document block directly corresponds to the error code and fault phenomenon in the user's query, and can also obtain high hit information.

[0126] The server rearranges multiple retrieval context fragments based on the hit information. The preset conditions may include at least one of the following: the number of hits exceeds a preset threshold, the hit score exceeds a preset score threshold, the hit score ranks within a preset number of hits, the hit score falls within a preset percentage range, or the hit information contains a context object corresponding to a strongly constrained query token. The server arranges retrieval context fragments that meet the preset conditions closer to the user's query, ensuring that evidence document blocks more relevant to the current user query are placed in a context region closer to the user's query during subsequent recalculation and decoding. For retrieval context fragments whose hit information does not meet the preset conditions, the server may maintain their original retrieval order or arrange them further away from the user's query.

[0127] For example, if the original search results are in the order of upgrade instructions document block, error code explanation document block, port configuration document block, and historical ticket document block, the server determines that the historical ticket document block and error code explanation document block have higher hit scores based on the hit information, followed by the port configuration document block, and then the upgrade instructions document block. If this request adopts a strategy of placing high-hit document blocks closer to the user query, the server can arrange the historical ticket document block and error code explanation document block with higher hit scores closer to the user query, and arrange the port configuration document block and upgrade instructions document block relatively farther away from the user query. Therefore, the rearranged context of this request can be represented as upgrade instructions document block, port configuration document block, error code explanation document block, historical ticket document block, and user query, or, depending on the system configuration, as other arrangements that place high-hit document blocks closer to the user query.

[0128] After rearranging the retrieval context fragments, the server forms a rearranged request context based on the rearranged fragments and the user query. Because the order of the retrieval context fragments has changed, the server can redetermine the start and end token positions of each fragment based on its actual position within the rearranged request context. It then performs position supplementation or correction on the corresponding reused context KV cache to ensure consistency between the reused context KV cache and the rearranged request context. For document block KV caches that support position-independent reuse or relocatable reuse, the server can recombine the representations of each document block KV cache based on the actual rearranged combination positions to obtain the reused context KV cache base corresponding to the rearranged request context.

[0129] During cache recalculation, the server not only recalculates the context tokens or consecutive token spans in the set of objects to be recalculated, but also includes the query token corresponding to the user query in the joint recalculation scope. Specifically, in each network layer of the Transformer model, the server performs joint recalculation on the set of objects to be recalculated and the query token corresponding to the user query, based on the rearranged context of the current request. This allows the context objects in the set of objects to be recalculated to absorb cross-document information between multiple rearranged retrieval context fragments, and allows the query token corresponding to the user query to recalculate its cache representation based on the rearranged retrieval context fragments. The cache representation of the query token includes the key and value representations of the query token at multiple network layers, or other intermediate cache representations that can be used for subsequent decoding.

[0130] For example, in the reordered context of the current request, historical work order document blocks and error code explanation document blocks are placed closer to the user's query. While the server performs a full cross-layer recalculation on objects awaiting recalculation such as "E504 indicates connection handshake timeout" and "re-enable remote management port and restart service," it also incorporates query tokens from the user's query, such as "Model A," "V3.2," "connection timeout," "E504," and "how to handle," into the joint recalculation scope. Therefore, the key and value representations of the query token are no longer determined solely by the original retrieval context fragment order or the original reuse cache state, but are recalculated based on the reordered context of the current request, allowing the user's query to more fully absorb evidence information from high-hit document blocks located nearby. The server then writes the cached representations of the objects to be recalculated and the query token obtained from the joint recalculation to the corresponding cache locations for subsequent decoding and generation.

[0131] Through the above processing, after determining the set of objects to be recalculated, the server can further identify high-hit document blocks by utilizing the distribution of the set of objects to be recalculated across multiple retrieval context fragments, and adjust these high-hit document blocks to positions closer to the user query. Since the user query is usually located at the end of the input sequence, placing retrieval context fragments with hit information that meet preset conditions close to the user query can enhance the contextual association between the query token and key evidence fragments. Simultaneously, jointly recalculating the set of objects to be recalculated and the query token corresponding to the user query during cache recalculation ensures that the cached representation of the query token remains consistent with the rearranged document block arrangement, avoiding the inconsistency caused by only updating the context token cache while the query token is still calculated based on the original arrangement. This further improves the adaptability of the updated cache to the current user query.

[0132] In this embodiment of the invention, the determination of the recalculation constraint can be performed through the following examples.

[0133] Obtain the request information and / or system operating status, which includes at least one of the following: context length, number of retrieved document blocks, user query length, query complexity, search result confidence, search result concentration, graphics processor memory remaining, batch size, service load, request accuracy level, and historical quality feedback.

[0134] Based on the current request information and / or system operating status, determine the recalculation ratio or the fixed number of recalculations to obtain the recalculation constraints.

[0135] In this embodiment of the invention, for example, when determining recalculation constraints, the server first obtains the current request information and the system operating status. The server receives the user query "After upgrading device A to V3.2, a connection timeout occurred, error code E504. How should this be handled?" and calculates the request context length, the number of retrieved document blocks, the user query length, the query complexity, the confidence level of the retrieved results, and the concentration of the retrieved results. The current request context consists of an upgrade instruction document block, an error code explanation document block, a port configuration document block, and a historical work order document block. The context length is relatively long, the number of retrieved document blocks is large, and the user query simultaneously includes the device model, software version, fault symptoms, error code, and processing purpose. The server identifies this request as a complex query with multiple constraints. The server also reads the remaining GPU memory, the current batch size, the online service load, the request accuracy requirement, and historical quality feedback. Currently, the remaining GPU memory is sufficient, the service load is at a normal level, and historical quality feedback shows that this type of upgrade fault problem requires answering based on evidence from multiple document blocks. Therefore, the server increases the recalculation budget for this request.

[0136] The server determines the recalculation ratio or a fixed number of recalculations based on the current request information and system operating status. For requests with long contexts, numerous document blocks, high query complexity, low search result confidence, or high accuracy requirements, the server determines a larger recalculation ratio, allowing more context objects relevant to the user's query to enter the subsequent selective recalculation process. For requests with short queries, concentrated search results, high system load, low GPU memory availability, or large batch sizes, the server determines a smaller recalculation ratio to reduce the online inference overhead caused by recalculation. The server also adjusts the budget based on historical quality feedback, increasing the fixed number of recalculations for request categories historically prone to incomplete responses due to missing cross-document evidence. Thus, the server obtains recalculation constraints adapted to the current request and operating environment, and provides these constraints to the budget selection module for subsequently determining the set of objects to be recalculated based on the importance of context objects.

[0137] In this embodiment of the invention, the step of performing cache recalculation on the set of objects to be recalculated based on the current request context, and updating the reuse context KV cache using the cache representation obtained from the recalculation to obtain the updated cache can be implemented through the following example.

[0138] When the set of objects to be recalculated includes tokens that are adjacent in position, the adjacent tokens are merged to form a continuous token span;

[0139] The continuous token span is modified based on at least one of the minimum span length, maximum span length, maximum quota per document block, and fragmentation constraint to obtain the modified set of objects to be recalculated.

[0140] Based on the number of the corrected set of objects to be recalculated in each retrieval context segment, the importance score determined by the importance information of the context objects, and / or the span length, the document block-level hit score of each retrieval context segment is determined;

[0141] The multiple search context fragments are rearranged according to the document block-level hit score, so that the search context fragments with higher document block-level hit scores are located closer to the user query, and the rearranged multiple search context fragments and the user query are used to form the rearranged current request context;

[0142] In each network layer of the Transformer model, based on the rearranged current request context, a complete cross-layer recalculation is performed on the corrected set of objects to be recalculated and the query token corresponding to the user query. This allows the context objects in the corrected set of objects to be recalculated to absorb cross-document information from the current combined context, obtaining the key and value representations corresponding to the context objects. The key and value representations of the query token are then recalculated based on the rearranged current request context.

[0143] Context objects that have not yet entered the revised set of objects to be recomputed continue to use the cached representation in the reused context KV cache;

[0144] The recalculated Key and Value representations of the context object, as well as the recalculated Key and Value representations of the query token, are written into the reused context KV cache at the positions corresponding to the rearranged current request context, thus obtaining the updated cache.

[0145] In this embodiment of the invention, for example, after determining the set of objects to be recalculated, the server further performs cache recalculation and cache update on the set of objects to be recalculated. The user query processed by the server is "After upgrading device A to V3.2, a connection timeout occurred, error code E504. How should this be handled?" The set of objects to be recalculated includes context tokens such as "Default security policy change" in the upgrade description document block, "E504 indicates connection handshake timeout" in the error code explanation document block, "Reopening the remote management port" in the port configuration document block, and "Restarting the remote management service" in the historical work order document block. The server detects adjacent tokens in the set of objects to be recalculated and merges consecutive high-scoring tokens to form a continuous token span. For example, the server merges multiple adjacent tokens corresponding to "The remote management port is closed by default after the upgrade" into one continuous token span, and merges multiple adjacent tokens corresponding to "Re-enable the port and restart the service" into another continuous token span.

[0146] The server modifies consecutive token spans based on at least one of the following: minimum span length, maximum span length, maximum quota per document block, and fragmentation constraints, resulting in a modified set of objects to be recomputed. For spans shorter than the minimum span length, the server extends them a small number of tokens to adjacent positions to meet the kernel execution granularity for subsequent batch recomputed calculations. For spans longer than the maximum span length, the server splits them according to semantic boundaries or a fixed window. For multiple spans appearing frequently in the same historical work order document block, the server retains the span with the higher importance score based on the maximum quota per document block, preventing a single document block from consuming the entire recomputed budget. Through these modifications, the server reduces the gather and scatter overhead caused by discrete token indexing and makes the modified set of objects to be recomputed more suitable for GPU parallel computing.

[0147] The server determines the document block-level hit score for each search context segment based on the number of the corrected set of objects to be recalculated in each search context segment, the importance score determined by the importance information of the context objects, and / or the span length. Specifically, the server can count at least one of the following in each search context segment: the number of tokens selected into the corrected set of objects to be recalculated, the number of consecutive token spans, the sum of the importance scores of each selected context object, the average importance score, the maximum importance score, and the total span length, and determine the document block-level hit score for the corresponding search context segment based on the above statistical results. For example, if the error code explanation document block hits the token corresponding to "E504 indicates connection handshake timeout," and the historical ticket document block hits the consecutive token span corresponding to "re-enable port and restart service," and the importance score of the above tokens or token spans is high, then the error code explanation document block and the historical ticket document block can obtain a high document block-level hit score.

[0148] The server rearranges multiple search context fragments based on document block-level hit scores, placing the fragments with higher hit scores closer to the user query. The rearranged request context is then formed based on these fragments and the user query. For example, if the original order is upgrade instructions, error code explanations, port configuration, and historical ticket documents, the server determines that the historical ticket and error code explanation documents have higher hit scores, followed by the port configuration document, and then the upgrade instructions. Since the user query is typically at the end of the input sequence, the server can adjust the historical ticket and error code explanation documents with higher hit scores to be closer to the user query, ensuring that the rearranged request context includes the upgrade instructions, port configuration, error code explanations, historical ticket documents, and the user query.

[0149] After forming the reordered request context, the server can redetermine the start and end token positions of each retrieval context fragment based on their actual positions within the reordered request context. It then performs position supplementation, position correction, or cache mapping updates on the reused context KV cache to ensure that the token positions correspond to those in the reordered request context. For document block KV caches that support position-independent reuse or relocatable reuse, the server can recombine the representations of each document block KV cache based on their actual combined positions after reordering, resulting in a reused context KV cache base corresponding to the reordered request context.

[0150] The server then performs a full cross-layer recalculation on the corrected set of objects to be recalculated and the query token corresponding to the user query, based on the rearranged context of the current request, across each network layer of the Transformer model. In this embodiment, full cross-layer recalculation means recalculating the cached representation of the corresponding object in each network layer of the model according to the rearranged context of the current request, rather than only correcting a single network layer or a single tensor. Through this full cross-layer recalculation, the context objects in the corrected set of objects to be recalculated can absorb cross-document information from the current combined context and obtain the corresponding Key and Value representations. For example, the "connection handshake timeout" token in the error code explanation document block can absorb the context impact of "security policy change" in the upgrade instructions document block, and the "remote management port" token in the port configuration document block can absorb the context impact of "restart service to restore connection" in the historical ticket document block, thereby repairing the cross-document connections that were missing in the original independent document block cache.

[0151] Simultaneously, the server incorporates the query token corresponding to the user query into the full cross-layer recalculation scope, ensuring that the key and value representations of the query token are recalculated based on the reordered context of the current request. For example, the server treats query tokens such as "Model A," "V3.2," "Connection Timeout," "E504," and "How to Handle" from the user query as joint recalculation objects. This allows these query tokens to re-obtain their context representations based on high-hit document blocks closer to the user's query position when recalculating their key and value representations. Therefore, the cached representation of the query token remains consistent with the rearranged order of the retrieval context fragments, avoiding inconsistencies in cache representation caused by only recalculating the context token while the query tokens retain their original order.

[0152] The server continues to use the cached representation in the reused context KV cache for context objects that are not included in the revised set of objects to be recomputed. For context tokens not selected for the revised set of objects to be recomputed, the server does not perform a full cross-layer recomputation, thus preserving the computational savings from document block KV cache reuse. The server writes the key and value representations corresponding to the recomputed context object, as well as the key and value representations of the recomputed query token, into the reused context KV cache at the positions corresponding to the reordered current request context, resulting in the updated cache. This updated cache retains most of the computational savings from the pre-computed cache and forms a cached representation consistent with the reordered current request context at key evidence positions and user query positions, thereby further improving the adaptability of subsequent decoding generation to the current user query.

[0153] To more clearly describe the solution provided by the present invention, a more specific implementation method is provided below.

[0154] In one specific embodiment of the present invention, the server is applied to a retrieval-enhanced question-and-answer generation system, which includes a knowledge base, a retrieval module, a document block caching module, a cache assembly module, a query attention evaluation module, a cross-layer fusion module, a budget selection module, a selective recalculation module, and a generation module. The document block caching module performs document segmentation, tokenization, independent prefilling, and position-independent caching on knowledge base documents, and outputs a document block KV cache library. The retrieval and assembly module retrieves relevant document blocks based on user queries, reads the corresponding cache, and combines them in the current request order to obtain a reused context cache. The query attention evaluation module calculates the layer-by-layer attention distribution from the user query token to the context token and outputs the importance score α_l(t) for each network layer. The cross-layer fusion module performs mean, weighted, or grouped fusion on multiple layers of α_l(t) and outputs a unified importance score α_bar(t). The budget selection module performs Top-K selection, span merging, quota control, and fragmentation constraints based on α_bar(t), the recomputation ratio p or a fixed number k, and the system state, and outputs a set S of objects to be recomputed. The selective recomputation module performs full cross-layer recomputation on the context objects in the set S and writes them back to the cache. The generation module decodes the updated KV cache and user query to generate the final answer. The knowledge base stores product manuals, business rule documents, interface descriptions, fault records, operation and maintenance logs, historical work orders, or other searchable text materials. In response to a user query, the server retrieves multiple relevant document blocks from the knowledge base and combines these document blocks with the user query to form the context of the current request. These document blocks can be denoted as D_1, D_2, ..., D_n, and the user query can be denoted as Q. Each document block is tokenized to form multiple context tokens, the total number of combined context tokens is denoted as s, and the set of query tokens corresponding to the user query is denoted as Q_s. The Transformer model includes L network layers. For the t-th context token in the l-th network layer, its Key representation and Value representation are denoted as K_l(t) and V_l(t), respectively.

[0155] The server performs preprocessing on the knowledge base documents during the offline phase or when a document block is first retrieved. Specifically, the server divides the knowledge base documents into multiple document blocks according to a fixed token length, chapter, paragraph, sentence window, or semantic boundary. The fixed token length can be 512 tokens, 1024 tokens, or other lengths adapted to the model context window and cache management strategy. The server independently performs prefill precomputation on the Transformer model for each document block, obtaining the key and value representations of the document block at multiple network layers. These key and value representations are then associated and saved with metadata such as document block identifier, source document identifier, token range, model version, cache validity period, location encoding type, and relocatable parameters, forming a document block KV cache library. Because each document block is processed independently during precomputation, the above document block KV cache representation mainly reflects the local context relationships within the document block and does not include the cross-document influence of other document blocks in the current request.

[0156] After receiving a user query, the retrieval module returns multiple target document blocks from the knowledge base based on the query. For example, if the user query is "How should I handle connection timeouts after upgrading device A to V3.2?", the server retrieves upgrade instruction document blocks, error code explanation document blocks, network configuration document blocks, and historical work order document blocks. The server reads the corresponding document block KV cache representation from the document block KV cache library based on the document block identifier of the target document block, and combines the document block KV cache representations according to the actual arrangement order of multiple target document blocks in the current request context. To adapt to the dynamic changes in document block order in the retrieval enhancement generation scenario, the document block KV cache representation supports position-independent reuse or relocatable reuse. During the combination process, the server supplements the read Key and Value representations with position information under the current request based on the start and end token positions of each target document block in the current request context, or performs position correction based on the position offset between the current position and the pre-calculated position. After the server completes the position supplementation or position correction, it concatenates the corresponding Key tensor and Value tensor in each network layer of the Transformer model according to the arrangement order of the target document blocks to obtain the reused context KV cache corresponding to the user query.

[0157] Reusing context key-value (KV) caches can reduce the computational overhead of the server performing full prefill on duplicate document blocks, but the cache obtained by direct combination is still a local cache representation, lacking cross-document attention interaction between multiple retrieved document blocks. The server does not aim to completely reconstruct the entire KV representation of the input sequence, but rather prioritizes repairing the KV representation of key context objects related to the current user query within a limited computational budget. To this end, the server enters a query-driven importance evaluation phase, using the query token corresponding to the user query as the attention query endpoint and the context object corresponding to the reused context KV cache as the evaluation object. The context object includes context tokens and also includes consecutive token spans obtained by merging the importance information of adjacent context tokens; in other embodiments, the context object can also be a sentence, paragraph, or fixed window.

[0158] In the query-driven importance assessment phase, the server does not perform full prefill precomputation on multiple retrieval context fragments. Instead, it only allows the user query token to access the context key cache in the reused context KV cache. The server computes the attention weights Attention_l(q,t) from each query token to each context object across multiple network layers of the Transformer model. For the user query "How to handle connection timeouts after upgrading device A to V3.2?", the server uses query tokens such as "Model A", "V3.2", "connection timeout", and "how to handle" as attention endpoints and calculates the degree of attention these query tokens receive from context tokens in the upgrade documentation block, error code explanation block, network configuration block, and historical ticket block at each network layer. The computational complexity of this phase is primarily related to the product of the number of user query tokens and the context length, rather than the square of the full context length. Therefore, it can obtain the context object value signal for the current user query with lower computational cost.

[0159] The server derives the importance score of the context object for each network layer based on the attention weights within the same network layer. Specifically, the server performs average aggregation, column sum aggregation, maximum aggregation, Top-m average aggregation, or weighted average aggregation on the attention weights of multiple query tokens regarding the same context object within the same network layer. Average aggregation reflects the degree of attention the context object receives from the overall query; column sum aggregation identifies evidence positions that are jointly attended by multiple query tokens; maximum aggregation retains context objects strongly attended by a few key query tokens; and Top-m average aggregation reduces the impact of noisy query tokens on the importance score. When using weighted average aggregation, the server identifies strongly constrained query tokens in the user query, such as entity names, timestamps, metric names, error codes, version numbers, relational terms, or constraints, and assigns these strongly constrained query tokens a higher weight than ordinary query tokens. For example, the server identifies "Model A" as an entity name, "V3.2" as a version constraint, "Connection Timeout" as a fault phenomenon, and "E504" as an error code, thus giving higher importance scores to the context evidence positions corresponding to these strongly constrained query tokens.

[0160] The server further performs cross-layer fusion of the context object importance scores obtained from multiple network layers. The scores output by shallow network layers reflect more local word literal matching and short-distance associations; the scores output by mid-level network layers reflect more semantic associations between error codes, configuration items, and fault phenomena; and the scores output by deep network layers reflect more entity relationships, fault cause chains, inference chains, or answer locations. The server obtains context object importance scores from all network layers or a representative network layer of the Transformer model. The representative network layer includes at least two groups of layers from shallow, mid, and deep layers. The server uses mean fusion, weighted average fusion, maximum value fusion, or group fusion to perform cross-layer fusion of the importance scores from multiple network layers to obtain unified context object importance information α_bar(t). In one implementation, the server uses α_bar(t) = 1 / L·Σ_{l=1}^{L}α_l(t) to calculate the importance score after cross-layer fusion. In scenarios with a large number of model layers or high online latency requirements, the server selects several representative network layers to participate in the fusion to reduce the computational overhead of Stage I lightweight evaluation.

[0161] The server determines the set of objects to be recalculated based on context object importance information and recalculation constraints. These constraints include at least one of the following: recalculation ratio, fixed recalculation quantity, budget allocated per document block, maximum quota per document block, minimum span length, maximum span length, and fragmentation constraints. The server obtains information about the current request and the system's operating status, including at least one of the following: context length, number of retrieved document blocks, user query length, query complexity, search result confidence, search result concentration, remaining GPU memory, batch size, service load, request accuracy requirements, and historical quality feedback. Based on this information, the server determines the recalculation ratio or fixed recalculation quantity to be used for the current request. For requests with longer contexts, more retrieved document blocks, higher query complexity, lower search result confidence, or higher request accuracy requirements, the server determines a larger recalculation budget; for requests with shorter queries, concentrated search results, higher system load, lower remaining GPU memory, or larger batch sizes, the server determines a smaller recalculation budget. For example, for low-latency requests or peak service periods, the server sets the recalculation ratio to 0.1 to 0.2; for high-accuracy requests, complex inference tasks, or high-value user requests, the server sets the recalculation ratio to 0.2 to 0.3 or higher.

[0162] When the recalculation constraint includes a recalculation ratio, the server proportionally converts the total amount of context tokens in the current request context according to the recalculation ratio and determines the target recalculation quantity according to a preset rounding rule. Specifically, when the total amount of context tokens is s and the recalculation ratio is p, the server determines the target recalculation quantity k according to k = ceil(p·s). When the recalculation constraint includes a fixed recalculation quantity, the server directly determines the fixed recalculation quantity as the target recalculation quantity. The server sorts the context objects corresponding to the reused context KV cache according to the importance information of the context objects, and selects the corresponding number of context objects from the sorting results according to the target recalculation quantity to form a set of objects to be recalculated. For the implementation method of allocating the budget by document block, the server allocates different numbers of recalculation objects to different document blocks according to the retrieval score of each retrieved document block, the semantic relevance to the user query, the document block length, and the historical hit quality, so as to avoid the recalculation budget being occupied by document blocks that are irrelevant to the current user query.

[0163] The server also performs query proximity rearrangement on multiple retrieval context fragments in the current request context based on the distribution of the set of objects to be recalculated across multiple target document blocks. Specifically, the server calculates the number of selected tokens or consecutive token spans, the sum of importance scores, the average importance score, the maximum importance score, and / or the span coverage length in each target document block to obtain document block-level hit information. The server determines the document block-level ranking result based on this hit information and arranges target document blocks with a larger number of selected tokens, higher importance scores, or more concentrated key spans closer to the user query. For context templates using "document block first, user query second," the server arranges the aforementioned target document blocks later in the multiple retrieval context fragments, making them closer to the user query in sequence position; for context templates using "user query first, document block second," the server arranges the aforementioned target document blocks earlier in the multiple retrieval context fragments. Thus, the server places evidence fragments that contribute more to the current user query in a more advantageous relative position during decoding and reduces interference from irrelevant document blocks on the attention distribution near the query.

[0164] To improve GPU computing efficiency, the server performs spanning on the set of objects to be recomputed. When adjacent context tokens exist in the set, the server merges consecutive high-scoring tokens into consecutive token spans. For example, the server merges multiple adjacent tokens corresponding to "remote management port closed by default after upgrade" into one consecutive token span, and merges multiple adjacent tokens corresponding to "re-enable remote management port and restart service" into another consecutive token span. The server adjusts consecutive token spans based on the minimum span length, maximum span length, maximum quota per document block, and fragmentation constraints. For spans shorter than the minimum span length, the server extends a small number of tokens to adjacent positions to meet the execution granularity of subsequent batch recomputed calculations; for spans longer than the maximum span length, the server splits them according to semantic boundaries or fixed windows; for multiple spans appearing together in the same document block, the server retains the span with the higher importance score based on the maximum quota per document block. Through this processing, the server reduces the gather and scatter overhead caused by discrete token indexing and improves the utilization of the underlying computing kernel.

[0165] The server performs selective full recalculation of the set of objects to be recalculated based on the current request context. This request context includes multiple retrieval context fragments and the user query. The server fixes the set of objects to be recalculated and, within each network layer of the Transformer model, recalculates the Key and Value representations of the context tokens or consecutive token spans within the set of objects to be recalculated, based on the actual document composition relationships in the current request. This full recalculation means that the selected context objects recalculate their K and V representations according to the current request context across all network layers of the model, rather than simply correcting a single network layer or a single tensor. Through this full cross-layer recalculation, the selected context objects can absorb cross-document information from the current composition context. For example, the "connection handshake timeout" token in the error code explanation document block can absorb the information "default security policy changed after upgrade" from the upgrade instructions document block, and the "port opening rules" token in the network configuration document block can absorb the information "re-enable remote management port after upgrade" from the historical ticket document block, thereby repairing cross-document connections that were missing in the cache of independent document blocks.

[0166] During selective full recalculation, the server also includes the query token corresponding to the user query in the recalculation scope. Specifically, based on the rearranged current request context, the server performs joint forward recalculation on the context objects in the set of objects to be recalculated and the query token, so that the key and value representations of the query token reabsorb information from key retrieval context fragments near the query position. For implementations that only recalculate the selected context objects without updating the query token representation, the query token may still carry a representation based on the context before rearrangement or the unrepaired context; this implementation improves the access quality to key evidence positions in the subsequent decoding stage by synchronously recalculating the query token, ensuring that the query-side representation is consistent with the context-side representation after rearrangement and selective repair.

[0167] After recalculation, the server writes the recalculated Key and Value representations to the corresponding positions in the context KV cache that correspond to the set of objects to be recalculated. Context objects not included in the set of objects to be recalculated continue to use the cached representations in the original reused context KV cache. The resulting updated cache retains most of the computational savings from document block cache reuse and closely approximates the fully prefilled representations at key evidence positions directly related to the current user query. The server then performs decoding based on the updated cache and the user query, enabling the model to access key evidence tokens repaired across document information when generating the answer, while continuing to reuse a large number of pre-computed caches of unrecalculated tokens. For the user query "How should I handle connection timeouts after upgrading device A to V3.2?", the server's generated answer revolves around processing steps such as confirming the error code, checking the upgraded network policy, re-enabling the remote management port, opening the specified port, restarting relevant services, and verifying the connection status.

[0168] In another specific implementation, the server is also applied to long-context, multi-evidence reasoning scenarios. In this request context, one retrieval document block records the location of an object, and another retrieval document block records the object's actions on a specific date. The user query requests to determine the city corresponding to the object on that date. When the server directly reuses the independently pre-computed KV caches of each document block, cross-document associations are lacking between the document blocks. The server identifies context tokens related to the object entity, date, and relational terms by using the attention weight of the query token on the context token, and performs a complete cross-layer recomputation on these context tokens. This allows the recomputed Key and Value representations to incorporate information from other document blocks in the current combined context, thereby repairing the missing cross-document connections in the independent caches.

[0169] In this embodiment of the invention, Stage I and Stage II have different responsibilities. Stage I is used to identify context objects worth recompiling with low computational cost, and its output is the importance information of context objects for the current user query. Stage II is used to actually repair the KV representation of the selected context objects, which is computationally more demanding but only applies to a small number of context objects. Through the collaboration of Stage I and Stage II, the server concentrates the limited recompiling budget on evidence positions directly related to the user query, rather than consuming it evenly across the entire long context. The above method can be used in conjunction with FlashAttention, PagedAttention, vLLM, SGLang, or other large language model inference frameworks. The underlying inference framework can be optimized for attention computation, KV cache paging management, batch processing scheduling, or GPU kernel execution, but it does not change the server's processing logic of computing context object importance information based on the user query, selecting the set of objects to be recomputed based on the cross-layer fusion results, and performing full cross-layer recompiling on the set of objects to be recomputed and writing it back to the KV cache. Through the above processing, the server can reduce the amount of full prefill computation, reduce the latency of the first token, and maintain the key evidence information required to generate the answer in long context retrieval enhancement generation scenarios.

[0170] After incorporating document block rearrangement and query token recalculation, the output of Stage I is used not only to select context objects to be recalculated but also to generate document block-level rearrangement signals. Stage II not only repairs the key-value representations of the selected context objects but also synchronously updates the cached representation of the user query token. This process brings key evidence fragments closer to the query in position and matches the query-side representation with the updated context-side representation.

[0171] In an experimental validation, with a 20% token recomputation ratio, the method can maintain near-full prefill generation accuracy. Compared to static position selection, low-level bias signal selection, or global saliency selection, this method more accurately identifies key evidence tokens related to the current user query through query attention and cross-layer fusion, thereby improving the utilization efficiency of part of the recomputation budget and improving the first token latency performance in long context retrieval enhancement generation scenarios.

[0172] Please refer to the following: Figure 4 , Figure 4 A selective recalculation device 110 for KV cache based on retrieval enhancement generation provided in this embodiment of the invention includes:

[0173] The acquisition module 1101 is used to acquire the reuse context key-value cache corresponding to the user query;

[0174] The calculation module 1102 is used to take the query token corresponding to the user query as the attention query endpoint, the context object corresponding to the reused context KV cache as the evaluation object, calculate the attention weight from the query token to the context object in multiple network layers of the Transformer model, and perform cross-layer fusion based on the attention weights of multiple network layers to obtain the importance information of the context object; the context object includes a context token, or includes a continuous token span obtained by merging the importance information of adjacent context tokens; determine the set of objects to be recalculated based on the importance information of the context object and recalculation constraints; perform cache recalculation on the set of objects to be recalculated based on the current request context, and update the reused context KV cache using the cache representation obtained by recalculation to obtain the updated cache; the current request context includes multiple retrieval context fragments and the user query;

[0175] The calculation module 1102 is also used to rearrange multiple retrieval context fragments according to the distribution of the set of objects to be recalculated in multiple retrieval context fragments, and to include the query token corresponding to the user query in the recalculation scope when performing cache recalculation.

[0176] Execution module 1103 is used to perform decoding and generation based on the updated cache and the user query.

[0177] It should be noted that the implementation principle of the aforementioned KV cache selective recalculation device 110 for retrieval enhancement generation can refer to the implementation principle of the aforementioned KV cache selective recalculation method for retrieval enhancement generation, and will not be repeated here. It should be understood that the division of the various modules in the above device is merely a logical functional division; in actual implementation, they can be fully or partially integrated into a single physical entity, or physically separated. Furthermore, these modules can all be implemented in software through processing element calls; they can all be implemented in hardware; or some modules can be implemented by processing element calls to software, and some modules can be implemented in hardware. For example, the KV cache selective recalculation device 110 for retrieval enhancement generation can be a separate processing element, or it can be integrated into a chip in the above device. Alternatively, it can be stored as program code in the memory of the above device, and called and executed by a processing element of the above device. The implementation of other modules is similar. Furthermore, these modules can be fully or partially integrated together, or implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step or module of the above method can be completed by the integrated logic circuit in the hardware of the processor element or by instructions in the form of software.

[0178] For example, these modules can be one or more integrated circuits configured to implement the above methods, such as one or more application-specific integrated circuits (ASICs), one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs). As another example, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together to implement a system-on-a-chip (SOC).

[0179] This invention provides a computer device 100, which includes a processor and a non-volatile memory storing computer instructions. When the computer instructions are executed by the processor, the computer device 100 performs the aforementioned KV cache selective recalculation method for retrieval enhancement generation. Figure 5 As shown, Figure 5This is a structural block diagram of a computer device 100 provided in an embodiment of the present invention. The computer device 100 includes a KV cache selective recalculation device 110 for retrieval enhancement generation, a memory 111, a processor 112, and a communication unit 113.

[0180] To enable data transmission or interaction, the memory 111, processor 112, and communication unit 113 are electrically connected to each other directly or indirectly. For example, these components can be electrically connected to each other via one or more communication buses or signal lines. The KV cache selective recalculation device 110 for retrieval enhancement generation includes at least one software function module that can be stored in the memory 111 or embedded in the operating system (OS) of the computer device 100 in the form of software or firmware. The processor 112 is used to execute the KV cache selective recalculation device 110 for retrieval enhancement generation stored in the memory 111, such as the software function module and computer program included in the KV cache selective recalculation device 110 for retrieval enhancement generation.

[0181] This invention provides a readable storage medium, which includes a computer program. When the computer program runs, it controls the computer device where the readable storage medium is located to execute the aforementioned KV cache selective recalculation device 110 for retrieval enhancement generation.

[0182] For illustrative purposes, the foregoing description has been made with reference to specific embodiments. However, the foregoing illustrative discussions are not intended to be exhaustive or to limit the present disclosure to the precise forms disclosed. Numerous modifications and variations are possible in accordance with the foregoing teachings. These embodiments were chosen and described in order to best illustrate the principles of the present disclosure and its practical application, thereby enabling those skilled in the art to best utilize the disclosure and to employ various embodiments with different modifications to suit a particular intended application.

Claims

1. A method for selective recalculation of key-value caches for retrieval enhancement generation, characterized in that, include: Retrieve the reused context key-value cache corresponding to the user query; The query token corresponding to the user query is used as the attention query end, and the context object corresponding to the reused context KV cache is used as the evaluation object. The attention weight from the query token to the context object is calculated in multiple network layers of the Transformer model, and cross-layer fusion is performed based on the attention weights of multiple network layers to obtain the importance information of the context object. The context object includes a context token, or a continuous token span obtained by merging adjacent context tokens based on their importance information; The set of objects to be recalculated is determined based on the context object importance information and recalculation constraints; Based on the current request context, perform cache recalculation on the set of objects to be recalculated, and use the cache representation obtained from the recalculation to update the reuse context KV cache to obtain the updated cache; The current request context includes multiple retrieval context fragments and the user query; Decoding and generation are performed based on the updated cache and the user query; After determining the set of objects to be recalculated based on the context object importance information and recalculation constraints, and before performing cache recalculation on the set of objects to be recalculated based on the current request context, the method further includes: Based on the distribution of the set of objects to be recalculated in the multiple retrieval context segments, the hit information of each retrieval context segment is determined; Based on the hit information, the multiple search context fragments are rearranged, and the search context fragments whose hit information meets the preset conditions are arranged in a position close to the user query. Based on the rearranged multiple retrieval context fragments and the user query, the rearranged current request context is formed; When performing the cache recalculation, a joint recalculation is performed on the set of objects to be recalculated and the query token corresponding to the user query, so that the cache representation of the query token is recalculated based on the rearranged current request context.

2. The method according to claim 1, characterized in that, The step of obtaining the reused context key-value cache corresponding to the user query includes: Divide the knowledge base document into multiple document blocks; For each document block, prefill is performed independently to obtain the document block KV cache representation corresponding to each document block. The document block KV cache representation includes key representations and value representations on multiple network layers. In response to the user query, multiple target document blocks are retrieved, and the document block key-value cache representations corresponding to the multiple target document blocks are read. The read document block KV cache representations are combined according to the order in which the multiple target document blocks are arranged in the current request context to obtain the reuse context KV cache.

3. The method according to claim 2, characterized in that, The step of combining the read document block KV cache representations according to the order of the multiple target document blocks in the current request context to obtain the reused context KV cache includes: Read document block key-value cached representations that support position-independent multiplexing or relocatable multiplexing; Based on the actual combined position of the multiple target document blocks in the context of this request, supplement the position information of the document block KV cache representation or perform position correction. The reused context KV cache is obtained by combining the document block KV cache representation based on supplementary location information or execution location correction.

4. The method according to claim 1, characterized in that, The process involves using the query token corresponding to the user query as the attention query endpoint, and the context object corresponding to the reused context key-value cache as the evaluation object. Attention weights from the query token to the context object are calculated across multiple network layers of the Transformer model. These attention weights are then fused across multiple network layers to obtain context object importance information, including: Without performing a full prefill precomputation on the multiple retrieval context fragments, the query token accesses the context key cache in the reused context KV cache; In multiple network layers of the Transformer model, the attention weights from the query token to the context object are calculated; The context object importance score for each network layer is obtained based on the attention weights. The importance scores of the context objects from multiple network layers are fused across layers to obtain the importance information of the context objects.

5. The method according to claim 4, characterized in that, The process involves obtaining a context object importance score for each network layer based on the attention weights, and then fusing the context object importance scores from multiple network layers across layers to obtain the context object importance information, including: For multiple query tokens in the same network layer, the attention weights about the same context object are aggregated by average, column sum, maximum value, Top-m average, or weighted average to obtain the context object importance score for each network layer. When using the weighted average aggregation, at least one strong constraint query token is identified from the entity name, time, indicator name, relational term, and constraint condition in the user query, and the strong constraint query token is assigned a weight higher than that of the ordinary query token. The context object importance score is obtained from all network layers or a representative network layer of the Transformer model, wherein the representative network layer includes at least two layers from shallow, medium and deep layers; The importance scores of the context objects are fused across layers using mean fusion, weighted average fusion, maximum value fusion, or group fusion to obtain the importance information of the context objects.

6. The method according to claim 1, characterized in that, The step of determining the set of objects to be recomputed based on the context object importance information and recomputed constraints includes: The recalculation constraint is determined, and the recalculation constraint includes a recalculation ratio or a fixed number of recalculations; When the recalculation constraint includes a recalculation ratio, the total amount of context tokens in the current request context is proportionally converted according to the recalculation ratio, and the target recalculation quantity is determined according to a preset rounding rule. When the recomputation constraint includes a fixed number of recomputations, the fixed number of recomputations is determined as the target number of recomputations; The context objects are sorted according to their importance information. Based on the target number of recalculations, select the corresponding number of context objects from the sorting results to obtain the set of objects to be recalculated.

7. The method according to claim 6, characterized in that, Determining the recalculation constraint includes: Obtain the request information and / or system operating status, which includes at least one of the following: context length, number of retrieved document blocks, user query length, query complexity, search result confidence, search result concentration, graphics processor memory remaining, batch size, service load, request accuracy level, and historical quality feedback. Based on the current request information and / or system operating status, determine the recalculation ratio or the fixed number of recalculations to obtain the recalculation constraint.

8. The method according to claim 1, characterized in that, The process of performing cache recalculation on the set of objects to be recalculated based on the current request context, and updating the reuse context key-value cache using the recalculated cache representation to obtain the updated cache, includes: When the set of objects to be recalculated includes tokens that are adjacent in position, the adjacent tokens are merged to form a continuous token span; The continuous token span is modified based on at least one of the minimum span length, maximum span length, maximum quota per document block, and fragmentation constraint to obtain the modified set of objects to be recalculated. Based on the number of the corrected set of objects to be recalculated in each retrieval context segment, the importance score determined by the importance information of the context objects, and / or the span length, the document block-level hit score of each retrieval context segment is determined; The multiple search context fragments are rearranged according to the document block-level hit score, so that the search context fragments with higher document block-level hit scores are located closer to the user query, and the rearranged multiple search context fragments and the user query are used to form the rearranged current request context; In each network layer of the Transformer model, based on the rearranged current request context, a complete cross-layer recalculation is performed on the corrected set of objects to be recalculated and the query token corresponding to the user query. This allows the context objects in the corrected set of objects to be recalculated to absorb cross-document information from the current combined context, obtaining the key and value representations corresponding to the context objects. The key and value representations of the query token are then recalculated based on the rearranged current request context. Context objects that have not yet entered the revised set of objects to be recomputed continue to use the cached representation in the reused context KV cache; The recalculated Key and Value representations of the context object, as well as the recalculated Key and Value representations of the query token, are written into the reused context KV cache at the positions corresponding to the rearranged current request context, thus obtaining the updated cache.

9. A selective recalculation device for KV cache for retrieval enhancement generation, characterized in that, include: The retrieval module is used to retrieve the reused context key-value cache corresponding to the user query; The calculation module is used to take the query token corresponding to the user query as the attention query end, take the context object corresponding to the reused context KV cache as the evaluation object, calculate the attention weight from the query token to the context object in multiple network layers of the Transformer model, and perform cross-layer fusion based on the attention weights of multiple network layers to obtain the importance information of the context object. The context object includes a context token, or a continuous token span obtained by merging adjacent context tokens based on their importance information; The set of objects to be recalculated is determined based on the importance information of the context objects and the recalculation constraints; cache recalculation is performed on the set of objects to be recalculated based on the current request context, and the cache representation obtained from the recalculation is used to update the reused context KV cache to obtain the updated cache; The current request context includes multiple retrieval context fragments and the user query; The execution module is used to perform decoding and generation based on the updated cache and the user query; The computing module is also used for: Based on the distribution of the set of objects to be recalculated in the multiple search context segments, the hit information of each search context segment is determined; based on the hit information, the multiple search context segments are rearranged, and the search context segments whose hit information meets the preset conditions are arranged in a position close to the user query; based on the rearranged multiple search context segments and the user query, the rearranged current request context is formed. When performing the cache recalculation, a joint recalculation is performed on the set of objects to be recalculated and the query token corresponding to the user query, so that the cache representation of the query token is recalculated based on the rearranged current request context.

Images