Large model reasoning cache optimization method and system based on power transaction system knowledge base

Abstract

According to the big model reasoning cache optimization method and system based on the power transaction system knowledge base, the big model reasoning efficiency is improved through cooperative work of the initialization stage and the background continuous optimization stage. In the initialization stage, according to document types, a knowledge base document and a time sequence data stream are subjected to fragment division, pre-filling calculation and key value cache storage, and an initial cache pool is constructed. The background continuous optimization stage comprises time sequence data optimization and self-adaptive cache elimination: performing incremental pre-filling on newly-added time sequence data, splicing the newly-added time sequence data with an existing cache, and keeping the cache effective through a sliding window and an overdue elimination strategy; meanwhile, a hotspot function value is calculated according to the access frequency and the recent access time, low-priority fragments are eliminated when the cache exceeds the limit, and cache metadata are periodically refreshed. According to the method, repeated calculation is effectively reduced, the first Token delay is reduced, the response speed is improved in a high-concurrency scene, and the method is suitable for large-scale knowledge base application of electric power, medical treatment and the like.

Description

Technical Field

[0001] This application relates to the field of caching technology, and in particular to a method and system for optimizing large-scale model inference caching based on a knowledge base of an electricity trading system. Background Technology

[0002] Existing technologies generally suffer from the following problems when using large language models to process the knowledge base of power trading systems: First, pre-filling computation is extremely expensive: In retrieval-enhanced power trading question-answering systems, input prompts for large language models are typically composed of user questions and relevant document fragments retrieved from power market rule documents, operation manuals, or historical trading data. Each new query request requires a complete pre-filling computation of the entire input sequence to build a key-value cache. Since power trading documents usually contain a large amount of time-series data, statistical charts, and clause details, this results in a quadratic increase in computational load during the pre-filling stage, significantly increasing GPU resource consumption and leading to a high "first token generation time," impacting the system's efficiency in responding to real-time trading inquiries.

[0003] Secondly, existing caching technologies suffer from low utilization: While technologies utilizing "prefix caching" to reuse pre-computed results exist in the industry, their effectiveness in power trading retrieval enhancement generation scenarios is very limited. This is because the set of document fragments retrieved by the retrieval enhancement generation system and their relative order can vary with each query, even with minor changes. For example, users may ask questions about trading policies, nodal pricing, or power dispatching in various ways, and the same core document (such as a clearing algorithm description) may be retrieved multiple times, but its position and relative order in the input sequence will differ. This uncertainty in contextual arrangement makes it difficult to hit the existing complete prefix cache, forcing the system to frequently calculate the key-value cache, resulting in an extremely low hit rate for prefix caching in actual power trading question-and-answer workloads. Statistics show that in actual production workloads, accurate prefix matching only applies to a small portion of requests, demonstrating very limited effectiveness.

[0004] Furthermore, indiscriminate caching leads to quality degradation: simple, indiscriminate reuse of key-value caches can disrupt the relative positional relationships between tokens (causal attention), resulting in decreased output quality, especially when the meaning of a document fragment strongly depends on its external context. This problem is particularly prominent in the power trading scenario, where the meaning of a dispatch instruction or market clause is highly dependent on its context (such as the chapter, publication time, or applicable region). Reusing caches solely based on textual surface matching, ignoring the external context (such as the current grid operating status or market stage), may cause large language models to generate incorrect answers that do not conform to the current trading cycle or operating environment, severely impacting the reliability of the question-answering system in power dispatch auxiliary decision-making.

[0005] Therefore, in the scenario of power trading systems, which have high requirements for response speed, effectiveness, and accuracy, how to effectively solve the problems of high pre-filling computational overhead, low utilization of caching technology, and quality degradation caused by blind caching in retrieval enhancement generation systems has become an urgent technical challenge. Summary of the Invention

[0006] The purpose of this application is to overcome the problems of high pre-filling computational overhead, low caching utilization, and quality degradation caused by blind caching in existing technologies when using large language models to process power trading system knowledge bases. Therefore, it provides a large model inference caching optimization method based on power trading system knowledge bases, including: Initialization phase and continuous backend optimization phase; The initialization phase initializes documents according to their type, including: When the document type is a knowledge base document, the document is divided into segments according to logic or semantics, and the segments are used as cache units. A globally unique identifier is calculated for each cache unit. Pre-filling calculation is performed on the segments to generate the corresponding document key-value cache. The fingerprint, metadata, access frequency, last access time, storage size and the key-value cache are entered into the database. When the document type is a time-series data stream, the time-series data stream is divided into time windows according to the time dimension or statistical period. Each time window is used as a segment to perform pre-filling calculation to obtain the key-value cache of the time-series data. The data is then stored in the database and the window time interval and version information are recorded. The initial cache pool is obtained by writing the key-value pairs stored in the database during the initialization phase into the cache pool. The background continuous optimization phase includes time-series data optimization and adaptive cache eviction. The time-series data optimization is as follows: When new time-series data is added, pre-filling calculation is performed only on the new time-series data to obtain the new key-value cache, and then concatenated with the key-value cache in the cache pool to obtain the updated key-value cache; a sliding time window and expiration eviction strategy are adopted to evict key-value caches that exceed the preset window range. For adjacent time windows, only the basic key-value cache generated by the previous window and the incremental key-value cache corresponding to the new data part are retained; the basic key-value cache is the key-value cache generated and retained by the pre-filling calculation of the previous time window, which is used to directly reuse in adjacent time windows to avoid duplicate calculations; The adaptive cache eviction process involves: real-time monitoring of the access frequency and recent access time of each segment in the cache pool, and calculation of the hot spot function value; when the cache occupancy of all segments exceeds the preset capacity, cache eviction is performed on the segment with the lowest hot spot function value, and the priority of the segments is judged and adjusted based on the final hot spot function value; and the cache metadata is periodically refreshed during system operation.

[0007] The globally unique identifier The calculation formula is as follows: ; in, For the first A segment, for The text and data contained in the fragment; for The metadata of the fragment includes the source document ID, version number, release time, and applicable region.

[0008] The cache pool includes key-value cache and metadata: ; in, For each Fragment key-value caching, for Fragmented fingerprints for The access frequency count value of the fragment. for The last access time of the fragment. for Storage size of the fragment.

[0009] The aforementioned reuse risk level The calculation formula is: ; in, As weight, for Fragmented fingerprints For fingerprints that are calculated or acquired in real time based on the current computing environment, Meta-information similarity, for Meta-information of the fragment, Metadata associated with the current user's session context. for The degree to which a fragment depends on its context.

[0010] The aforementioned position remapping is based on the starting position of the key-value cache. Placement location Displacement between The key vector of the corresponding cached fragment Sum value vector Perform a position transformation; The displacement The calculation formula is: ; The position transformation is as follows: ; in, and For position transformation operators, The key vector after position transformation. This is the value vector after the position transformation.

[0011] The updated key-value cache is obtained by concatenating the newly added key-value cache with the time-series data key-value cache. ; in, For the first Time series data for each time window, To add new time series data, for Key-value cache; This indicates that a new key-value cache has been added.

[0012] The sliding time window and expired elimination strategy include, Based on the preset sliding window length and the latest time window To obtain an effective cache set Remove key-value caches that do not belong to the time window of the valid cache set from the cache pool.

[0013] The hotspot function value The calculation formula is: ; in, and These are the weighting coefficients. for Fragment access frequency, for The most recent access time of the fragment; The aforementioned method of judging and adjusting the priority of segments based on the final hotspot function value is achieved by changing the magnitude of the adjustment coefficient, thereby altering the final hotspot score and thus adjusting the priority. The calculation formula is: ; in, For adjustment coefficients, The larger the segment, the higher its final hotspot score, and the less likely it is to be eliminated; Preset quantization values ​​are used to measure segments. Priority.

[0014] This application also proposes a large-model inference caching optimization system based on the knowledge base of an electricity trading system, including, The document type determination module is used to determine the document type. The knowledge base document initialization module is used to divide documents into segments according to logic or semantics, use the segments as cache units, calculate a globally unique identifier for each cache unit, perform pre-filling calculation on the segments, generate key-value caches for the corresponding documents, and perform database entry operations on fingerprints, metadata, access frequency, last access time, storage size, and the key-value caches. The time series data initialization module divides the time series data stream into time windows according to the time dimension or statistical period, pre-fills each time window as a segment to obtain the key-value cache of the time series data, performs the storage operation and records the window time interval and version information; The cache entry module is used to write the key-value cached during the initialization phase into the cache pool; The time-series data optimization module is used to perform pre-filling calculations only on the newly added time-series data to obtain a new key-value cache, and then concatenate it with the key-value cache in the cache pool to obtain an updated key-value cache. The sliding time window and expiration elimination module is used to eliminate key-value caches that exceed the preset window range. For adjacent time windows, only the basic key-value cache generated by the previous window and the incremental key-value cache corresponding to the newly added data are retained. The adaptive cache eviction module is used to monitor the access frequency and recent access time of each segment in the cache pool in real time and calculate the hot spot function value. When the cache occupancy of all segments exceeds the preset capacity, the segment with the lowest hot spot function value is cached and evicted. The priority of the segments is judged and adjusted according to the final hot spot function value. The cache metadata is also refreshed periodically during system operation.

[0015] The advantages of this application are: 1. This application constructs a content-address-based segment-level caching and reuse mechanism, enabling the same document fragments to be reliably retrieved directly from the cache pool using a globally unique identifier in different queries, orders, and contexts. After passing a security reuse check, the pre-calculated key-value cache is reused, removing the strong positional binding restriction of traditional caching. This reduces redundant pre-filling calculations, minimizes wasted computing resources and system response latency, and improves cache effectiveness.

[0016] 2. This application introduces a mechanism of "time-series segmented caching, incremental cache update, and sliding window strategy." By segmenting data according to time dimension or statistical period, and building an independent key-value cache for each segment, incremental pre-filling calculation is performed only on newly added data and concatenated with the existing cache. Combined with the sliding time window strategy, the key-value cache within the effective cache set is maintained, avoiding repeated pre-filling calculations for large-scale historical time-series data, significantly reducing GPU computing resource consumption. Through incremental and windowed caching, the system can quickly respond to high-frequency tasks such as real-time electricity price fluctuations and market clearing result queries. It is optimized for the time-series and dynamic characteristics of electricity trading, improving the accuracy and timeliness of question answering and analysis results.

[0017] 3. This application identifies "hot data" by recording the access frequency and time of each segment in real time, and adopts different cache eviction mechanisms to dynamically evict infrequently used caches, providing space for high-frequency data; it dynamically adjusts cache priority according to the importance and urgency of the query; by dynamically adjusting the cache content, it improves the utilization rate of cache resources, ensures that hot data and key information can be accessed quickly, and improves the overall system performance. Attached Figure Description

[0018] Figure 1 This is a schematic diagram illustrating the key-value cache filling process in this application. Figure 2 This is a schematic diagram of the online query stage of this application. Detailed Implementation

[0019] The technical solutions provided in this application are further illustrated below with reference to the embodiments.

[0020] Existing technologies for processing knowledge bases in power trading systems using large language models (LLMs) generally suffer from the following problems: huge computational overhead for prefilling, low utilization of existing caching techniques, and quality degradation caused by blind caching.

[0021] To address the aforementioned issues, this application proposes a large-model reasoning caching optimization method and system based on the knowledge base of the power trading system.

[0022] Symbols and Conventions: Document chunk collection: .

[0023] Excerpt Number of tokens : .

[0024] Number of model layers: .

[0025] No. Layer key-value representation: for fragments have: Block The number of tokens. It represents... and The number of rows in the matrix represents the number of elements (tokens) contained in that block.

[0026] The dimension of the key vector. It represents... The number of columns in the matrix, i.e., the feature dimension of the key vector for each token.

[0027] Value: The dimension of the vector. It represents... The number of columns in the matrix is ​​the feature dimension of the Value vector for each Token.

[0028] Globally unique identifier: .

[0029] Cache pool: (Mapping ID) (Cache item). The cache item contains key-value caches of all fragments across all model layers. Meta-information ,fingerprint Access frequency count Last visit time Storage size wait.

[0030] Prefilling operation: Its computational cost is expressed as .

[0031] Position transformation operator (relative position remapping): This is used to offset cached key-value pairs from their original locations to new locations.

[0032] Query the sequence of fragments obtained from the retrieval: .

[0033] Cache hit indicator: If Otherwise, 0.

[0034] The purpose of this application is to overcome the problems of high pre-filling computational overhead, low caching utilization, and quality degradation caused by blind caching in existing technologies when using large language models to process power trading system knowledge bases. Therefore, it provides a large model inference caching optimization method based on power trading system knowledge bases, including: Initialization phase and continuous backend optimization phase; The initialization phase initializes documents according to their type, including: When the document type is a knowledge base document, the document is divided into segments according to logic or semantics, and the segments are used as cache units. A globally unique identifier is calculated for each cache unit. Pre-filling calculation is performed on the segments to generate the corresponding document key-value cache. The fingerprint, metadata, access frequency, last access time, storage size and the key-value cache are entered into the database. When the document type is a time-series data stream, the time-series data stream is divided into time windows according to the time dimension or statistical period. Each time window is used as a segment to perform pre-filling calculation to obtain the key-value cache of the time-series data. The data is then stored in the database and the window time interval and version information are recorded. The initial cache pool is obtained by writing the key-value pairs stored in the database during the initialization phase into the cache pool. The background continuous optimization phase includes time-series data optimization and adaptive cache eviction. The time-series data optimization is as follows: When new time-series data is added, pre-filling calculation is performed only on the new time-series data to obtain a new key-value cache, which is then concatenated with the key-value cache in the cache pool to obtain an updated key-value cache; a sliding time window and expiration eviction strategy are adopted to evict key-value caches that exceed the preset window range. For adjacent time windows, only the basic key-value cache generated by the previous window and the incremental key-value cache corresponding to the new data part are retained; the adaptive cache eviction is as follows: the access frequency and recent access time of each segment in the cache pool are monitored in real time, and the hot spot function value is calculated; when the cache occupancy of all segments exceeds the preset capacity, the segment with the lowest hot spot function value is cached and the priority of the segment is judged and adjusted according to the final hot spot function value; during system operation, cache metadata (such as electricity price curves, clearing results, load forecasts, etc.) are periodically refreshed. During a query, a list of ordered fragments is obtained using vector retrieval. Priority is given to retrieving key-value caches with higher priority. A cache hit flag is used to check if each query input fragment is in the cache pool. When a cache hit occurs, a reuse risk is calculated. If the reuse risk is less than a preset reuse risk threshold, reuse is allowed. During reuse, position remapping is performed based on the fragment position difference, and the key-value caches of all transformed fragments are concatenated sequentially. If multiple window sets within multiple time intervals are involved, the key-value caches of the window sets are concatenated according to the time sequence to generate the final result.

[0035] The globally unique identifier The calculation formula is as follows: ; in, For the first A segment, for The text and data contained in the fragment; for The metadata of the fragment includes the source document ID, version number, release time, and applicable region.

[0036] The cache pool includes key-value cache and metadata: ; in, For each Fragment key-value caching, for Fragmented fingerprints for The access frequency count value of the fragment. for The last access time of the fragment. for Storage size of the fragment.

[0037] The aforementioned reuse risk level The calculation formula is: ; in, As weight, for Fragmented fingerprints For fingerprints that are calculated or acquired in real time based on the current computing environment, Meta-information similarity, for Meta-information of the fragment, Metadata associated with the current user's session context. for The degree to which a fragment depends on its context.

[0038] The aforementioned position remapping is based on the starting position of the key-value cache. Placement location Displacement between The key vector of the corresponding cached fragment Sum value vector Perform a position transformation; The displacement The calculation formula is: ; The position transformation is as follows: ; in, and For position transformation operators, The key vector after position transformation. This is the value vector after the position transformation.

[0039] The updated key-value cache is obtained by concatenating the newly added key-value cache with the time-series data key-value cache. ; in, For the first Time series data for each time window, To add new time series data, for Key-value cache; This indicates that a new key-value cache has been added.

[0040] The sliding time window and expired elimination strategy include, Based on the preset sliding window length and the latest time window To obtain an effective cache set Remove key-value caches that do not belong to the time window of the valid cache set from the cache pool.

[0041] The hotspot function value The calculation formula is: ; in, and These are the weighting coefficients. for Fragment access frequency, for The most recent access time of the fragment; The aforementioned method of judging and adjusting the priority of segments based on the final hotspot function value is achieved by changing the magnitude of the adjustment coefficient, thereby altering the final hotspot score and thus adjusting the priority. The calculation formula is: ; in, For adjustment coefficients, The larger the segment, the higher its final hotspot score, and the less likely it is to be eliminated; Preset quantization values ​​are used to measure segments. Priority is determined by the following: High-priority fragments include: the latest real-time electricity price data, essential safety rule manuals, or documents currently under intense discussion. Low-priority fragments include: outdated historical records and infrequently accessed supplementary documents.

[0042] The adjustment coefficient Automatic updates are based on historical access behavior. High-priority segments that consistently demonstrate high importance or high access activity are prioritized. The value will be dynamically increased during system operation, thereby strengthening the preset quantization value. The weighting effect on this segment; while for low-priority segments with low historical access frequency or declining importance, its The value will decrease accordingly to reduce the preset quantization value. The extent of its influence. Automatic adjustment based on continuous accumulation and analysis of historical data. This allows the system to adaptively reflect the priority changes of various segments during actual operation, thereby more stably implementing the segment optimization retention and elimination strategy.

[0043] Example 1 like Figure 1 The diagram shows how the cache pool of this application is filled. Under the premise of maintaining the generation quality (causal attention correctness), this application realizes efficient reuse of key-value caching of the fragments retrieved in any query (unaffected by changes in the position and order of the fragments in the prompt words), and provides precise steps and judgment criteria for the splicing / transformation process.

[0044] A method for optimizing large-scale model inference caching based on a knowledge base of a power trading system can be broken down into several sub-tasks performed collaboratively, including: 1. Content-identifier-based segment-level location-independent caching: Traditional key-value caching mechanisms suffer from two major bottlenecks when processing requests for large language models (LLMs): Computational redundancy and high latency: The system needs to re-pre-calculate for document fragments referenced in each query (even if they are duplicates), resulting in a significant waste of computing resources and response delays.

[0045] Location-bound limitations: Traditional key-value caching is strongly bound to the absolute position of the token in the input sequence, which means that a cached document fragment (Chunk) cannot be reused in another location or in a different context, greatly limiting the effectiveness of the cache.

[0046] To address the aforementioned issues, this application proposes a caching strategy that combines segment-level partitioning with location-independent principles: 1. Chunk-based Caching: First, knowledge base materials such as technical documents and operation manuals related to the power system are logically or semantically divided into independent and meaningful paragraphs or segments (Chunks) using a method based on embedding similarity (SemanticSplitterNodeParser). Each Chunk is considered as the basic unit of caching.

[0047] 2. Content-addressing & Unique ID: Abandoning location-based caching indexes, we calculate a unique global ID for each fragment that is strongly associated with its content. This means that as long as the content of the fragment remains unchanged, its global ID remains unchanged, completely ignoring which document or query it appears in.

[0048] 3. Caching and Reuse Mechanism: When a fragment is retrieved and processed by the system for the first time, its corresponding key-value cache is calculated and stored in the cache pool using its unique global identifier as the key. When any subsequent query (even with completely different context combinations) needs to use the same fragment, the system will directly retrieve and hit it from the cache pool using its global identifier. Once hit, the system can directly reuse its pre-calculated key-value cache, regardless of the fragment's position in the new suggestion word concatenation, thus completely avoiding redundant pre-filling calculations.

[0049] The steps of the segment-level location-independent caching based on content identifiers are as follows: Step 1: Document segmentation and unique global identifier (UID) generation: The power trading knowledge base documents are divided into several segments based on semantics or logic. The segment is The token count for the first segment is... :

[0050] For each Calculate the globally unique identifier (UID):

[0051] It is a fragment The text and data included. For example: "All market participants must submit their electricity and price declaration curves for the following day before 15:00 each day. The declared price range is [-100, 1000] yuan / MWh." Meta information This includes the source document ID, version number, release date, and applicable region, which are used for subsequent security reuse determination. For example: {"Source Document ID":"Rulebook-2024","Version Number":"V1.0.1","Release Date":"2024-01-01","Applicable Region":"East China Market"}.

[0052] Step 2: Fragment pre-filling calculation (offline / on-demand): For each Perform pre-filling to obtain the key-value cache at each level:

[0053] And define the pre-padding cost for a single fragment:

[0054] As an improvement to this method, static documents can be calculated offline in batches and written to the cache; newly updated or real-time streaming data can be calculated on demand.

[0055] Step 3: Cache loading and metadata: Will Inputting metadata into the database:

[0056] Among them, fingerprints Used for context-sensitive matching. This refers to the "external context" when calculating or using this fragment, belonging to the runtime environment. For example, the version number of an LLM model. This prevents key-value cache invalidation due to inconsistencies in the computing environment.

[0057] The fingerprint definition is used to quickly determine whether the external context matches.

[0058] Step 4: Query, Retrieval, and Decomposition (Online): like Figure 2 As shown, receiving user queries Using vector retrieval, an ordered list of fragments is obtained:

[0059] in, (Process): This arrow and the text above it... "(The search engine) represents a processing procedure." It is an ordered list that contains The document fragments that the search engine considers most relevant. It is the most relevant segment. It is the second most relevant, and so on, until... .

[0060] The system first processes the user query This is converted into a vector (i.e., an embedding). Then, it searches a pre-built "vector database" (which stores all document fragments from step 1). (Vector of the query). Objective: To find the vector that matches the user query. The document fragments that are most similar and related in semantics.

[0061] Each search term should have a target placement location. (The starting position of the Token index range in the prompt), used for subsequent position remapping.

[0062] Step 5: Cache hit / miss determination and safe reuse check: For each Check if it's in the cache pool: It is a cache hit flag, where 1 indicates a hit and 0 indicates a miss.

[0063] It is an indicator function. Its function is: If the condition within the parentheses is true (i.e., the element is in the set), the function returns 1.

[0064] If the condition within the parentheses is false (i.e., the element is not in the set), the function returns 0.

[0065] like Further context / fingerprint trustworthiness checks are performed: the reuse risk level is calculated. :

[0066] in, As weight, For fingerprints, A fingerprint that is calculated or acquired in real time based on the current computing environment.

[0067] Meta-information similarity (the higher the similarity, the better the match).

[0068] This indicates the strength of the segment's dependence on external context. Using a regression model, the model can learn whether the segment contains language features that strongly depend on context, such as pronouns (e.g., "it", "this"), indicator words (e.g., "the aforementioned", "in this case"), or relative time / space descriptions.

[0069] Set a reuse risk threshold. ,like If the condition is met, reuse is allowed (hit); otherwise, the segment is marked as a forced recalculation (miss).

[0070] Step 6: Location remapping ( ): If reuse is allowed, calculate the starting position of the original cache. Placement location Displacement:

[0071] For each layer For the corresponding cached fragment , value Perform position transformation:

[0072] in , For the position transformation operator, the model uses This is achieved by using a rotation operator, which reduces complexity.

[0073] Define the transformation cost as .

[0074] like ( It is a tolerance factor or scaling factor used to adjust the comparison between the cost of "transformation" and the cost of "recalculation". The system uses this rule to decide: "Is it more cost-effective to spend effort transforming the old cache, or is it more cost-effective to simply recalculate?", and directly recalculate to avoid the costs or risks of complex transformation.

[0075] Step 7: Key-value concatenation (in the order of the final suggested words): Indexing all fragments (Hit or just calculated fragments), construct each level of merged key-value pairs in the order of their final appearance: ; ; in , It refers to the pre-filled key-value pairs in the query token of the prompt (only key-value vectors and value vectors with reusability across time steps are worth caching). It is generated only from the token currently being generated; if Query Tokens are always generated in real time, only the token being calculated is needed. ).

[0076] The performance and cost analysis formula is as follows: Cache hit rate (for a given query):

[0077] It is a Boolean condition (true / false) that determines the first... Retrieved fragments Whether its cache is allowed to be reused. The following conditions must be met simultaneously: according to step 5 Cache hit; and according to step 5 Safety Check.

[0078] Actual pre-fill cost (for this query): in For the pre-padding cost of a single fragment, This represents the transformation cost for position remapping. To avoid reusing the cache for this segment (recalculation is required). This transformation is used to reuse the cache of the segment and perform position remapping. .

[0079] Estimated savings (relative to a full recalculation): .

[0080] 2. Optimization of pre-filling cache for electricity trading time-series data: The input sequences of time-series data are often lengthy and frequently updated. If the pre-filling is recalculated for each query, the overhead is huge, leading to a significant increase in response latency. To address the problem that general caching strategies cannot effectively handle the high-frequency query and update issues of large-scale time-series data (such as electricity price curves, clearing results, and load forecasts) in power trading scenarios, this paper proposes a caching solution.

[0081] This application employs time-series segmented caching: Power trading time-series data is segmented by time dimension or statistical period (e.g., intraday, day-ahead, real-time market), and an independent key-value cache (KV-cache) is built for each segment, supporting flexible reuse in queries. Incremental cache updates: When only a small amount of the latest time-series data is added, the system only performs pre-filling calculations on the newly added portion and concatenates it with the existing cache, avoiding redundant calculations of the entire data segment. A windowing strategy is also used: Considering the characteristics of power market operation, a sliding time window caching strategy is adopted, maintaining only the key-value cache (KV-cache) within a certain time range, ensuring real-time query performance while reducing storage and computational overhead.

[0082] It avoids repetitive pre-filling calculations for large-scale historical time-series data, significantly reducing GPU computing resource consumption and computational redundancy; through incremental and windowed caching, the system can quickly respond to high-frequency tasks such as real-time electricity price fluctuations and market clearing result queries, improving query real-time performance; it is optimized for the time-series and dynamic characteristics of electricity trading, improving the accuracy and timeliness of question answering and analysis results, and is in line with the characteristics of electricity trading.

[0083] For highly time-series, frequently updated data in electricity trading scenarios (such as real-time clearing curves, electricity price time series, and load forecasts), windowing, incremental pre-filling, and differential storage are used to significantly reduce the cost of repetitive pre-filling while maintaining real-time performance. The steps are as follows: Step 1: Timing Segmentation: Time series data stream Divided into time periods based on fixed or business-driven factors:

[0084] Time series data for each window Treat it as a fragment (or further fragments if the document in the window is large), and generate a UID:

[0085] in For the first Time window The starting time point, For the first Time window The end time point.

[0086] Step 2: Window-level pre-population and caching: Perform pre-filling calculations on the time-series data for each window: ; Store in cache pool It also records the window time range and version information.

[0087] Step 3: Delta-Prefill; When new time series data arrives (Usually, several tokens are added at the end of the time frame) only applies to Calculate key values: ; The increment was originally (like ) Step 4: Sliding window and expiration policy: Let the length of the sliding window be... (For example, recently) Hours / Recent (a market-ticks) This is the latest time window. The set of key-value caches that are currently deemed "valid" and "must be retained" by the system:

[0088] Time series data within a certain time window The key-value cache no longer belongs to the current valid set. When the elimination criteria are met, the process will proceed. Operation, will The corresponding key-value cache is removed from the cache pool.

[0089] .

[0090] Step 5: Timing Differentiation and Compression: For adjacent windows that are highly correlated in terms of business (such as two intermediate bids within the same clearing cycle or a continuous period of stable load), the baseline plus the difference can be saved. This represents the formation of a new window. Number of "incremental update blocks":

[0091] Only save and (Save space) The key-value difference operation is incremented (appended) by the token.

[0092] Step 6: Multi-window query concatenation (time-series aggregation): If the query involves a time range The time series data set corresponding to the window Then, concatenate the key-value pairs according to the time series (and perform a ReMap to ensure consistent positions):

[0093] The incremental computation cost is only the sum of the pre-fill costs of the missed windows.

[0094] (2) Revenue estimation formula for the time series module: For a query within a certain time interval, the set of windows that are hit. (Hit), Missed Collection (Requires recalculation), actual pre-fill cost:

[0095] Expected savings (relative to a full recalculation): .

[0096] 3. Adaptive caching strategy: To address the issue that cached content is static and cannot adapt to dynamic changes in power system data and query patterns.

[0097] This application proposes an adaptive caching strategy, which dynamically retains the most valuable segments (those that save the most on future queries and are less prone to invalidation) within a limited cache capacity, prioritizing hot data and suppressing frequent cache misses, including: Access frequency monitoring: Records the access frequency and time of each segment in real time and identifies "hot data"; Cache eviction mechanism: Employs a cache eviction mechanism based on strategies such as Least Recently Used (LRU) or Least Frequently Used (LFU) to dynamically evict infrequently used cache data, freeing up space for high-frequency data. Priority Adjustment: Adjust cache priority based on the importance of the query (e.g., queries for troubleshooting manuals take precedence over daily maintenance guidelines) and urgency to ensure that critical data can be accessed quickly.

[0098] By dynamically adjusting cached content, the utilization of limited cache resources is maximized, thus optimizing resource utilization.

[0099] Ensure that frequently accessed data and critical information can be accessed quickly, thereby improving the overall performance of the system.

[0100] The adaptive caching strategy steps are as follows: Step 1: Cache monitoring: The system monitors the access frequency of each segment in the cache pool in real time. With the most recent visit time .

[0101] Step 2: Hotspot Identification: Define hotspot functions based on access frequency and time: , , These are the weighting coefficients.

[0102] Step 3: Capacity Detection like This will trigger cache eviction.

[0103] in The total cache space occupied by all segments. This is the maximum cache capacity.

[0104] Step 4: Cache eviction: Based on the hot function value-based elimination strategy, the LRU strategy is adopted to eliminate fragments. hotspot function values Minimum cache entry.

[0105] in, For the eviction operation, it represents the segment that is ultimately selected to be removed from the cache. ; The function returns the input value (the argument) that makes the function achieve its minimum value; here, it means returning the value that makes the function achieve its minimum value. The one that reaches the minimum value .

[0106] Step 5: Priority Adjustment: Assign higher weights to mission-critical segments Update hotspot functions.

[0107] It is an adjustment coefficient (usually a positive number) used for control. Final Hotspot Score The extent of the impact. The larger the value, the higher the "score bonus" that high-priority fragments receive in the elimination competition, making them more difficult to eliminate and thus more likely to be retained in the cache.

[0108] It is a preset quantization value used to measure segments. Importance to system-critical tasks or business processes. High priority: For example, the latest real-time electricity price data, mandatory safety rule manuals, or documents currently under intense discussion. Low priority: For example, outdated historical records or infrequently accessed supplementary documents.

[0109] Step 6: Dynamic Updates: During system operation, cached metadata is periodically refreshed to ensure that cached content matches the latest access patterns.

[0110] Step 7: Execute the query: When a new query is executed, high-priority caches are retrieved first to improve response speed.

[0111] Example 2 This application also proposes a large-model inference caching optimization system based on the knowledge base of an electricity trading system, including, The document type determination module is used to determine the document type. The knowledge base document initialization module is used to divide documents into segments according to logic or semantics, use the segments as cache units, calculate a globally unique identifier for each cache unit, perform pre-filling calculation on the segments, generate key-value caches for the corresponding documents, and perform database entry operations on fingerprints, metadata, access frequency, last access time, storage size, and the key-value caches. The time series data initialization module divides the time series data stream into time windows according to the time dimension or statistical period, pre-fills each time window as a segment to obtain the key-value cache of the time series data, performs the storage operation and records the window time interval and version information; The cache entry module is used to write the key-value cached during the initialization phase into the cache pool; The time-series data optimization module is used to perform pre-filling calculations only on the newly added time-series data to obtain a new key-value cache, and then concatenate it with the key-value cache in the cache pool to obtain an updated key-value cache. The sliding time window and expiration elimination module is used to eliminate key-value caches that exceed the preset window range. For adjacent time windows, only the basic key-value cache generated by the previous window and the incremental key-value cache corresponding to the newly added data are retained. The adaptive cache eviction module is used to monitor the access frequency and recent access time of each segment in the cache pool in real time, and calculate the hot spot function value. When the cache occupancy of all segments exceeds the preset capacity, the segment with the lowest hot spot function value is cached and evicted. The priority of the segments is judged and adjusted according to the final hot spot function value. The cache metadata is also refreshed periodically during system operation. The retrieval and hit module is used to perform a hit determination in the cache pool based on the globally unique identifier of the fragment when a query request is received; The security check and reuse module is used to perform security checks on the cached segments that have been hit, and to allow the reuse of cached segments that meet the conditions. The location remapping module is used to remap the key-value cache of the corresponding fragment based on the difference between the target position and the original position of the fragment in the query when the cache reuse condition is met. The cache concatenation and result generation module is used to concatenate the key-value caches of each fragment according to the final order in the query and generate the final result.

[0112] This application achieves significant performance improvements by implementing the aforementioned caching optimization strategy in the power system knowledge base question-and-answer system.

[0113] 1. Significantly reduce computational overhead: By reusing key-value caches of the same document fragments, repeated prefill calculations are effectively avoided, significantly reducing the system's computational load (FLOPs).

[0114] 2. Significantly Improved Response Speed: Direct reuse of the cache reduces end-to-end processing time, especially in high-concurrency scenarios, significantly accelerating query response and improving overall system throughput. Compared to traditional large-model inference systems in RAG knowledge base scenarios, when handling duplicate queries or queries involving the same document fragments (taking 10K input as an example), the system's first token latency (TTFT) response time is reduced from 6.3 seconds to 2.1 seconds.

[0115] 3. Fundamentally improve cache hit rate: Because the cache is decoupled from its location, a single chunk of cache can be reused in various query scenarios, greatly improving cache hit rate and utilization efficiency. In the power system knowledge base question-and-answer system scenario, the reuse hit rate has increased by more than 30% compared to before.

[0116] 4. Enhance system flexibility and adaptability: Enables the system to efficiently process complex queries that are dynamically spliced ​​from different document fragments, enhancing its flexibility and practicality in ever-changing real-world application scenarios.

[0117] Application scenarios: This method is not only applicable to technical document Q&A in power systems, but can also be extended to other fields that require processing large-scale knowledge bases, such as medicine and law.

[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to the embodiments, those skilled in the art should understand that modifications or equivalent substitutions to the technical solutions of this application do not depart from the spirit and scope of the technical solutions of this application, and should all be covered within the scope of the claims of this application.

Claims

1. A method for optimizing large-scale model inference caching based on a knowledge base of an electricity trading system, comprising: Initialization phase and continuous backend optimization phase; The initialization phase initializes documents according to their type, including: When the document type is a knowledge base document, the document is divided into segments according to logic or semantics, and the segments are used as cache units. A globally unique identifier is calculated for each cache unit. Pre-filling calculation is performed on the segments to generate the corresponding document key-value cache. The fingerprint, metadata, access frequency, last access time, storage size and the key-value cache are entered into the database. When the document type is a time-series data stream, the time-series data stream is divided into time windows according to the time dimension or statistical period. Each time window is used as a segment to perform pre-filling calculation to obtain the key-value cache of the time-series data. The data is then stored in the database and the window time interval and version information are recorded. The initial cache pool is obtained by writing the key-value pairs stored in the database during the initialization phase into the cache pool. The background continuous optimization phase includes time-series data optimization and adaptive cache eviction. The time-series data optimization is as follows: When new time-series data is added, pre-filling calculation is performed only on the new time-series data to obtain the new key-value cache, and then concatenated with the key-value cache in the cache pool to obtain the updated key-value cache; a sliding time window and expiration eviction strategy are adopted to evict key-value caches that exceed the preset window range. For adjacent time windows, only the basic key-value cache generated by the previous window and the incremental key-value cache corresponding to the new data part are retained; the basic key-value cache is the key-value cache generated and retained by the pre-filling calculation of the previous time window, which is used to directly reuse in adjacent time windows to avoid duplicate calculations; The adaptive cache eviction process involves: real-time monitoring of the access frequency and recent access time of each segment in the cache pool, and calculation of the hot spot function value; when the cache occupancy of all segments exceeds the preset capacity, cache eviction is performed on the segment with the lowest hot spot function value, and the priority of the segments is judged and adjusted based on the final hot spot function value; and the cache metadata is periodically refreshed during system operation.

2. The large model inference cache optimization method according to claim 1, characterized in that, The globally unique identifier The calculation formula is as follows: ； in, For the first A segment, for The text and data contained in the fragment; for The metadata of the fragment includes the source document ID, version number, release time, and applicable region.

3. The large model inference cache optimization method according to claim 2, characterized in that, The cache pool includes key-value cache and metadata: ； in, For each Fragment key-value caching, for Fragmented fingerprints for The access frequency count value of the fragment. for The last access time of the fragment. for Storage size of the fragment.

4. The large model inference cache optimization method according to claim 1, characterized in that, The aforementioned reuse risk level The calculation formula is: ； in, As weight, for Fragmented fingerprints For fingerprints that are calculated or acquired in real time based on the current computing environment, Meta-information similarity, for Meta-information of the fragment, Metadata associated with the current user's session context. for The degree to which a fragment depends on its context.

5. The large model inference cache optimization method according to claim 1, characterized in that, The aforementioned position remapping is based on the starting position of the key-value cache. Placement location Displacement between The key vector of the corresponding cached fragment Sum value vector Perform a position transformation; The displacement The calculation formula is: ； The position transformation is as follows: ； in, and For position transformation operators, The key vector after position transformation. This is the value vector after the position transformation.

6. The large model inference cache optimization method according to claim 1, characterized in that, The updated key-value cache is obtained by concatenating the newly added key-value cache with the time-series data key-value cache. ； in, For the first Time series data for each time window, To add new time series data, for Key-value cache; This indicates the addition of a key-value cache.

7. The large model inference cache optimization method according to claim 1, characterized in that, The sliding time window and expired elimination strategy include, Based on the preset sliding window length and the latest time window To obtain an effective cache set Remove key-value caches that do not belong to the time window of the valid cache set from the cache pool.

8. The large model inference cache optimization method according to claim 1, characterized in that, The hotspot function value The calculation formula is: ； in, and These are the weighting coefficients. For fragments Access frequency, for The most recent access time of the fragment; The aforementioned method of judging and adjusting the priority of segments based on the final hotspot function value is achieved by changing the magnitude of the adjustment coefficient, thereby altering the final hotspot score and thus adjusting the priority. The calculation formula is: ； in, For adjustment coefficients, The larger the segment, the higher its final hotspot score, and the less likely it is to be eliminated; Preset quantization values ​​are used to measure segments. Priority.

9. A large-model inference cache optimization system based on a knowledge base of a power trading system, implemented according to the method described in any one of claims 1-8, characterized in that, include, The document type determination module is used to determine the document type. The knowledge base document initialization module is used to divide documents into segments according to logic or semantics, use the segments as cache units, calculate a globally unique identifier for each cache unit, perform pre-filling calculation on the segments, generate key-value caches for the corresponding documents, and perform database entry operations on fingerprints, metadata, access frequency, last access time, storage size, and the key-value caches. The time series data initialization module divides the time series data stream into time windows according to the time dimension or statistical period, pre-fills each time window as a segment to obtain the key-value cache of the time series data, performs the storage operation and records the window time interval and version information; The cache entry module is used to write the key-value cached during the initialization phase into the cache pool; The time-series data optimization module is used to perform pre-filling calculations only on the newly added time-series data to obtain a new key-value cache, and then concatenate it with the key-value cache in the cache pool to obtain an updated key-value cache. The sliding time window and expiration elimination module is used to eliminate key-value caches that exceed the preset window range. For adjacent time windows, only the basic key-value cache generated by the previous window and the incremental key-value cache corresponding to the newly added data are retained. and The adaptive cache eviction module is used to monitor the access frequency and recent access time of each segment in the cache pool in real time and calculate the hot spot function value. When the cache occupancy of all segments exceeds the preset capacity, the segment with the lowest hot spot function value is cached and evicted. The priority of the segments is judged and adjusted according to the final hot spot function value. The cache metadata is also refreshed periodically during system operation.

Images