A large model key-value cache directory layering targeted dormancy control method and system

Abstract

The application discloses a large model key value cache directory layering targeted dormancy control method and system, and belongs to the technical field of large model internal bottom layer reasoning scheduling. The system comprises a directory classification module and a dormancy wake-up execution module. The directory classification module classifies the key value cache of the large model according to calling characteristics, and establishes a mapping relationship between cache characteristics and directory nodes. The dormancy wake-up execution module executes targeted dormancy wake-up based on the mapping relationship, and does not execute global traversal retrieval and equivalent global traversal retrieval in the dormancy wake-up process. The corresponding method realizes large model key value cache management and control optimization based on directory classification and targeted dormancy wake-up. The application can reduce invalid power loss, improve key value cache management and control efficiency, reduce running power consumption, adapt to various large models and various implementation and deployment forms, and can be applied to the training and reasoning stages of large models.

Description

Technical Field

[0001] This invention relates to the field of low-level inference scheduling technology within large models, specifically to a hierarchical targeted sleep control method and system for key-value caching in large models, which falls under the category of native low-level architecture technology for large models.

[0002] 2. Definition of terms The atomic-level targeted execution described in this invention refers to a sleep-wake method that directly reaches the target directory node point-to-point based on mapping relationships without any software or hardware intermediaries. The intelligent fallback retrieval described in this invention refers to a local precise matching retrieval that is triggered only when the atomic-level targeted matching fails, and is performed using a non-global traversal method, without performing global or equivalent global traversal retrieval. The directory nodes and structured directory forms described in this invention include at least one of single directory structure, multi-directory structure, and hierarchical nested structure. The number of directories and the structure form are not limited and can be dynamically adapted and adjusted according to the running requirements of large models. Background Technology

[0004] In current large-scale models, the management of key-value cache generally adopts global traversal retrieval or equivalent global traversal retrieval methods, which suffers from low management efficiency, serious waste of computing resources, and high power consumption. Existing optimization solutions in the industry mostly rely on top-level algorithm adjustments without changing the core management logic of global traversal at the bottom layer, resulting in significant limitations in optimization effects. At the same time, existing solutions have a single classification management method for key-value cache, lack a generalized directory classification mechanism, have insufficient accuracy in sleep and wake-up, and cannot achieve precise targeted management, making it difficult to adapt to the high-efficiency and low-power operation requirements of next-generation large-scale models. Summary of the Invention

[0006] This invention aims to solve the technical problems of wasted computing power, low efficiency of global traversal, and insufficient accuracy of sleep / wake-up in existing large model key-value cache management, and provides a directory-based hierarchical targeted sleep control method and system for large model key-value cache.

[0007] This invention employs a hardware or software architecture that separates directory classification from sleep / wake-up execution. The directory classification module performs generalized directory classification of key-value caches based on cache call characteristics, establishing standardized mapping relationships and achieving ordered management of key-value cache data. The sleep / wake-up execution module performs targeted sleep / wake-up based on the mapping relationships. During sleep / wake-up, global traversal retrieval and equivalent global traversal retrieval are not performed, reducing computational power consumption and overall power management at the source. The system supports various implementation forms and deployment methods, possessing good universal adaptability and underlying compatibility.

[0008] This invention is a native underlying management and control technology solution for large models. It is not limited to specific model parameters, training frameworks or business scenarios. Its protection scope covers all implementation methods that use large model key-value caching catalog classification, targeted sleep-wake, and non-global or equivalent global traversal retrieval. Detailed Implementation

[0010] This system supports multiple implementation forms, including pure software, pure hardware, and hybrid software and hardware. It can be non-intrusively embedded into the kernel of a large model or deployed externally, achieving universal adaptation without modifying the top-level algorithm logic of the large model.

[0011] The directory classification module classifies the key-value cache of the large model into a directory based on the calling features. It can adopt a structured directory form and establish a mapping relationship between cache features and directory nodes. Directory nodes can be in the form of physical partitions or logical partitions.

[0012] This system preferably adopts a multi-directory classification architecture, which can achieve precise classification based on cache call status, and achieve optimal technical effects in targeted sleep and wake-up and overall power consumption control. The directory structure can be dynamically expanded or switched according to the running status.

[0013] The hibernation / wake-up execution module only receives large model running instructions and directory mapping signals. It performs targeted hibernation / wake-up based on the mapping relationship, does not traverse non-target directories, and does not process redundant data. The hibernation / wake-up action can be executed in hardware-synchronized timing with large model training or inference instructions. During the hibernation / wake-up process, global traversal retrieval and equivalent global traversal retrieval are not performed.

[0014] Targeted sleep wake-up prioritizes atomic-level directed execution. When atomic-level directed execution fails to match, it automatically switches to intelligent fallback retrieval for backup control. The backup retrieval process still does not perform global or equivalent global traversal retrieval.

[0015] This system is compatible with various large-scale models and vertical-specific large-scale models. It can be applied to the training and inference stages of large models. During the training stage, it completes key-value caching and pre-classification, and during the inference stage, it achieves efficient and accurate sleep-wake-up, adapting to the full lifecycle operation requirements of large models.

[0016] 6. Technical Effects Description When applied to various large-scale base models and vertical-scale models, this system can reduce the unnecessary computational power loss caused by global traversal retrieval and equivalent global traversal retrieval during the management process, and improve the efficiency of key-value cache management; the catalog-based classification mechanism can improve the accuracy of sleep / wake-up and reduce unnecessary computational power occupation; the targeted sleep / wake-up mechanism can reduce power consumption and optimize the overall operating efficiency of large models; the characteristics of multi-form implementation and multi-mode deployment can ensure seamless compatibility between the system and various large models, and have stable underlying adaptation capabilities.

[0017] 7. Conclusion on Technical Effectiveness This invention reconstructs the key-value cache management logic of large models from the bottom layer by classifying them according to the call feature directory and using targeted sleep and wake-up, thus solving the problems of wasted computing power and high power consumption caused by global traversal retrieval during the management process.

[0018] This invention improves the accuracy of sleep / wake-up while reducing power consumption, optimizing the running efficiency of large models, and reducing computing costs.

[0019] This invention has good universality and adaptability, is compatible with various implementation forms, deployment methods and directory structures, and can be adapted to various large models. It is the basic technical support for the underlying inference and scheduling of the next generation of large models.

[0020] 8. List of Existing Technologies Comparison Point 1: Conventional Global Cache Management Architecture Core mechanism: Large model-based caching architecture; cache management requires a full global traversal query, which is a forced global traversal retrieval. Objective differences: No catalog-based classification was used; no targeted sleep / wake-up mechanism was employed; global traversal retrieval was still performed. Comparison Point 2: Cache Capacity Threshold Control Architecture Core mechanism: Cleanup is triggered based on a cache capacity threshold, requiring coordination with multi-layered gating and control mechanisms and the basic retrieval mechanism. Objective differences: Controlled by capacity threshold, not categorized by call characteristics, involves multiple layers of computation and relay, and global search has not been abandoned. Comparison Point 3: Dynamic Memory Allocation Optimized Architecture Core mechanism: Optimizes dynamic memory allocation logic to reduce memory usage, while still using the traditional cache retrieval mechanism. Objective differences: Only memory allocation is optimized, no catalog-based categorization mechanism is set, targeted sleep / wake-up is not adopted, and global search logic is retained. Comparison Point 4: Dynamic Computing Power Scheduling and Cache Adaptation Architecture Core mechanism: Dynamically adapts cache scheduling based on computing power load, with scheduling and feature fusion intermediaries. Objective differences: dynamic computing power adaptation is used, no fixed directory mapping is established, there is a scheduling transfer link, and local global retrieval is retained. Comparison Point 5: General History Cache Reuse Technology Core mechanism: Reusing historical cached data reduces redundant calculations, without changing the core logic of sleep / wake-up and retrieval. Objective differences: Only cache reuse is implemented, no classification based on call characteristics is implemented, there is no targeted sleep / wake design, and global and equivalent global traversal retrieval are not abandoned.

Claims

1. A hierarchical targeted sleep control system for a large model key-value cache directory, characterized in that, Includes a directory classification module and a hibernation / wake-up execution module; The directory classification module categorizes the key-value cache of the large model into directories based on the calling features, and establishes a mapping relationship between cache features and directory nodes. The hibernation wake-up execution module performs targeted hibernation wake-up based on the mapping relationship. During the hibernation wake-up process, global traversal retrieval is not performed, nor is an equivalent global traversal retrieval.

2. The system according to claim 1, characterized in that, The directory node can be in the form of a physical partition or a logical partition.

3. The system according to claim 1, characterized in that, The targeted sleep-wake-up is an atomic operation without any software or hardware intermediaries.

4. The system according to claim 1, characterized in that, The catalog classification adopts a structured catalog format.

5. The system according to claim 1, characterized in that, The system can be implemented as pure software, pure hardware, or a hybrid of software and hardware.

6. The system according to claim 1, characterized in that, The system can be non-intrusively embedded or externally scheduled and deployed.

7. The system according to claim 1, characterized in that, The hibernation / wake-up execution module only performs hibernation / wake-up operations and does not participate in the computation and processing of core business data of the large model.

8. The system according to claim 1, characterized in that, The targeted sleep-wake-up mechanism uses atomic-level targeted execution as the primary execution path. When atomic-level targeted execution fails to match, intelligent fallback retrieval is automatically triggered as a backup execution path.

9. A hierarchical targeted sleep control method for a large model key-value cache directory, characterized in that, include: (1) Classify the key-value cache of the large model into a directory based on the calling features, and establish a mapping relationship between cache features and directory nodes; (2) Targeted sleep wake-up is performed based on the mapping relationship. During the sleep wake-up process, global traversal retrieval is not performed, and equivalent global traversal retrieval is not performed.

10. The method according to claim 9, characterized in that, Catalog-based classification and targeted sleep / wake-up operations are triggered in real time by the large model's execution instructions.