Resource processing methods, devices, equipment, and storage media for large language model services

By detecting the request status and cache size of the large language model service and dynamically scheduling cache resources, the problem of KV cache consuming a large amount of GPU memory is solved, achieving efficient resource management and performance assurance.

CN121349656BActive Publication Date: 2026-11-17THE HONG KONG UNIV OF SCI & TECH
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202511389484.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2024-10-08
Filing Date
2025-09-26
Publication Date
2026-11-17
Estimated Expiration
2045-09-26

AI Technical Summary

Technical Problem

When dealing with long contexts, existing large language models consume a lot of GPU memory for key-value caching (KV caching), resulting in low resource utilization efficiency. Existing management solutions, such as compression or offloading to CPU memory, may affect performance or efficiency.

Method used

By detecting the request status and cache size of the large language model service, cache resources are dynamically scheduled, including cache allocation, release, and update tasks, thereby optimizing the resource utilization of the multi-GPU cluster.

Benefits of technology

It enables efficient and flexible management of cached resources for large language model services, improves resource utilization efficiency, and ensures service performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN121349656B_ABST
    Figure CN121349656B_ABST
Patent Text Reader

Abstract

This application discloses a resource processing method, apparatus, device, and storage medium for a large language model service. It detects the request status and required cache size of each service request within the large language model service and performs dynamic resource scheduling accordingly. Specifically, for requests in a new state, a cache allocation task is established based on their required cache size; for requests in a completed state, a cache release task is established based on their occupied cache resources; for requests in a running state whose required cache size has changed, a corresponding cache update task is established. The system uniformly schedules the cache resources of the large language model service based on these cache allocation, release, and update tasks to optimize memory resource usage. This application enables efficient and flexible management of cache resources for a large language model service, thereby improving cache resource utilization efficiency. The technical solution of this application can be widely applied in the field of artificial intelligence technology.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] Cross-referencing of related patent applications This invention claims priority to U.S. Provisional Patent Application No. 63 / 704,563, filed October 8, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence technology, and in particular to a resource processing method, apparatus, device, and storage medium for a large language model service. Background Technology

[0003] Large Language Models (LLMs) have become a core component of artificial intelligence due to their ability to understand and generate human-like text. To improve the inference efficiency of LLMs on GPUs, key-value (KV) caching is a crucial module. It stores the keys and values ​​of all previous tokens in GPU memory during each LLM inference process to avoid redundancy and repeated computation. Despite this advantage, GPU memory faces some challenges when generating long contexts. Unlike model weights, the size of the KV cache grows with increasing sequence length and batch size, consuming significant amounts of GPU memory. This issue becomes increasingly prominent as the demand for longer sequence lengths (and larger batch sizes) grows.

[0004] In related technologies, existing key-value (KV) cache management solutions mainly focus on compressing the KV cache or offloading the KV cache to CPU memory. The effectiveness of these implementations is generally limited. Compressing the KV cache often requires quantization or sparsification, which inevitably reduces the performance of large language models; while offloading the KV cache to CPU memory is inefficient due to the I / O bottleneck between CPU memory and GPU.

[0005] In summary, the relevant technologies lack effective storage strategies for caching large language models, resulting in low utilization efficiency and high overhead of caching resources. Summary of the Invention

[0006] This application provides a resource processing method, apparatus, device, and storage medium for a large language model service, which can achieve efficient and flexible management of cached resources for the large language model service, thereby improving the utilization efficiency of cached resources.

[0007] One aspect of this application provides a resource processing method for a large language model service, the method comprising: The system detects the request status and required cache size of each service request in the large language model service; wherein the request status includes new status, completed status, and running status. In response to the service request being in a new state, a cache allocation task is created based on the cache size required by the service request. In response to the service request being in a completed state, a cache release task is created based on the cache resources occupied by the service request; In response to the fact that the service request is in a running state and the cache size required by the service request has changed, a cache update task corresponding to the service request is established; The cache resources of the large language model service are scheduled according to the cache allocation task, the cache release task, and the cache update task.

[0008] Specifically, in some embodiments, the step of responding to the service request as a new request by establishing a cache allocation task based on the cache size required by the service request includes: In response to the service request being in a new state, the cache size required for the service request is compared with a pre-set first cache threshold, second cache threshold, and third cache threshold; wherein the first cache threshold is greater than the second cache threshold, and the second cache threshold is greater than the third cache threshold. If the cache size required for the service request is greater than the first cache threshold, the service request is determined to be a first request, and a first allocation task is established; In response to the fact that the cache size required for the service request is greater than the second cache threshold and less than or equal to the first cache threshold, the service request is determined to be a second request, and a second allocation task is established; If the cache size required for the service request is greater than the third cache threshold and less than or equal to the second cache threshold, the service request is determined to be a third request, and a third allocation task is established. If the cache size required for the service request is less than or equal to the third cache threshold, the service request is determined to be a fourth request, and multiple fourth requests are combined to establish a third allocation task.

[0009] Specifically, in some embodiments, scheduling the cache resources of the large language model service according to the cache allocation task includes: For the first allocation task, a new first cache device is activated, the first allocation task is processed through the new first cache device, and cached data of at least one second request is pulled from the activated second cache device; For the second allocation task, it is detected whether there is redundant space in each of the currently activated first cache devices to support the second allocation task. If there is redundant space to support the second allocation task, the second allocation task is processed through the first cache device. If there is no redundant space to support the second allocation task, a new second cache device is activated and the second allocation task is processed through the new second cache device. For the third allocation task, it is detected whether there is redundant space to support the third allocation task among the currently activated first cache devices. If there is redundant space to support the third allocation task, the third allocation task is processed by the first cache device. If there is no redundant space to support the third allocation task, a new third cache device is activated and the third allocation task is processed by the new third cache device. Wherein, the cached data of the service request with the largest value stored in the first cache device is greater than the first cache threshold, the cached data of the service request with the largest value stored in the second cache device is greater than the second cache threshold and less than or equal to the first cache threshold, and the cached data of the service request with the largest value stored in the third cache device is greater than the third cache threshold and less than or equal to the second cache threshold.

[0010] Specifically, in some embodiments, detecting whether each of the currently activated first cache devices has redundant space to support the second allocation task includes: Detect the first available space of the currently active first cache device; If the first available space is greater than or equal to the cache size required by the second allocation task, it is determined that the first cache device has redundant space to support the second allocation task; If the first available space is less than the cache size required by the second allocation task, the cache data of the third and fourth requests in the currently activated first cache device are statistically reduced, and the second available space of the currently activated first cache device after reduction is determined. If the second available space is greater than or equal to the cache size required by the second allocation task, the cached data of the third and fourth requests in the currently activated first cache device are removed, and it is determined that the first cache device has redundant space to support the second allocation task. Alternatively, if the second available space is less than the cache size required by the second allocation task, it is determined that the first cache device does not have redundant space to support the second allocation task.

[0011] Specifically, in some embodiments, scheduling the cache resources of the large language model service according to the cache release task includes: Based on the cache release task, determine the target cache device and request type of the service request that needs to be released; The service requests that need to be cached are released on the target cache device, and service requests of the same type are migrated from other recently activated cache devices to the target cache device.

[0012] Specifically, in some embodiments, scheduling the cache resources of the large language model service according to the cache allocation task, the cache release task, and the cache update task includes: The first migration operation to be executed is determined based on the cache release task, and the first migration operation is added to the operation cache area; Based on the cache update task, a second migration operation that needs to be performed is determined, the second migration operation is added to the operation cache area, and the operation cache area is deduplicated. The third migration operation to be executed is determined based on the cache allocation task, the third migration operation is added to the operation cache area, and the operation cache area is deduplicated. Execute all migration operations in the operation cache.

[0013] Specifically, in some embodiments, The execution of all migration operations in the operation cache includes: Boundary analysis is performed on the communication links between the cache resources of the large language model service to determine the boundary conditions; Based on the aforementioned boundary conditions, a hybrid migration model for the two-box packing problem is established; wherein, the hybrid migration includes migration based on key-value caching and migration based on lexical representation. The two-box packing problem model is solved using a greedy algorithm, and the migration operation is performed based on the solution results.

[0014] On the other hand, embodiments of this application provide a resource processing apparatus for a large language model service, the apparatus comprising: The detection unit is used to detect the request status and required cache size of each service request of the large language model service; wherein, the request status includes new status, completed status and running status; The first response unit is used to respond to the service request being in a new state by establishing a cache allocation task based on the cache size required by the service request. The second response unit is used to respond to the service request being in a completed state by establishing a cache release task based on the cache resources occupied by the service request. The third response unit is used to establish a cache update task corresponding to the service request in response to the service request being in a running state and the cache size required by the service request changing. The scheduling unit is used to schedule the cache resources of the large language model service according to the cache allocation task, the cache release task, and the cache update task.

[0015] On the other hand, embodiments of this application provide an electronic device, including a processor and a memory; The memory is used to store computer programs; The processor executes the computer program to implement the aforementioned resource processing method for the large language model service.

[0016] On the other hand, embodiments of this application provide a computer-readable storage medium storing a computer program, which is executed by a processor to implement the aforementioned resource processing method for the large language model service.

[0017] On the other hand, embodiments of this application also provide a computer program product, which includes a computer program stored in a computer-readable storage medium. The processor of an electronic device reads the computer program from the computer-readable storage medium and executes the computer program, causing the electronic device to execute the resource processing method that implements the aforementioned large language model service.

[0018] The embodiments of this application include at least the following beneficial effects: This application provides a resource processing method, apparatus, device, and storage medium for a large language model service, aiming to improve the problem of large language model cache data consuming a large amount of memory as the sequence length and batch size increase. This application detects the request status (including new, completed, and running status) of each service request in the large language model service and its required cache size, and performs dynamic resource scheduling accordingly. Specifically, for new requests, a cache allocation task is established based on its required cache size; for completed requests, a cache release task is established based on the cache resources it occupies; for running requests whose required cache size has changed, a corresponding cache update task is established; finally, the system uniformly schedules the cache resources of the large language model service based on these cache allocation tasks, cache release tasks, and cache update tasks to optimize the use of memory resources. Through a state-driven dynamic scheduling mechanism, this application can achieve efficient and flexible management of cache resources for the large language model service, thereby improving the utilization efficiency of cache resources and helping to ensure the service performance of the large language model. Attached Figure Description

[0019] The accompanying drawings are used to provide a further understanding of the technical solutions of this application and constitute a part of the specification. They are used together with the embodiments of this application to explain the technical solutions of this application and do not constitute a limitation on the technical solutions of this application.

[0020] Figure 1 This is a system architecture diagram of a resource processing method for a large language model service provided in the embodiments of this application. Figure 2 A flowchart illustrating a resource processing method for a large language model service provided in this application embodiment; Figure 3 This is a schematic diagram of a migration operation provided in an embodiment of this application; Figure 4 This is a structural block diagram of a resource processing device for a large language model service provided in the embodiments of this application; Figure 5 This is a structural block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0022] It is understood that the terms “first,” “second,” etc., used in this application may be used to describe various concepts herein, but unless otherwise stated, these concepts are not limited by these terms. These terms are used only to distinguish one concept from another.

[0023] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0024] Before introducing the technical solution of this application, the technical terms involved in the embodiments of this application will be introduced as follows: 1) Artificial intelligence (AI) is a new technological science that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. AI also utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to obtain optimal results—theories, methods, technologies, and application systems for this purpose.

[0025] 2) Large Language Model (LLM) is an artificial intelligence program based on deep learning. It learns the patterns, grammar, knowledge, and contextual relationships of human language by training on massive amounts of text data. Its core capability is to understand and generate natural language.

[0026] Large Language Models (LLMs) have become a core component of artificial intelligence due to their ability to understand and generate human-like text. To improve the inference efficiency of LLMs on GPUs, key-value (KV) caching is a crucial module. It stores the keys and values ​​of all previous tokens in GPU memory during each LLM inference process to avoid redundancy and repeated computation. Despite this advantage, GPU memory faces some challenges when generating long contexts. Unlike model weights, the size of the KV cache grows with increasing sequence length and batch size, consuming significant amounts of GPU memory. This issue becomes increasingly prominent as the demand for longer sequence lengths (and larger batch sizes) grows.

[0027] In related technologies, existing key-value (KV) cache management solutions mainly focus on compressing the KV cache or offloading the KV cache to CPU memory. The effectiveness of these implementations is generally limited. Compressing the KV cache often requires quantization or sparsification, which inevitably reduces the performance of large language models; while offloading the KV cache to CPU memory is inefficient due to the I / O bottleneck between CPU memory and GPU.

[0028] In view of this, this application provides a resource processing method, apparatus, device, and storage medium for a large language model service, aiming to improve the problem of large language model cache data consuming a large amount of memory as the sequence length and batch size increase. This application detects the request status (including new, completed, and running status) of each service request in the large language model service and its required cache size, and performs dynamic resource scheduling accordingly. Specifically, for new requests, a cache allocation task is established based on its required cache size; for completed requests, a cache release task is established based on the cache resources they occupy; for running requests whose required cache size has changed, a corresponding cache update task is established; finally, the system uniformly schedules the cache resources of the large language model service based on these cache allocation tasks, cache release tasks, and cache update tasks to optimize memory resource usage. This application, through a state-driven dynamic scheduling mechanism, can achieve efficient and flexible management of cache resources for the large language model service, thereby improving the utilization efficiency of cache resources and helping to ensure the service performance of the large language model.

[0029] System architecture and scenario description used in the embodiments of this application Please refer to Figure 1 , Figure 1 The diagram shows a system architecture diagram of a resource processing method for a large language model service provided in an embodiment of this application. It includes a terminal device 140, an Internet 130, a gateway 120, a backend server 110, etc.

[0030] In this embodiment, the terminal device 140 may include various forms such as a desktop computer, laptop computer, PDA (personal digital assistant), mobile phone, vehicle terminal, home theater terminal, and dedicated terminal. Furthermore, it can be a single device or a collection of multiple devices. The terminal device 140 can communicate with the Internet 130 via wired or wireless means to exchange data.

[0031] Backend server 110 refers to a computer system that can provide certain services to terminal device 140. Compared with ordinary terminal device 140, backend server 110 has higher requirements in terms of stability, security, and performance. Backend server 110 can be a single high-performance computer in a network platform, a cluster of multiple high-performance computers, a portion of a single high-performance computer (e.g., a virtual machine), or a combination of portions of multiple high-performance computers (e.g., virtual machines).

[0032] Gateway 120, also known as an internetwork connector or protocol converter, is a computer system or device that acts as a translator, enabling network interconnection at the transport layer. It bridges the gap between two systems using different communication protocols, data formats, languages, or even completely different architectures. Gateways can also provide filtering and security functions. Messages sent from terminal device 140 to backend server 110 are forwarded to the corresponding backend server 110 via gateway 120. Messages sent from backend server 110 to terminal device 140 are also forwarded to the corresponding terminal device 140 via gateway 120.

[0033] The backend server 110 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0034] The resource processing method for the large language model service provided in this application embodiment can be executed separately on the backend server 110, separately on the terminal device 140, or based on the data interaction between the terminal device 140 and the backend server 110.

[0035] Of course, it is understood that the implementation environment corresponding to the method in the embodiments of this application is not limited to that of the implementation environment. Figure 1 As shown, those skilled in the art can flexibly select the specific implementation environment according to actual needs, and this application does not impose any restrictions on this.

[0036] General Description of Embodiments in this Application Please refer to Figure 2 , Figure 2 This illustration shows a flowchart of a resource processing method for a large language model service provided in an embodiment of this application. Figure 2 As shown, a resource processing method for a large language model service according to an embodiment of this application includes, but is not limited to, the following steps: Step 210: Detect the request status and required cache size of each service request of the large language model service; where the request status includes new status, completed status and running status; Step 220: In response to the service request being in a new state, create a cache allocation task based on the cache size required by the service request; Step 230: In response to the service request being in a completed state, create a cache release task based on the cache resources occupied by the service request; Step 240: In response to a service request being in a running state and a change in the cache size required by the service request, create a cache update task corresponding to the service request; Step 250: Schedule the cache resources of the large language model service according to the cache allocation task, cache release task, and cache update task.

[0037] This application provides an efficient method for processing large language model service resources. It can dynamically and finely manage the lifecycle of each service request. By monitoring changes in request status and resource requirements in real time and triggering corresponding scheduling tasks, it can achieve global optimization scheduling of KV cache resources in a multi-GPU cluster, thereby saving the number of GPUs and improving resource utilization.

[0038] Specifically, in this embodiment, the large language model service can use GPUs as computing and storage hardware. In the large language model service, the GPU acts as both the computing engine and memory container. The large language model itself is a massive neural network, requiring extensive matrix operations to generate each token. The thousands of cores and highly parallelized architecture of the GPU make it well-suited for this computing model, thus enabling it to handle the majority of computational tasks. Furthermore, the large language model requires significant memory during runtime, such as for pre-trained model parameters and key-value caches. The size of the pre-trained model parameters is fixed; however, the key-value cache is dynamic. To accelerate the generation process, the large language model service caches the key and value vectors of all tokens generated before each request, avoiding redundant computation. The size of the key-value cache generally depends on the length of the request sequence, i.e., the context length and the length of generated tokens. Since the large language model service typically handles high-concurrency requests, the number of requests also affects the overall size of the key-value cache.

[0039] In this embodiment, a large language model service platform needs to provide request services to a large number of users simultaneously. Each user request (such as engaging in a conversation) is represented as a service request in the system. For each service request, an independent KV Cache (cache data of the service request) is created and maintained on the GPU. It is easy to understand that a large number of concurrent service requests will compete for limited GPU memory resources. Moreover, service requests are dynamic, with their own lifecycle (added, running, completed), and their resource requirements (KV Cache size) will continuously grow as they are generated. Therefore, how to efficiently place tens of thousands of dynamically changing service requests and their cache data in a limited number of multi-GPU clusters is the core technical problem that this application aims to solve.

[0040] To address the aforementioned technical issues, the main difficulties are as follows: On the demand side, the arrival, completion, and release of service requests are often unpredictable. This can lead to some GPUs becoming memory full due to processing multiple long-context requests, while other GPUs may be idle due to processing short requests. Furthermore, the release of small requests on some GPUs leaves many small, discontinuous free fragments in GPU memory, which cannot be utilized by large requests, resulting in reduced GPU utilization efficiency. On the supply side, GPUs are expensive and limited hardware resources; simply purchasing more GPUs to handle peak loads would result in significant cost waste and resource idleness.

[0041] In this regard, the technical solution in this application embodiment does not change the configuration of the GPU device, but uses an optimized scheduling algorithm to achieve efficient utilization of GPU resources by dynamically considering the cache load of service requests and the overall high cost of request migration.

[0042] The methods in the embodiments of this application will be described in detail below with specific steps.

[0043] In step 210, the request status of each service request of the large language model service and the required cache size are detected.

[0044] In this application, the large language model service refers to a software platform deployed on a cloud or local server that can simultaneously provide large language model functions (such as dialogue, text generation, code completion, etc.) to multiple users. For example, it could be an online dialogue service similar to ChatGPT or a similar system deployed within an enterprise. In this application, the large language model service is the object of management and optimization.

[0045] For the large language model service, individual users can initiate service requests. A single, independent interactive task initiated by a user to the large language model service constitutes a service request. For example, when a user opens a chat window and enters a question, this generates a service request. This service request includes the user's input (prompt words) and requests the model to generate a corresponding response. In this embodiment, the service request is the basic unit of resource scheduling, and each service request independently occupies computing and memory resources on the GPU.

[0046] Each service request has a corresponding request state, which describes a specific stage in the lifecycle of a service request. Specifically, request states can include new, completed, and running states. The new state indicates that a brand new service request has just arrived in the system and is about to be processed; at this point, the system needs to allocate resources for it. The completed state indicates that a service request has been processed (e.g., the model has generated a complete response and returned it to the user). At this point, the resources it occupied can be reclaimed for use by other new requests. The running state indicates that a service request is being processed by the GPU, for example, the model is generating token-by-token output.

[0047] Furthermore, for service requests in the newly added or running states, this embodiment also detects the required cache size, i.e., the memory size occupied by the key-value cache. For each service request, the large language model service stores the key and value vectors of all previously generated tokens during the generation process to avoid redundant calculations when generating new tokens. The size of this cache is not fixed and changes as the service request processing progresses.

[0048] Step 220 is triggered when the system detects a newly arrived service request (i.e., the service request is in the "new" state). Its core task is to create a "cache allocation task" for this new request, with the goal of finding and allocating a suitable execution location for it within the multi-GPU cluster. This process is not simply "finding a GPU with free memory," but rather an intelligent decision based on fine-grained management and global optimization strategies.

[0049] For example, in this embodiment, the cache size required for a service request can be assessed in real time. Generally, the initial cache size for a service request is mainly determined by the length of the user input (prompt word), which is not only a quantitative indicator of memory requirements but also the basis for classifying requests. In this embodiment, requests can be precisely classified into T (very small), S (small), M (medium), or L (large) types based on a preset threshold (e.g., the ratio to the total memory of a single GPU device). This classification is the cornerstone of the entire allocation strategy, aiming to achieve differentiated resource scheduling.

[0050] For service requests with newly added statuses, the corresponding decision logic is encapsulated into a specific "cache allocation task." This task explicitly includes the instruction to "allocate a specific request to a specific GPU." This task is output and added to the system's scheduling queue, where it is implemented by the subsequent scheduling execution module (step 250), thus completing the closed loop from decision to action.

[0051] In step 230, this step is activated when the system detects that a service request has been processed and entered the "completed state". The core task of step 230 is not just to simply reclaim the memory resources released by the request, but to create a "cache release task". The aim is to take the departure of the service request as an opportunity to proactively trigger a global resource reorganization and optimization. This is a process of transforming resource reclamation into a system optimization opportunity.

[0052] Specifically, step 230 begins by identifying and confirming the details of the "resource gap" released by the completion request, including its location (which GPU) and its size (the released cache space). In this embodiment, the system does not passively wait for new requests to fill this gap, as that could lead to uneven load distribution and resource fragmentation. Instead, it initiates an active rebalancing process based on the cache release task.

[0053] For example, the system determines whether the newly freed space is worth reusing. Ideally, if the last request on a GPU completes, the entire GPU should be released and taken offline to save costs. However, more commonly, there are still other requests on the GPU. In this case, the system will select one or more similar requests from other GPUs with similar workloads to migrate and fill the newly available free space.

[0054] In step 240, when the system detects a service request in a "running" state whose required key-value cache (KV Cache) size has increased (i.e., the model is generating output, and each new token generated causes the cache to expand), this step will be triggered. The core task of step 240 is to create a "cache update task" to address the issue that the request's resource demands may be mismatched with its current GPU resources, thereby avoiding local overload and maintaining global load balance.

[0055] In this embodiment, the system monitors the cache usage of each running service request in real time. Once an increase in cache size is detected (an inevitable result of the large language model generation process), the system immediately assesses the impact of this change. At this point, the type of request may change. For example, a request originally classified as "medium" may have its cache expanded to the point where it can be redefined as a "large" request as more content is generated. This type of transition means that its original location (e.g., a GPU specifically designed to handle small and medium-sized requests) may no longer be able to meet its new resource requirements, or remaining in its original location may overload the GPU and affect other requests on the same GPU.

[0056] Therefore, the "cache update task" established in this application embodiment is essentially a relocation plan. This task does not simply expand memory in its original location, but rather plans a complex migration operation: first, the request is "released" from its current GPU (but its state remains running, not completed), and then immediately treated as a "new" request with a changed type, re-executing the complete allocation process. This means the system will reassess the global resource state for this "updated" request and find a new, more suitable GPU (e.g., migrating a request that has just been upgraded from M-type to L-type to a GPU specifically for handling L-type requests). In extreme cases, if the cache growth of the request directly causes its current GPU to overload, the task may also plan a more thorough operation, namely, releasing and reallocating all requests on that GPU for a complete reorganization.

[0057] In step 250, the tasks generated in the previous steps—cache allocation tasks, cache release tasks, and cache update tasks—are processed in a comprehensive manner. The cache resources in the multi-GPU cluster are scheduled globally and dynamically, thereby transforming discrete task requests into precise control of hardware resources. Ultimately, this enables the efficient and stable handling of dynamically changing large language model service loads with a relatively small number of GPUs.

[0058] It is understood that the resource processing method for a large language model service provided in this application embodiment detects the request status (including new, completed, and running status) of each service request in the large language model service and its required cache size, and performs dynamic resource scheduling accordingly. Specifically, for requests in the new status, a cache allocation task is established based on its required cache size; for requests in the completed status, a cache release task is established based on the cache resources it occupies; for requests in the running status and whose required cache size has changed, a corresponding cache update task is established; finally, the system uniformly schedules the cache resources of the large language model service based on these cache allocation tasks, cache release tasks, and cache update tasks to optimize the use of memory resources. This application, through a state-driven dynamic scheduling mechanism, can achieve efficient and flexible management of cache resources for the large language model service, thereby improving the utilization efficiency of cache resources and helping to ensure the service performance of the large language model.

[0059] Specifically, in some embodiments, in response to a service request being in a new state, a cache allocation task is established based on the cache size required by the service request, including: In response to a new service request, the cache size required by the service request is compared with a pre-set first cache threshold, a second cache threshold, a third cache threshold, and the cache size required by the service request; wherein the first cache threshold is greater than the second cache threshold, and the second cache threshold is greater than the third cache threshold. If the cache size required for a service request is greater than the first cache threshold, the service request is determined to be the first request, and the first allocation task is established. If the cache size required for a service request is greater than the second cache threshold but less than or equal to the first cache threshold, the service request is determined to be a second request, and a second allocation task is established. If the cache size required for a service request is greater than the third cache threshold but less than or equal to the second cache threshold, the service request is determined to be a third request, and a third allocation task is established. If the cache size required for a service request is less than or equal to the third cache threshold, the service request is determined to be a fourth request, and multiple fourth requests are combined to establish a third allocation task.

[0060] In this embodiment of the application, when establishing a cache allocation task, the cache requests can be classified based on the cache size required by the service requests. For example, in this embodiment of the application, service requests can be divided into four categories: L (large), M (medium), S (small), and T (very small). Of course, in some embodiments, they can also be divided into other categories, such as merging M (medium) and S (small) into one category.

[0061] For example, in this embodiment, cache thresholds can be pre-defined. For instance, if the total memory capacity of a single GPU is C, requests requiring cache sizes in the range (C / 2, C] are classified as L-type requests; those in the range (C / 3, C / 2] are classified as M-type requests; those in the range (C / 4, C / 3] are classified as S-type requests; and those in the range (C / 8, C / 4] are classified as T-type requests. For requests requiring cache sizes smaller than C / 4, multiple requests are merged into a single group, and this group, falling within the range (C / 8, C / 4], is also classified as a T-type request. In the above embodiments, multiple cache thresholds are set. In this embodiment, the number of cache thresholds set can be arbitrary, and this application does not impose any restrictions on this.

[0062] Correspondingly, in this embodiment, GPUs can be classified according to the type of the largest request they process: if the service request with the largest cached data in a GPU belongs to type L, M, S, or T, it is marked as an L, M, S, or T type GPU. In this embodiment, a priority relationship between GPU j and GPU j' is also defined, which is determined by factors such as the workload of GPU j' (e.g., the number of requests and free GPU memory) and the distance between GPU j and GPU j'. For example, GPU j', which processes fewer requests, has more GPU memory, or is located on the same physical machine as GPU j, will be given a higher priority.

[0063] For example, in some embodiments, three cache thresholds can be set, namely C / 2, C / 4, and C / 8, denoted as the first cache threshold, the second cache threshold, and the third cache threshold. If the cache size required by a service request is greater than the first cache threshold, it is determined to belong to the first request (i.e., L-type request), and a first allocation task is created for it; if the required cache size is greater than the second cache threshold but less than or equal to the first cache threshold, it is determined to belong to the second request (i.e., M-type or S-type request), and a second allocation task is created for it; if the required cache size is greater than the third cache threshold but less than or equal to the second cache threshold, it is determined to belong to the third request (i.e., T-type request), and a third allocation task is created for it. For service requests with a required cache size less than the third cache threshold, they are identified as fourth requests, and multiple fourth requests can be packaged to create a third allocation task.

[0064] In this embodiment of the application, this hierarchical task design effectively avoids the additional overhead caused by frequent scheduling of a large number of micro requests, thereby significantly improving the overall scheduling efficiency and resource utilization of the system.

[0065] The method in this application embodiment dynamically manages service requests for large language models in a multi-GPU cluster through cache allocation task Allocate(i), cache release task Depart(i), and cache update task Update(i). Its ultimate goal is to maximize GPU utilization and minimize the total number of GPUs required.

[0066] The overall strategy included in this application is as follows: Priority-based classification scheduling: The algorithm in this application is based on classifying service requests into three categories according to the required cache size: T (very small), S / M (small / medium), and L (large), with GPUs also classified accordingly. This allows the system to tailor allocation strategies for requests of different sizes, avoiding large requests blocking small requests, thereby reducing memory fragmentation.

[0067] The clever use of "most recently activated" GPUs: The algorithm frequently uses the concept of "most recently activated" GPUs, which is actually a load balancing and resource pooling strategy. Prioritizing the allocation of new service requests to "most recently activated" GPUs of the same type allows a smaller number of GPUs to be fully utilized, thus freeing up other GPUs and enabling dynamic scaling of the number of GPUs.

[0068] Cascaded Re-allocation: Any operation (allocate, release, update) can trigger a series of cascaded "move" and "re-allocate" operations. It ensures that after each system state change, requests are proactively rearranged to the optimal GPU, rather than passively accepting a fragmented state, thus always maintaining a near-optimal global allocation.

[0069] The following is a detailed introduction and explanation: Specifically, in some embodiments, scheduling cache resources for the large language model service according to cache allocation tasks includes: For the first allocation task, activate a new first cache device, process the first allocation task through the new first cache device, and pull at least one cached data of the second request from the activated second cache device; For the second allocation task, check whether there is redundant space in each of the currently activated first cache devices to support the second allocation task. If there is redundant space to support the second allocation task, the second allocation task is processed through the first cache device. If there is no redundant space to support the second allocation task, activate a new second cache device and process the second allocation task through the new second cache device. For the third allocation task, check whether there is redundant space in each of the currently activated first cache devices that supports the third allocation task. If there is redundant space that supports the third allocation task, the third allocation task is processed through the first cache device. If there is no redundant space that supports the third allocation task, activate a new third cache device and process the third allocation task through the new third cache device. Among them, the cached data of the service request with the largest value stored in the first cache device is greater than the first cache threshold, the cached data of the service request with the largest value stored in the second cache device is greater than the second cache threshold and less than or equal to the first cache threshold, and the cached data of the service request with the largest value stored in the third cache device is greater than the third cache threshold and less than or equal to the second cache threshold.

[0070] In this embodiment, for a newly arriving service request, different caching allocation strategies can be executed according to the type of the corresponding allocation task. Here, for the caching devices, the following rules can be followed: the largest cached data of the service request stored in the first caching device is greater than the first cache threshold; the largest cached data of the service request stored in the second caching device is greater than the second cache threshold and less than or equal to the first cache threshold; and the largest cached data of the service request stored in the third caching device is greater than the third cache threshold and less than or equal to the second cache threshold.

[0071] In other words, the type of caching device is determined by the type of service request for the largest amount of cached data it carries. For example, a caching device that stores cached data for L (large) type service requests is a first-class caching device; a caching device that stores cached data for S / M (small / medium) type service requests but not for L (large) type service requests is a second-class caching device; and a caching device that stores cached data for T (very small) type service requests but not for L (large) or S / M (small / medium) type service requests is a third-class caching device.

[0072] Specifically, for a service request requiring L (large) class cache resources (i.e., the first request), its corresponding task is the first allocation task. Since it requires a large amount of cache resources, a new first cache device will be activated specifically to handle the first allocation task. Furthermore, in this embodiment, an attempt will be made to move cached data of an S / M class service request from the S / M-GPU (i.e., the second cache device) to fill the new first cache device as much as possible, and fill the gaps in the source GPU with cached data of other S / M class service requests. This allows for load balancing; by utilizing the opportunity of needing to activate a new GPU for allocating L class requests (i.e., the first request), the load of S / M class requests (i.e., the second request) within the system can be rebalanced, potentially causing an S / M-GPU to be cleared and released.

[0073] For service requests that require S / M type cache resources (i.e., the second request), the system will first look for an L-GPU (i.e., the first cache device) that can accommodate it. If such a cache device exists, the cached data of the second service request can be directly placed into the first cache device to process the second allocation task. If the cache device does not exist, a new second cache device can be created to process the second allocation task.

[0074] For service requests requiring cached resources of type T (i.e., third requests or combined fourth requests), the system prioritizes finding an L-GPU (i.e., the first cache device) that can accommodate it. If such a L-GPU exists, it can be placed in a free space within the first cache device. Since the first cache device has a larger memory gap, there is a higher probability of redundant space. This improves fragmented memory utilization and avoids activating a new cache device specifically for small requests. If the first cache device does not have redundant space to support the third allocation task, a new third cache device can be activated to handle the third allocation task, thus allowing for centralized management of small requests.

[0075] Specifically, in some embodiments, detecting whether each of the currently activated first cache devices has redundant space to support the second allocation task includes: Detect the first available space of the currently active first cache device; If the first available space is greater than or equal to the cache size required by the second allocation task, it is determined that the first cache device has redundant space to support the second allocation task; If the first available space is less than the cache size required by the second allocation task, the cache data of the third and fourth requests in the currently activated first cache device are statistically reduced, and the second available space of the currently activated first cache device after reduction is determined. If the second available space is greater than or equal to the cache size required for the second allocation task, the cached data of the third and fourth requests in the currently activated first cache device are removed, and it is determined that the first cache device has redundant space to support the second allocation task. Alternatively, if the second available space is less than the cache size required for the second allocation task, it is determined that the first cache device does not have redundant space to support the second allocation task.

[0076] In this embodiment of the application, when looking for suitable storage space for S / M type service requests (i.e., the second request), it is not simply necessary to check whether the current free memory (first available space) of the first cache device is sufficient. Instead, active space reorganization methods can be used to try to create space to accommodate medium-sized service requests.

[0077] Specifically, when checking whether each currently activated first cache device has redundant space to support the second allocation task, the available space of the currently activated first cache device can be directly checked first, and recorded as the first available space. If the current first available space is sufficient to meet the requirements of the second allocation task, it is immediately confirmed that the first cache device has the required redundant space and can be prepared to receive the request.

[0078] Conversely, if the first available space is insufficient, the system will not immediately abandon the first cache device, but will initiate a more aggressive space reclamation process. At this time, the system will detect the third and fourth requests that already exist on the first cache device and have relatively low priority, then perform statistical deletion of the cached data of these requests, and determine the available space of the first cache device after deleting this cached data, which is recorded as the second available space.

[0079] If the second available space after pruning is greater than or equal to the cache size required for the second allocation task, it means that removing low-priority requests can free up enough space for the second request. At this point, the cached data of the third and fourth requests in the currently active first cache device can be formally purged, and it can be determined that the first cache device has redundant space to support the second allocation task. If the second available space is less than the cache size required for the second allocation task, it means that even after removing low-priority requests, the space in the first cache device is still insufficient, indicating that there is indeed no required redundant space. The system will then look for other targets for the second allocation task (e.g., select another GPU or perform other scheduling operations).

[0080] In this embodiment, the relevant task processing is achieved through layered and progressively deeper spatial detection and optimization logic. It prioritizes direct allocation at zero cost, and if that is not supported, it tries to migrate smaller requests with lower priority as much as possible to free up space for medium-sized requests, thereby maximizing the utilization of GPU resources and the scheduling efficiency of the overall system.

[0081] Specifically, in some embodiments, scheduling of cache resources for the large language model service is performed based on cache release tasks, including: Based on the cache release task, determine the target cache device and request type of the service request that needs to be released; Release the cached service requests that need to be released on the target cache device, and migrate service requests of the same type from other recently activated cache devices to the target cache device.

[0082] In this embodiment, for a cache release task, when the system executes it, the operation is not simply to clear the cached data of completed requests on the target GPU (i.e., the "target cache device"). Instead, it views this as an opportunity to optimize the overall system resource layout.

[0083] Specifically, the system first identifies two key pieces of information based on the release task: which GPU (i.e., the target cache device) and what type of request (request type) has completed its task and is about to release its cache resources. Then, after the specified cache resource release is completed on the target cache device, this application will not passively wait for new service requests to fill the suddenly idle resources on the target cache device. Instead, it will proactively migrate a request of the same type from another recently activated GPU that handles the same type of request. For example, if a medium-sized request is released on the target cache device, the system will select a medium-sized request from another recently activated GPU handling medium-sized requests to migrate and fill the gap. In this way, by concentrating scattered requests of the same type onto fewer GPUs as much as possible, some GPUs can be completely cleared, ensuring efficient utilization of cluster resources and avoiding resource fragmentation and uneven load due to request completion.

[0084] For cache update tasks, generally, when the growth of cached data for service requests leads to a significant change in their type (e.g., a T-type request becomes an S-type request, or an M-type request becomes an L-type request), rescheduling is required. In specific scheduling, this embodiment removes the cached data of the changed service requests from the original cache device (this process may trigger cache device rebalancing). Then, the cached data of the changed service requests is treated as a new task, and the corresponding cache allocation task logic is re-executed, thereby achieving cache data migration.

[0085] In some cases, a caching device for a certain type L storage request may become overloaded due to the increasing size of the cached data for the service request. In such cases, all other requests on that GPU can be released and reallocated.

[0086] In this embodiment of the application, when scheduling the cache resources of the large language model service, an adaptive migration method is adopted to adjust the cache resources. Specifically, the process mitigates the negative impact of request migration by balancing communication and computing costs. The process mainly includes: first, identifying idle computing and communication resources in the system that can be used for request migration without affecting the normal operation of the system; then, transferring each request by migrating tokens or key-value caches, and coordinating all requests to be migrated to keep their resource consumption within a predetermined boundary.

[0087] Specifically, in some embodiments, the cache resources of the large language model service are scheduled according to cache allocation tasks, cache release tasks, and cache update tasks, including: The first migration operation to be executed is determined based on the cache release task, and the first migration operation is added to the operation cache. Based on the cache update task, determine the second migration operation that needs to be performed, add the second migration operation to the operation cache, and perform deduplication processing on the operation cache. The third migration operation to be executed is determined based on the cache allocation task, the third migration operation is added to the operation cache, and the operation cache is deduplicated. Perform all migration operations in the operation cache.

[0088] In this embodiment, the foregoing description refers to the processing logic of a single task. However, generally speaking, a large language model service corresponds to multiple service requests. When multiple service requests need to execute these operations simultaneously, overlapping operations may occur, leading to redundant migrations. To solve this problem, this embodiment introduces a request operation batch processing mechanism. This mechanism optimizes operation combinations into unified groups rather than discrete execution, thereby minimizing unnecessary resource allocation and migration. Implementing operation batch processing is crucial to ensuring efficient migration within the framework, especially in high-concurrency scenarios.

[0089] Specifically, when processing related tasks, the system first handles cache release tasks. The migration operations that this task may trigger (i.e., the first migration operation, such as migrating requests from other GPUs to fill resource gaps) are not executed immediately but are temporarily stored in the operation cache. This avoids inconsistencies in intermediate states and conflicts in subsequent operations that might result from immediate execution. Next, the system processes cache update tasks. Similarly, the required migration operations (i.e., the second migration operation, such as migrating requests to a new GPU due to increased request volume) are also added to the operation cache. At this stage, the system performs a crucial initial deduplication process, such as identifying and removing migrations that cancel each other out due to shared goals, thereby eliminating unnecessary overhead.

[0090] Subsequently, the system processes the cache allocation task, adding any potential migration operations (i.e., third migration operations, such as migrating an old request from the current GPU to accommodate a new request) to the cache, and performing deduplication again to further purify the scheduling instruction set. Finally, after all possible scheduling impacts in the first three stages have been calculated and optimized, the system executes all the cleaned migration operations in the operation cache in batches.

[0091] It is readily understood that, in this embodiment, the delayed execution strategy provides the system with a global perspective, enabling it to perceive the inherent relationships between different scheduled tasks and thus intelligently eliminate redundant operations. This not only significantly reduces the number of actual migrations but also alleviates the pressure on inter-GPU communication links, avoids performance fluctuations caused by frequent migrations, and ensures the global optimality of scheduling decisions.

[0092] Specifically, in some embodiments, performing all migration operations in the operation buffer includes: Perform boundary analysis on the communication links between cached resources of the large language model service to determine boundary conditions; Based on boundary conditions, a hybrid transfer model for the two-box packing problem is established; the hybrid transfer includes transfer based on key-value caching and transfer based on lexical representation. A greedy algorithm is used to solve the double-box packing problem model, and a transfer operation is performed based on the solution results.

[0093] In this embodiment, during the migration operation, the communication performance between GPUs (e.g., via NVLink, PCIe, or network switches) is analyzed in advance or in real-time to determine the boundary conditions of each link, i.e., the upper limit of the amount of data that can be stably transmitted per unit time. This quantitative analysis provides crucial physical constraints for subsequent decisions, ensuring that the migration operation does not encroach on the communication bandwidth required for normal services. For example, in some embodiments, during system initialization, the communication boundary of each link and the computational boundary of each instance can be determined through offline analysis, and this boundary information is shared with all instances in the system.

[0094] Based on these boundary conditions, this application embodiment establishes a hybrid migration two-box packing problem model. Here, "two boxes" refers to two distinct migration methods: 1. Key-value cache-based migration: The complete KV cache data is directly copied to the target GPU. This method has high communication overhead (large data transfer volume) but low computational overhead (the target GPU does not need to recalculate). 2. Token-based migration: Only the original input tokens of the request are transferred, which are then pre-processed by the target GPU to generate the KV cache. This method has low communication overhead (small data transfer volume) but high computational overhead (the target GPU needs to consume computing resources to regenerate).

[0095] Here, each service request to be migrated can be viewed as an "item," which can be packaged into either a key-value cache or a lexical migration "box." The optimization objective can be to select a hybrid migration scheme that minimizes the total cost (communication cost and computation cost) as much as possible, without exceeding the communication link boundary conditions (i.e., box capacity). The system can use a greedy algorithm (such as the first-fit or best-fit algorithm) to efficiently approximate the solution to this NP-hard problem. This algorithm quickly assigns a suitable migration method to each migration request and outputs an executable migration plan. The system then executes the final migration operation in parallel or serially based on the result of this plan.

[0096] In this embodiment, a global collaborative approach can be used to implement the migration operation, allowing multiple instances to use the same (communication) link and migrate requests to the same target instance. For example, refer to... Figure 3 , Figure 3 This illustration shows a schematic diagram of a migration operation provided in an embodiment of this application. Figure 3 In this scenario, GPU1 and GPU2 use an inter-server switch to migrate service requests to GPU4. If each instance performs this (request) partitioning independently without collaboration, their migration operations may exceed the communication boundaries of the link. To avoid this, each instance must consider all requests to be migrated in the system, not just its own, when running the scheduling algorithm.

[0097] It is understood that the global request migration method used in the embodiments of this application can transform the execution action into a decision-making process that seeks the optimal solution between communication and computing resources, thus ensuring the stability and high resource utilization efficiency of the large language model service during dynamic scheduling.

[0098] Reference Figure 4 In this embodiment of the application, a resource processing device for a large language model service is also provided, which includes: The detection unit 410 is used to detect the request status and required cache size of each service request of the large language model service; wherein, the request status includes new status, completed status and running status; The first response unit 420 is used to respond to a service request in a new state by creating a cache allocation task based on the cache size required by the service request. The second response unit 430 is used to create a cache release task based on the cache resources occupied by the service request when the service request is in a completed state. The third response unit 440 is used to create a cache update task corresponding to the service request in response to a service request being in a running state and a change in the cache size required by the service request. The scheduling unit 450 is used to schedule the cache resources of the large language model service according to the cache allocation task, cache release task and cache update task.

[0099] It is understandable that, such as Figure 2 The content of the resource processing method embodiments of the large language model service shown herein is applicable to the resource processing device embodiments of this large language model service. The specific functions implemented by the resource processing device embodiments of this large language model service are as follows: Figure 2 The resource processing method for the large language model service shown in the example is the same, and the beneficial effects achieved are the same as those described above. Figure 2The beneficial effects achieved by the resource processing method embodiment of the large language model service shown are also the same.

[0100] Reference Figure 5 This application provides an electronic device, including: At least one processor 510; At least one memory 520 is used to store at least one program; When at least one program is executed by at least one processor 510, the at least one processor 510 enables... Figure 2 This illustrates a resource processing method for a large language model service.

[0101] Similarly, the content of the above method embodiments is applicable to the embodiments of this electronic device. The specific functions implemented by the embodiments of this electronic device are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0102] This application embodiment also provides a computer-readable storage medium storing a program executable by a processor 510, which, when executed by the processor 510, is used to perform the resource processing method for the large language model service described above.

[0103] It is understandable that, such as Figure 2 The content of the resource processing method embodiment for a large language model service shown is applicable to the embodiment of this computer-readable storage medium. The specific functions implemented by the embodiment of this computer-readable storage medium are the same as those shown below. Figure 2 The resource processing method for a large language model service shown is the same as that implemented in this example, and the beneficial effects achieved are the same as those described above. Figure 2 The beneficial effects achieved by the resource processing method embodiment of the large language model service shown are also the same.

[0104] This application also discloses a computer program product or computer program, which includes computer instructions stored in the aforementioned computer-readable storage medium. Figure 5 The processor of the illustrated electronic device can read the computer instructions from the aforementioned computer-readable storage medium, and the processor executes the computer instructions, causing the electronic device to perform... Figure 2 The example shown illustrates a resource processing method for a large language model service.

[0105] Understandable, Figure 2 The content of the resource processing method embodiments of the large language model service shown are all applicable to this computer program product or computer program embodiment. The specific functions implemented by this computer program product or computer program embodiment are the same as those shown in the embodiments. Figure 2The resource processing method for the large language model service shown in the embodiment is the same, and the beneficial effects achieved are the same. Figure 2 The beneficial effects achieved by the resource processing method embodiment of the large language model service shown are also the same.

[0106] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this application are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and sub-operations described as part of a larger operation are executed independently.

[0107] Furthermore, although this application is described in the context of functional modules, it should be understood that, unless otherwise stated to the contrary, one or more of the functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding this application. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional technology for an engineer. Therefore, those skilled in the art can implement the application set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of this application, which is determined by the full scope of the appended claims and their equivalents.

[0108] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0109] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0110] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable storage medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0111] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0112] In the foregoing description of this specification, the references to terms such as "one embodiment," "another embodiment," or "some embodiments," etc., indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0113] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

[0114] The above is a detailed description of the preferred embodiments of this application, but this application is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A resource processing method for a large language model service, characterized in that, The method includes: The system detects the request status and required cache size of each service request of the large language model service; wherein, the request status includes new status, completed status and running status, and the cache size is the size of an independent KV cache created and maintained on the GPU for the service request; In response to the service request being in a new state, a cache allocation task is created based on the cache size required by the service request. In response to the service request being in a completed state, a cache release task is created based on the cache resources occupied by the service request; In response to the fact that the service request is in a running state and the cache size required by the service request has changed, a cache update task corresponding to the service request is established. The cache resources of the large language model service are scheduled according to the cache allocation task, the cache release task, and the cache update task; The scheduling of cache resources for the large language model service based on the cache allocation task, the cache release task, and the cache update task includes: The first migration operation to be executed is determined based on the cache release task, and the first migration operation is added to the operation cache area; Based on the cache update task, a second migration operation that needs to be performed is determined, the second migration operation is added to the operation cache area, and the operation cache area is deduplicated. The third migration operation to be executed is determined based on the cache allocation task, the third migration operation is added to the operation cache area, and the operation cache area is deduplicated. Execute all migration operations in the operation buffer; The execution of all migration operations in the operation cache includes: Boundary analysis is performed on the communication links between the cache resources of the large language model service to determine the boundary conditions; wherein, the boundary conditions are used to characterize the upper limit of the amount of data that the communication link can stably transmit per unit time. Based on the aforementioned boundary conditions, a hybrid migration two-box packing problem model is established; wherein, the hybrid migration includes migration based on key-value caching and migration based on lexical forms, and the two-box packing problem model is a packing decision model constructed by treating each service request to be migrated as an item to be packed, using migration based on key-value caching or migration based on lexical forms as packing methods, using the aforementioned boundary conditions as constraints, and minimizing the sum of communication cost and computation cost as the optimization objective; The two-box packing problem model is solved using a greedy algorithm, and the migration operation is performed based on the solution results.

2. The resource processing method for large language model services according to claim 1, characterized in that, The step of responding to a new service request by establishing a cache allocation task based on the cache size required by the service request includes: In response to the service request being in a new state, the cache size required for the service request is compared with a pre-set first cache threshold, second cache threshold, and third cache threshold; wherein the first cache threshold is greater than the second cache threshold, and the second cache threshold is greater than the third cache threshold. If the cache size required for the service request is greater than the first cache threshold, the service request is determined to be a first request, and a first allocation task is established; If the cache size required for the service request is greater than the second cache threshold and less than or equal to the first cache threshold, the service request is determined to be a second request, and a second allocation task is established. If the cache size required for the service request is greater than the third cache threshold and less than or equal to the second cache threshold, the service request is determined to be a third request, and a third allocation task is established. If the cache size required for the service request is less than or equal to the third cache threshold, the service request is determined to be a fourth request, and multiple fourth requests are combined to establish a third allocation task.

3. The resource processing method for large language model services according to claim 2, characterized in that, According to the cache allocation task, the cache resources of the large language model service are scheduled, including: For the first allocation task, a new first cache device is activated, the first allocation task is processed through the new first cache device, and at least one cached data of the second request is pulled from the activated second cache device; For the second allocation task, it is detected whether there is redundant space in each of the currently activated first cache devices to support the second allocation task. If there is redundant space to support the second allocation task, the second allocation task is processed through the first cache device. If there is no redundant space to support the second allocation task, a new second cache device is activated and the second allocation task is processed through the new second cache device. For the third allocation task, it is detected whether there is redundant space to support the third allocation task among the currently activated first cache devices. If there is redundant space to support the third allocation task, the third allocation task is processed by the first cache device. If there is no redundant space to support the third allocation task, a new third cache device is activated and the third allocation task is processed by the new third cache device. Wherein, the cached data of the service request with the largest value stored in the first cache device is greater than the first cache threshold, the cached data of the service request with the largest value stored in the second cache device is greater than the second cache threshold and less than or equal to the first cache threshold, and the cached data of the service request with the largest value stored in the third cache device is greater than the third cache threshold and less than or equal to the second cache threshold.

4. The resource processing method for large language model services according to claim 3, characterized in that, The step of detecting whether each of the currently activated first cache devices has redundant space to support the second allocation task includes: Detect the first available space of the currently active first cache device; If the first available space is greater than or equal to the cache size required by the second allocation task, it is determined that the first cache device has redundant space to support the second allocation task; If the first available space is less than the cache size required by the second allocation task, the cache data of the third and fourth requests in the currently activated first cache device are statistically reduced, and the second available space of the currently activated first cache device after reduction is determined. If the second available space is greater than or equal to the cache size required by the second allocation task, the cached data of the third and fourth requests in the currently activated first cache device are removed, and it is determined that the first cache device has redundant space to support the second allocation task. Alternatively, if the second available space is less than the cache size required by the second allocation task, it is determined that the first cache device does not have redundant space to support the second allocation task.

5. The resource processing method for large language model services according to claim 2, characterized in that, According to the cache release task, the cache resources of the large language model service are scheduled, including: Based on the cache release task, determine the target cache device and request type of the service request that needs to be released; The service requests that need to be cached are released on the target cache device, and service requests of the same type are migrated from other recently activated cache devices to the target cache device.

6. A resource processing device for a large language model service, characterized in that, The device includes: The detection unit is used to detect the request status and required cache size of each service request of the large language model service; wherein, the request status includes new status, completed status and running status, and the cache size is the size of an independent KV cache created and maintained on the GPU for the service request; The first response unit is used to respond to the service request being in a new state by establishing a cache allocation task based on the cache size required by the service request. The second response unit is used to respond to the service request being in a completed state by establishing a cache release task based on the cache resources occupied by the service request. The third response unit is used to establish a cache update task corresponding to the service request in response to the service request being in a running state and the cache size required by the service request changing. The scheduling unit is used to schedule the cache resources of the large language model service according to the cache allocation task, the cache release task, and the cache update task; The scheduling of cache resources for the large language model service based on the cache allocation task, the cache release task, and the cache update task includes: The first migration operation to be executed is determined based on the cache release task, and the first migration operation is added to the operation cache area; Based on the cache update task, a second migration operation that needs to be performed is determined, the second migration operation is added to the operation cache area, and the operation cache area is deduplicated. The third migration operation to be executed is determined based on the cache allocation task, the third migration operation is added to the operation cache area, and the operation cache area is deduplicated. Execute all migration operations in the operation buffer; The execution of all migration operations in the operation cache includes: Boundary analysis is performed on the communication links between the cache resources of the large language model service to determine the boundary conditions; wherein, the boundary conditions are used to characterize the upper limit of the amount of data that the communication link can stably transmit per unit time. Based on the aforementioned boundary conditions, a hybrid migration two-box packing problem model is established; wherein, the hybrid migration includes migration based on key-value caching and migration based on lexical forms, and the two-box packing problem model is a packing decision model constructed by treating each service request to be migrated as an item to be packed, using migration based on key-value caching or migration based on lexical forms as packing methods, using the aforementioned boundary conditions as constraints, and minimizing the sum of communication cost and computation cost as the optimization objective; The two-box packing problem model is solved using a greedy algorithm, and the migration operation is performed based on the solution results.

7. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the resource processing method for the large language model service as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the resource processing method for the large language model service as described in any one of claims 1 to 5.

Citation Information

Patent Citations

  • Memory monitoring method and device, equipment and medium

    CN118585401A