Resource processing method and device for large language model service, equipment and storage medium

Abstract

The invention discloses a resource processing method, device and equipment for a large language model service and a storage medium, which are used for carrying out dynamic resource scheduling by detecting the request state of each service request in the large language model service and the required cache size thereof. Specifically, for a request in a newly-added state, a cache allocation task is established according to the size of a cache required by the request; for the request in the completion state, establishing a cache release task according to the cache resources occupied by the request; establishing a corresponding cache updating task for the request that the state in operation and the size of the required cache are changed; and the system carries out unified scheduling on cache resources of the large language model service according to the cache allocation tasks, the cache release tasks and the cache update tasks so as to optimize the use of memory resources. According to the method and the device, the cache resources of the large language model service can be efficiently and flexibly managed, so that the utilization efficiency of the cache resources is improved. The technical scheme of the invention can be widely applied to the technical field of artificial intelligence.

Description

[0001] Cross Reference to Related Patent Applications This application is claiming priority to U.S. Provisional Patent Application No. 63 / 704,563, filed October 8, 2024, the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the field of artificial intelligence, in particular to a resource processing method and device for large language model service, equipment and storage medium. BACKGROUND

[0003] The ability of large language models (LLMs) to understand and generate human-like text has made them a central component of artificial intelligence. To improve the inference efficiency of LLMs on GPUs, key-value (KV) caching is one of the key modules that stores the keys and values of all previous tokens in GPU memory during each LLM inference process to avoid redundancy and repeated calculations. Despite this advantage, GPU memory faces some problems when processing long contexts. Unlike model weights, the size of KV cache grows with the increase of sequence length and batch size, requiring a large amount of GPU memory. As the demand for longer sequence lengths (and larger batch sizes) grows, the size problem of KV cache becomes increasingly prominent.

[0004] In related technologies, existing KV cache management schemes mainly focus on compressing KV cache or offloading KV cache to CPU memory. The use effect of these implementation methods is general, and compressing KV cache often needs to be quantized or sparsified, which inevitably reduces the performance of large language models; while offloading KV cache to CPU memory is limited by the I / O bottleneck between CPU memory and GPU, and the efficiency is low.

[0005] In summary, the related technology lacks a better storage strategy for the cache data of large language models, resulting in low utilization efficiency and high overhead of cache resources. SUMMARY

[0006] The embodiments of the present application provide a resource processing method and device for large language model service, equipment and storage medium, which can realize efficient and flexible management of the cache resources of large language model service, thereby improving the utilization efficiency of cache resources.

[0007] One aspect of the embodiments of the present application provides a resource processing method for large language model service, the method comprising: detecting the request state and required cache size of each service request of the large language model service; wherein the request state comprises a new state, a completed state and a running state; in response to the service request being in the new state, establishing a cache allocation task according to a cache size required by the service request; in response to the service request being in the completed state, establishing a cache release task according to cache resources occupied by the service request; in response to the service request being in the running state and a change in the cache size required by the service request, establishing a cache update task corresponding to the service request; According to the cache allocation task, the cache release task and the cache update task, the cache resources of the large language model service are scheduled.

[0008] Specifically, in some embodiments, in response to the service request being in the new state, a cache allocation task is established according to the cache size required by the service request, comprising: in response to the service request being in the new state, comparing the cache size required by the service request through a pre-set first cache threshold, a second cache threshold and a third cache threshold; wherein the first cache threshold is greater than the second cache threshold, and the second cache threshold is greater than the third cache threshold; in response to the cache size required by the service request being greater than the first cache threshold, determining that the service request is a first request, and establishing a first allocation task; in response to the cache size required by the service request being greater than the second cache threshold and less than or equal to the first cache threshold, determining that the service request is a second request, and establishing a second allocation task; in response to the cache size required by the service request being greater than the third cache threshold and less than or equal to the second cache threshold, determining that the service request is a third request, and establishing a third allocation task; in response to the cache size required by the service request being less than or equal to the third cache threshold, determining that the service request is a fourth request, and combining a plurality of fourth requests to establish a third allocation task.

[0009] Specifically, in some embodiments, according to the cache allocation task, the cache resources of the large language model service are scheduled, comprising: for the first allocation task, activating a new first cache device, processing the first allocation task through the new first cache device, and pulling at least one cache data of a second request from an activated second cache device; For the second allocation task, it is detected whether the currently activated first cache device has redundant space supporting the second allocation task, if the first cache device has redundant space supporting the second allocation task, the second allocation task is processed through the first cache device, if the first cache device does not have redundant space supporting the second allocation task, a new second cache device is activated, and the second allocation task is processed through the new second cache device. For the third allocation task, it is detected whether the currently activated first cache device has redundant space supporting the third allocation task, if the first cache device has redundant space supporting the third allocation task, the third allocation task is processed through the first cache device, if the first cache device does not have redundant space supporting the third allocation task, a new third cache device is activated, and the third allocation task is processed through the new third cache device. Among them, the service request maximum cache data stored in the first cache device is greater than the first cache threshold, the service request maximum cache data stored in the second cache device is greater than the second cache threshold and less than or equal to the first cache threshold, and the service request maximum cache data stored in the third cache device is greater than the third cache threshold and less than or equal to the second cache threshold.

[0010] Specifically, in some embodiments, the detection of whether the currently activated first cache device has redundant space supporting the second allocation task includes: Detecting the first available space of the currently activated first cache device; If the first available space is greater than or equal to the cache size required by the second allocation task, it is determined that the first cache device has redundant space supporting the second allocation task; If the first available space is less than the cache size required by the second allocation task, the cache data of the third request and the fourth request in the currently activated first cache device is counted and pruned, and the second available space of the currently activated first cache device after pruning is determined; If the second available space is greater than or equal to the cache size required by the second allocation task, the cache data of the third request and the fourth request in the currently activated first cache device is pruned, and it is determined that the first cache device has redundant space supporting the second allocation task, or if the second available space is less than the cache size required by the second allocation task, it is determined that the first cache device does not have redundant space supporting the second allocation task.

[0011] Specifically, in some embodiments, according to the cache release task, the cache resources of the large language model service are scheduled, including: According to the cache release task, a target cache device and a request type to which a service request requiring cache release belongs are determined; The cache release task is performed on the target cache device, and service requests belonging to the request type are migrated from other recently activated cache devices to the target cache device.

[0012] Specifically, in some embodiments, the scheduling of the cache resources of the large language model service according to the cache allocation task, the cache release task and the cache update task comprises: A first migration operation required to be performed is determined according to the cache release task, and the first migration operation is added to an operation cache area; A second migration operation required to be performed is determined according to the cache update task, the second migration operation is added to the operation cache area, and the operation cache area is de-duplicated; A third migration operation required to be performed is determined according to the cache allocation task, the third migration operation is added to the operation cache area, and the operation cache area is de-duplicated; All migration operations in the operation cache area are executed.

[0013] Specifically, in some embodiments, The execution of all migration operations in the operation cache area comprises: A boundary condition is determined by performing boundary analysis on communication links between cache resources of the large language model service; Based on the boundary condition, a two-bin bin packing problem model of hybrid migration is established; wherein the hybrid migration comprises migration based on key-value cache form and migration based on word form; The two-bin bin packing problem model is solved by a greedy algorithm, and the migration operation is executed according to the result of the solution.

[0014] On the other hand, the embodiments of the present application provide a resource processing device of a large language model service, the device comprises: A detection unit is configured to detect a request state and a required cache size of each service request of a large language model service; wherein the request state comprises an added state, a completed state and a running state; A first response unit is configured to, in response to the service request being in the added state, establish a cache allocation task according to the required cache size of the service request; A second response unit is configured to, in response to the service request being in the completed state, establish a cache release task according to cache resources occupied by the service request; a third response unit configured to, in response to the service request being in a running state and a change in the cache size required by the service request, establish a cache update task corresponding to the service request; a scheduling unit configured to schedule cache resources of the large language model service according to the cache allocation task, the cache release task, and the cache update task.

[0015] In another aspect, an electronic device is provided, including a processor and a memory. The memory is configured to store a computer program. The processor executes the computer program to implement the resource processing method of the large language model service.

[0016] In another aspect, a computer readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement the resource processing method of the large language model service.

[0017] In another aspect, a computer program product is provided, which includes a computer program stored in a computer readable storage medium. A processor of an electronic device reads the computer program from the computer readable storage medium. The processor executes the computer program to cause the electronic device to implement the resource processing method of the large language model service.

[0018] The embodiments of the present application at least have the following beneficial effects: The present application provides a resource processing method, device, equipment and storage medium of a large language model service, aiming to improve the problem that cache data of a large language model consumes a large amount of memory as the sequence length and batch size grow. The present application detects the request state (including new, complete and running state) of each service request in the large language model service and the cache size required by the service request, and dynamically schedules resources accordingly. Specifically, for a request in a new state, a cache allocation task is established according to the cache size required by the request; for a request in a complete state, a cache release task is established according to the cache resources occupied by the request; for a request in a running state and a change in the cache size required by the request, a corresponding cache update task is established; finally, the system schedules the cache resources of the large language model service according to the cache allocation task, the cache release task and the cache update task, to optimize the use of memory resources. The present application can realize efficient and flexible management of cache resources of the large language model service through a state-driven dynamic scheduling mechanism, thereby improving the utilization efficiency of cache resources and being conducive to guaranteeing the service performance of the large language model. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings are used to provide a further understanding of the technical solutions of the present application, constitute a part of the specification, and are used to explain the technical solutions of the present application together with the embodiments of the present application, and do not constitute a limitation on the technical solutions of the present application.

[0020] Figure 1 A system architecture diagram applied to a resource processing method for providing a large language model service in embodiments of the present application; Figure 2 A flowchart of the resource processing method for providing a large language model service in embodiments of the present application; Figure 3 A schematic diagram of a migration operation provided in embodiments of the present application; Figure 4 A structural block diagram of a resource processing device for providing a large language model service in embodiments of the present application; Figure 5 A structural block diagram of an electronic device provided in embodiments of the present application. DETAILED DESCRIPTION

[0021] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0022] It can be understood that the terms "first", "second", and the like used in the present application can be used herein to describe various concepts, but unless specifically stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another.

[0023] The terms "at least one", "multiple", "each", "any" and the like used in the present application include one, two or more than two, multiple includes two or more than two, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0024] Before the technical solutions of the present application are introduced, the technical terms involved in the embodiments of the present application are first introduced as follows: 1) Artificial intelligence is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. Artificial intelligence is a branch of computer science. Artificial intelligence aims to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. Artificial intelligence can simulate the information process of human consciousness and thinking. Artificial intelligence is also a theory, method, technology and application system that uses digital computers or digital computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to achieve the best results.

[0025] 2) Large Language Model (LLM) is a deep learning-based artificial intelligence program. It learns the patterns, grammar, knowledge and context relationships of human language by training on massive amounts of text data, and its core capability is to understand and generate natural language.

[0026] The ability of Large Language Model (LLM) to understand and generate human-like text has made it a core component of artificial intelligence. To improve the inference efficiency of LLM on GPU, key-value (KV) cache is one of the key modules that stores the keys and values of all previous tokens in GPU memory during each LLM inference process to avoid redundancy and repeated calculations. Despite this advantage, GPU memory faces some problems when processing long contexts. Unlike model weights, the size of KV cache grows with the increase of sequence length and batch size, requiring a large amount of GPU memory. As the demand for longer sequence lengths (and larger batch sizes) grows, the size problem of KV cache becomes increasingly prominent.

[0027] In related technologies, existing KV cache management solutions mainly focus on compressing KV cache or offloading KV cache to CPU memory. The use effect of these implementation methods is general. Compressing KV cache often needs to be quantized or sparsified, which inevitably reduces the performance of large language models; while offloading KV cache to CPU memory is limited by the I / O bottleneck between CPU memory and GPU, and the efficiency is low.

[0028] Therefore, the embodiments of the present application provide a resource processing method, device and equipment for large language model service and a storage medium, aiming at solving the problem that the cache data of the large language model consumes a large amount of memory as the sequence length and batch size increase. The present application detects the request state (including new, completed and running state) of each service request in the large language model service and the cache size required by the service request, and dynamically schedules resources according to the request state. Specifically, for the request in the new state, a cache allocation task is established according to the required cache size; for the request in the completed state, a cache release task is established according to the occupied cache resources; for the request in the running state and the request whose required cache size changes, a corresponding cache update task is established; finally, the system schedules the cache resources of the large language model service according to the cache allocation task, the cache release task and the cache update task, so as to optimize the use of memory resources. Through the state-driven dynamic scheduling mechanism, the present application can realize efficient and flexible management of the cache resources of the large language model service, thereby improving the utilization efficiency of the cache resources and being beneficial to guaranteeing the service performance of the large language model.

[0029] System architecture and scenario of the present application Please refer to Figure 1 , Figure 1 A system architecture diagram to which a resource processing method for large language model service provided by the embodiments of the present application is applied is shown, which includes a terminal device 140, an Internet 130, a gateway 120, a background server 110, etc.

[0030] In the embodiments of the present application, the terminal device 140 can include desktop computers, laptop computers, PDAs (personal digital assistants), mobile phones, vehicle-mounted terminals, home theater terminals, special-purpose terminals, etc. In addition, it can be a single device or a collection of multiple devices. The terminal device 140 can communicate with the Internet 130 in a wired or wireless manner to exchange data.

[0031] The background server 110 refers to a computer system that can provide certain services to the terminal device 140. Compared with ordinary terminal devices 140, the background server 110 requires higher stability, security and performance. The background server 110 can be a high-performance computer in a network platform, a cluster of multiple high-performance computers, a part of a high-performance computer (such as a virtual machine), a combination of parts of multiple high-performance computers (such as virtual machines), etc.

[0032] The gateway 120 is also called an internetworking connector or a protocol converter. The gateway implements network interconnection at the transport layer and is a computer system or device that acts as a conversion role. The gateway is a translator between two systems that use different communication protocols, data formats or languages, or even have completely different architectures. At the same time, the gateway can also provide filtering and security functions. The messages sent by the terminal device 140 to the background server 110 are sent to the corresponding background server 110 through the gateway 120. The messages sent by the background server 110 to the terminal device 140 are also sent to the corresponding terminal device 140 through the gateway 120.

[0033] The background server 110 can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, and basic cloud computing services such as big data and artificial intelligence platforms.

[0034] The resource processing method of the large language model service provided by the embodiments of the present application can be executed separately on the background server 110 side, or on the terminal device 140 side, or based on data interaction between the terminal device 140 and the background server 110.

[0035] Of course, it can be understood that the implementation environment corresponding to the method in the embodiments of the present application is not limited to Figure 1 the case shown, and those skilled in the related art of the present application can flexibly select a specific implementation environment according to actual needs, and the present application does not limit this.

[0036] General description of the embodiments of the present application Please refer to Figure 2 , Figure 2 A flowchart of a resource processing method of a large language model service according to an embodiment of the present application is shown. As Figure 2 shown, according to the resource processing method of the large language model service according to an embodiment of the present application, the following steps are included but not limited to: Step 210, detecting the request state and the required cache size of each service request of the large language model service; wherein the request state includes an added state, a completed state and a running state; Step 220, in response to the service request being in the added state, establishing a cache allocation task according to the required cache size of the service request; Step 230, in response to the service request being in the completed state, establishing a cache release task according to the cache resources occupied by the service request; Step 240, in response to the service request being in a running state and the cache size required by the service request changing, establishing a cache update task corresponding to the service request; Step 250, scheduling the cache resources of the large language model service according to the cache allocation task, the cache release task and the cache update task.

[0037] In the embodiments of the present application, an efficient large language model service resource processing method is provided, which can dynamically and finely manage the life cycle of each service request, trigger corresponding scheduling tasks by monitoring the changes of request state and resource demand in real time, and realize global optimization scheduling of KV cache resources in a multi-GPU cluster, so as to save the number of GPUs and improve resource utilization.

[0038] Specifically, in the embodiments of the present application, the large language model service can use GPU as a computing and storage hardware device. In the large language model service, GPU can be used as a computing engine and a memory container. The large language model itself is a huge neural network, and each token (word element) generated by the large language model needs to perform a large matrix operation. The thousands of cores of GPU and its highly parallel architecture make it very suitable for this computing mode, so it can undertake most of the computing tasks. In addition, the large language model runtime needs to occupy a large amount of memory, such as pre-trained model parameters, key-value cache, etc. For pre-trained model parameters, its size is fixed; while the key-value cache is dynamic. In order to speed up the generation process, the large language model service will cache the Key and Value vectors of all tokens (word elements) generated before each request, avoiding repeated calculation. The size of the key-value cache generally depends on the sequence length of the request, i.e. the context length and the generated length. Since the large language model service generally processes high-concurrency requests, the number of requests will also affect the overall size of the key-value cache.

[0039] In the embodiments of the present application, a large language model service platform needs to provide request services for a large number of users at the same time. Each user request (such as a dialogue) is represented as a service request in the system. For each service request, an independent KV Cache (cache data of the service request) will be created and maintained on the GPU. It is easy to understand that a large number of concurrent service requests will competitively consume limited GPU memory resources. The service request is dynamic and has a life cycle (new, running, completed), and its resource demand (KV Cache size) will continue to grow as the generation proceeds. Therefore, how to efficiently place thousands of dynamically changing service requests and their cache data in a limited number of multi-GPU clusters is the core technical problem to be solved by the present application.

[0040] For the above technical problems, there are mainly the following difficulties: for the demand side, the arrival, completion and release of service requests are often unpredictable, which can cause some GPUs to run out of memory due to processing multiple long context requests, while other GPUs can have idle memory due to processing short requests; in addition, after small requests in some GPUs are released, many small and discontinuous free fragments are left in the GPU memory, which cannot be utilized by large requests, resulting in reduced utilization efficiency of the GPU. For the supply side, GPU itself is an expensive and limited hardware resource, and simply buying more GPUs to cope with peak load will cause huge cost waste and resource idling.

[0041] To this end, the technical solution in the embodiments of the present application does not change the configuration of the GPU device, but uses an optimized scheduling algorithm to realize efficient utilization of GPU resources by dynamically considering the cache load of service requests and comprehensive high-cost request migration.

[0042] The method in the embodiments of the present application will be described in detail below in combination with specific steps.

[0043] In step 210, the request state and required cache size of each service request of the large language model service are detected.

[0044] In the embodiments of the present application, the large language model service refers to a software platform deployed on a cloud or local server that can provide large language model functions (such as dialogue, text generation, code completion, etc.) to multiple users simultaneously. For example, an online dialogue service similar to ChatGPT or an internally deployed similar system. In the application of the present application, the large language model service is the object of management and optimization.

[0045] For the large language model service, each user can initiate a service request. An independent interaction task initiated by a user to the large language model service is a service request. For example, when the user opens a chat window and inputs a question, a service request is generated. The service request contains the user's input (prompt word) and requires the model to generate a corresponding reply. In the embodiments of the present application, the service request is the basic unit of resource scheduling, and each service request will independently occupy computing and memory resources on the GPU.

[0046] For each service request, there is a corresponding request state, which describes a specific stage of a service request in its life cycle. Specifically, the request state can include a new state, a completed state, and a running state. Among them, the new state indicates that a brand new service request has just arrived in the system and is ready to be processed. At this time, the system needs to allocate resources for it. The completed state indicates that a service request has been processed (e.g., the model has generated a complete reply and returned to the user). At this time, the resources it occupies can be recycled for other new requests. The running state indicates that a service request is being processed by the GPU, for example, the model is generating Token-by-Token output.

[0047] In addition, for service requests in the new state and the running state, the application embodiments also detect the required cache size, that is, the memory size occupied by the Key-Value Cache. For each service request, the large language model service stores the Key and Value vectors of all previously generated Tokens during the generation process to avoid repeated calculations when generating new Tokens. The size of this cache is not fixed and will change with the processing process of the service request.

[0048] In step 220, when the system detects a newly arrived service request (i.e., the service request is in the "new state"), this step is triggered. Its core task is to create a "cache allocation task" for this new request, aiming to find and allocate a suitable execution location for it in the multi-GPU cluster. This process is not simply "finding a GPU with free memory", but an intelligent decision based on fine-grained management and global optimization strategies.

[0049] Illustratively, in the embodiments of the present application, the required cache size of the service request can be evaluated in real time. Generally speaking, the initial cache size of the service request is mainly determined by the length of the user input (prompt word), which is not only a quantitative indicator of memory demand, but also a basis for classifying requests. In the embodiments of the present application, it can be accurately classified as T (extremely small), S (small), M (medium) or L (large) request according to the preset threshold (e.g., the ratio to the total memory of a single GPU device). This classification is the cornerstone of the entire allocation strategy, aiming to achieve differentiated resource scheduling.

[0050] For service requests in the new state, the corresponding decision logic is encapsulated into a specific "cache allocation task". This task explicitly includes the instruction "allocate a specific request to a specific GPU". This task is output and added to the scheduling queue of the system, and is implemented by the subsequent scheduling execution module (step 250), thereby completing the closed loop from decision to action.

[0051] Step 230 is activated when the system detects that a certain service request has been processed and enters the "completed state". The core task of step 230 is not just to simply reclaim the memory resources released by the request, but to create a "cache release task" aimed at proactively triggering a global resource restructuring and optimization as the service request leaves. This is a process that turns resource recovery into an opportunity for system optimization.

[0052] Specifically, the starting point of step 230 is to identify and confirm the specific situation of the "resource hole" released by the completed request, including its location (which GPU) and its size (released cache space). In the embodiments of the present application, the system does not passively wait for new requests to fill this hole, because that may lead to uneven load and resource fragmentation. Instead, it initiates an active rebalancing process based on the cache release task.

[0053] For example, if the last request on a GPU is completed, the ideal state is that the entire GPU should be released and taken offline to save costs. However, more commonly, there are other requests on the GPU, and at this time the system will select one or more similar requests from other GPUs with similar loads to migrate to fill the newly available idle space.

[0054] In step 240, when the system monitors a service request that is in the "running" state, the size of its key-value cache (KV Cache) increases (i.e., the model is generating output, and each new Token generated will cause the cache to expand), this step will be triggered. The core task of step 240 is to create a "cache update task" aimed at solving the problem that the request may not match the current GPU resources due to resource demand expansion, thereby avoiding local load overload and maintaining global balance.

[0055] In the embodiments of the present application, the system monitors the cache occupancy of each running service request in real time, and once it detects that the size increases (which is a necessary result of the generation process of large language models), it will immediately evaluate the impact of this change. At this time, the type of the request may change, for example, a request that was originally classified as "medium" may expand its cache to be redefined as a "large" request as the generated content increases. This type of transition means that its original accommodation place (e.g., a GPU dedicated to handling small and medium requests) may no longer meet its new resource needs, or staying in the original place will cause the GPU to overload and affect other requests on the same GPU.

[0056] Therefore, the "cache update task" established in the embodiments of the present application is essentially a relocation plan. Instead of simply expanding the memory at the original location, the task plans a complex migration operation: first, the request is "released" from its current GPU (but its state remains running, not completed), and then it is treated as a "new" request but with a changed type, and the complete allocation process is re-executed. This means that the system will re-evaluate the global resource state for this "updated" request and find a new, more suitable GPU for it (for example, a request that has just been upgraded from M to L is migrated to a GPU that specializes in handling L requests). In extreme cases, if the cache growth of the request directly causes the current GPU to overload, the task may also plan a more thorough operation, i.e., release and reallocate all requests on the GPU for a complete overhaul.

[0057] In step 250, the tasks generated in the previous steps, cache allocation tasks, cache release tasks, and cache update tasks, are integrated to perform global and dynamic scheduling of cache resources in the multi-GPU cluster, thereby converting discrete task requests into precise manipulation of hardware resources, and ultimately achieving efficient and stable loading of dynamically changing large language model service loads with fewer GPUs.

[0058] It can be understood that the resource processing method for large language model services provided in the embodiments of the present application detects the request state (including new, completed, and running states) and the required cache size of each service request in the large language model service, and performs dynamic resource scheduling accordingly. Specifically, for a request in the new state, a cache allocation task is established according to the required cache size; for a request in the completed state, a cache release task is established according to the occupied cache resources; for a request in the running state and with a changed required cache size, a corresponding cache update task is established; finally, the system performs unified scheduling of the cache resources of the large language model service according to these cache allocation tasks, cache release tasks, and cache update tasks to optimize the use of memory resources. Through the state-driven dynamic scheduling mechanism, the present application can achieve efficient and flexible management of the cache resources of the large language model service, thereby improving the utilization efficiency of the cache resources and being beneficial to ensuring the service performance of the large language model.

[0059] Specifically, in some embodiments, in response to the service request being in the new state, a cache allocation task is established according to the required cache size of the service request, including: In response to the service request being in the new state, a comparison is made among a first cache threshold value, a second cache threshold value, a third cache threshold value, and a cache size required by the service request, wherein the first cache threshold value is greater than the second cache threshold value, and the second cache threshold value is greater than the third cache threshold value; In response to the cache size required by the service request being greater than the first cache threshold value, it is determined that the service request is a first request, and a first allocation task is established; In response to the cache size required by the service request being greater than the second cache threshold value and less than or equal to the first cache threshold value, it is determined that the service request is a second request, and a second allocation task is established; In response to the cache size required by the service request being greater than the third cache threshold value and less than or equal to the second cache threshold value, it is determined that the service request is a third request, and a third allocation task is established; In response to the cache size required by the service request being less than or equal to the third cache threshold value, it is determined that the service request is a fourth request, and a third allocation task is established by combining multiple fourth requests.

[0060] In the embodiments of the present application, when establishing a cache allocation task, the service request can be classified based on the cache size required by the service request. For example, in the embodiments of the present application, the service request can be classified into four categories: L (large), M (medium), S (small), and T (extremely small). Of course, in some embodiments, other numbers of categories can be classified, such as merging M (medium) and S (small) into one category.

[0061] For example, in the embodiments of the present application, the classified cache threshold values can be set in advance. For example, the total memory capacity of a single GPU is C, the cache size required by the service request is set in the interval (C / 2, C] as L class request, in the interval (C / 3, C / 2] as M class request, in the interval (C / 4, C / 3] as S class request, and in the interval (C / 8, C / 4] as T class request. For the cache size required being less than C / 4, multiple are merged into one combination, the combination is located between (C / 8, C / 4], and the whole is also classified as T class request. In the above embodiments, multiple cache threshold values are set. In the embodiments of the present application, the number of set cache threshold values can be arbitrary, which is not limited by the present application.

[0062] Correspondingly, in the embodiments of the present application, the GPU can be classified according to the maximum request type it processes: if the service request with the largest cached data in the GPU belongs to L, M, S or T type, it is marked as L, M, S or T type GPU. In the embodiments of the present application, the priority relationship of GPU j to GPU j' is also defined, which is jointly determined by the workload of GPU j' (such as the number of requests and the idle GPU memory) and the distance between GPU j and GPU j'. For example, GPU j' that processes fewer requests, has more GPU memory or is located in the same physical machine as GPU j will be assigned a higher priority.

[0063] For example, in some embodiments, three cache thresholds can be set, C / 2, C / 4 and C / 8, respectively, denoted as the first cache threshold, the second cache threshold and the third cache threshold. If the cache size required by a certain service request is greater than the first cache threshold, it is determined to belong to the first request (i.e. L type request), and the first allocation task is established for it; if the cache size required is greater than the second cache threshold and less than or equal to the first cache threshold, it is determined to belong to the second request (i.e. M type or S type request), and the second allocation task is established for it; if the cache size required is greater than the third cache threshold and less than or equal to the second cache threshold, it is determined to belong to the third request (i.e. T type request), and the third allocation task is established for it. For service requests with a required cache size less than the third cache threshold, they are identified as fourth requests, and multiple fourth requests can be packaged to establish the third allocation task.

[0064] In the embodiments of the present application, this hierarchical task design better avoids the additional overhead generated by frequent scheduling for a large number of micro requests, thereby significantly improving the overall scheduling efficiency and resource utilization of the system.

[0065] The method in the embodiments of the present application dynamically manages the service requests of large language models in a multi-GPU cluster through the cache allocation task Allocate(i), the cache release task Depart(i) and the cache update task Update(i), and the ultimate goal is to maximize GPU utilization and minimize the total number of GPUs required.

[0066] The overall strategy contained in the present application is as follows: Priority-based classification scheduling: The basis of the algorithm of the present application is to divide service requests into T (extremely small), S / M (small / medium) and L (large) three categories according to the required cache size, and the GPU is also classified accordingly. This allows the system to tailor allocation strategies for requests of different sizes, avoiding large requests from blocking small requests, thereby reducing memory fragmentation.

[0067] The clever use of "most recently activated" GPUs: The algorithm frequently uses the concept of "most recently activated" GPUs, which is actually a load balancing and resource pooling strategy. Prioritizing the allocation of new service requests to "most recently activated" GPUs of the same type allows a smaller number of GPUs to be fully utilized, thus freeing up other GPUs and enabling dynamic scaling of the number of GPUs.

[0068] Cascaded Re-allocation: Any operation (allocate, release, update) can trigger a series of cascaded "move" and "re-allocate" operations. It ensures that after each system state change, requests are proactively rearranged to the optimal GPU, rather than passively accepting a fragmented state, thus always maintaining a near-optimal global allocation.

[0069] The following is a detailed introduction and explanation: Specifically, in some embodiments, scheduling cache resources for the large language model service according to cache allocation tasks includes: For the first allocation task, activate a new first cache device, process the first allocation task through the new first cache device, and pull at least one cached data of the second request from the activated second cache device; For the second allocation task, check whether there is redundant space in each of the currently activated first cache devices to support the second allocation task. If there is redundant space to support the second allocation task, the second allocation task is processed through the first cache device. If there is no redundant space to support the second allocation task, activate a new second cache device and process the second allocation task through the new second cache device. For the third allocation task, check whether there is redundant space in each of the currently activated first cache devices that supports the third allocation task. If there is redundant space that supports the third allocation task, the third allocation task is processed through the first cache device. If there is no redundant space that supports the third allocation task, activate a new third cache device and process the third allocation task through the new third cache device. Among them, the cached data of the service request with the largest value stored in the first cache device is greater than the first cache threshold, the cached data of the service request with the largest value stored in the second cache device is greater than the second cache threshold and less than or equal to the first cache threshold, and the cached data of the service request with the largest value stored in the third cache device is greater than the third cache threshold and less than or equal to the second cache threshold.

[0070] In the embodiments of the present application, for a newly-arrived service request, different cache allocation strategies can be executed according to the type of the allocation task corresponding to the service request. Here, the cache devices can be divided according to the following rules: the maximum cache data of the service requests stored in the first cache device is greater than a first cache threshold, the maximum cache data of the service requests stored in the second cache device is greater than a second cache threshold and less than or equal to the first cache threshold, and the maximum cache data of the service requests stored in the third cache device is greater than a third cache threshold and less than or equal to the second cache threshold.

[0071] In other words, the type of the cache device is determined according to the type of the maximum cache data of the service requests carried by the cache device. For example, if a cache device stores cache data of L (large) class service requests, it belongs to the first cache device; if a cache device stores cache data of S / M (small / medium) class service requests and does not store cache data of L (large) class service requests, it belongs to the second cache device; and if a cache device stores cache data of T (tiny) class service requests and does not store cache data of L (large) class and S / M (small / medium) class service requests, it belongs to the third cache device.

[0072] Specifically, for a service request (i.e., a first request) requiring L (large) class cache resources, the corresponding task is a first allocation task. Since the service request requires more cache resources, a new first cache device is activated to process the first allocation task. In addition, in the embodiments of the present application, an attempt is made to move cache data of an S / M class service request from an S / M-GPU (i.e., a second cache device) to fill the new first cache device as much as possible, and to fill the vacancy of the source GPU with cache data of other S / M class service requests. In this way, the load can be rearranged, and the opportunity of allocating L class requests (i.e., first requests) to new GPUs can be used to rebalance the load of S / M class requests (i.e., second requests) in the system, which can promote the emptying of a certain S / M-GPU to release.

[0073] For a service request (i.e., a second request) requiring S / M class cache resources, an L-GPU (i.e., a first cache device) capable of accommodating the service request is searched first. If the L-GPU exists, the cache data of the second service request can be directly placed in the first cache device to process a second allocation task. If the L-GPU does not exist, a new second cache device can be created to process the second allocation task.

[0074] For the service request (i.e. the third request or the combined fourth request) requiring the cache resource of the T class, a L-GPU (i.e. the first cache device) that can accommodate the service request is searched first, and if the L-GPU exists, the service request can be inserted into a free corner of the first cache device. Since the memory gap of the first cache device is large, there is a high possibility that a redundant space exists. In this way, the fragmented memory utilization rate can be improved, and a new cache device is not activated for a small request. If the first cache device does not have a redundant space supporting the third allocation task, a new third cache device is activated to process the third allocation task, so that small requests can be managed centrally.

[0075] Specifically, in some embodiments, detecting whether a currently activated first cache device has a redundant space supporting a second allocation task includes: detecting a first available space of the currently activated first cache device; if the first available space is greater than or equal to a cache size required by the second allocation task, it is determined that the first cache device has a redundant space supporting the second allocation task; if the first available space is less than the cache size required by the second allocation task, cached data of third requests and fourth requests in the currently activated first cache device is counted and pruned, and a second available space of the currently activated first cache device after pruning is determined; if the second available space is greater than or equal to the cache size required by the second allocation task, the cached data of the third requests and the fourth requests in the currently activated first cache device is removed, and it is determined that the first cache device has a redundant space supporting the second allocation task, or if the second available space is less than the cache size required by the second allocation task, it is determined that the first cache device does not have a redundant space supporting the second allocation task.

[0076] In the embodiments of the present application, for the service request (i.e. the second request) of the S / M class, when searching for a suitable storage space, not only the current free memory (the first available space) of the first cache device can be checked, but also a space reorganization method can be used to create a space for accommodating the medium-sized service request.

[0077] Specifically, when detecting whether a currently activated first cache device has a redundant space supporting a second allocation task, the available space of the currently activated first cache device can be directly detected first, which is recorded as a first available space. If the current first available space itself is sufficient to meet the demand of the second allocation task, it is immediately confirmed that the first cache device has the required redundant space, and the request can be prepared to be received.

[0078] If the first available space is insufficient, the system does not immediately give up the first cache device, but starts a more aggressive space recycling process. At this time, the system detects the third request and the fourth request with relatively low priority that already exist on the first cache device, and then performs statistical pruning on the cache data of these requests, and determines the available space of the first cache device after the cache data is pruned, which is recorded as the second available space.

[0079] If the second available space after pruning is greater than or equal to the cache size required by the second allocation task, it means that eliminating low-priority requests can free up enough space for the second request. At this time, the cache data of the third request and the fourth request in the currently activated first cache device can be formally eliminated, and it is determined that the first cache device has redundant space to support the second allocation task. If the second available space is less than the cache size required by the second allocation task, it means that even if the low-priority request is eliminated, the space in the first cache device is still insufficient, which means that there is no redundant space at this time. The system will find other targets (such as selecting another GPU or performing other scheduling operations) for the second allocation task.

[0080] In the embodiments of the present application, the space detection and optimization logic is implemented by layering and gradually deepening the related task processing. It preferentially attempts zero-cost direct allocation, and if it is not supported, it tries to migrate small requests with low priority as much as possible to free up space for medium requests, thereby maximizing the utilization of GPU resources and the scheduling efficiency of the overall system.

[0081] Specifically, in some embodiments, according to the cache release task, the cache resources of the large language model service are scheduled, including: According to the cache release task, determining the target cache device and the request type of the service request that needs to be released. Releasing the cache of the service request that needs to be released on the target cache device, and migrating the service request belonging to the request type from the other cache device that is recently activated to the target cache device.

[0082] In the embodiments of the present application, for the cache release task, when the system is executed, its operation is not as simple as only clearing the cache data of the completed request on the target GPU (i.e., the "target cache device"). On the contrary, it regards this as an opportunity to optimize the overall system resource layout.

[0083] Specifically, the system will first identify two key information according to the release task, which GPU (i.e., the target cache device) and which type of request (request type) on the GPU completes the task and is about to release its cache resources. Then, after completing the specified cache resource release on the target cache device, the application will not passively wait for a new service request to fill the suddenly idle resources on the target cache device, but will actively migrate a request of the same type from the other GPU that was recently activated to handle the same type of request. For example, if a medium-sized request is released on the target cache device, the system will select a medium-sized request from another GPU that was recently activated to handle medium-sized requests to migrate to fill the vacancy. In this way, by concentrating as many similar requests as possible on fewer GPUs, some GPUs can be completely emptied, ensuring that cluster resources can be used efficiently and avoiding resource fragmentation and uneven load due to request completion.

[0084] For cache update tasks, generally, when the cache data of a service request grows significantly, causing a change in its type, such as a T-class request becoming an S-class request or an M-class request becoming an L-class request, a new scheduling is needed. In specific scheduling, the application embodiment will remove the cache data of the service request that has changed from the original cache device (this process may trigger rebalancing of the cache device), and then perform the corresponding cache allocation task logic as a new task for the cache data of the service request that has changed, thereby realizing migration of the cache data.

[0085] In some cases, for a cache device storing an L-class request, it may be overloaded due to the growth of the cache data size of the service request, in which case all other requests on the GPU can be released and reallocated.

[0086] In the embodiment of the application, when scheduling the cache resources of the large language model service, an adaptive migration method is used to adjust the cache resources. Specifically, the process balances communication and computing costs to mitigate the negative impact of request migration. The process mainly includes: first, identifying idle computing and communication resources that can be used for request migration without affecting the normal operation of the system; then, transferring each request by migrating tokens or key-value caches, and coordinating all requests to be migrated to control their resource consumption within a certain boundary.

[0087] Specifically, in some embodiments, the cache resources of the large language model service are scheduled according to the cache allocation task, the cache release task, and the cache update task, including: determining a first migration operation to be performed according to the cache release task, and adding the first migration operation to the operation cache area; determine a second migration operation needed to be executed according to the cache update task, add the second migration operation to the operation cache area, and perform deduplication processing on the operation cache area; determine a third migration operation needed to be executed according to the cache allocation task, add the third migration operation to the operation cache area, and perform deduplication processing on the operation cache area; perform all migration operations in the operation cache area.

[0088] In the embodiments of the present application, the foregoing description is directed to the processing logic of a single task, but generally, a large language model service corresponds to multiple service requests, but when multiple service requests need to execute these operations at the same time, operation overlap may cause redundant migration. To solve this problem, in the embodiments of the present application, a request operation batch processing mechanism is introduced, which optimizes operations into a unified group rather than discrete execution, thereby minimizing unnecessary resource allocation and migration. Implementing operation batch processing is crucial to ensuring efficient migration within the framework, especially in high-concurrency scenarios.

[0089] Specifically, when processing related tasks, the system first processes the cache release task, the migration operation triggered by which (i.e., the first migration operation, such as a request migrated from another GPU to fill a resource vacancy) will not be executed immediately but will be temporarily stored in the operation cache area, which avoids intermediate state inconsistency and subsequent operation conflicts that may be caused by immediate execution. Next, the system processes the cache update task, and similarly, the migration operation calculated by it (i.e., the second migration operation, such as migrating a request to a new GPU due to the growth of the request size) is also added to the operation cache area. At this stage, the system performs critical first deduplication processing, such as identifying and removing migration operations that cancel each other out due to the same target, thereby eliminating unnecessary overhead.

[0090] Subsequently, the system processes the cache allocation task, and the migration operation generated by it (i.e., the third migration operation, such as migrating an old request on the current GPU to make room for a new request) is also added to the cache area, and deduplication processing is performed again to further purify the scheduling instruction set. Finally, after all possible scheduling effects in the previous three stages are calculated and optimized, the system batch-executes all migration operations in the operation cache area that have been sorted.

[0091] As can be easily understood, in the embodiments of the present application, through the delayed execution strategy, the system is provided with a global view, enabling it to understand the internal relationship between different scheduling tasks and intelligently eliminate redundant operations. This not only greatly reduces the number of actual migrations but also reduces the pressure on the communication links between GPUs, avoids performance jitter caused by frequent migration, and ensures the global optimality of scheduling decisions.

[0092] Specifically, in some embodiments, performing all migration operations in the operation cache area includes: boundary conditions are determined by performing a boundary analysis on the communication links between the cache resources of the large language model service; Based on the boundary conditions, a two-bin bin packing problem model of hybrid migration is established; wherein, the hybrid migration includes migration based on key-value cache form and migration based on token form; The two-bin bin packing problem model is solved by a greedy algorithm, and the migration operation is performed according to the result of the solution.

[0093] In the embodiments of the present application, when performing the migration operation, the communication performance between GPUs (such as through NVLink, PCIe or network switch) is analyzed in advance or in real time, and the boundary conditions of each link, i.e. the upper limit of the amount of data that can be stably transmitted per unit time, are determined. This amount analysis provides a key physical constraint for subsequent decision-making, ensuring that the migration operation does not squeeze the communication bandwidth required for normal service. Illustratively, in some embodiments, during system initialization, the communication boundary of each link and the calculation boundary of each instance can be determined by offline analysis, and this boundary information is shared with all instances in the system.

[0094] Based on this boundary condition, in the embodiments of the present application, a two-bin bin packing problem model of hybrid migration is established. The two bins here refer to two completely different migration methods: 1. Migration based on key-value cache (KV Cache): copy the complete KV cache data of the request directly to the target GPU. This method has large communication overhead (large amount of data transmission), but small calculation overhead (the target GPU does not need to recalculate). 2. Migration based on token (Token): only transmit the original input token of the request, and the target GPU receives it and performs prefill to generate the KV cache. This method has small communication overhead (small amount of data transmission), but large calculation overhead (the target GPU needs to consume calculation resources to regenerate).

[0095] Here, each service request to be migrated can be regarded as an "item", which can be packaged into two different "boxes" of key-value cache or token migration. The optimization goal can be to select a hybrid migration scheme that minimizes the total cost (communication cost and calculation cost) as much as possible without exceeding the boundary conditions of the communication link (i.e. the capacity of the box). The system can use a greedy algorithm (such as the first-fit or best-fit algorithm) to efficiently approximate the solution of this NP-hard problem, which will quickly assign an appropriate migration method to each migration request and output an executable migration plan. The system then executes the final migration operation in parallel or serially according to the result of the plan.

[0096] In the embodiments of the present application, the migration operation can be implemented in a global cooperative manner, multiple instances can use the same (communication) link and migrate the request to the same target instance. For example, referring to Figure 3 , Figure 3 a schematic diagram of a migration operation provided in the embodiments of the present application is shown, in which Figure 3 GPU1 and GPU2 use an inter-machine switch to migrate the service request to GPU4. If each instance independently performs the above (request) division without cooperation, their migration operations may exceed the communication boundary of the link. To avoid this situation, each instance must consider all requests to be migrated in the system when running the scheduling algorithm, not just its own request.

[0097] It can be understood that the global request migration method used in the embodiments of the present application can convert the execution action into a decision-making process that seeks the optimal solution between communication and computing resources, ensuring the stability and high resource utilization efficiency of large language model services in the dynamic scheduling process.

[0098] Referring to Figure 4 , the embodiments of the present application also provide a resource processing device for a large language model service, which comprises: a detection unit 410 configured to detect the request state and required cache size of each service request of the large language model service; wherein the request state includes an added state, a completed state and a running state; a first response unit 420 configured to, in response to the service request being in the added state, establish a cache allocation task according to the required cache size of the service request; a second response unit 430 configured to, in response to the service request being in the completed state, establish a cache release task according to the cache resources occupied by the service request; a third response unit 440 configured to, in response to the service request being in the running state and the required cache size of the service request changing, establish a cache update task corresponding to the service request; a scheduling unit 450 configured to schedule the cache resources of the large language model service according to the cache allocation task, the cache release task and the cache update task.

[0099] It can be understood that the contents in the resource processing method embodiment of the large language model service as shown in Figure 2 are all applicable to the resource processing device embodiment of the large language model service, the resource processing device embodiment of the large language model service specifically implements the same functions as the resource processing method embodiment of the large language model service as shown in Figure 2 , and achieves the same beneficial effects as the resource processing method embodiment of the large language model service as shown in Figure 2The beneficial effects achieved by the resource processing method embodiment of the large language model service shown are the same.

[0100] Referring to Figure 5 The electronic device embodiment provided by the present application comprises: at least one processor 510; at least one memory 520 for storing at least one program; When the at least one program is executed by the at least one processor 510, the at least one processor 510 implements Figure 2 the resource processing method of the large language model service shown.

[0101] Similarly, the contents in the above method embodiments are applicable to the present electronic device embodiment, the functions specifically implemented by the present electronic device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved by the present electronic device embodiment are the same as those achieved by the above method embodiments.

[0102] The present application embodiment further provides a computer readable storage medium, wherein the computer readable storage medium stores a program executable by the processor 510, and the program executable by the processor 510 is used to execute the above-mentioned resource processing method of the large language model service when executed by the processor 510.

[0103] It can be understood that the contents in the above-mentioned resource processing method embodiment of the large language model service are applicable to the present computer readable storage medium embodiment, the functions specifically implemented by the present computer readable storage medium embodiment are the same as those of the above-mentioned resource processing method embodiment of the large language model service, and the beneficial effects achieved by the present computer readable storage medium embodiment are the same as those achieved by the above-mentioned resource processing method embodiment of the large language model service. Figure 2 Figure 2 Figure 2

[0104] The present application embodiment further discloses a computer program product or computer program, which comprises computer instructions stored in the above-mentioned computer readable storage medium; Figure 5 The processor of the electronic device can read the computer instructions from the above-mentioned computer readable storage medium, and the processor executes the computer instructions, so that the electronic device executes Figure 2 the resource processing method embodiment of the large language model service shown.

[0105] It can be understood that the contents in the above-mentioned resource processing method embodiment of the large language model service are applicable to the present computer program product or computer program embodiment, the functions specifically implemented by the present computer program product or computer program embodiment are the same as those of the above-mentioned resource processing method embodiment of the large language model service, and the beneficial effects achieved by the present computer program product or computer program embodiment are the same as those achieved by the above-mentioned resource processing method embodiment of the large language model service. Figure 2 Figure 2 ​​​​The resource processing method embodiments of the large language model service shown are the same, and the beneficial effects achieved are the same. Figure 2 The resource processing method embodiments of the large language model service shown are the same, and the beneficial effects achieved are the same.

[0106] In some alternative embodiments, the functions / operations mentioned in the block diagram can not occur in the order mentioned in the operation diagram. For example, depending on the functions / operations involved, two blocks shown in succession can actually be executed substantially simultaneously or the blocks can sometimes be executed in reverse order. In addition, the embodiments presented and described in the flowcharts of the present application are provided by way of example, with the purpose of providing a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logical flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and in which sub-operations described as part of a larger operation are independently executed.

[0107] In addition, although the present application is described in the context of functional modules, it should be understood that, unless otherwise stated to the contrary, one or more of the functions and / or features can be integrated in a single physical device and / or software module, or one or more functions and / or features can be implemented in separate physical devices or software modules. It can also be understood that a detailed discussion of the actual implementation of each module is unnecessary for an understanding of the present application. Rather, given the properties, functions, and internal relationships of the various functional modules disclosed in the devices herein, the actual implementation of the module will be within the routine skill of an engineer, given the properties, functions, and internal relationships of the various functional modules disclosed in the devices herein. Therefore, those skilled in the art, using ordinary skill, can implement the present application as set forth in the claims without undue experimentation, given the benefit of this disclosure. It can also be understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present application, which is defined by the full scope of the claims and their equivalents.

[0108] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the embodiments of the method of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0109] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program having a predetermined function, and works together with other related parts to achieve a predetermined target, and can be implemented in whole or in part by using software, hardware (such as a processing circuit or a memory), or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an integral module or unit that includes the functions of the module or unit.

[0110] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a list of executable instructions for implementing logic functions, and can be embodied in any computer-readable storage medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, a processor-based system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions, or in conjunction with such an instruction execution system, apparatus, or device. For the purposes of this specification, a "computer-readable storage medium" can be any device that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device, or in conjunction with such an instruction execution system, apparatus, or device.

[0111] It should be understood that parts of the present application can be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, a plurality of steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and as in another embodiment, it can be implemented using any one or a combination of the following technologies known in the art: discrete logic circuit having logic gates for implementing logic functions on data signals, application specific integrated circuit (ASIC) having appropriate combinational logic gates, programmable gate array (PGA), field programmable gate array (FPGA), etc.

[0112] In the above description of the present specification, the description of the terms "one embodiment / one example", "another embodiment / another example", or "certain embodiments / certain examples" and the like means that the specific features, structures, materials, or characteristics described in connection with the embodiments or examples are included in at least one embodiment or example of the present application. In the present specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or more embodiments or examples in an appropriate manner.

[0113] While the embodiments of the application have been shown and described, it is to be understood that the embodiments can be varied, modified, substituted and changed by those skilled in the art without departing from the principles and spirit of the application, the scope of which is defined by the claims and their equivalents.

[0114] The above is a specific description of the preferred embodiments of the application, but the application is not limited to the embodiments, and those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the application, and these equivalent modifications or substitutions are all included in the scope defined by the claims of the application.

Claims

1. A resource processing method of a large language model service, characterized by, The method comprises: detecting the request state and the required cache size of each service request of the large language model service; wherein the request state comprises an added state, a completed state and a running state; in response to the service request being in the added state, establishing a cache allocation task according to the required cache size of the service request; in response to the service request being in the completed state, establishing a cache release task according to the cache resource occupied by the service request; in response to the service request being in the running state and the required cache size of the service request changing, establishing a cache update task corresponding to the service request; scheduling the cache resources of the large language model service according to the cache allocation task, the cache release task and the cache update task. 2.The method of Claim 1, wherein, In response to the service request being in the added state, establishing a cache allocation task according to the required cache size of the service request, comprising: in response to the service request being in the added state, comparing the required cache size of the service request through a pre-set first cache threshold, a second cache threshold and a third cache threshold; wherein the first cache threshold is greater than the second cache threshold, and the second cache threshold is greater than the third cache threshold; in response to the required cache size of the service request being greater than the first cache threshold, determining that the service request is a first request, and establishing a first allocation task; in response to the required cache size of the service request being greater than the second cache threshold and less than or equal to the first cache threshold, determining that the service request is a second request, and establishing a second allocation task; in response to the required cache size of the service request being greater than the third cache threshold and less than or equal to the second cache threshold, determining that the service request is a third request, and establishing a third allocation task; in response to the required cache size of the service request being less than or equal to the third cache threshold, determining that the service request is a fourth request, and combining a plurality of fourth requests to establish a third allocation task.

3. The resource processing method of a large language model service according to claim 2, characterized in that, Scheduling the cache resources of the large language model service according to the cache allocation task, comprising: for the first allocation task, activating a new first cache device, processing the first allocation task through the new first cache device, and pulling at least one second request cache data from the activated second cache device; for the second allocation task, detecting whether there is redundant space supporting the second allocation task in each first cache device currently activated, if there is redundant space supporting the second allocation task, processing the second allocation task through the first cache device, if there is no redundant space supporting the second allocation task, activating a new second cache device, and processing the second allocation task through the new second cache device; For the third allocation task, it is detected whether each currently activated first cache device has redundant space supporting the third allocation task, if the first cache device has redundant space supporting the third allocation task, the third allocation task is processed by the first cache device, if the first cache device does not have redundant space supporting the third allocation task, a new third cache device is activated, and the third allocation task is processed by the new third cache device. The first cache device stores cache data with the largest service request greater than the first cache threshold, the second cache device stores cache data with the largest service request greater than the second cache threshold and less than or equal to the first cache threshold, and the third cache device stores cache data with the largest service request greater than the third cache threshold and less than or equal to the second cache threshold.

4. The resource processing method of a large language model service according to claim 3, characterized in that, The detection of whether each currently activated first cache device has redundant space supporting the second allocation task includes: Detecting a first available space of the currently activated first cache device; If the first available space is greater than or equal to the cache size required by the second allocation task, it is determined that the first cache device has redundant space supporting the second allocation task; If the first available space is less than the cache size required by the second allocation task, the cache data of the third request and the fourth request in the currently activated first cache device is counted and pruned, and a second available space of the currently activated first cache device after pruning is determined; If the second available space is greater than or equal to the cache size required by the second allocation task, the cache data of the third request and the fourth request in the currently activated first cache device is pruned, and it is determined that the first cache device has redundant space supporting the second allocation task, or if the second available space is less than the cache size required by the second allocation task, it is determined that the first cache device does not have redundant space supporting the second allocation task. 5.The method of Claim 2, wherein, According to the cache release task, the cache resources of the large language model service are scheduled, including: According to the cache release task, determining a target cache device and a request type of a service request requiring cache release; Performing cache release on the target cache device and migrating service requests belonging to the request type from other recently activated cache devices to the target cache device. 6.The method of Claim 1, wherein, According to the cache release task, the cache resources of the large language model service are scheduled, including: According to the cache release task, a first migration operation is determined to be executed, and the first migration operation is added to an operation cache area; According to the cache update task, a second migration operation is determined to be executed, the second migration operation is added to the operation cache area, and the operation cache area is de-duplicated. determining a third migration operation to be executed according to the cache allocation task, adding the third migration operation to the operation cache area, and performing deduplication processing on the operation cache area; performing all migration operations in the operation cache area.

7. The resource processing method of a large language model service according to claim 6, characterized in that, The performing all migration operations in the operation cache area comprises: performing boundary analysis on a communication link between cache resources of the large language model service to determine a boundary condition; based on the boundary condition, establishing a double bin bin packing problem model of hybrid migration; wherein the hybrid migration includes migration based on a key-value cache form and migration based on a word form; solving the double bin bin packing problem model by a greedy algorithm, and performing the migration operation according to the solving result.

8. A resource processing apparatus of a large language model service, characterized by, The device comprises: a detection unit configured to detect a request state and a required cache size of each service request of a large language model service; wherein the request state comprises an added state, a completed state and a running state; a first response unit configured to, in response to the service request being in the added state, establish a cache allocation task according to the required cache size of the service request; a second response unit configured to, in response to the service request being in the completed state, establish a cache release task according to cache resources occupied by the service request; a third response unit configured to, in response to the service request being in the running state and the required cache size of the service request changing, establish a cache update task corresponding to the service request; a scheduling unit configured to schedule cache resources of the large language model service according to the cache allocation task, the cache release task and the cache update task. 9.An electronic device comprising a memory and a processor, the memory storing a computer program, wherein, The processor executes the computer program to implement the resource processing method of the large language model service according to any one of claims 1 to 7.

10. A computer readable storage medium, the storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the resource processing method of the large language model service according to any one of claims 1 to 7.

Images