Request rescheduling method and device in large language model service, equipment and medium

Abstract

The invention relates to the technical field of large language models, in particular to a method, a device and equipment for requesting rescheduling in large language model service and a medium. The method comprises the steps that the scheduling amount of each instance is calculated according to the queuing condition of large language model service requests in each instance and key value cache, and at least one source instance to be scheduled and a target instance corresponding to each source instance to be scheduled are determined; and when a migration instruction is received, based on the type of the large language model service request and a preset migration strategy, migrating the token and / or key value cache in each to-be-scheduled source instance to a target instance corresponding to each to-be-scheduled source instance, and feeding back a response result of the large language model service request to a user. Therefore, efficient request allocation is realized in a dynamic environment, and the problem that effective scheduling cannot be performed when a traditional scheduling strategy faces uncertainty and dynamics of large language model requests is solved, so that the resource utilization rate and the response performance of a system are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of large language models, and in particular to a method, device, equipment and medium for requesting rescheduling in a large language model service. Background Art

[0002] In recent years, Large Language Models (LLM) have been widely used in multiple fields due to their excellent general performance and wide applicability. However, compared with traditional Deep Neural Network (DNN) services, LLM requests show high uncertainty in input length, output length, and latency requirements, which significantly increases the complexity of scheduling and resource management.

[0003] Traditional DNN services are usually designed for specific tasks, with relatively fixed input and output sizes, and relatively uniform task complexity and latency requirements. Therefore, the request scheduling and resource management of traditional DNN services are relatively simple, most requests are homogeneous, and the scheduling strategy is designed based on deterministic factors such as request processing time and resource usage.

[0004] However, the reasoning process of large language models is usually generated step by step. Each request requires multiple iterations to generate output tokens. The generation process of each token is dynamic, and the total generation length is difficult to determine in advance, making the execution time and resource requirements of large language models highly uncertain. In addition, during the execution of large language models, due to the characteristics of step-by-step generation, the generation of each token requires dynamic allocation of GPU memory, and the memory demand increases with the number of tokens generated, resulting in significant memory fragmentation problems and affecting resource scheduling between multiple instances.

[0005] In order to deal with the above problems, the scheduling strategies commonly used in related technologies include polling scheduling, scheduling based on resource demand, and load balancing based on the number of connections. However, these strategies are often unable to effectively schedule when faced with the uncertainty of large language model requests. For example, polling scheduling and load balancing methods based on the number of connections assume that the resource requirements and processing time of requests are static, which may cause some instances to be overloaded while other instances are idle. Scheduling based on resource demand relies on accurate prediction of request resource consumption before scheduling, which is difficult to achieve for large language model requests and needs to be solved urgently. Summary of the invention

[0006] The present invention provides a method, apparatus, device and medium for requesting rescheduling in a large language model service, so as to solve the problem that traditional scheduling strategies cannot effectively schedule when facing the uncertainty and dynamics of large language model requests, thereby improving the resource utilization and response performance of the system.

[0007] To achieve the above object, a first aspect of the present invention provides a method for requesting rescheduling in a large language model service, comprising the following steps: Obtain the queuing status and key-value cache of large language model service requests in each instance, and calculate the scheduling amount of each instance based on the queuing status and key-value cache of large language model service requests in each instance; Determine at least one source instance to be scheduled and a target instance corresponding to each source instance to be scheduled according to the scheduling amount of each instance; When a migration instruction is received, based on the type of the large language model service request and the preset migration strategy, the tokens and / or key-value caches in each source instance to be scheduled are migrated to the target instance corresponding to each source instance to be scheduled, and the response result of the large language model service request is fed back to the user.

[0008] Through the above technical means, by realizing efficient request allocation in a dynamic environment, the problem that traditional scheduling strategies cannot effectively schedule when faced with the uncertainty and dynamics of large language model requests is solved, thereby improving the system's resource utilization and responsiveness.

[0009] According to the method for rescheduling requests in a large language model service proposed in an embodiment of the present invention, the scheduling amount of each instance is calculated according to the queuing status of the large language model service request and the key-value cache, and each source instance to be scheduled and its corresponding target instance are determined; when a migration instruction is received, based on the type of the large language model service request and the preset migration strategy, the token and / or key-value cache in each source instance to be scheduled is migrated to the corresponding target instance, and the response result of the large language model service request is fed back to the user. Thus, by realizing efficient request allocation in a dynamic environment, the problem that the traditional scheduling strategy cannot effectively schedule when facing the uncertainty and dynamics of large language model requests is solved, thereby improving the resource utilization and response performance of the system.

[0010] To achieve the above object, a second embodiment of the present invention provides a device for requesting rescheduling in a large language model service, comprising: An acquisition module, used to obtain the queuing status and key value cache of large language model service requests in each instance, and calculate the scheduling amount of each instance according to the queuing status and key value cache of large language model service requests in each instance; A determination module, used to determine at least one source instance to be scheduled and a target instance corresponding to each source instance to be scheduled according to the scheduling amount of each instance; The migration module is used to migrate the tokens and / or key-value caches in each source instance to be scheduled to the target instance corresponding to each source instance to be scheduled based on the type of the large language model service request and the preset migration strategy when receiving the migration instruction, and to feed back the response result of the large language model service request to the user.

[0011] According to the device for requesting rescheduling in a large language model service proposed in an embodiment of the present invention, the scheduling amount of each instance is calculated according to the queuing status of the large language model service request and the key-value cache, and each source instance to be scheduled and its corresponding target instance are determined; when a migration instruction is received, based on the type of the large language model service request and the preset migration strategy, the token and / or key-value cache in each source instance to be scheduled is migrated to the corresponding target instance, and the response result of the large language model service request is fed back to the user. Thus, by realizing efficient request allocation in a dynamic environment, the problem that the traditional scheduling strategy cannot effectively schedule when facing the uncertainty and dynamic nature of large language model requests is solved, thereby improving the resource utilization and response performance of the system.

[0012] To achieve the above-mentioned objectives, an embodiment of the third aspect of the present invention proposes an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for requesting rescheduling in a large language model service as described in the above-mentioned embodiment.

[0013] To achieve the above objectives, a fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the method for requesting rescheduling in a large language model service as described in the above embodiments.

[0014] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 Serving scheduling methods for traditional deep neural networks; Figure 2 A schematic diagram of a system for requesting rescheduling in a large language model service according to an embodiment of the present invention; Figure 3A flowchart of a method for requesting rescheduling in a large language model service provided according to an embodiment of the present invention; Figure 4 A schematic diagram of an example migration process according to an embodiment of the present invention; Figure 5 A block diagram of an apparatus for requesting rescheduling in a large language model service provided according to an embodiment of the present invention; Figure 6 It is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present invention.

[0017] Figure numerals: 10 - device for requesting rescheduling in a large language model service, 100 - acquisition module, 200 - determination module, 300 - migration module; 601 - memory, 602 - processor, 603 - communication interface. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] It should be noted that, in the description of the present invention, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in the present invention are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0020] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0021] The following describes a method, apparatus, device, and medium for requesting rescheduling in a large language model service according to an embodiment of the present invention with reference to the accompanying drawings.

[0022] Before introducing the method for requesting rescheduling in a large language model service according to an embodiment of the present invention, the scheduling strategy for deep neural network service requests in the related art and the system for requesting rescheduling in a large language model service involved in the method for requesting rescheduling in a large language model service according to the present invention are briefly introduced.

[0023] Specifically, if Figure 1As shown in the figure, the traditional deep neural network service scheduling method in the related art is that the user (Actor) sends a request to the traditional scheduling strategy module, which assigns the request to the traditional service for processing according to the preset strategy. The traditional service is generally composed of multiple instances (or multiple copies), each of which has the same function and can process requests from users. The scheduling strategies for traditional deep neural network service requests mainly include polling scheduling, scheduling strategies based on resource requirements, and load balancing scheduling strategies based on the number of connections.

[0024] Among them, polling scheduling is a simple scheduling strategy. During scheduling, requests are assigned to service instances in a sequential and cyclic manner, so each request is equally assigned for execution. This scheduling strategy is suitable for situations where the processing time difference of each request is small.

[0025] The resource demand-based scheduling strategy is to schedule according to the resource requirements of each request (such as GPU memory, CPU, etc.). Therefore, the resource requirements of the request need to be accurately modeled, otherwise it may lead to inefficient use of resources.

[0026] The load balancing scheduling strategy based on the number of connections means that in a multi-instance service, the scheduling system will choose which instance to assign a new request to based on the number of connections of each instance (that is, the number of requests being processed). Instances with fewer request connections will receive new requests first. This method can prevent some nodes from being overloaded due to too many requests, which helps improve the overall performance of the system.

[0027] The above three scheduling strategies represent the design patterns of current scheduling strategies for traditional deep neural network services. Their scheduling strategies based on predictability are feasible in solving the request scheduling problems of traditional deep neural network services.

[0028] Those skilled in the art can understand that, in order to improve efficiency during the large language model inference process, the large language model uses a key-value cache mechanism to help the model avoid recalculating previous input information when generating, thereby accelerating the inference process. The following briefly describes the implementation of the key-value cache mechanism: In the Transformer's self-attention mechanism, each input token generates three vectors: query, key, and value. In the standard reasoning process, all inputs need to recalculate these three vectors each time they are generated. When using key-value caching, the model caches the calculated keys and values. In this way, when generating each token, the model can directly use these cached keys and values ​​without recalculating all inputs. This caching mechanism is very efficient, especially when generating long texts, reducing the amount of calculation.

[0029] Prefilling in the large language model inference process means processing the input sequence in the user request into a key-value cache through the model at one time for use in the decoding stage. Decoding is based on the context of the prefilling stage. The model gradually generates new tokens until the target length or the end symbol is reached. Decoding uses the key-value cache (KV Cache) to avoid repeated calculations and improve efficiency.

[0030] Large language model inference usually involves multiple iterations, each of which generates a token, and the total length of generation is usually unpredictable, which makes the memory requirements of each large language model request change dynamically with the generation process. In addition, large language model requests may have different processing times for each request due to different processing tasks (such as text summarization tasks that require a longer processing time and real-time conversation tasks that require a shorter processing time).

[0031] Since the scheduling strategy of traditional deep neural network service requests is based on request determinism, it faces difficulties when processing large language model requests whose processing time, resource usage, input, and output are uncertain. For example, polling scheduling and load balancing scheduling strategies based on the number of connections usually assume that the resources and processing time required for each connection are relatively static, which may lead to unreasonable request scheduling. For example, when scheduling based on the number of connections, multiple requests with high processing times are scheduled together, resulting in some instances being overloaded while other instances are idle. Scheduling based on resource demand requires accurate prediction of request resource consumption during scheduling, which is difficult to achieve for large language model requests.

[0032] Based on the fact that the above-mentioned traditional scheduling strategy cannot effectively schedule when facing the uncertainty of large language model requests, the method for rescheduling requests in large language model services proposed in the present invention rescheduling the requests according to the current status in the instance after the traditional scheduling strategy schedules the requests to the instance, and schedules the requests to other instances, thereby solving the problem that the traditional scheduling strategy cannot effectively schedule when facing the uncertainty of large language model requests.

[0033] Furthermore, if Figure 2 As shown, Figure 2 The method for requesting rescheduling in a large language model service in an embodiment of the present invention involves a system for requesting rescheduling in a large language model service. The system for requesting rescheduling in a large language model service includes a scheduling module, a monitoring module and a migration module, wherein the scheduling module is located globally, and each instance includes a monitoring module and a migration module.

[0034] Among them, the scheduling module comprehensively analyzes and determines the instances that need to be scheduled based on the load data collected by the monitoring module in all instances, and evaluates the instances with lower load and the ability to receive more requests. Then, the scheduling module establishes the association between the source instance and the target instance, and sends the migration instruction to the migration module. It should be noted that the scheduling module does not directly determine the migration details of the specific request. Instead, after the migration instruction is issued, the migration module in the instance is responsible for controlling and executing the specific request migration operation. In addition, the monitoring module is responsible for real-time monitoring of the instance load, and regularly reports the load data to the scheduling module at the set time interval to support its decision-making process.

[0035] In addition, the scheduling module of the present invention collects monitoring data in real time and dynamically adjusts the scheduling plan. Therefore, during the request processing, as the key value cache of the request increases, the requests that need to be scheduled in time can be quickly determined, thereby promptly repairing the irrationality of the traditional scheduling strategy.

[0036] The following is a detailed description of the method for requesting rescheduling in a large language model service proposed by the present invention using the system for requesting rescheduling in a large language model service.

[0037] Figure 3 The figure is a flowchart of a method for requesting rescheduling in a large language model service according to an embodiment of the present invention.

[0038] For example, Figure 3 As shown, the method for requesting rescheduling in the large language model service includes the following steps: In step S301, the queuing status and key value cache of the large language model service request in each instance are obtained, and the scheduling amount of each instance is calculated according to the queuing status and key value cache of the large language model service request in each instance.

[0039] Specifically, before obtaining the queuing status and key-value cache of the large language model service request in each instance, it is first determined whether a large language model service request is received, and when a large language model service request is received, the large language model service request is scheduled to multiple instances.

[0040] The above technical solution ensures that the system can respond to large language model service requests in a timely manner and achieve load balancing through multi-instance scheduling, providing a basis for subsequent resource management and optimized scheduling.

[0041] Furthermore, in some embodiments, the scheduling amount of each instance is calculated based on the queuing status of the large language model service requests in each instance and the key value cache, including: obtaining the key value cache occupied by the running requests, the virtual key value cache expected to be occupied by the queued requests, the total video memory capacity of the current instance, the video memory capacity occupied by the parameters inherent in the current large language model inference model, and the batch size of the current large language model inference model; calculating the scheduling amount of each instance based on the key value cache occupied by the running requests, the virtual key value cache expected to be occupied by the queued requests, the total video memory capacity of the current instance, the video memory capacity occupied by the parameters inherent in the current large language model inference model, and the batch size of the current large language model inference model.

[0042] Among them, the key value cache occupied by the running requests refers to the key value cache occupied by the requests currently being processed, the virtual key value cache expected to be occupied by the queued requests refers to the key value cache expected to be occupied by the requests currently waiting to be processed, the total video memory capacity of the current instance refers to the total GPU video memory capacity of the current large language model service instance, the video memory capacity occupied by the inherent parameters of the current large language model inference model refers to the video memory capacity occupied by the parameters of the large language model inference model (such as weights, biases, etc.), and the batch size of the current large language model inference model refers to the number of requests processed at one time during the large language model inference process.

[0043] Specifically, the embodiments of the present invention can obtain the key value cache occupied by the running requests and the virtual key value cache expected to be occupied by the queued requests through the monitoring module in the request rescheduling system in the large language model service, and can directly read the total video memory capacity allocated to the current instance from the hardware configuration or system resource management tool. The video memory capacity occupied by the inherent parameters of the current large language model inference model can be calculated based on the parameter scale when the model is loaded, and the batch size of the current large language model inference model can be obtained through the configuration file or API interface of the model running framework, which is not specifically limited here.

[0044] Furthermore, for the running request set R, it is necessary to calculate the amount of key value cache occupied by these requests. The calculation formula for the key value cache of the running request set R is: ; ; in, A key-value cache for the running request set R. is the sum of the cache in the pre-filling phase and the cache in the decoding phase, For the cache in the pre-fill phase, It is the cache for the decoding stage.

[0045] For the queued request set Q, the virtual key value cache that is expected to be occupied is calculated according to the priority P of the requests in the queued request set Q: ; ; in, The virtual key value cache that is expected to be occupied by the queued request set Q, It is the virtual key value cache of each request in the queued request set. At this time, the request is in the queue and has not yet been actually executed on the physical machine and occupied the video memory, so it is a theoretical data in a statistical sense, so it is called "virtual" cache. It is calculated in two cases. The first is when the request is not executed at all. At this time, decoding has not started. In this case, it is only necessary to count the input sequence of the user request. The second is when the request is suspended due to insufficient remaining space in the video memory during execution, and then enters the queue set. Since the request has entered the decoding stage, then at this time is equivalent to . For requests in the queued request set, is the queuing time of the request, is the parameter affecting the queuing time, is the priority of the request, is the priority influencing parameter, is the threshold value, Corresponding to the prefill phase in the LLM reasoning process, this phase will calculate the prefilled key value cache once through the model based on the input sequence requested by the user. This only happens in the second case, that is, the request that has been suspended has entered the decoding stage, so when it counts the cache, it is necessary to count the key-value cache that has been generated. Because when the request is scheduled for execution again, the key-value caches of the pre-filling and decoding stages will be swapped into the video memory, and the request will continue to execute from where it was suspended.

[0046] The queuing time of the request A number greater than 0. When requests start queuing, the virtual key value cache that the queued requests are expected to occupy is a linear growth, when the request queue time When the virtual key value cache is greater than the threshold T, the virtual key value cache that is expected to be occupied by the queued requests is an exponential growth. Queuing time affects parameters Used to balance the queue time against the final The influence of queuing time on parameters A number greater than or equal to 1. When the large language model service is highly sensitive to the user's waiting time, you need to increase the queue time impact parameter . Priority affects parameters Used to balance priorities against final The degree of influence, priority influence parameters A number greater than or equal to 1. When the large language model service focuses on responding quickly to high-priority user requests, you need to increase the priority impact parameter. , It is understandable that because large language model services need to consider the impact of delays on users, as the waiting time increases Increase when queue time When it is greater than the set threshold T, the virtual key value cache that the queued request set Q is expected to occupy Rapidly increase, thereby triggering scheduling to avoid excessive delays in user requests. The threshold T can be a pre-set threshold, and the principle of setting is the user's tolerance for delays. For high-priority requests, queuing should be avoided, so the design is to rapidly increase the virtual key value cache expected to be occupied by the queued requests when queuing them. The priority P is an integer greater than or equal to 0, and the larger the priority P value is, the higher the level is.

[0047] Therefore, by comprehensively considering the queueing time and priority of the requests in the queue, it is possible to quickly find out which requests need to be scheduled in a timely manner, thereby increasing user-friendliness.

[0048] Furthermore, in some embodiments, the scheduling amount of each instance is calculated based on the key value cache occupied by the running requests, the virtual key value cache expected to be occupied by the queued requests, the total video memory capacity of the current instance, the video memory capacity occupied by the parameters inherent in the current large language model inference model, and the batch size of the current large language model inference model, including: obtaining the load value of the current instance based on the key value cache occupied by the running requests, the virtual key value cache expected to be occupied by the queued requests, and the video memory capacity occupied by the parameters inherent in the current large language model inference model; obtaining the remaining space of the current instance based on the difference between the total video memory capacity of the current instance and the load value of the current instance; and obtaining the scheduling amount of the current instance based on the ratio between the remaining space of the current instance and the batch size of the current large language model inference model.

[0049] Among them, the load value of the current instance is obtained by the sum of the key value cache occupied by the running requests, the virtual key value cache expected to be occupied by the queued requests, and the video memory capacity occupied by the number of parameters inherent in the current large language model inference model.

[0050] Specifically, the load value of the current instance can be expressed as: ; in, is the load value of the current instance, The key-value cache occupied by the running request, The virtual key value cache that is expected to be occupied by the queued requests, This is the memory capacity occupied by the inherent parameters of the current large language model inference model.

[0051] Furthermore, let C be the total memory capacity of the instance. The remaining space of the current instance can be obtained based on the difference between the total memory capacity of the current instance and the load value of the current instance, that is, The remaining space of the current instance can be calculated. Since tokens are generated in batches, the remaining space of the current instance can be divided by the batch size B of the current large language model inference model to calculate the consumption rate of the remaining space, that is, the scheduling amount. For ease of understanding, the scheduling amount can be expressed as: ; in, is the scheduling quantity, is the total video memory capacity of the instance, The key-value cache occupied by the running request, The virtual key value cache that is expected to be occupied by the queued requests, is the memory capacity occupied by the inherent parameters of the current large language model inference model, and B is the batch size of the current large language model inference model.

[0052] It is understandable that the scheduling amount S may be negative. When the scheduling amount S is negative, it means that there are queued requests for video memory. , the memory capacity occupied by the inherent parameters of the current large language model inference model and the key-value cache occupied by the running requests The occupied video memory is close to the total video memory capacity C of the instance, so new requests cannot obtain enough video memory and can only queue. Therefore, the purpose of designing the virtual cache in the present invention is to make S smaller, or even a negative value, so that in the next scheduling module, these instances can be given priority in scheduling.

[0053] Therefore, by comprehensively calculating the load value and remaining space of the current instance and determining the scheduling amount in combination with the model batch size, the resource usage and scheduling capacity of the instance can be accurately evaluated, thereby achieving efficient use of resources.

[0054] In step S302, at least one source instance to be scheduled and a target instance corresponding to each source instance to be scheduled are determined according to the scheduling amount of each instance.

[0055] Furthermore, in some embodiments, at least one source instance to be scheduled and a target instance corresponding to each source instance to be scheduled are determined according to the scheduling amount of each instance, including: determining a set of source instances to be scheduled and a set of candidate target instances based on the scheduling amount of each instance; judging whether the set of candidate target instances is an empty set; if the set of candidate target instances is not an empty set, then based on the set of candidate target instances, screening out instances that are at the same node as each source instance to be scheduled in the set of source instances to be scheduled and have the largest scheduling amount, to obtain the target instance corresponding to each source instance to be scheduled.

[0056] The source instance set to be scheduled refers to the instance set that needs to migrate requests to other instances due to excessive load, and the candidate target instance set refers to the instance set with low load and sufficient resources to receive migration requests.

[0057] Furthermore, in some embodiments, based on the scheduling amount of each instance, a set of source instances to be scheduled and a set of candidate target instances are determined, including: based on the scheduling amount of each instance, screening out instance combinations with scheduling amounts less than a first preset threshold to obtain a set of source instances to be scheduled; based on the scheduling amount of each instance, judging whether there are instances with scheduling amounts greater than a second preset threshold; if there are instances with scheduling amounts greater than the second preset threshold, combining the instances with scheduling amounts greater than the second preset threshold to obtain a set of candidate target instances.

[0058] The first preset threshold and the second preset threshold may be thresholds set by those skilled in the art according to actual conditions, and are not specifically limited here.

[0059] Specifically, the scheduling module collects and monitors the scheduling amount S from the instance set of the large language model service in real time, and sorts the scheduling amount S from small to large. As shown in Table 1, , indicating that the k-th instance after sorting is located on the j-th node.

[0060] Table 1

[0061] Further, take out the instance k in order from small to large, and judge Is the value less than the first preset threshold G? If the value is less than the first preset threshold G, the scheduling is started, instance k is used as the source instance, and the source instance is added to the set of source instances to be scheduled, and then a suitable target instance is searched from large to small: .

[0062] It should be noted that when looking for the target instance When If the scheduling amount S values ​​of multiple instances are greater than the second preset threshold H, the instances that meet the conditions are added to the candidate target instance set, and the instance that meets the conditions is preferentially selected from the candidate target instance set. The candidate target instance on the same node with the largest scheduling amount S value is used as the target instance corresponding to the current source instance.

[0063] Therefore, the embodiment of the present invention takes into account the problem that the communication bandwidth across nodes is smaller than the communication bandwidth between GPUs within a node, and gives priority to instances within the same node during scheduling, thereby saving migration time and being able to accurately identify instances with insufficient resources and abundant resources, providing a basis for subsequent scheduling decisions, thereby optimizing resource allocation.

[0064] Furthermore, in some embodiments, after determining whether the candidate target instance set is an empty set, it also includes: if the candidate target instance set is an empty set, creating a new instance for each source instance to be scheduled; and using the new instance corresponding to each source instance to be scheduled as the target instance corresponding to each source instance to be scheduled.

[0065] Specifically, when the set of candidate target instances is an empty set, it means that the scheduling module finds that the target instance cannot be found. In order to ensure that the scheduling can proceed smoothly and avoid the source instance affecting the service quality due to excessive load, it is necessary to create a new instance for each source instance to be scheduled, associate each source instance to be scheduled with its corresponding new instance, and set the target address to the new instance address, that is, designate the newly created instance as the target instance.

[0066] Therefore, through the above technical means, it is ensured that in the absence of a suitable target instance, the system can still meet the scheduling requirements by dynamically creating a new instance, ensuring the continuous operation of the system and the reasonable allocation of resources.

[0067] In step S303, when the migration instruction is received, based on the type of the large language model service request and the preset migration strategy, the tokens and / or key-value cache in each source instance to be scheduled are migrated to the target instance corresponding to each source instance to be scheduled, and the response result of the large language model service request is fed back to the user.

[0068] Specifically, after the scheduling module determines the source instance to be scheduled and its corresponding target instance, the migration module starts migration or refuses migration according to the preset migration strategy. First, the priority of the request is sorted, and the request is scheduled according to the priority sorting result, and the request with the same priority is scheduled on a first-come-first-served basis.

[0069] Through the above technical means, the migration process is simplified and the migration efficiency is improved. At the same time, it is ensured that the target instance can quickly take over the unexecuted requests to ensure the efficient operation of the system.

[0070] Further, in some embodiments, the type of the large language model service request is a completely unexecuted large language model service request, and based on the type of the large language model service request and a preset migration strategy, the tokens and / or key-value cache in each source instance to be scheduled are migrated to the target instance corresponding to each source instance to be scheduled, including: based on the preset migration strategy, migrating the tokens in each source instance to be scheduled to the target instance corresponding to each source instance to be scheduled.

[0071] Specifically, for large language model service requests that have not been executed at all, since the pre-filling and decoding phases have not yet started, there is no key-value cache. At this time, the input tokens of these requests are extracted from the source instance and the requested tokens are directly migrated to the target instance.

[0072] Through the above technical means, by focusing on meeting the video memory requirements, it is ensured that the target instance can seamlessly take over the executing request, ensuring the continuity of the task while optimizing resource allocation.

[0073] Further, in some embodiments, the type of the large language model service request is a large language model service request waiting for video memory during execution, and based on the type of the large language model service request and a preset migration strategy, the tokens and / or key-value cache in each source instance to be scheduled are migrated to the target instance corresponding to each source instance to be scheduled, including: based on the preset migration strategy, migrating the key-value cache in each source instance to be scheduled to the target instance corresponding to each source instance to be scheduled.

[0074] Specifically, for requests waiting for video memory during execution, since a key-value cache already exists, the existing key-value cache is directly migrated. If this request waits for video memory during the migration process and starts execution, it is migrated based on the migration strategy of the large language model service request being executed based on the type of the large language model service request.

[0075] Further, in some embodiments, the type of the large language model service request is the large language model service request being executed, and based on the type of the large language model service request and the preset migration strategy, the tokens and / or key-value caches in each source instance to be scheduled are migrated to the target instance corresponding to each source instance to be scheduled, including: based on the preset migration strategy, copying the existing key-value cache of each source instance to be scheduled to the target instance corresponding to each source instance to be scheduled, marking the end position of this copy of each source instance to be scheduled, and decoding and calculating each source instance to be scheduled, and appending the newly generated key-value cache to the existing key-value cache; repeating the above The existing key-value cache of each source instance to be scheduled is copied to the target instance corresponding to each source instance to be scheduled, and the end position of this copy of each source instance to be scheduled is marked, and each source instance to be scheduled is decoded and calculated to append the newly generated key-value cache to the existing key-value cache, until there is a key-value cache of one token remaining in each source instance to be scheduled; the key-value cache of the remaining one token is copied to the target instance corresponding to each source instance to be scheduled, the key-value cache in each source instance to be scheduled is cleared, and the key-value cache that has been migrated to the target instance corresponding to each source instance to be scheduled is spliced, and the calculation is resumed after the splicing is completed.

[0076] Specifically, for requests being executed, a progressive migration strategy is adopted. Because the key-value cache of the large language model is appended, that is, after the pre-filling stage, each iteration generates a cache of tokens and appends them to the existing key-value cache. Therefore, when migrating requests, the strategy of overwriting the existing key-value cache by using the calculation time of iteratively generated tokens is executed.

[0077] That is to say, after the migration is started, the existing key-value cache is copied from the source instance to be scheduled to the target instance, and the end position W of this copy is marked. At the same time, the source instance to be scheduled continues to perform decoding calculations and appends them to the key-value cache. When the existing key-value cache is copied, the newly calculated cache after the end position W of the last copy is copied to the target instance, and the source instance to be scheduled continues to perform decoding calculations.

[0078] Repeat the above-mentioned copying and calculation process. Since the speed of copying is much faster than the speed of calculation, there is only a key-value cache of one token left in the source instance to be scheduled. In this case, the last key-value cache is copied. The migration module stops copying after the last key-value cache is copied, and the source instance to be scheduled stops calculating. The key-value cache in the source instance to be scheduled is cleared, and the copied key-value cache is spliced ​​in the target instance. After the splicing is completed, the calculation is resumed, and the scheduling module is notified that the migration is successful. After the migration is successful, the scheduling module resets the route of the user request to the target node.

[0079] As a possible implementation, during the migration process, if the source instance to be scheduled has completed its computation or an error occurs during the replication process, the migration is terminated, the key-value cache that has been migrated to the target instance is cleared, and the scheduling module is notified to reject the migration.

[0080] Through the above technical means, by gradually processing the key-value cache, it is ensured that the ongoing requests can smoothly transition to the target instance, while minimizing the impact on the source instance, thus guaranteeing the continuity of the task and the stability of the system.

[0081] Furthermore, as Figure 4 shown, although the migration duration of the entire request depends on the size of the key-value cache, thanks to the migration method of the present invention, the time loss during the migration process is very small, only being the replication time of the key-value cache of one token, which is almost negligible.

[0082] Thus, when migrating the key-value cache between the source instance and the target instance, the present invention adopts a progressive migration strategy for the ongoing requests, so that the time loss during the migration can be ignored.

[0083] Furthermore, in some embodiments, the method for rescheduling requests in the large language model service further includes: determining whether there are multiple instances with a scheduling volume less than a third preset threshold within a preset duration; if there are multiple instances with a scheduling volume less than the third preset threshold within the preset duration, then based on a preset merging strategy, migrating and merging the multiple instances with a scheduling volume less than the third preset threshold within the preset duration, and deleting the idle instances.

[0084] Among them, the preset duration can be a duration preset by those skilled in the art, and the third preset threshold can be a threshold preset by those skilled in the art, which are not specifically limited herein.

[0085] Specifically, when the scheduling volume S value of a certain instance is lower than the third preset threshold L within a certain preset time D, that is < L, then a merging operation is triggered, merging the requests of multiple modules with too low loads, and deleting the idle instances, thereby releasing resources such as GPUs occupied by the instances and freeing up the system resources.

[0086] In detail, when performing the merging operation on multiple instances with too low loads, the source instances to be scheduled are arranged in descending order of the scheduling volume S value in sequence, and the target instances are arranged in ascending order of the scheduling volume S value. When the scheduling volume S value of the target instance is greater than the third preset threshold H, the migration is performed.

[0087] Therefore, the migration times are reduced through the above migration method, and the requests in the instance with lower load are completely migrated to the instance with relatively higher load, thereby releasing the resources of the instance with lower load.

[0088] In order to enable those skilled in the art to further understand the method for requesting rescheduling in the large language model service according to the embodiment of the present invention, it is described in detail below in conjunction with specific embodiments.

[0089] Specifically, the method for requesting rescheduling in the large language model service may include the following steps: Step 1: Deploy the scheduling module and large language model service in the cluster, and initialize each instance of the service. When initializing the instance, you need to start the large language model inference model as well as the instance's monitoring module and migration module; Step 2: The user starts using the large language model service and sends a request to the large language model service. The request is scheduled to the instance using the traditional scheduling strategy. Step 3: The monitoring module in the instance monitors each request entering the instance, including its queue status, the running key-value cache, and calculates the scheduling amount S value; Step 4: The scheduling module obtains the scheduling amount S value from the monitoring module of each instance in real time, and sorts the scheduling amount S value of each instance. First, check whether the scheduling amount S value of each instance is less than the first preset threshold from small to large. When the scheduling amount S value of the instance is less than the first preset threshold, it is determined to be an instance that needs to be scheduled (source instance). Secondly, start looking for the target instance from large to small, and when the scheduling amount S value of the instance is greater than the second preset threshold, add it to the candidate target instance set. Finally, when the candidate target instance set is greater than 1, the instance on the same node as the source instance and with the largest scheduling amount S value is preferentially selected as the target instance; Step 5: The scheduling module sends the migration instruction to the migration modules of the source instance and the target instance; Step 6: After receiving the request, the migration modules in the source and target instances will start the migration process based on the highest priority and first-come-first-served principle; Step 7: The migration module determines the status of the request. If the request is not executed at all or is waiting for video memory during execution, the token and key value cache are directly migrated to the target instance. If the request is being executed, the key value cache is gradually migrated and the calculation is finally resumed on the target instance. Step 8: After the request is successfully migrated, the scheduling module resets the route of the user request from the source instance to the target instance, so that the target instance returns the response result of the large language model request to the user.

[0090] According to the method for rescheduling requests in a large language model service proposed in an embodiment of the present invention, the scheduling amount of each instance is calculated according to the queuing status of the large language model service request and the key-value cache, and each source instance to be scheduled and its corresponding target instance are determined; when a migration instruction is received, based on the type of the large language model service request and the preset migration strategy, the token and / or key-value cache in each source instance to be scheduled is migrated to the corresponding target instance, and the response result of the large language model service request is fed back to the user. Thus, by realizing efficient request allocation in a dynamic environment, the problem that the traditional scheduling strategy cannot effectively schedule when facing the uncertainty and dynamics of large language model requests is solved, thereby improving the resource utilization and response performance of the system.

[0091] Next, the apparatus for requesting rescheduling in a large language model service according to an embodiment of the present invention will be described with reference to the accompanying drawings.

[0092] Figure 5 The figure is a block diagram of an apparatus for requesting rescheduling in a large language model service according to an embodiment of the present invention.

[0093] like Figure 5 As shown, the apparatus 10 for requesting rescheduling in the large language model service includes: an acquisition module 100 , a determination module 200 and a migration module 300 .

[0094] Among them, the acquisition module 100 is used to obtain the queuing status and key value cache of the large language model service request in each instance, and calculate the scheduling amount of each instance based on the queuing status and key value cache of the large language model service request in each instance; the determination module 200 is used to determine at least one source instance to be scheduled and the target instance corresponding to each source instance to be scheduled according to the scheduling amount of each instance; the migration module 300 is used to migrate the tokens and / or key value cache in each source instance to be scheduled to the target instance corresponding to each source instance to be scheduled based on the type of the large language model service request and the preset migration strategy when receiving the migration instruction, and feedback the response result of the large language model service request to the user.

[0095] Further, in some embodiments, the acquisition module 100 is specifically used to: obtain the key value cache occupied by the running requests, the virtual key value cache expected to be occupied by the queued requests, the total video memory capacity of the current instance, the video memory capacity occupied by the parameters inherent in the current large language model inference model, and the batch size of the current large language model inference model; calculate the scheduling amount of each instance based on the key value cache occupied by the running requests, the virtual key value cache expected to be occupied by the queued requests, the total video memory capacity of the current instance, the video memory capacity occupied by the parameters inherent in the current large language model inference model, and the batch size of the current large language model inference model.

[0096] Furthermore, in some embodiments, the acquisition module 100 is specifically used to: obtain the load value of the current instance based on the key value cache occupied by the running requests, the virtual key value cache expected to be occupied by the queued requests, and the video memory capacity occupied by the parameter amount inherent in the current large language model inference model; obtain the remaining space of the current instance based on the difference between the total video memory capacity of the current instance and the load value of the current instance; obtain the scheduling amount of the current instance based on the ratio between the remaining space of the current instance and the batch size of the current large language model inference model.

[0097] Further, in some embodiments, the load value of the current instance is obtained by the sum of the key value cache occupied by the running requests, the virtual key value cache expected to be occupied by the queued requests, and the video memory capacity occupied by the number of parameters inherent in the current large language model inference model.

[0098] Further, in some embodiments, the determination module 200 is used to: determine a set of source instances to be scheduled and a set of candidate target instances based on the scheduling amount of each instance; determine whether the set of candidate target instances is an empty set; if the set of candidate target instances is not an empty set, then based on the set of candidate target instances, filter out the instances that are at the same node as each source instance to be scheduled in the set of source instances to be scheduled and have the largest scheduling amount, and obtain the target instance corresponding to each source instance to be scheduled.

[0099] Further, in some embodiments, the determination module 200 is used to: based on the scheduling amount of each instance, screen out instance combinations with scheduling amounts less than a first preset threshold to obtain a set of source instances to be scheduled; based on the scheduling amount of each instance, determine whether there are instances with scheduling amounts greater than a second preset threshold; if there are instances with scheduling amounts greater than the second preset threshold, combine the instances with scheduling amounts greater than the second preset threshold to obtain a set of candidate target instances.

[0100] Further, in some embodiments, after determining whether the candidate target instance set is an empty set, the determination module 200 is also used to: if the candidate target instance set is an empty set, create a new instance for each source instance to be scheduled; and use the new instance corresponding to each source instance to be scheduled as the target instance corresponding to each source instance to be scheduled.

[0101] Further, in some embodiments, the type of large language model service request is a completely unexecuted large language model service request, and the migration module 300 is specifically used to: based on a preset migration strategy, migrate the tokens in each source instance to be scheduled to the target instance corresponding to each source instance to be scheduled.

[0102] Further, in some embodiments, the type of large language model service request is a large language model service request waiting for video memory during execution, and the migration module 300 is specifically used to: based on a preset migration strategy, migrate the key-value cache in each source instance to be scheduled to the target instance corresponding to each source instance to be scheduled.

[0103] Further, in some embodiments, the type of the large language model service request is the large language model service request being executed, and the migration module 300 is specifically used to: based on a preset migration strategy, copy the existing key value cache of each source instance to be scheduled to the target instance corresponding to each source instance to be scheduled, and mark the end position of this copy of each source instance to be scheduled, and decode and calculate each source instance to be scheduled, and append the newly generated key value cache to the existing key value cache; repeat the above-mentioned copying of the existing key value cache of each source instance to be scheduled to each target instance to be scheduled; The target instance corresponding to the source instance is found, and the end position of this replication of each source instance to be scheduled is marked, and each source instance to be scheduled is decoded and calculated, and the newly generated key value cache is appended to the existing key value cache, until there is a key value cache of one token left in each source instance to be scheduled; the key value cache of the remaining one token is copied to the target instance corresponding to each source instance to be scheduled, the key value cache in each source instance to be scheduled is cleared, and the key value cache that has been migrated to the target instance corresponding to each source instance to be scheduled is spliced, and the calculation is resumed after the splicing is completed.

[0104] Furthermore, in some embodiments, the device 10 for requesting rescheduling in the large language model service is also used to: determine whether there are multiple instances whose scheduling amounts are less than a third preset threshold within a preset time length; if there are multiple instances whose scheduling amounts are less than the third preset threshold within the preset time length, then based on a preset merging strategy, migrate and merge the multiple instances whose scheduling amounts are less than the third preset threshold within the preset time length, and delete the idle instances.

[0105] Furthermore, in some embodiments, before obtaining the queuing status and key-value cache of the large language model service request in each instance, the acquisition module 100 is also used to: determine whether a large language model service request is received; when a large language model service request is received, schedule the large language model service request to multiple instances.

[0106] It should be noted that the above explanation of the embodiment of the method for requesting rescheduling in the large language model service is also applicable to the device for requesting rescheduling in the large language model service of this embodiment, and will not be repeated here.

[0107] According to the device for requesting rescheduling in a large language model service proposed in an embodiment of the present invention, the scheduling amount of each instance is calculated according to the queuing status of the large language model service request and the key-value cache, and each source instance to be scheduled and its corresponding target instance are determined; when a migration instruction is received, based on the type of the large language model service request and the preset migration strategy, the token and / or key-value cache in each source instance to be scheduled is migrated to the corresponding target instance, and the response result of the large language model service request is fed back to the user. Thus, by realizing efficient request allocation in a dynamic environment, the problem that the traditional scheduling strategy cannot effectively schedule when facing the uncertainty and dynamic nature of large language model requests is solved, thereby improving the resource utilization and response performance of the system.

[0108] Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. The electronic device may include: A memory 601 , a processor 602 , and a computer program stored in the memory 601 and executable on the processor 602 .

[0109] When the processor 602 executes the program, the method for requesting rescheduling in the large language model service provided in the above embodiment is implemented.

[0110] Furthermore, the electronic device further comprises: The communication interface 603 is used for communication between the memory 601 and the processor 602 .

[0111] The memory 601 is used to store computer programs that can be executed on the processor 602 .

[0112] The memory 601 may include a high-speed RAM (Random Access Memory) memory, and may also include a non-volatile memory, such as at least one disk memory.

[0113] If the memory 601, the processor 602 and the communication interface 603 are implemented independently, the communication interface 603, the memory 601 and the processor 602 can be connected to each other through a bus and communicate with each other. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0114] Optionally, in a specific implementation, if the memory 601, the processor 602 and the communication interface 603 are integrated on a chip, the memory 601, the processor 602 and the communication interface 603 can communicate with each other through an internal interface.

[0115] The processor 602 may be a CPU (Central Processing Unit), or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present invention.

[0116] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for requesting rescheduling in a large language model service as described above.

[0117] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0118] The above is a detailed introduction to the method for requesting rescheduling in a large language model service provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A method for requesting rescheduling in a large language model service, characterized in that: The following steps are involved: Obtaining the queuing status and key value cache of the large language model service request in each instance, and calculating the scheduling amount of each instance according to the queuing status and key value cache of the large language model service request in each instance; Determine at least one source instance to be scheduled and a target instance corresponding to each source instance to be scheduled according to the scheduling amount of each instance; When a migration instruction is received, based on the type of the large language model service request and the preset migration strategy, the tokens and / or key-value caches in each source instance to be scheduled are migrated to the target instance corresponding to each source instance to be scheduled, and the response result of the large language model service request is fed back to the user.

2. The method for requesting rescheduling in a large language model service according to claim 1, characterized in that: The calculating the scheduling amount of each instance according to the queuing status of the large language model service request in each instance and the key-value cache includes: Obtaining the key value cache occupied by the running request, the virtual key value cache expected to be occupied by the queued request, the total video memory capacity of the current instance, the video memory capacity occupied by the inherent parameters of the current large language model inference model, and the batch size of the current large language model inference model; The scheduling amount of each instance is calculated based on the key value cache occupied by the running request, the virtual key value cache expected to be occupied by the queued request, the total video memory capacity of the current instance, the video memory capacity occupied by the parameters inherent in the current large language model inference model, and the batch size of the current large language model inference model.

3. The method for requesting rescheduling in a large language model service according to claim 2, characterized in that: The calculating the scheduling amount of each instance according to the key value cache occupied by the running request, the virtual key value cache expected to be occupied by the queued request, the total video memory capacity of the current instance, the video memory capacity occupied by the inherent parameter amount of the current large language model inference model, and the batch size of the current large language model inference model includes: Obtaining the load value of the current instance according to the key value cache occupied by the running request, the virtual key value cache expected to be occupied by the queued request, and the video memory capacity occupied by the inherent parameter amount of the current large language model inference model; Obtaining the remaining space of the current instance according to the difference between the total video memory capacity of the current instance and the load value of the current instance; The scheduling amount of the current instance is obtained according to a ratio between the remaining space of the current instance and the batch size of the current large language model inference model.

4. The method for requesting rescheduling in a large language model service according to claim 3, characterized in that: The load value of the current instance is obtained by the sum of the key value cache occupied by the running request, the virtual key value cache expected to be occupied by the queued request, and the video memory capacity occupied by the parameter quantity inherent in the current large language model inference model.

5. The method for requesting rescheduling in a large language model service according to claim 1, characterized in that: The determining, according to the scheduling amount of each instance, at least one source instance to be scheduled and a target instance corresponding to each source instance to be scheduled includes: Based on the scheduling amount of each instance, determining a set of source instances to be scheduled and a set of candidate target instances; Determining whether the candidate target instance set is an empty set; If the candidate target instance set is not an empty set, based on the candidate target instance set, the instance with the largest scheduling amount and the same node as each source instance to be scheduled in the source instance set to be scheduled is screened out to obtain the target instance corresponding to each source instance to be scheduled.

6. The method for requesting rescheduling in a large language model service according to claim 5, characterized in that: The determining, based on the scheduling amount of each instance, a set of source instances to be scheduled and a set of candidate target instances includes: Based on the scheduling amount of each instance, filter out instance combinations whose scheduling amounts are less than a first preset threshold to obtain the set of source instances to be scheduled; Based on the scheduling amount of each instance, determining whether there is an instance whose scheduling amount is greater than a second preset threshold; If there are instances whose scheduling amount is greater than the second preset threshold, the instances whose scheduling amount is greater than the second preset threshold are combined to obtain a set of candidate target instances.

7. The method for requesting rescheduling in a large language model service according to claim 5, characterized in that: After determining whether the candidate target instance set is an empty set, the method further includes: If the candidate target instance set is an empty set, a new instance is created for each source instance to be scheduled; The new instance corresponding to each source instance to be scheduled is used as the target instance corresponding to each source instance to be scheduled.

8. The method for requesting rescheduling in a large language model service according to claim 1, characterized in that: The type of the large language model service request is a completely unexecuted large language model service request, and the step of migrating the token and / or key-value cache in each source instance to be scheduled to the target instance corresponding to each source instance to be scheduled based on the type of the large language model service request and a preset migration strategy includes: Based on the preset migration strategy, the token in each source instance to be scheduled is migrated to the target instance corresponding to each source instance to be scheduled.

9. The method for requesting rescheduling in a large language model service according to claim 1, characterized in that: The type of the large language model service request is a large language model service request waiting for video memory during execution, and the token and / or key value cache in each source instance to be scheduled is migrated to the target instance corresponding to each source instance to be scheduled based on the type of the large language model service request and a preset migration strategy, including: Based on the preset migration strategy, the key-value cache in each source instance to be scheduled is migrated to the target instance corresponding to each source instance to be scheduled.

10. The method for requesting rescheduling in a large language model service according to claim 1, characterized in that: The type of the large language model service request is the large language model service request being executed, and the migration of the token and / or key-value cache in each source instance to be scheduled to the target instance corresponding to each source instance to be scheduled based on the type of the large language model service request and a preset migration strategy includes: Based on the preset migration strategy, the existing key-value cache of each source instance to be scheduled is copied to the target instance corresponding to each source instance to be scheduled, and the end position of this copy of each source instance to be scheduled is marked, and the newly generated key-value cache calculated by decoding each source instance to be scheduled is appended to the existing key-value cache; Repeat the above steps of copying the existing key-value cache of each source instance to be scheduled to the target instance corresponding to each source instance to be scheduled, marking the end position of this copy of each source instance to be scheduled, and appending the newly generated key-value cache by decoding and calculating each source instance to be scheduled to the existing key-value cache, until a key-value cache of one token remains in each source instance to be scheduled; The key-value cache of the remaining token is copied to the target instance corresponding to each source instance to be scheduled, the key-value cache in each source instance to be scheduled is cleared, and the key-value cache that has been migrated to the target instance corresponding to each source instance to be scheduled is spliced, and the calculation is resumed after the splicing is completed.

11. The method for requesting rescheduling in a large language model service according to claim 1, characterized in that: Also includes: Determine whether there are multiple instances in which the scheduling amount is less than a third preset threshold within a preset time period; If there are multiple instances whose scheduling amounts are less than the third preset threshold within the preset duration, based on the preset merging strategy, the multiple instances whose scheduling amounts are less than the third preset threshold within the preset duration are migrated and merged, and the idle instances are deleted.

12. The method for requesting rescheduling in a large language model service according to claim 1, characterized in that: Before obtaining the queue status and key-value cache of large language model service requests in each instance, it also includes: Determine whether a large language model service request is received; When a large language model service request is received, the large language model service request is dispatched to multiple instances.

13. A device for requesting rescheduling in a large language model service, characterized in that: include: An acquisition module, used to acquire the queuing status and key value cache of the large language model service request in each instance, and calculate the scheduling amount of each instance according to the queuing status and key value cache of the large language model service request in each instance; A determination module, configured to determine at least one source instance to be scheduled and a target instance corresponding to each source instance to be scheduled according to the scheduling amount of each instance; A migration module is used to migrate the tokens and / or key-value caches in each source instance to be scheduled to the target instance corresponding to each source instance to be scheduled based on the type of the large language model service request and a preset migration strategy when a migration instruction is received, and to feed back the response result of the large language model service request to the user.

14. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for requesting rescheduling in a large language model service as described in any one of claims 1 to 12.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program is executed by a processor to implement the method for requesting rescheduling in a large language model service according to any one of claims 1 to 12.

Images