Multi-low-rank adapter distributed reasoning scheduling method based on resource decoupling

Abstract

The invention discloses a multi-low-rank adapter distributed reasoning scheduling method, device and equipment based on resource decoupling and a storage medium, and is applied to task sending equipment, and the method comprises the following steps: obtaining a request queue containing a plurality of reasoning task requests and request record data containing completed historical task requests; determining a sending priority according to the output data and the timestamp corresponding to the reasoning task request and the request record data; wherein the sending priority is in negative correlation with the estimated deadline of the reasoning task request; selecting the reasoning task request with the highest sending priority in the request queue as a target reasoning task request; and sending the target reasoning task request to task execution equipment to execute the target reasoning task request. According to the method, resource scrambling and mutual interference between tasks are avoided, and the problem that the service satisfaction rate is reduced due to mutual resource scrambling and mutual interference between services in the resource decoupling reasoning process is solved.

Description

Technical Field

[0001] The present application belongs to the field of distributed reasoning, and specifically relates to a multi-low-rank adapter distributed reasoning scheduling method, apparatus, device and storage medium based on resource decoupling. Background Art

[0002] With the rapid development of Large Language Model (LLM) and its related applications, the scale and number of models are increasing. How to efficiently perform resource decoupled reasoning has become an important issue in distributed machine learning.

[0003] In the prior art, the Low-Rank Adapter (LoRA) significantly reduces the amount of trainable parameters for fine-tuning tasks by fixing the basic model parameters and adding small trainable matrices to the model. At the same time, it can achieve model effects equivalent to full fine-tuning while reducing costs. In this way, LLM can share the same set of basic model parameters and conveniently perform reasoning services in the same reasoning instance. This method can not only share the basic model and reduce the memory usage of the graphics processing unit (GPU), but also increase the batch size during reasoning through efficient parallel computing, thereby improving the utilization of computing resources.

[0004] However, simply co-locating these inference services will cause the services to compete for resources and interfere with each other, leading to serious head-of-line blocking problems and a decrease in the service level objective (SLO). Summary of the invention

[0005] The present application aims to provide a multi-low-rank adapter distributed reasoning scheduling method, device, electronic device and computer-readable storage medium based on resource decoupling, at least to solve the problem of decreased service satisfaction rate caused by competition for resources and interference between services during the resource decoupling reasoning process.

[0006] In a first aspect, an embodiment of the present application discloses a multi-low-rank adapter distributed reasoning scheduling method based on resource decoupling, which is applied to a task sending device, including:

[0007] Acquire a request queue including multiple reasoning task requests and request record data including completed historical task requests; the reasoning task request records output data and execution mode corresponding to the reasoning task request, and multiple timestamps of the reasoning task request in the process of flowing between the task sending device and the task execution device; the request record data records multiple timestamps of the historical task request in the process of flowing between the task sending device and the task execution device;

[0008] Determine the sending priority of the reasoning task request according to the output data and timestamp corresponding to the reasoning task request and the request record data; the sending priority is negatively correlated with the estimated deadline of the corresponding reasoning task request;

[0009] The reasoning task request with the highest sending priority in the request queue is selected as the target reasoning task request;

[0010] The target reasoning task request is sent to the task execution device, so that the target reasoning task request is executed on the task execution device in an execution mode corresponding to the target reasoning task request.

[0011] In a second aspect, the embodiment of the present application further discloses a multi-low-rank adapter distributed reasoning scheduling device based on resource decoupling, which is applied to a task sending device, including:

[0012] An information acquisition module, used to acquire a request queue containing multiple reasoning task requests, and request record data containing completed historical task requests; the reasoning task request records output data and execution methods corresponding to the reasoning task request, and multiple timestamps of the reasoning task request in the process of flowing between the task sending device and the task execution device; the request record data records multiple timestamps of the historical task request in the process of flowing between the task sending device and the task execution device;

[0013] A priority calculation module, used to determine the sending priority of the reasoning task request according to the output data and timestamp corresponding to the reasoning task request and the request record data; the sending priority is negatively correlated with the estimated deadline of the corresponding reasoning task request;

[0014] A task selection module, used for selecting the reasoning task request with the highest sending priority in the request queue as the target reasoning task request;

[0015] The task sending module is used to send the target reasoning task request to the task execution device, so that the target reasoning task request is executed on the task execution device according to the execution mode corresponding to the target reasoning task request.

[0016] In a third aspect, an embodiment of the present application further discloses an electronic device, comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0017] In a fourth aspect, an embodiment of the present application further discloses a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0018] In summary, in the embodiment of the present application, the sending priority of each task determined by analyzing the output data and timestamp of the reasoning task request is made so that the sending priority of the task is negatively correlated with the estimated deadline of the corresponding reasoning task request. Based on the characteristic that the earlier the estimated deadline, the faster the corresponding reasoning task request is completed, the model is ensured to execute the reasoning task as quickly as possible. At the same time, by establishing the sending priority, the resource competition and mutual interference between tasks are effectively avoided, and the overall operation efficiency and stability of the system are improved. Then, according to the sending priority of each reasoning task request, the target reasoning task request is selected in the request queue, and finally it is ensured that the high-priority task can be processed first, thereby maximizing the use of system resources and improving the timeliness and accuracy of task processing. Therefore, the method based on the embodiment of the present application avoids resource competition and mutual interference between tasks, and solves the problem of decreased service satisfaction rate caused by competition for resources and interference between services in the resource decoupling reasoning process. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In the attached picture:

[0020] Figure 1 It is a flowchart of the steps of a multi-low-rank adapter distributed reasoning scheduling method based on resource decoupling provided by an embodiment of the present application;

[0021] Figure 2 It is a flowchart of another method for distributed reasoning scheduling of multiple low-rank adapters based on resource decoupling provided by an embodiment of the present application;

[0022] Figure 3 It is a program execution step under the method provided in the embodiment of the present application;

[0023] Figure 4 It is a block diagram of a multi-low-rank adapter distributed reasoning scheduling device based on resource decoupling provided by an embodiment of the present application;

[0024] Figure 5 is a block diagram of an electronic device according to an embodiment of the present application;

[0025] Figure 6 It is a block diagram of an electronic device of another embodiment provided by the embodiments of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0027] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0028] Figure 1 This embodiment provides a multi-low-rank adapter distributed reasoning scheduling method based on resource decoupling, which is applied to a task sending device.

[0029] The method may include the following steps:

[0030] Step 101 , obtaining a request queue including multiple reasoning task requests and request record data including completed historical task requests.

[0031] Among them, the inference task request records the output data and execution method corresponding to the inference task request, as well as multiple timestamps of the inference task request in the process of flowing between the task sending device and the task execution device; the request record data records multiple timestamps of historical task requests in the process of flowing between the task sending device and the task execution device.

[0032] In some embodiments of the present application, a request queue containing multiple reasoning task requests is obtained. Each reasoning task request records the output data and execution mode corresponding to the request, as well as multiple timestamps of the request during the process of flowing between the task sending device and the task execution device. These timestamps include the arrival time, start execution time, and completion time of the request. By obtaining this information, the system can effectively manage and schedule each request.

[0033] For example, in a distributed reasoning system, a user submits multiple LoRA reasoning task requests. These requests are received by the system and stored in a request queue. Each request contains the following information: output data: for example, a generated text paragraph or code snippet; execution mode: for example, using a specific LoRA model for reasoning; timestamp: including request arrival time, start execution time, and completion time. By obtaining these requests and their related information, the system can perform reasonable scheduling according to the specific needs and time requirements of each request, ensuring efficient resource utilization and service satisfaction.

[0034] Step 102, determining the sending priority of the reasoning task request according to the output data and timestamp corresponding to the reasoning task request, and the request record data.

[0035] Among them, the sending priority is negatively correlated with the estimated deadline of the corresponding reasoning task request.

[0036] In some embodiments of the present application, the sending priority of each reasoning task request is determined based on the output data and timestamp corresponding to the reasoning task request. The sending priority is used to characterize the service satisfaction rate (SLO) of the corresponding task. Specifically, the system analyzes the output data and timestamp information of each request, including the arrival time, start execution time, and completion time of the request, and uses this information to evaluate the urgency and importance of each request, thereby determining its sending priority.

[0037] For example, in a distributed reasoning system, suppose there are two LoRA reasoning task requests A and B: the output data is a text summary, and has the corresponding request arrival time and expected completion time. The system will evaluate the SLO satisfaction rate of each request based on the output data and timestamp information of these requests. For example, if the SLO requirement of request A is more stringent, the system will assign a higher sending priority to request A to ensure that it can be completed within the specified time. In this way, the system can effectively manage and schedule multiple reasoning task requests, ensure that high-priority tasks can be processed first, and improve the operating efficiency and stability of the overall system.

[0038] Step 103 : Select the inference task request with the highest sending priority in the request queue as the target inference task request.

[0039] In some embodiments of the present application, the reasoning task request with the highest sending priority in the request queue is selected as the target reasoning task request. The system first evaluates the sending priority of each request, which is based on the service satisfaction rate determined in the previous step. Then, the system arranges the requests in order from high to low priority to ensure that high-priority tasks can be processed first. This process helps to optimize the overall performance and stability of the system and ensure the timely execution of key tasks.

[0040] For example, in a distributed reasoning system, suppose there are three LoRA reasoning task requests in the request queue: A, B, and C, where request A has the highest sending priority and request C has the lowest sending priority. The system will sort them according to these priorities and first select request A with the highest priority for processing. In this way, the system ensures that high-priority tasks can be processed first, thereby improving the timeliness and accuracy of task processing, maximizing the use of system resources, and avoiding resource competition and mutual interference between tasks.

[0041] Step 104 , sending the target reasoning task request to the task execution device, so that the task execution device executes the target reasoning task request in an execution mode corresponding to the target reasoning task request.

[0042] In some embodiments of the present application, the selected target reasoning task request is sent to the task execution device so that these target reasoning tasks can be executed on the task execution device. The task execution device is usually a high-performance computing device, such as a GPU cluster or a distributed computing system, which can efficiently process complex reasoning tasks. By sending the target reasoning task request to these devices, the system can use its powerful computing power to ensure that the reasoning task can be completed quickly and accurately.

[0043] For example, in a distributed reasoning system, assume that the system has selected the target reasoning task request A based on priority. Request A requires the use of a specific LoRA model for text generation. The system sends request A to the task execution device, which may be a cluster of multiple GPUs. After receiving request A, the task execution device starts to execute the corresponding reasoning task and generates the required text output. In this way, the system can efficiently process multiple reasoning task requests and ensure that each request can be completed within the specified time, thereby improving the operating efficiency and stability of the overall system.

[0044] In summary, in the embodiment of the present application, the sending priority of each task is determined by analyzing the output data and timestamp of the reasoning task request, so that the sending priority of the task is negatively correlated with the service satisfaction rate of the corresponding reasoning task request. Based on the characteristic that the higher the service satisfaction rate, the faster the corresponding reasoning task request is completed, it is ensured that the model executes the reasoning task as quickly as possible. At the same time, by establishing the sending priority, the resource competition and mutual interference between tasks are effectively avoided, and the overall operation efficiency and stability of the system are improved. Then, according to the sending priority of each reasoning task request, the target reasoning task request is selected in the request queue, and finally it is ensured that the high-priority task can be processed first, thereby maximizing the use of system resources and improving the timeliness and accuracy of task processing. Therefore, the method based on the embodiment of the present application avoids resource competition and mutual interference between tasks, and solves the problem of decreased service satisfaction rate caused by competition for resources and interference between services in the resource decoupling reasoning process.

[0045] Figure 2 Another multi-low-rank adapter distributed reasoning scheduling method based on resource decoupling provided in the application embodiment is applied to a task sending device:

[0046] Step 201 , obtaining a request queue including multiple reasoning task requests and request record data including completed historical task requests.

[0047] Among them, the inference task request records the output data and execution method corresponding to the inference task request, as well as multiple timestamps of the inference task request in the process of flowing between the task sending device and the task execution device; the request record data records multiple timestamps of historical task requests in the process of flowing between the task sending device and the task execution device.

[0048] The method shown in this step has been explained in step 101 and will not be repeated here.

[0049] Step 202: Determine the sending priority of the reasoning task request according to the output data and timestamp corresponding to the reasoning task request and the request record data.

[0050] Among them, the sending priority is negatively correlated with the estimated deadline of the corresponding reasoning task request.

[0051] The method shown in this step has been explained in step 102 and will not be repeated here.

[0052] Optionally, step 202 includes the following sub-steps:

[0053] Sub-step 2021, determining an estimated deadline for sending the reasoning task request based on the output data and timestamp corresponding to the reasoning task request, as well as the request record data.

[0054] In some embodiments of the present application, the estimated deadline for sending the reasoning task request will be determined based on the output data and timestamp corresponding to the reasoning task request. The estimated deadline for sending refers to the time point at which each request is estimated by the system to be completed. By analyzing the output data and timestamp information of the request (such as the request arrival time, the start execution time, etc.), the system can calculate the estimated deadline for each request. This estimated deadline will be used to determine the sending priority of the request in subsequent steps.

[0055] For example, in a distributed reasoning system, assuming there is a LoRA reasoning task request A, the system will calculate its estimated deadline for sending based on the output data and timestamp information of request A. In this way, the system can determine a reasonable estimated deadline for each request, thereby providing a basis for subsequent priority determination.

[0056] Optionally, sub-step 2021 includes the following sub-steps:

[0057] In sub-step 20211, when the timestamp indicates that the inference task request arrives at the task sending device for the first time, the sum of the time when the inference task request first arrives at the task sending device and the time required for the inference task request to generate the first token is used as the estimated deadline for sending.

[0058] In some embodiments of the present application, when the timestamp characterizes the reasoning task request as arriving at the task sending device for the first time, the sum of the time when the reasoning task request first arrives at the task sending device and the time required for the reasoning task request to generate the first token is used as the estimated deadline for sending. Specifically, when an reasoning task request first arrives at the task sending device, the system records the arrival time of the request, and calculates the estimated deadline for sending the request in combination with the time required to generate the first token. This estimated deadline is used to determine the sending priority of the request in subsequent steps. The corresponding formula for the above process is:

[0059] DDL = arrival ts +TTFT,

[0060] Among them, DDL represents the estimated deadline, arrival ts Indicates the time when the inference task request first arrives at the task sending device, and TTFT indicates the time required for the inference task request to generate the first token. It should be noted that since the solution requirements of this application can only be met when the actual generation time does not exceed TTFT, the TTFT here is actually an estimated value, that is, the TTFT here is the definition of the maximum time required to generate the first token.

[0061] For example, in a distributed reasoning system, suppose there is a LoRA reasoning task request F: the time of first arrival at the task sending device is 00:00:05, and the time required to generate the first token is 5 seconds. The system will add the first arrival time of request F 00:00:05 and the time required to generate the first token (5 seconds), and the estimated deadline for sending is 00:00:10. In this way, the system can determine a reasonable estimated deadline for each request, thereby providing a basis for subsequent priority determination.

[0062] In sub-step 20212, when the timestamp represents the reasoning task request as outputting the first token, the product of the average time required for the reasoning task request to generate a token and the total output length of the reasoning task request minus one, and the timestamp of outputting the first token are taken as the estimated deadline for sending.

[0063] In some embodiments of the present application, when the timestamp characterizes the reasoning task request as outputting the first token, the product of the average time required for the reasoning task request to generate a token and the total output length of the request minus one, and the timestamp of the output of the first token are used as the estimated deadline for sending. Specifically, when the reasoning task request has output the first token, the system records the timestamp and calculates the estimated deadline for sending the request based on the average time required to generate each token and the total output length of the request. This estimated deadline is used to determine the sending priority of the request in subsequent steps. The corresponding formula for the above process is:

[0064]

[0065] Among them, DDL means estimated deadline, prefill ts It indicates the timestamp of the first token output, and TPOT indicates the average time required for an inference task request to generate a token; Indicates the estimated total output length of the request. It should be noted that the TPOT here is actually an estimated value, because the solution requirements of this application can only be met when the actual generation time does not exceed TPOT. That is, TPOT here is the definition of the average time required to generate a token.

[0066] For example, in a distributed reasoning system, suppose there is a LoRA reasoning task request G: the timestamp of the output of the first token is 00:00:05, the average time required to generate a token is 2 seconds, and the total output length is 10 tokens. Then the system will add the timestamp of the output of the first token of request G (00:00:05) to the average time required to generate a token (2 seconds) multiplied by the total output length minus one (10-1=9), and the estimated deadline for sending is 00:00:05+(2 seconds*9)=00:00:23. In this way, the system can determine a reasonable estimated deadline for each request, thereby providing a basis for subsequent priority determination.

[0067] Sub-step 20213, when the length of the output data is greater than the output length threshold estimated for the inference task request, the sum of the estimated sending deadline and the average time required for the inference task request to generate a token is updated as the new estimated sending deadline.

[0068] In some embodiments of the present application, when the length of the output data is greater than the estimated output length threshold for the reasoning task request, the sum of the estimated sending deadline and the average time required for the reasoning task request to generate a token is updated to the new estimated sending deadline. Specifically, when the system detects that the actual output length of the reasoning task request exceeds the estimated output length threshold, it recalculates the estimated sending deadline to ensure that the new estimated deadline can more accurately reflect the actual completion time of the task. This adjustment helps to improve the scheduling accuracy and SLO satisfaction rate of the system. The corresponding formula for the above process is:

[0069] DDL t =DDL0+TPOT,

[0070] Among them, DDL t It represents the estimated deadline after the update, DDL0 represents the estimated deadline before the update, and TPOT represents the average time required for an inference task request to generate a token.

[0071] For example, in a distributed reasoning system, suppose there is a LoRA reasoning task request H, and the initial estimated output length is 10 tokens. When the output length is 11 tokens, assume that the average time required to generate a token is 2 seconds. Then assume that the system initially calculated the estimated sending deadline to be 00:00:20 (based on the output length of 10 tokens). However, when the system detects that the actual output length is 11 tokens, it recalculates the estimated sending deadline. The new estimated sending deadline is 00:00:20+(2 seconds*(11-10))=00:00:22. In this way, the system can dynamically adjust the estimated deadline to ensure the accuracy and effectiveness of the scheduling strategy.

[0072] Sub-step 2022, using the value of the estimated deadline to be sent as the sending priority of the corresponding inference task request.

[0073] In some embodiments of the present application, the value of the estimated deadline for sending is used as the sending priority of the corresponding reasoning task request. The estimated deadline for sending is the expected completion time point calculated by the system based on the output data and timestamp of the request. By directly using this estimated deadline as the sending priority, the system can ensure that tasks that need to be completed as soon as possible are processed first, thereby improving the service satisfaction rate of the overall system.

[0074] For example, in a distributed reasoning system, suppose there are two LoRA reasoning task requests D and E. Suppose the system determines the estimated deadline for sending request D as 00:00:10, and the estimated deadline for sending request E as 00:00:15 as its sending priority. Then the system can prioritize request D because its estimated deadline is earlier, ensuring that it can be completed as soon as possible, thereby improving the timeliness and accuracy of task processing.

[0075] Step 203: adjusting the sending priority of the reasoning task request according to the output data, execution mode and timestamp corresponding to the reasoning task request.

[0076] In some embodiments of the present application, the sending priority of the reasoning task request will be adjusted according to the output data, execution mode and timestamp corresponding to the reasoning task request. The system will comprehensively consider the output data, execution mode and timestamp information of each request, and dynamically adjust its sending priority to ensure that the service of the task can be better satisfied. Through this adjustment mechanism, the system can respond to different types of requests more flexibly, optimize resource allocation, and improve the operating efficiency and stability of the overall system.

[0077] For example, in a distributed reasoning system, suppose there are three LoRA reasoning task requests: requests A, B, and C. The system will evaluate the current priority of each request based on the output data, execution mode, and timestamp information of these requests. For example, suppose the actual execution time of request A is lower than expected, and the system has determined a low priority for it in the previous step, which will cause request A to be unable to be executed for a long time (requests B and C are always executed due to their higher priority). At this time, by adjusting the priority of request A and advancing it, A will not always be placed in the queue and cannot be executed. Through this dynamic adjustment, the system can more effectively manage and schedule multiple reasoning task requests, ensuring that high-priority tasks are given priority while all priority tasks are given priority.

[0078] Optionally, step 203 includes the following sub-steps:

[0079] Sub-step 2031, calculating the violation parameters of the reasoning task request according to the output data, execution mode and timestamp corresponding to the reasoning task request.

[0080] In some embodiments of the present application, the violation parameter of the reasoning task request is calculated based on the output data, execution mode and timestamp corresponding to the reasoning task request. The violation parameter is used to quantify the extent to which the reasoning task request fails to meet the service during the execution process. Specifically, the system analyzes the output data, execution mode and timestamp information of each request, calculates the difference between the actual execution time and the expected execution time of the request, and thus obtains the violation parameter. This parameter will be used to correct the estimated deadline in subsequent steps.

[0081] For example, in a distributed reasoning system, suppose there is a LoRA reasoning task request L. The system calculates its violation parameters based on the output data, execution mode, and timestamp information of request L. In this way, the system can quantify the extent to which each request fails to meet the SLO, providing a basis for subsequent priority adjustments.

[0082] Optionally, sub-step 2031 includes the following sub-steps:

[0083] Sub-step 20311, determining the target sub-queue of the request queue according to the execution mode of the reasoning task request.

[0084] The execution mode corresponding to the subqueue reasoning task request in the target subqueue is the same as the execution mode of the reasoning task request.

[0085] In some embodiments of the present application, the target subqueue of the request queue is determined according to the execution mode of the reasoning task request. The target subqueue refers to the subqueue containing all reasoning task requests with the same execution mode as the current reasoning task request. By classifying requests with the same execution mode into the same subqueue, the system can manage and schedule these requests more effectively, ensuring that tasks of the same type can be processed consistently.

[0086] For example, in a distributed reasoning system, suppose there are multiple LoRA reasoning task requests, and each request is executed in a different way: request K: the execution mode is to use LoRA model 1; request L: the execution mode is to use LoRA model 2; request M: the execution mode is to use LoRA model 1. Then the system will classify request K and request M into target subqueue 1 (execution mode is LoRA model 1) according to the execution mode of each request, and classify request L into target subqueue 2 (execution mode is LoRA model 2).

[0087] Sub-step 20312, determining a first number of first sub-queue reasoning task requests and a second number of second sub-queue reasoning task requests in the target sub-queue according to the timestamps respectively corresponding to the sub-queue reasoning task requests in the target sub-queue.

[0088] The first subqueue reasoning task request is a subqueue reasoning task request that has been selected in the current task round, and the second subqueue reasoning task request is a subqueue reasoning task request that has not been selected in the current task round.

[0089] In some embodiments of the present application, a first number of first subqueue reasoning task requests and a second number of second subqueue reasoning task requests in the target subqueue are determined based on the timestamps respectively corresponding to the subqueue reasoning task requests in the target subqueue. The first subqueue reasoning task request refers to a request that has been selected in the current task round, and the second subqueue reasoning task request refers to a request that has not been selected in the current task round. By distinguishing these two types of requests, the system can better understand the execution status of the current task and provide a basis for subsequent scheduling decisions.

[0090] For example, in a distributed reasoning system, suppose that in the current task round, there are the following reasoning task requests in the target subqueue: request N has been selected, request O has not been selected, request P has been selected, and request Q has not been selected. Then the system will determine the number of reasoning task requests in the first subqueue in the target subqueue (the number of selected requests) as 2 (request N and request P), and the number of reasoning task requests in the second subqueue (the number of unselected requests) as 2 (request O and request Q) based on these timestamp information. In this way, the system can more clearly understand the execution status of the current task and provide accurate data support for subsequent scheduling decisions.

[0091] Sub-step 20313, taking the difference between the total number of selected inference task requests and the first number, the minimum value of the second number, and the ratio of the second number as the interference parameter of the inference task request.

[0092] In some embodiments of the present application, the difference between the total number of selected reasoning task requests and the first number, the minimum value of the second number, and the ratio of the second number are used as interference parameters of the reasoning task request. The interference parameter is used to quantify the degree to which a reasoning task request is interfered with by other requests during the scheduling process. By calculating this parameter, the system can more accurately evaluate the priority adjustment requirements of each request, thereby optimizing the scheduling strategy and ensuring the service satisfaction rate. The corresponding formula for the above process is:

[0093]

[0094] Among them, interference cnt represents the disturbed parameter, x represents the first quantity, y represents the second quantity, |scheduled reqs |Indicates the total number of selected inference task requests.

[0095] For example, in a distributed reasoning system, suppose that in the current task round, 8 reasoning task requests are selected (that is, |scheduled reqs |=8). For the unselected request, there are 2 task requests with the same execution mode (that is, the same LoRA type) in the selected task requests, that is, the first number is 2; there are 3 task requests with the same execution mode (that is, the same LoRA type) in the unselected task requests, that is, the second number is 3. Then the difference between the total number of selected inference task requests and the first number is 8-2=6, and the minimum value between it and the second number is min(6,3)=3. At this time, the interference parameter calculation is the ratio of this minimum value to the second number 3: 3 / 3=1. In this way, the system can quantify the degree of interference each request receives during the scheduling process, provide a basis for subsequent priority adjustments, and ensure the fairness and effectiveness of the scheduling strategy.

[0096] Sub-step 20314, taking the ratio of the difference between the last timestamp of the inference task request and the estimated deadline of the inference task request, the maximum value among the zero values, and the single batch running time of the inference task request as the estimated batch number of the inference task request.

[0097] In some embodiments of the present application, the difference between the last timestamp of the reasoning task request and the estimated deadline of the reasoning task request, the maximum value among the zero values, and the ratio of the single batch running time of the reasoning task request are used as the estimated batch number of the reasoning task request. The estimated batch number is used to quantify the number of batches required for an reasoning task request during execution. By calculating this parameter, the system can more accurately evaluate the execution time requirements of each request, thereby optimizing the scheduling strategy and ensuring the service satisfaction rate. The corresponding formula for the above process is:

[0098]

[0099] Among them, advance cnt Indicates the estimated number of batches, completion ts Indicates the last timestamp, DDL indicates the estimated deadline, t iter Indicates the single batch running time.

[0100] For example, in a distributed reasoning system, suppose there is a LoRA reasoning task request V: the last timestamp is 00:00:30, the estimated deadline is 00:00:20, and the single batch running time is 2 seconds. Then the system will calculate the estimated number of batches for request V as follows: the difference between the last timestamp and the estimated deadline: 00:00:30-00:00:20 = 10 seconds. This value and the maximum value of the zero value max(10 seconds,0) are 10 seconds. Then the ratio: 10 seconds / 2 seconds = 5. Therefore, the estimated number of batches is 5. In this way, the system can quantify the number of batches required for each request during execution, provide a basis for subsequent priority adjustments, and ensure the accuracy and effectiveness of the scheduling strategy.

[0101] In sub-step 20315, the ratio of the estimated number of batches requested by the inference task to the disturbed parameter is used as the violation parameter of the inference task request.

[0102] In some embodiments of the present application, the ratio of the estimated number of batches of the reasoning task request and the disturbed parameter is used as the violation parameter of the reasoning task request. The violation parameter is used to quantify the degree to which the reasoning task request fails to meet the service during the execution process. By calculating the ratio of the estimated number of batches and the disturbed parameter, the system can more accurately evaluate the priority adjustment requirements of each request, thereby optimizing the scheduling strategy and ensuring the satisfaction rate of the SLO. The corresponding formula for the above process is:

[0103]

[0104] For example, in a distributed reasoning system, suppose there is a LoRA reasoning task request L with an estimated batch number of 2 and a interference parameter of 5. Then the system will calculate the ratio of the estimated batch number (2) and the interference parameter (5) of request W as 2 / 5=0.4. This ratio is the violation parameter of request W. In this way, the system can quantify the extent to which each request fails to meet the SLO during execution, provide a basis for subsequent priority adjustments, and ensure the fairness and effectiveness of the scheduling strategy.

[0105] Sub-step 2032, using the violation parameter as a delay correction coefficient to correct the estimated deadline of the reasoning task request, and using the corrected estimated deadline as the adjusted sending priority.

[0106] In some embodiments of the present application, the violation parameter will be used as a delay correction coefficient to correct the estimated deadline of the reasoning task request, and the corrected estimated deadline will be used as the adjusted sending priority. Specifically, the system will use the violation parameter calculated in sub-step 2031 to correct the estimated deadline of each request. The corrected estimated deadline will reflect the delay encountered by the request during the actual execution process, so as to more accurately determine its sending priority. This adjustment helps to optimize the system's scheduling strategy and improve service satisfaction.

[0107] For example, in a distributed reasoning system, suppose there is a LoRA reasoning task request L with an estimated deadline of 00:00:12 and a violation parameter of 0.2. Then the system will introduce the violation parameter 0.2 as a delay correction coefficient into the estimated deadline of request L to obtain a new estimated deadline. The corrected estimated deadline will be used as the adjusted sending priority of request L, ensuring that the system can schedule requests more accurately and balance the timeliness of task processing.

[0108] Optionally, in order to use the violation parameter as a delay correction coefficient to correct the estimated deadline of the reasoning task request, sub-step 2032 includes the following sub-steps:

[0109] Sub-step 20321, the product of the difference between the number one and the violation parameter and the difference between the estimated deadline of the reasoning task request and the time when it first arrives at the task sending device is taken as the arrival time correction of the reasoning task request.

[0110] In some embodiments of the present application, the difference between the number one and the violation parameter, and the product of the difference between the estimated deadline of the reasoning task request and the time of first arrival at the task sending device, are used as the arrival time correction of the reasoning task request. Specifically, the system calculates a correction to adjust the estimated deadline of the reasoning task request. This correction is obtained by multiplying the difference between a constant value and the violation parameter by the difference between the estimated deadline and the first arrival time. In this way, the system can dynamically adjust the estimated deadline of the request to more accurately reflect its actual execution requirements.

[0111] For example, in a distributed reasoning system, suppose there is a LoRA reasoning task request X: the time of first arrival at the task sending device is 00:00:00, the estimated deadline is 00:00:20, and the violation parameter is 0.2. Then the system will calculate the arrival time correction of request X as follows: the difference between the estimated deadline and the first arrival time: 00:00:20-00:00:00 = 20 seconds; the difference between the constant value and the violation parameter: 1 -0.2 = 0.8; arrival time correction: 0.8*20 seconds = 16 seconds. In this way, the system can calculate the arrival time correction of request X, providing a basis for the subsequent adjustment of the estimated deadline.

[0112] Sub-step 20322, updating the sum of the time when the reasoning task request first arrives at the task sending device and the arrival time correction amount as the corrected estimated deadline time of the reasoning task request.

[0113] In some embodiments of the present application, the sum of the time when the reasoning task request first arrives at the task sending device and the arrival time correction amount is updated as the corrected estimated deadline time of the reasoning task request. Specifically, the system adds the time when the reasoning task request first arrives at the task sending device to the arrival time correction amount calculated in the previous step to obtain the corrected estimated deadline time. This corrected estimated deadline time will be used for subsequent priority adjustments and scheduling decisions to ensure that the system can more accurately meet the service.

[0114] For example, in a distributed reasoning system, suppose there is a LoRA reasoning task request X with an arrival time correction of 16 seconds. Then the system will add the first arrival time of request X, 00:00:02, to the arrival time correction (16 seconds), and the corrected estimated deadline is 00:00:18. In this way, the system can dynamically adjust the estimated deadline of the request to ensure the accuracy and effectiveness of the scheduling strategy, thereby improving the timeliness and accuracy of task processing.

[0115] The corresponding formulas of step 20321 and step 20322 are:

[0116] priority=arrival ts +(1-PoIiV)·(DDL-arrival ts ),

[0117] Among them, priority indicates the estimated deadline after the update, DDL indicates the estimated deadline before the update, and arrival ts It indicates the time when the inference task request first arrives at the task sending device, and PoIiV indicates the violation parameter.

[0118] Step 204 : Select the inference task request with the highest sending priority in the request queue as the target inference task request.

[0119] The method shown in this step has been explained in step 103 and will not be repeated here.

[0120] Step 205 , sending the target reasoning task request to the task execution device, so that the task execution device executes the target reasoning task request in an execution mode corresponding to the target reasoning task request.

[0121] The method shown in this step has been described in step 104 and will not be repeated here.

[0122] Figure 3 The program execution steps under the method provided in the embodiment of the present application are:

[0123] Step S1: Calculate the request deadline of each reasoning task request according to the historical data of the reasoning task requests contained in the request queue and the service satisfaction rate requirement.

[0124] Step S2: Send the task with the requested deadline to the priority adjuster. At this time, the task is in the waiting-to-send state.

[0125] Step S3: Adjust the priority through the interference detector.

[0126] Step S4: Send the task with the highest priority in the adjusted inference task request in the to-be-sent state to the inference engine so that the task can be executed.

[0127] In summary, in the embodiment of the present application, the sending priority of each task is determined by analyzing the output data and timestamp of the reasoning task request, so that the sending priority of the task is negatively correlated with the service satisfaction rate of the corresponding reasoning task request. Based on the characteristic that the higher the service satisfaction rate, the faster the corresponding reasoning task request is completed, it is ensured that the model executes the reasoning task as quickly as possible. At the same time, by establishing the sending priority, the resource competition and mutual interference between tasks are effectively avoided, and the overall operation efficiency and stability of the system are improved. Then, according to the sending priority of each reasoning task request, the target reasoning task request is selected in the request queue, and finally it is ensured that the high-priority task can be processed first, thereby maximizing the use of system resources and improving the timeliness and accuracy of task processing. Therefore, the method based on the embodiment of the present application avoids resource competition and mutual interference between tasks, and solves the problem of decreased service satisfaction rate caused by competition for resources and interference between services in the resource decoupling reasoning process.

[0128] refer to Figure 4, which shows a multi-low-rank adapter distributed reasoning scheduling device 30 based on resource decoupling provided by an embodiment of the present application, applied to a task sending device, including:

[0129] The information acquisition module 301 is used to acquire a request queue including multiple reasoning task requests and request record data including completed historical task requests; the reasoning task request records the output data and execution mode corresponding to the reasoning task request, and multiple timestamps of the reasoning task request in the process of flowing between the task sending device and the task execution device; the request record data records multiple timestamps of the historical task request in the process of flowing between the task sending device and the task execution device;

[0130] The priority calculation module 302 is used to determine the sending priority of the reasoning task request according to the output data and timestamp corresponding to the reasoning task request and the request record data; the sending priority is negatively correlated with the estimated deadline of the corresponding reasoning task request;

[0131] The task selection module 303 is used to select the reasoning task request with the highest sending priority in the request queue as the target reasoning task request;

[0132] The task sending module 304 is used to send the target reasoning task request to the task execution device, so that the task execution device executes the target reasoning task request according to the execution mode corresponding to the target reasoning task request.

[0133] Optionally, the priority calculation module 302 includes:

[0134] The sending estimated deadline submodule is used to determine the sending estimated deadline of the reasoning task request according to the output data and timestamp corresponding to the reasoning task request and the request record data;

[0135] The priority submodule is used to use the value of the estimated deadline for sending as the sending priority of the corresponding inference task request.

[0136] Optionally, the sending estimated deadline submodule includes:

[0137] A first cutoff unit is used to take the sum of the time when the reasoning task request first arrives at the task sending device and the time required for the reasoning task request to generate a first token as the estimated sending deadline when the timestamp indicates that the reasoning task request arrives at the task sending device for the first time;

[0138] A second cutoff unit is used for, when the timestamp characterizes the reasoning task request as outputting the first token, taking the sum of the product of the time required for the reasoning task request to generate one token on average and the value of the total output length of the reasoning task request minus one, and the timestamp of outputting the first token as the estimated deadline for sending;

[0139] The third cutoff unit is used to update the sum of the estimated sending deadline and the average time required for the inference task request to generate one token to a new estimated sending deadline when the length of the output data is greater than the output length threshold estimated for the inference task request.

[0140] Optionally, the multi-low-rank adapter distributed reasoning scheduling device 30 based on resource decoupling further includes:

[0141] The priority adjustment module is used to adjust the sending priority of the reasoning task request according to the output data, execution mode and timestamp corresponding to the reasoning task request.

[0142] Optionally, the priority adjustment module includes:

[0143] A violation parameter submodule, used to calculate the violation parameter of the reasoning task request according to the output data, execution mode and timestamp corresponding to the reasoning task request;

[0144] The priority adjustment submodule is used to use the violation parameter as a delay correction coefficient to correct the estimated deadline of the reasoning task request, and use the corrected estimated deadline as the adjusted sending priority.

[0145] Optionally, the parameter violation submodules include:

[0146] The subqueue unit is used to determine the target subqueue of the request queue according to the execution mode of the inference task request; the execution mode corresponding to the subqueue inference task request in the target subqueue is the same as the execution mode of the inference task request;

[0147] a classification unit, configured to determine a first number of first subqueue reasoning task requests and a second number of second subqueue reasoning task requests in the target subqueue according to timestamps respectively corresponding to the subqueue reasoning task requests in the target subqueue; the first subqueue reasoning task requests are subqueue reasoning task requests that have been selected in the current task round, and the second subqueue reasoning task requests are subqueue reasoning task requests that have not been selected in the current task round;

[0148] An interference parameter unit, used for using the difference between the total number of selected reasoning task requests and the first number, the minimum value of the total number of selected reasoning task requests and the ratio of the total number of selected reasoning task requests to the second number as the interference parameter of the reasoning task request;

[0149] The estimated batch number unit is used to take the ratio of the difference between the last timestamp of the inference task request and the estimated deadline of the inference task request, the maximum value among the zero values, and the single batch running time of the inference task request as the estimated batch number of the inference task request;

[0150] The violation parameter unit is used to use the ratio of the estimated batch number of the reasoning task request and the disturbed parameter as the violation parameter of the reasoning task request.

[0151] Optionally, the priority adjustment submodule includes:

[0152] A correction calculation unit, used to take the product of the difference between the number one and the violation parameter and the difference between the estimated deadline time of the reasoning task request and the time when the reasoning task request first arrives at the task sending device as the arrival time correction of the reasoning task request;

[0153] The time correction unit is used to update the sum of the time when the reasoning task request first arrives at the task sending device and the arrival time correction amount to the corrected estimated deadline time of the reasoning task request.

[0154] In summary, in the embodiment of the present application, the sending priority of each task is determined by analyzing the output data and timestamp of the reasoning task request, so that the sending priority of the task is negatively correlated with the service satisfaction rate of the corresponding reasoning task request. Based on the characteristic that the higher the service satisfaction rate, the faster the corresponding reasoning task request is completed, it is ensured that the model executes the reasoning task as quickly as possible. At the same time, by establishing the sending priority, the resource competition and mutual interference between tasks are effectively avoided, and the overall operation efficiency and stability of the system are improved. Then, according to the sending priority of each reasoning task request, the target reasoning task request is selected in the request queue, and finally it is ensured that the high-priority task can be processed first, thereby maximizing the use of system resources and improving the timeliness and accuracy of task processing. Therefore, the method based on the embodiment of the present application avoids resource competition and mutual interference between tasks, and solves the problem of decreased service satisfaction rate caused by competition for resources and interference between services in the resource decoupling reasoning process.

[0155] Reference Figure 5 , the electronic device 500 may include one or more of the following components: a processing component 502 , a memory 504 , a power component 506 , a multimedia component 508 , an audio component 510 , an input / output (I / O) interface 512 , a sensor component 514 , and a communication component 516 .

[0156] The processing component 502 generally controls the overall operation of the electronic device 500, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 502 may include one or more processors 520 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 502 may include one or more modules to facilitate the interaction between the processing component 502 and other components. For example, the processing component 502 may include a multimedia module to facilitate the interaction between the multimedia component 508 and the processing component 502.

[0157] The memory 504 is used to store various types of data to support the operation of the electronic device 500. Examples of such data include instructions for any application or method operating on the electronic device 500, contact data, phone book data, messages, pictures, multimedia, etc. The memory 504 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0158] The power supply component 506 provides power to the various components of the electronic device 500. The power supply component 506 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 500.

[0159] The multimedia component 508 includes an interface that provides an output interface between the electronic device 500 and the user. In some embodiments, the interface may include a liquid crystal display (LCD) and a touch panel (TP). If the interface includes a touch panel, the interface may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 508 includes a front camera and / or a rear camera. When the electronic device 500 is in an operating mode, such as a shooting mode or a multimedia mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0160] The audio component 510 is used to output and / or input audio signals. For example, the audio component 510 includes a microphone (MIC), and when the electronic device 500 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is used to receive an external audio signal. The received audio signal can be further stored in the memory 504 or sent via the communication component 516. In some embodiments, the audio component 510 also includes a speaker for outputting audio signals.

[0161] The input / output I / O interface 512 provides an interface between the processing component 502 and the peripheral interface module, which may be a keyboard, a click wheel, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0162] The sensor assembly 514 includes one or more sensors for providing various aspects of status assessment for the electronic device 500. For example, the sensor assembly 514 can detect the open / closed state of the electronic device 500, the relative positioning of components, such as the display and keypad of the electronic device 500, and the sensor assembly 514 can also detect the position change of the electronic device 500 or a component of the electronic device 500, the presence or absence of user contact with the electronic device 500, the orientation or acceleration / deceleration of the electronic device 500, and the temperature change of the electronic device 500. The sensor assembly 514 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 514 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 514 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0163] The communication component 516 is used to facilitate wired or wireless communication between the electronic device 500 and other devices. The electronic device 500 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 516 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 516 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0164] In an exemplary embodiment, the electronic device 500 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to implement the methods provided in the embodiments of the present application.

[0165] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 504 including instructions, and the instructions can be executed by a processor 520 of an electronic device 500 to perform the above method. For example, the non-transitory storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0166] Figure 6 600 is a block diagram of an electronic device 600 according to another embodiment of the present invention. For example, the electronic device 600 may be provided as a server. Figure 6 , the electronic device 600 includes a processing component 622, which further includes one or more processors, and a memory resource represented by a memory 632, for storing instructions that can be executed by the processing component 622, such as an application. The application stored in the memory 632 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 622 is configured to execute instructions to perform the method provided in the embodiment of the present application.

[0167] The electronic device 600 may also include a power supply component 626 configured to perform power management of the electronic device 600, a wired or wireless network interface 650 configured to connect the electronic device 600 to a network, and an input / output (I / O) interface 658. The electronic device 600 may operate based on an operating system stored in the memory 632, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.

[0168] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the application disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0169] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A multi-low-rank adapter distributed reasoning scheduling method based on resource decoupling, applied to a task sending device, characterized in that: include: Acquire a request queue including multiple reasoning task requests and request record data including completed historical task requests; the reasoning task request records output data and execution methods corresponding to the reasoning task request, and multiple timestamps of the reasoning task request in the process of flowing between the task sending device and the task execution device; The request record data records multiple timestamps of the historical task request during the process of flowing between the task sending device and the task execution device; Determine the sending priority of the reasoning task request according to the output data and timestamp corresponding to the reasoning task request and the request record data; the sending priority is negatively correlated with the estimated deadline of the corresponding reasoning task request; The reasoning task request with the highest sending priority in the request queue is selected as the target reasoning task request; The target reasoning task request is sent to the task execution device, so that the target reasoning task request is executed on the task execution device in an execution mode corresponding to the target reasoning task request.

2. The method according to claim 1, characterized in that The determining the sending priority of the reasoning task request according to the output data and timestamp corresponding to the reasoning task request and the request record data includes: Determining an estimated deadline for sending the reasoning task request according to the output data and timestamp corresponding to the reasoning task request and the request record data; The value of the estimated sending deadline is used as the sending priority of the corresponding reasoning task request.

3. The method according to claim 2, characterized in that The determining, according to the output data and timestamp corresponding to the reasoning task request and the request record data, an estimated deadline for sending the reasoning task request includes: In the case where the timestamp characterizes the reasoning task request as arriving at the task sending device for the first time, the sum of the time when the reasoning task request first arrives at the task sending device and the time required for the reasoning task request to generate a first token is used as the estimated deadline for sending; In the case where the timestamp characterizes the reasoning task request as outputting the first token, the sum of the product of the time required for the reasoning task request to generate one token on average and the total output length of the reasoning task request minus one and the timestamp of outputting the first token is taken as the estimated deadline for sending; When the length of the output data is greater than the output length threshold estimated for the reasoning task request, the sum of the estimated sending deadline and the average time required for the reasoning task request to generate one token is updated as the new estimated sending deadline.

4. The method according to claim 1, characterized in that The method further comprises: The sending priority of the reasoning task request is adjusted according to the output data, execution mode and timestamp corresponding to the reasoning task request.

5. The method according to claim 4, characterized in that The adjusting the sending priority of the reasoning task request according to the output data, execution mode and timestamp corresponding to the reasoning task request includes: Calculate the violation parameter of the reasoning task request according to the output data, execution mode and timestamp corresponding to the reasoning task request; The violation parameter is used as a delay correction coefficient to correct the estimated deadline of the reasoning task request, and the corrected estimated deadline is used as the adjusted sending priority.

6. The method according to claim 5, characterized in that The calculating the violation parameter of the reasoning task request according to the output data, execution mode and timestamp corresponding to the reasoning task request includes: Determine the target subqueue of the request queue according to the execution mode of the inference task request; the execution mode corresponding to the subqueue inference task request in the target subqueue is the same as the execution mode of the inference task request; Determine, according to the timestamps respectively corresponding to the subqueue reasoning task requests in the target subqueue, a first number of the first subqueue reasoning task requests and a second number of the second subqueue reasoning task requests in the target subqueue; the first subqueue reasoning task requests are subqueue reasoning task requests that have been selected in the current task round, and the second subqueue reasoning task requests are subqueue reasoning task requests that have not been selected in the current task round; taking the difference between the total number of the selected reasoning task requests and the first number, the minimum value of the second number, and the ratio of the second number as the interference parameter of the reasoning task request; The estimated batch number of the reasoning task request is taken as the ratio of the difference between the last timestamp of the reasoning task request and the estimated deadline of the reasoning task request, the maximum value of the zero value, and the single batch running time of the reasoning task request; The ratio of the estimated batch number of the reasoning task request to the disturbed parameter is used as the violation parameter of the reasoning task request.

7. The method according to claim 5, characterized in that The using the violation parameter as a delay correction coefficient to correct the estimated deadline of the reasoning task request includes: The product of the difference between the number 1 and the violation parameter and the difference between the estimated deadline of the reasoning task request and the time when the reasoning task request first arrives at the task sending device is used as the arrival time correction amount of the reasoning task request; The sum of the time when the inference task request first arrives at the task sending device and the arrival time correction amount is updated as the corrected estimated deadline time of the inference task request.

8. A multi-low-rank adapter distributed reasoning scheduling device based on resource decoupling, applied to a task sending device, characterized in that: include: An information acquisition module, used to acquire a request queue containing multiple reasoning task requests and request record data containing completed historical task requests; The inference task request records the output data and execution mode corresponding to the inference task request, as well as multiple timestamps of the inference task request in the process of flowing between the task sending device and the task execution device; The request record data records multiple timestamps of the historical task request during the process of flowing between the task sending device and the task execution device; A priority calculation module, used to determine the sending priority of the reasoning task request according to the output data and timestamp corresponding to the reasoning task request and the request record data; the sending priority is negatively correlated with the estimated deadline of the corresponding reasoning task request; A task selection module, used for selecting the reasoning task request with the highest sending priority in the request queue as the target reasoning task request; The task sending module is used to send the target reasoning task request to the task execution device, so that the target reasoning task request is executed on the task execution device according to the execution mode corresponding to the target reasoning task request.

9. An electronic device, characterized in that: include: A processor, a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method as claimed in any one of claims 1 to 7.

Images