State awareness scheduling method and device for large model reasoning service

Abstract

The invention relates to a state awareness scheduling method and device for large model reasoning service, and the method comprises the steps: responding to the completion of the (i-1) th iteration, updating a request state and a system state, and determining a scheduling result according to the request state, the system state and a cost model, the scheduling result is used for determining the number and the execution sequence of the target requests in the execution queue of the ith iteration. According to the embodiment of the invention, the scheduling strategy can be adjusted in real time on the iteration level according to the monitored state.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a state-aware scheduling method and apparatus for large model reasoning services. Background Technology

[0002] With the continuous evolution of artificial intelligence technology, Large Language Models (LLMs) have demonstrated powerful reasoning capabilities in certain scenarios, and more and more users are starting to use large models for content generation. The typical usage involves users sending requests to large model service providers and then waiting for the results. Continuous batch processing is a fundamental technique for large model inference services. Differences in output length mean that requests within the same batch cannot be completed simultaneously, resulting in completed requests not being immediately returned to the user. New incoming requests also need to wait until the current batch is fully processed before being added. This "request"-level batching not only impairs system utilization but also causes a significant increase in request latency. To address this issue, continuous batch processing technology has been proposed. This technology batches requests at the "iteration" level, where completed requests can be promptly removed in each iteration, while new incoming requests can be added to the current batch.

[0003] However, the work in the related technologies only reveals part of the scheduling design space and does not take into account the changes in latency during the scheduling process. Therefore, it is not optimal in terms of achieving the Service Level Objective (SLO) and needs further improvement. Summary of the Invention

[0004] In view of this, this disclosure proposes a scheduling technology solution.

[0005] According to one aspect of this disclosure, a scheduling method is provided, comprising: the scheduling method including multiple iterations, each iteration being used to select a target request from a waiting queue to an execution queue, the method comprising: in response to completing the (i-1)th iteration, updating a request state and a system state, the request state including at least one first-word delay time, a word-by-word delay time, a request input length, an output length, and a remaining length, the system state including first delay constraint information determined based on the first-word delay time, and second delay constraint information determined based on the word-by-word delay time; determining a scheduling result based on the request state, the system state, and a cost model, the scheduling result being used to determine the number and execution order of target requests in the execution queue in the i-th iteration, where i is an integer greater than or equal to 2.

[0006] In one possible implementation, updating the request state and system state in response to the completion of the (i-1)th iteration includes: obtaining the request state corresponding to the i-th iteration in response to the completion of the (i-1)th iteration; and determining the system state based on the statistical data of the request state.

[0007] In one possible implementation, determining the scheduling result based on the request state, the system state, and the cost model includes: if the second delay constraint information is greater than the first delay constraint information but less than a preset threshold, using a maximum first-word delay time less than or equal to a preset first-word delay threshold as a constraint condition, and determining the scheduling result by solving for the minimum value of the maximum word-by-word delay time; or, if the first delay constraint information is greater than the second delay constraint information but less than a preset threshold, using a maximum word-by-word delay time less than or equal to a preset word-by-word delay threshold as a constraint condition, and determining the scheduling result by solving for the minimum value of the maximum first-word delay time.

[0008] In one possible implementation, determining the scheduling result based on the request state, the system state, and the cost model includes: if a preset threshold is greater than the first delay constraint information and less than the second delay constraint information, sorting the first-letter delay times of target requests in the execution queue from smallest to largest, and determining a first sorting position with the maximum first-letter delay time less than or equal to the preset first-letter delay threshold; using the first-letter delay time of the first sorting position after sorting from smallest to largest as a constraint condition, and determining the scheduling result by solving for the minimum value of the word-by-word delay time of the first sorting position after sorting from smallest to largest; or, if the preset threshold is greater than the second delay constraint information and less than the first delay constraint information, sorting the word-by-word delay times of target requests in the execution queue from smallest to largest, and determining a second sorting position with the maximum word-by-word delay time less than or equal to the preset word-by-word delay threshold; using the word-by-word delay time of the second sorting position after sorting from smallest to largest as a constraint condition, and determining the scheduling result by solving for the minimum value of the first-letter delay time of the second sorting position after sorting from smallest to largest.

[0009] In one possible implementation, the method further includes: if the second delay constraint information is greater than the first delay constraint information, placing requests in the decoding stage at the front of the waiting queue and requests in the pre-filling stage at the back of the waiting queue; or, if the first delay constraint information is greater than the second delay constraint information, placing requests in the pre-filling stage at the front of the waiting queue and requests in the decoding stage at the back of the waiting queue; sorting the requests in the pre-filling stage according to the first-word delay time and the cost model of the pre-filling stage to obtain a first sort; and sorting the requests in the decoding stage according to the word-by-word delay time, output length, remaining length, and the cost model of the decoding stage to obtain a second sort.

[0010] In one possible implementation, the execution process of any request includes a pre-filling stage and a decoding stage. The method further includes: if the second delay constraint information is greater than the first delay constraint information, selecting all requests in the decoding stage in the waiting queue as target requests and adding them to the execution queue; and determining the number of requests in the pre-filling stage selected as target requests in the waiting queue based on the first-word delay time, the cost model of the decoding stage, the cost model of the pre-filling stage, and the first-word delay threshold; or, if the first delay constraint information is greater than the second delay constraint information, determining the number of requests in the pre-filling stage selected as target requests in the waiting queue based on the word-by-word delay time, the output length, the cost model of the pre-filling stage, and the word-by-word delay threshold.

[0011] In one possible implementation, the scheduling method is used to schedule different requests from different users to a neural network model, which is a trained neural network model for at least one task among text response, image classification, object detection, image segmentation, and speech recognition.

[0012] According to one aspect of this disclosure, a scheduling device is provided, comprising: the scheduling device for implementing multiple iterations, each iteration for selecting target requests from a waiting queue to an execution queue, the device comprising: a state detection module for updating a request state and a system state in response to the completion of the (i-1)th iteration, the request state including at least one first-word delay time, a word-by-word delay time, a request input length, an output length, and a remaining length, the system state including first delay constraint information determined based on the first-word delay time, and second delay constraint information determined based on the word-by-word delay time; and a strategy generation module for determining a scheduling result based on the request state, the system state, and a cost model, the scheduling result being used to determine the number and execution order of target requests in the execution queue in the i-th iteration, where i is an integer greater than or equal to 2.

[0013] In one possible implementation, the state detection module is used to: obtain the request state corresponding to the i-th iteration in response to the completion of the (i-1)th iteration; and determine the system state based on the statistical data of the request state.

[0014] In one possible implementation, the strategy generation module is configured to: when the second delay constraint information is greater than the first delay constraint information and less than a preset threshold, use a maximum first-word delay time less than or equal to a preset first-word delay threshold as a constraint condition, and determine the scheduling result by solving for the minimum value of the maximum word-by-word delay time; or, when the first delay constraint information is greater than the second delay constraint information and less than a preset threshold, use a maximum word-by-word delay time less than or equal to a preset word-by-word delay threshold as a constraint condition, and determine the scheduling result by solving for the minimum value of the maximum first-word delay time.

[0015] In one possible implementation, the strategy generation module is configured to: sort the first-word delay times of target requests in the execution queue from smallest to largest when a preset threshold is greater than the first delay constraint information and less than the second delay constraint information; determine a first sorting position with the maximum first-word delay time less than or equal to the preset first-word delay threshold; use the first-word delay time of the first sorting position after sorting from smallest to largest as a constraint condition, and determine the scheduling result by solving for the minimum value of the word-by-word delay time of the first sorting position after sorting from smallest to largest; or, when the preset threshold is greater than the second delay constraint information and less than the first delay constraint information, sort the word-by-word delay times of target requests in the execution queue from smallest to largest, determine a second sorting position with the maximum word-by-word delay time less than or equal to the preset word-by-word delay threshold; use the word-by-word delay time of the second sorting position after sorting from smallest to largest as a constraint condition, and determine the scheduling result by solving for the minimum value of the first-word delay time of the second sorting position after sorting from smallest to largest.

[0016] In one possible implementation, the strategy generation module is further configured to: if the second delay constraint information is greater than the first delay constraint information, prioritize requests in the decoding stage in the waiting queue and prioritize requests in the pre-filling stage in the waiting queue; or, if the first delay constraint information is greater than the second delay constraint information, prioritize requests in the pre-filling stage in the waiting queue and prioritize requests in the decoding stage in the waiting queue; sort the requests in the pre-filling stage according to the first-word delay time and the cost model of the pre-filling stage to obtain a first sort; and sort the requests in the decoding stage according to the word-by-word delay time, output length, remaining length, and the cost model of the decoding stage to obtain a second sort.

[0017] In one possible implementation, the execution process of any request includes a pre-filling stage and a decoding stage. The strategy generation module is further configured to: when the second delay constraint information is greater than the first delay constraint information, select all requests in the decoding stage in the waiting queue as target requests and enter them into the execution queue; and determine the number of requests in the pre-filling stage selected as target requests in the waiting queue based on the first-word delay time, the cost model of the decoding stage, the cost model of the pre-filling stage, and the first-word delay threshold; or, when the first delay constraint information is greater than the second delay constraint information, determine the number of requests in the pre-filling stage selected as target requests in the waiting queue based on the word-by-word delay time, the output length, the cost model of the pre-filling stage, and the word-by-word delay threshold.

[0018] In one possible implementation, the scheduling device is used to schedule different requests from different users to a neural network model, which is a trained neural network model for at least one task among text response, image classification, object detection, image segmentation, and speech recognition.

[0019] According to one aspect of this disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to perform the method described above.

[0020] According to one aspect of this disclosure, a computer-readable storage medium is provided that stores computer program instructions thereon, which, when executed by a processor, implement the above-described method.

[0021] In this embodiment of the disclosure, in response to the completion of the (i-1)th iteration, the request status and system status are updated, and the scheduling result is determined based on the request status, the system status, and the cost model. The scheduling result is used to determine the number and execution order of target requests in the execution queue of the i-th iteration, and the scheduling strategy can be adjusted in real time at the iteration level based on the monitored status.

[0022] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the specification, serve to illustrate the technical solutions of this disclosure.

[0024] Figure 1 A schematic diagram illustrating the application of the large model inference stage according to an embodiment of this disclosure is shown.

[0025] Figure 2 A flowchart illustrating a scheduling method according to an embodiment of the present disclosure is shown.

[0026] Figure 3 A schematic diagram of the framework of a scheduling method according to an embodiment of the present disclosure is shown.

[0027] Figure 4 A schematic diagram illustrating the process of solving a constraint optimization problem according to an embodiment of the present disclosure is shown.

[0028] Figure 5 A block diagram of a scheduling apparatus according to an embodiment of the present disclosure is shown.

[0029] Figure 6 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0030] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0031] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0032] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0033] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0034] With the booming development of applications related to large models (such as neural network models with extremely large parameters), more and more users are starting to use large models for content generation. The typical usage is for users to send a request to a large model service provider and then wait for the result. Figure 1 This diagram illustrates the application of the large model inference stage according to embodiments of the present disclosure, such as... Figure 1 As shown, from the request's perspective, there are often two latency metrics to consider. First is the Time-To-First-Token (TTFT), which represents the time required from the initiation of the request to the return of the first token. TTFT is the sum of the waiting time for scheduling and the execution time of the pre-filling phase. Second is the Time-Per-Output-Token (TPOT), which represents the average latency of each output token from the time the request outputs the first token until the end of the output process.

[0035] Service providers typically describe latency requirements from a request perspective using Service-Level Objectives (SLOs). For large-scale services, the SLO can be a tuple consisting of first-word latency and word-by-word latency. From a user's perspective, lower first-word latency and word-by-word latency are always better. However, for a specific application scenario, it's sufficient if the first-word latency and word-by-word latency meet the SLO requirements. For example, in text responses, the first-word latency is typically 500ms, and the word-by-word latency is typically 150ms. Furthermore, the specific SLO settings can have other variations, which are not limited in the embodiments disclosed herein.

[0036] For large model service providers, the goal is to serve the most user requests with the least amount of computing and storage resources. Therefore, their primary concern is maximizing the system's request throughput (the number of requests processed per second) while meeting quality of service (QoS) targets. Consequently, the trade-off between latency and throughput has a significant impact on large model scheduling. Figure 1 As shown, the execution of the large language model includes two stages: prefill and decoding. Each stage may include one or more iterations. In the prefill stage, the large model processes the initial input text sequence all at once, generating the corresponding hidden states or context representations. In the decoding stage, the model generates the output text sequence token by token based on the context information generated in the prefill stage.

[0037] On the one hand, there is a relatively complex dependency and competition relationship between the two stages. In terms of dependency, only when enough requests complete the pre-filling stage can enough requests enter the decoding stage. In terms of competition, computing and storage resources are limited, and the pre-filling stage and the decoding stage will compete for computing and storage resources, and different requests will also compete for computing resources. On the other hand, compared with requests with a single service quality objective, the service quality objective of a large model service can include two indicators (such as first-word latency and word-by-word latency), which significantly increases the scheduling difficulty.

[0038] Continuous batch processing is a fundamental technique for large-model inference services. Differences in output length prevent requests within the same batch from completing simultaneously, resulting in completed requests not being immediately returned to the user. Newly arriving requests also must wait until the current batch is fully processed before being added. This "request"-level batching impairs system utilization and significantly increases request latency. To address this, continuous batch processing technology has been proposed. This technology processes requests at the "iteration" level, allowing completed requests to exit promptly during each iteration, while new incoming requests are added to the current batch.

[0039] Based on batch processing at the "iterative" level, related technologies employ different scheduling strategies. For example, some technologies prioritize the pre-filling phase of newly arriving requests over the decoding phase of currently running requests. Alternatively, some technologies do not insert the calculation of the pre-filling phase for newly arriving requests while executing the decoding phase of all currently running requests. Intuitively, this single-phase priority strategy always benefits either first-word latency or literal latency at the expense of the other. Even if the scheduling methods sampled by related technologies process requests from different phases in the same batch and balance first-word latency and literal latency by adjusting the mutual blocking between the two phases, they cannot guarantee the correction of the distribution deviation between first-word latency and literal latency. Furthermore, even if related technologies involve prioritizing requests, they do not address the latency constraints in the quality of service (QoS) objectives. Moreover, related technologies only reveal a portion of the scheduling design space and do not recognize the latency variations during the scheduling process, thus they are not optimal in terms of achieving QoS objectives.

[0040] It is evident that the scheduling methods of related technologies lead to two imbalances, resulting in a lower service quality target achievement rate during inference service, namely:

[0041] 1) Distribution deviation of request latency between first-word latency and word-by-word latency: The success rate of a request for one latency constraint, either first-word latency or word-by-word latency, is much lower than the other;

[0042] 2) Variance of request latency distribution among different requests: The latency distribution of different requests is relatively dispersed. Some of these requests have far exceeded the service quality target limit, while others are still far from the service quality target limit.

[0043] The relevant technologies lack the ability to perceive and control these two imbalances, thus making it impossible to correct them through adjustments to scheduling strategies. In other words, in large-model inference services, computation involves two heterogeneous phases, each with its own service-level objectives. Related scheduling technologies struggle to balance different requests and service-level objectives, resulting in a low overall service achievement rate.

[0044] In view of this, in order to achieve real-time adjustment of the scheduling strategy at the iterative level, the scheduling method proposed in this disclosure can effectively achieve the above balance, obtain a higher overall service achievement rate, and thus improve the request rate that the service system can support. Figure 2 A flowchart illustrating a scheduling method according to an embodiment of this disclosure is shown, such as... Figure 2 As shown, the scheduling method includes multiple iterations, each iteration selecting a target request from the waiting queue to the execution queue. The scheduling method includes:

[0045] In step S11, in response to the completion of the (i-1)th iteration, the request state and the system state are updated. The request state includes at least one first-word delay time, word-by-word delay time, request input length, output length, and remaining length. The system state includes first delay constraint information determined based on the first-word delay time and second delay constraint information determined based on the word-by-word delay time.

[0046] In step S12, a scheduling result is determined based on the request status, the system status, and the cost model. The scheduling result is used to determine the number and execution order of target requests in the execution queue for the i-th iteration, where i is an integer greater than or equal to 2.

[0047] In one possible implementation, the scheduling method can be executed by an electronic device such as a terminal device or a server. The terminal device can be a user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. The method can be implemented by a processor calling computer-readable instructions stored in memory. Alternatively, the method can be executed by a server.

[0048] In one possible implementation, the scheduling method is used to schedule different requests from different users to a neural network model, which is a trained neural network model for at least one task among text response, image classification, object detection, image segmentation, and speech recognition.

[0049] In one possible implementation, the neural network model may include multiple convolutional layers, pooling layers, fully connected layers, etc., and may be at least one of Convolutional Neural Networks (CNN), Deep Learning Neural Networks (DNN), Recurrent Neural Networks (RNN), Residual Networks (ResNets), Back Propagation Neural Networks (BP), Backbone Neural Networks, Large Language Models (LLM), etc. This disclosure does not impose specific limitations on the network structure of the neural network model.

[0050] Optionally, the trained neural network model can be a neural network model used for any task in image classification, object detection, image segmentation, speech recognition, natural language processing, machine translation, or question answering systems, and this disclosure does not limit it.

[0051] Figure 3 A schematic diagram of the framework of the scheduling method according to an embodiment of the present disclosure is shown, such as... Figure 3 As shown, the scheduling method of this disclosure can record a state monitor that records real-time states (e.g., request states and system states) and a strategy generator that optimizes the scheduling strategy in each iteration based on the real-time states (e.g., request states and system states).

[0052] In one possible implementation, step S11 may include: in response to the completion of the (i-1)th iteration, obtaining the request status corresponding to the i-th iteration; and determining the system status based on the statistical data of the request status.

[0053] For example, in response to the completion of the (i-1)th iteration, that is, after the execution process of the (i-1)th iteration ends, the request status can be updated through the status monitor to obtain the request status corresponding to the i-th iteration. Here, i is an integer greater than or equal to 2, and the request status may include the real-time first-word delay time of request r. Word-by-word delay time Request input length Existing output length Predict the remaining length to be output

[0054] After obtaining the request status corresponding to the i-th iteration, the system status can be determined based on the statistical data of the request status. The system status may include the overall system latency constraint achievement level, such as based on the first character delay time. Determined first delay constraint information and based on word-by-word delay time Determined second delay constraint information Right now:

[0055]

[0056]

[0057] in, This represents the first delay constraint information for the i-th iteration. This represents the second delay constraint information for the i-th iteration. Let represent the delay time of the first word of request r among multiple requests in the waiting queue during the i-th iteration. This represents the word-by-word delay time of request r among multiple requests in the waiting queue during the i-th iteration. `max` is a maximum value function used to compare the maximum value among multiple values. TTFT and T TPOT The specific value of the latency constraint for the Service Level Objective (SLO) set according to the deployment scenario can be set according to the actual scenario, and the embodiments disclosed herein do not impose any restrictions on this.

[0058] In this way, the request status and system status of each iteration can be updated in real time by the status monitor, which helps to more accurately determine the number of target requests and the execution order in the execution queue of each iteration.

[0059] In step S11, the request status and system status of the i-th iteration are updated. In step S12, the scheduling result is determined based on the request status, the system status, and the cost model. This scheduling result is used to determine the number of target requests and their execution order in the execution queue of the i-th iteration.

[0060] Optionally, based on the updated request state and system state in the state monitor, the policy generator will construct a constrained optimization problem to generate an optimized scheduling policy to achieve balance management in the i-th iteration.

[0061] For example, the policy generator may include a cost model that predicts the execution delay of the prefill phase. And the cost model for the execution latency of the decoding phase. Cost model of pre-filling stage Cost model for the decoding stage The algorithm can be optimized based on the actual execution time of the (i-1)th iteration to achieve higher prediction accuracy.

[0062] Among them, the cost model of the pre-filling stage Cost model for the decoding stage The initialization process is performed offline. Given the model and parallel strategy, the cost model can be initialized through offline inference.

[0063] For example, given certain execution requests, a cost model can be introduced to predict processing latency. Due to the differences in data streams, a cost model for the pre-filling stage can be constructed for requests r in the pre-filling stage and the decoding stage, respectively. Cost model for the decoding stage like Figure 3As shown, the cost model can be initialized using analytical data from offline inference. For example, a linear regression method can be used to initialize the cost model, and prediction accuracy can be improved through dynamic adjustments. The continuity of the service process can be leveraged (e.g., the trend of KV cache length changes, where the KV cache is used to avoid recalculating the key-value vector each time a token is sampled), and for the i-th iteration, a scaling ratio γ can be used. i To adjust the cost model linearly.

[0064] Scaling ratio γ i The delay can be adjusted in real time by the status monitor, i.e.:

[0065]

[0066] in, Cost model representing the initial pre-filling stage and the cost model for the initial decoding phase The cost model for the i-th iteration pre-filling stage. and the cost model for the i-th iteration decoding stage t(r) is the record delay for requesting r, α represents the confidence level of the tuning result, and γ i γ represents the scaling ratio corresponding to the i-th iteration. i-1 This represents the scaling ratio corresponding to the (i-1)th iteration.

[0067] The generated scheduling policy has a design space, which mainly includes two dimensions: request execution order control and iterative load control. These two dimensions are orthogonal to each other, and the output of the constrained optimization problem is the scheduling result, which is the policy based on these two dimensions.

[0068] To optimize the achievement rate of overall service quality goals, optimization can be performed separately on the achievement rate of first-character delay time or character-by-character delay time, while also considering the other delay constraint to achieve a balance between the two. For example, optimization can be performed on the achievement rate of first-character delay time while also considering the constraint based on character-by-character delay time. Alternatively, optimization can be performed on the achievement rate of character-by-character delay time while also considering the constraint based on first-character delay time.

[0069] Therefore, in step S12 of the embodiments of this disclosure, a constraint optimization-based balanced scheduling strategy is proposed based on the request status, the system status, and the cost model, so as to determine the scheduling result through the balanced scheduling strategy, which is beneficial to simultaneously achieve a balance between the first-word delay time and the word-by-word delay time, as well as a balance between different requests.

[0070] In one possible implementation, step S12 may include: if the second delay constraint information is greater than the first delay constraint information and less than a preset threshold (e.g., 1), taking the maximum first-word delay time as less than or equal to a preset first-word delay threshold as a constraint condition, and determining the scheduling result by solving for the minimum value of the maximum word-by-word delay time, i.e.:

[0071]

[0072] Wherein, the value 1 represents the preset threshold. This represents the second delay constraint information for the i-th iteration. This represents the first delay constraint information for the i-th iteration. Let represent the delay time of the first word of request r among multiple requests in the waiting queue during the i-th iteration. Max represents the word-by-word delay time of request r among multiple requests in the waiting queue during the i-th iteration. r (·) is the maximum value function, used to compare the maximum value among multiple values; min is the minimum value function, used to compare the minimum value among multiple values; T TTFT The specific value of the latency constraint for the Service Level Objective (SLO) set according to the deployment scenario can be set according to the actual scenario, and the embodiments disclosed herein do not impose any restrictions on this.

[0073] In one possible implementation, if a preset threshold (e.g., 1) is greater than the first delay constraint information and less than the second delay constraint information, the first-word delay times of the target requests in the execution queue are sorted from smallest to largest to determine the first sorting position of the maximum first-word delay time that is less than or equal to the preset first-word delay threshold. The first-word delay time of the first sorting position after sorting from smallest to largest is less than or equal to the preset first-word delay threshold, which is used as a constraint condition. The scheduling result is determined by solving for the minimum value of the word-by-word delay time of the first sorting position after sorting from smallest to largest.

[0074]

[0075]

[0076] Wherein, the value 1 represents the preset threshold. This represents the second delay constraint information for the i-th iteration. This represents the first delay constraint information for the i-th iteration. Let represent the delay time of the first word of request r among multiple requests in the waiting queue during the i-th iteration. a%-max represents the word-by-word delay time of request r among multiple requests in the waiting queue during the i-th iteration. r() means taking the value at the a% position after sorting from smallest to largest, min is the minimum value function used to compare the minimum value among multiple values, T TTFT The specific value of the latency constraint for the Service Level Objective (SLO) set according to the deployment scenario can be set according to the actual scenario, and the embodiments disclosed herein do not impose any restrictions on this.

[0077] In this way, we can take the value at position a% after sorting from smallest to largest, and continuously decrease a (a<100) until the condition is met. Less than or equal to T TTFT To obtain the first sorting position a%, then based on this first sorting position a%, ... Optimize the results to satisfy the relaxed constraints. We impose constraints to create a new constrained optimization problem.

[0078] In one possible implementation, if the first delay constraint information is greater than the second delay constraint information but less than a preset threshold, the maximum word-by-word delay time being less than or equal to the preset word-by-word delay threshold is used as a constraint condition. The scheduling result is determined by solving for the minimum value of the maximum first-word delay time.

[0079]

[0080] Wherein, the value 1 represents the preset threshold. This represents the second delay constraint information for the i-th iteration. This represents the first delay constraint information for the i-th iteration. Let represent the delay time of the first word of request r among multiple requests in the waiting queue during the i-th iteration. Max represents the word-by-word delay time of request r among multiple requests in the waiting queue during the i-th iteration. r (·) is the maximum value function, used to compare the maximum value among multiple values; min is the minimum value function, used to compare the minimum value among multiple values; T TPOT The specific value of the latency constraint for the Service Level Objective (SLO) set according to the deployment scenario can be set according to the actual scenario, and the embodiments disclosed herein do not impose any restrictions on this.

[0081] In one possible implementation, when a preset threshold is greater than the second delay constraint information and less than the first delay constraint information, the word-by-word delay times of the target requests in the execution queue are sorted from smallest to largest to determine a second sorting position for the maximum word-by-word delay time that is less than or equal to the preset word-by-word delay threshold. The word-by-word delay time of the second sorting position after sorting from smallest to largest is taken as a constraint condition. The scheduling result is determined by solving for the minimum value of the first word delay time of the second sorting position after sorting from smallest to largest.

[0082]

[0083]

[0084] Wherein, the value 1 represents the preset threshold. This represents the second delay constraint information for the i-th iteration. This represents the first delay constraint information for the i-th iteration. Let represent the delay time of the first word of request r among multiple requests in the waiting queue during the i-th iteration. a%-max represents the word-by-word delay time of request r among multiple requests in the waiting queue during the i-th iteration. r () means taking the value at the a% position after sorting from smallest to largest, min is the minimum value function used to compare the minimum value among multiple values, T TPOT The specific value of the latency constraint for the Service Level Objective (SLO) set according to the deployment scenario can be set according to the actual scenario, and the embodiments disclosed herein do not impose any restrictions on this.

[0085] In this way, we can take the value at position a% after sorting from smallest to largest, and continuously decrease a (a<100) until the condition is met. Less than or equal to T TPOT To obtain the second sorting position a%, and then based on this second sorting position a%, ... Optimize the results to satisfy the relaxed constraints. We impose constraints to create a new constrained optimization problem.

[0086] It can be seen that when the requests in the waiting queue can satisfy the Service Level Objective (SLO) for either the first-word delay time or the literal delay time, the policy generator will choose the one with the larger ratio of the corresponding maximum value to the SLO for optimization, while using the other as a constraint. However, when neither the first-word delay time nor the literal delay time can satisfy the corresponding SLO, the constraint can be relaxed. Specifically, this involves adjusting the "max" value... r The "(·)" operator should be changed to "a%-max". r The "(·)" operator takes the value at position a% after sorting from smallest to largest, and continuously decreases a (a<100) until either the first-character delay time or the character-by-character delay time satisfies "a%-max". r The constraints (·) are less than or equal to the Service Level Objective (SLO). Then, the unmet delays are optimized, and the satisfied delays are constrained, thus forming a new constraint optimization problem.

[0087] Figure 4 A schematic diagram illustrating the process of solving a constrained optimization problem according to an embodiment of the present disclosure is shown, such as... Figure 4 As shown, this paper introduces the control of request execution order and load size in the process of solving constrained optimization problems.

[0088] In one possible implementation, the method further includes: if the second delay constraint information is greater than the first delay constraint information, placing requests in the decoding stage at the front of the waiting queue and requests in the pre-filling stage at the back of the waiting queue; or, if the first delay constraint information is greater than the second delay constraint information, placing requests in the pre-filling stage at the front of the waiting queue and requests in the decoding stage at the back of the waiting queue; sorting the requests in the pre-filling stage according to the first-word delay time and the cost model of the pre-filling stage to obtain a first sort; and sorting the requests in the decoding stage according to the word-by-word delay time, output length, remaining length, and the cost model of the decoding stage to obtain a second sort.

[0089] For example, such as Figure 4 As shown, request sorting can be divided into a "stage" level and a "request" level. At the stage level, when the strategy generator bases its decisions on the overall delay state of the i-th iteration... After selecting the corresponding constraint optimization problem, when optimizing the first-word delay time and constraining the word-by-word delay time, requests in the pre-filling stage (e.g.) will be optimized. Figure 4 Request A on the left p Request B p Request C p ) are placed at the front of the waiting queue, and requests during the decoding phase (e.g.) Figure 4 Request D on the left d Request E d Request F d Request G d They are placed at the back of the waiting queue.

[0090] Alternatively, when constraining the first-word delay time and optimizing the word-by-word delay time, requests in the decoding stage (e.g.) Figure 4 Request D on the right d Request E d Request F d Request G d ) are placed at the front of the waiting queue, and requests in the pre-filling phase (e.g.) Figure 4 Request A on the right p Request B p Request C p (It is placed at the back.)

[0091] Then, at the request level, requests in the same stage are internally sorted. For requests in the pre-population stage, use... Sort the requests in descending order to obtain the first sort. For example... Figure 4 The first order shown in the pre-population phase at the request level is: Request B p Request A p Request C p .in, Let represent the delay time of the first word of request r among multiple requests in the waiting queue during the i-th iteration. Let represent the cost model of request r among multiple requests in the pre-filled waiting queue corresponding to the i-th iteration.

[0092] For requests in the decoding phase, the cost model can be based on the word-by-word delay time, output length, remaining length, and decoding phase cost for each request, for example... The requests are sorted to obtain a second sort. For example... Figure 4 The second ordering shown in the decoding phase at the request level is: Request E d Request G d Request F d Request D d .in, Let represent the word-by-word delay time of request r among multiple requests in the waiting queue during the i-th iteration. This represents the output length of request r among the multiple requests in the waiting queue during the i-th iteration. This represents the execution queue for the (i-1)th iteration. This represents the cost model for the decoding stage corresponding to the i-th iteration. This indicates that among the multiple requests in the waiting queue for the i-th iteration, request r predicts the remaining length to be output.

[0093] In one possible implementation, the method further includes: when the second delay constraint information is greater than the first delay constraint information, selecting all requests in the decoding stage in the waiting queue as target requests and entering them into the execution queue; and determining the number of requests in the pre-filling stage in the waiting queue as target requests to be entered into the execution queue based on the first word delay time, the cost model of the decoding stage, the cost model of the pre-filling stage, and the first word delay threshold.

[0094] For example, in the second delay constraint information Greater than the first delay constraint information In the case of optimizing the character-by-character delay time while constraining the first-character delay time, the insertion of pre-filling stage requests should be minimized to avoid impacting the execution efficiency of decoding stage requests while satisfying the first-character delay time constraint. Specifically, all decoding stage requests are placed in the execution queue. On this basis, a suitable number of pre-filling stage requests are added. It is necessary to control the number of pre-filling stage requests still waiting to prevent excessive accumulation of waiting time that could break the first-character delay time constraint. The constraint on the number of pre-filling stage requests in the waiting queue is as follows:

[0095]

[0096] in, Represents the waiting queue for the i-th iteration. The delay time of the first word of request 'r' in multiple requests. This represents the execution queue for the i-th iteration. This represents the cost model for the decoding stage corresponding to the i-th iteration. This represents the cost model for the pre-filling stage corresponding to the i-th iteration. This is a maximum value function used to compare waiting queues. The maximum value among multiple requests, where r is the waiting queue. For any request in T TTFT The specific value of the latency constraint for the Service Level Objective (SLO) set according to the deployment scenario can be set according to the actual scenario, and the embodiments disclosed herein do not impose any restrictions on this.

[0097] Since the full word-by-word delay only occurs after the pre-filling phase of a request is completed, the load control here is based on the number of requests rather than the number of tokens. That is, the number of requests executed in the pre-filling phase is the minimum number of requests that satisfy the above constraints.

[0098] In one possible implementation, when the first delay constraint information is greater than the second delay constraint information, the number of pre-filling stage requests selected as target requests to be included in the execution queue in the waiting queue is determined based on the word-by-word delay time, output length, cost model of the pre-filling stage, and word-by-word delay threshold.

[0099] For example, in the first delay constraint information Greater than the second delay constraint information In the case of optimizing the first-word delay time and constraining the word-by-word delay time, as many pre-filling phase requests as possible should be executed while satisfying the word-by-word delay time constraint. Since the insertions performed in the pre-filling phase will increase the word-by-word delay time, the word-by-word delay time constraint can be achieved by constraining the load size of the pre-filling phase requests, as follows:

[0100]

[0101] in, Represents the waiting queue for the i-th iteration. The word-for-word delay time of request r in multiple requests. This represents the execution queue for the i-th iteration. This represents the cost model for the pre-filling stage corresponding to the i-th iteration. Represents the waiting queue for the i-th iteration. The length of the output of r is requested in multiple requests. This is a maximum value function used to satisfy the word-by-word delay time. If the condition is greater than 0, compare the waiting queue. In multiple requests The maximum value, r is the waiting queue. For any request in T TPOT The specific value of the latency constraint for the Service Level Objective (SLO) set according to the deployment scenario can be set according to the actual scenario, and the embodiments disclosed herein do not impose any restrictions on this.

[0102] Considering that requests in the pre-filling phase can be chunked, allowing a single request to be divided into many sub-chunks for execution along a sequence dimension, the load on pre-filling phase requests can be controlled at the token level. Furthermore, considering the slicing quantization effect of token control, the largest integer multiple of the slice is chosen as the load limit for the number of tokens, thereby controlling the number of pre-filling requests added and the total number of tokens.

[0103] In summary, the scheduling method of this disclosure implements fine-grained scheduling with state awareness, comprising a fine-grained scheduling framework with state awareness and a constraint optimization-based balanced scheduling strategy. The fine-grained scheduling framework with state awareness enables the scheduling strategy to change along the iteration dimension, while the constraint optimization-based balanced scheduling strategy achieves resource balance among different requests and service-level objectives by constructing a constraint optimization problem. In this way, the scheduling method of this disclosure enables fine-grained scheduling optimization of a large model at the iteration level. The optimization objective is to improve the overall service achievement rate by balancing resources among different requests and service-level objectives, thereby serving more requests within the same time frame.

[0104] The scheduling method of this disclosure not only adjusts the scheduling strategy in real time at the iterative level based on the monitored status, but also provides a constraint-optimized balanced scheduling strategy. This strategy abstracts the achievement rate optimization of first-word delay time and word-by-word delay time into two constraint optimization problems, with each iteration selecting one to solve to obtain the optimized scheduling strategy. This significantly improves the overall service achievement rate and service request rate; for example, it can increase the overall service achievement rate by up to 50% and achieve an approximately 25% improvement in the service request rate.

[0105] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0106] In addition, this disclosure also provides scheduling devices, electronic devices, computer-readable storage media, and programs, all of which can be used to implement any of the scheduling methods provided in this disclosure. The corresponding technical solutions and descriptions are described in the corresponding records in the method section and will not be repeated here.

[0107] Figure 5 A block diagram of a scheduling apparatus according to an embodiment of the present disclosure is shown, such as Figure 5As shown, the scheduling device is used to implement multiple iterations, each iteration selecting a target request from the waiting queue to the execution queue. The device includes:

[0108] The state detection module 51 is used to update the request state and the system state in response to the completion of the (i-1)th iteration. The request state includes at least one first-word delay time, word-by-word delay time, request input length, output length, and remaining length. The system state includes first delay constraint information determined based on the first-word delay time and second delay constraint information determined based on the word-by-word delay time.

[0109] The strategy generation module 52 is used to determine the scheduling result based on the request status, the system status, and the cost model. The scheduling result is used to determine the number and execution order of target requests in the execution queue in the i-th iteration, where i is an integer greater than or equal to 2.

[0110] This method is specifically technically related to the internal structure of computer systems and can solve technical problems of how to improve hardware computing efficiency or execution performance (including reducing data storage, reducing data transmission, and increasing hardware processing speed), thereby achieving technical effects that improve the internal performance of computer systems in accordance with natural laws.

[0111] In one possible implementation, the state detection module 51 is used to: in response to the completion of the (i-1)th iteration, obtain the request state corresponding to the i-th iteration; and determine the system state based on the statistical data of the request state.

[0112] In one possible implementation, the strategy generation module 52 is configured to: when the second delay constraint information is greater than the first delay constraint information and less than a preset threshold, use the maximum first-word delay time being less than or equal to a preset first-word delay threshold as a constraint condition, and determine the scheduling result by solving for the minimum value of the maximum word-by-word delay time; or, when the first delay constraint information is greater than the second delay constraint information and less than a preset threshold, use the maximum word-by-word delay time being less than or equal to a preset word-by-word delay threshold as a constraint condition, and determine the scheduling result by solving for the minimum value of the maximum first-word delay time.

[0113] In one possible implementation, the strategy generation module 52 is configured to: sort the first-word delay times of target requests in the execution queue from smallest to largest when a preset threshold is greater than the first delay constraint information and less than the second delay constraint information; determine a first sorting position with the maximum first-word delay time less than or equal to the preset first-word delay threshold; use the first-word delay time of the first sorting position after sorting from smallest to largest as a constraint condition, and determine the scheduling result by solving for the minimum value of the word-by-word delay time of the first sorting position after sorting from smallest to largest; or, when a preset threshold is greater than the second delay constraint information and less than the first delay constraint information, sort the word-by-word delay times of target requests in the execution queue from smallest to largest, determine a second sorting position with the maximum word-by-word delay time less than or equal to the preset word-by-word delay threshold; use the word-by-word delay time of the second sorting position after sorting from smallest to largest as a constraint condition, and determine the scheduling result by solving for the minimum value of the first-word delay time of the second sorting position after sorting from smallest to largest.

[0114] In one possible implementation, the strategy generation module 52 is further configured to: when the second delay constraint information is greater than the first delay constraint information, prioritize requests in the decoding stage in the waiting queue and prioritize requests in the pre-filling stage in the waiting queue; or, when the first delay constraint information is greater than the second delay constraint information, prioritize requests in the pre-filling stage in the waiting queue and prioritize requests in the decoding stage in the waiting queue; sort the requests in the pre-filling stage according to the first-word delay time and the cost model of the pre-filling stage to obtain a first sort; and sort the requests in the decoding stage according to the word-by-word delay time, output length, remaining length, and the cost model of the decoding stage to obtain a second sort.

[0115] In one possible implementation, the execution process of any request includes a pre-filling stage and a decoding stage. The strategy generation module 52 is further configured to: select all requests in the decoding stage in the waiting queue as target requests and enter them into the execution queue when the second delay constraint information is greater than the first delay constraint information; and determine the number of requests in the pre-filling stage selected as target requests in the execution queue based on the first-word delay time, the cost model of the decoding stage, the cost model of the pre-filling stage, and the first-word delay threshold; or, when the first delay constraint information is greater than the second delay constraint information, determine the number of requests in the pre-filling stage selected as target requests in the execution queue based on the word-by-word delay time, the output length, the cost model of the pre-filling stage, and the word-by-word delay threshold.

[0116] In one possible implementation, the scheduling device is used to schedule different requests from different users to a neural network model, which is a trained neural network model for at least one task among text response, image classification, object detection, image segmentation, and speech recognition.

[0117] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0118] This disclosure also proposes a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the above-described method. The computer-readable storage medium can be volatile or non-volatile.

[0119] This disclosure also proposes an electronic device, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to execute the above-described method.

[0120] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described method.

[0121] Electronic devices can be provided as terminals, servers, or other forms of devices.

[0122] Figure 6 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. For example, the electronic device 1900 may be provided as a server or a terminal device. (Refer to...) Figure 6 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0123] Electronic device 1900 may also include a power supply component 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input / output interface 1958. Electronic device 1900 can operate on an operating system stored in memory 1932, such as Microsoft Server operating system (Windows Server). TM Apple's graphical user interface-based operating system (Mac OS X) TM ), a multi-user, multi-process computer operating system (Unix) TM Linux is a free and open-source Unix-like operating system. TM ), the open-source Unix-like operating system (FreeBSD) TM (or similar.)

[0124] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of an electronic device 1900 to perform the above-described method.

[0125] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0126] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, (but not limited to) electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical decoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0127] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0128] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0129] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0130] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0131] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0132] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0133] The computer program product can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0134] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0135] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0136] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0137] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A scheduling method, characterized in that, The scheduling method includes multiple iterations, each iteration selecting a target request from the waiting queue to the execution queue. The scheduling method includes: In response to the completion of the (i-1)th iteration, the request state and system state are updated. The request state includes at least one first-word delay time, word-by-word delay time, request input length, output length, and remaining length. The system state includes first delay constraint information determined based on the first-word delay time and second delay constraint information determined based on the word-by-word delay time. Based on the request status, the system status, and the cost model, a scheduling result is determined. The scheduling result is used to determine the number and execution order of target requests in the execution queue for the i-th iteration, where i is an integer greater than or equal to 2.

2. The method according to claim 1, characterized in that, In response to the completion of the (i-1)th iteration, update the request state and system state, including: In response to the completion of the (i-1)th iteration, obtain the request status corresponding to the i-th iteration; The system status is determined based on the statistical data of the requested status.

3. The method according to claim 1, characterized in that, Based on the request status, the system status, and the cost model, the scheduling result is determined, including: If the second delay constraint information is greater than the first delay constraint information but less than a preset threshold, the maximum first-word delay time is less than or equal to the preset first-word delay threshold as a constraint condition. The scheduling result is determined by solving for the minimum value of the maximum word-by-word delay time. or, If the first delay constraint information is greater than the second delay constraint information but less than a preset threshold, the maximum word-by-word delay time is less than or equal to the preset word-by-word delay threshold as a constraint condition. The scheduling result is determined by solving for the minimum value of the maximum first-word delay time.

4. The method according to claim 1, characterized in that, Based on the request status, the system status, and the cost model, the scheduling result is determined, including: If the preset threshold is greater than the first delay constraint information and less than the second delay constraint information, the first word delay time of the target request in the execution queue is sorted from smallest to largest, and the first sorting position of the maximum first word delay time that is less than or equal to the preset first word delay threshold is determined. The scheduling result is determined by finding the minimum value of the delay time of the first character in the first position after sorting from smallest to largest, which is less than or equal to a preset delay threshold for the first character. or, If the preset threshold is greater than the second delay constraint information and less than the first delay constraint information, the word-by-word delay time of the target request in the execution queue is sorted from smallest to largest, and the second sorting position of the maximum word-by-word delay time that is less than or equal to the preset word-by-word delay threshold is determined. The scheduling result is determined by taking the word-by-word delay time of the second sorted position after sorting from smallest to largest as a constraint condition that is less than or equal to a preset word-by-word delay threshold, and by solving for the minimum value of the first word delay time of the second sorted position after sorting from smallest to largest.

5. The method according to claim 3 or 4, characterized in that, The method further includes: If the second delay constraint information is greater than the first delay constraint information, requests in the decoding stage will be placed at the front of the waiting queue and requests in the pre-filling stage will be placed at the back of the waiting queue; or, if the first delay constraint information is greater than the second delay constraint information, requests in the pre-filling stage will be placed at the front of the waiting queue and requests in the decoding stage will be placed at the back of the waiting queue. Based on the first word delay time of each request in the pre-filling phase and the cost model of the pre-filling phase, the requests in the pre-filling phase are sorted to obtain the first sort; Based on the word-by-word delay time, output length, remaining length, and cost model of each request in the decoding stage, the requests in the decoding stage are sorted to obtain the second sort.

6. The method according to any one of claims 1-4, characterized in that, The execution process of any request includes a pre-filling phase and a decoding phase, and the method further includes: If the second delay constraint information is greater than the first delay constraint information, all requests in the decoding stage in the waiting queue are selected as target requests and entered into the execution queue. The number of requests in the pre-filling stage selected as target requests and entered into the execution queue in the waiting queue is determined based on the first word delay time, the cost model of the decoding stage, the cost model of the pre-filling stage, and the first word delay threshold. Alternatively, if the first delay constraint information is greater than the second delay constraint information, the number of requests in the pre-filling stage selected as target requests to be included in the execution queue in the waiting queue is determined based on the word-by-word delay time, output length, cost model of the pre-filling stage, and word-by-word delay threshold.

7. The method according to any one of claims 1-4, characterized in that, The scheduling method is used to schedule different requests from different users to a neural network model, wherein the neural network model is a trained neural network model for at least one task among text response, image classification, object detection, image segmentation, and speech recognition.

8. A scheduling device, characterized in that, The scheduling device is used to implement multiple iterations, each iteration selecting a target request from the waiting queue to the execution queue. The scheduling device includes: The state detection module is used to update the request state and system state in response to the completion of the (i-1)th iteration. The request state includes at least one first-word delay time, word-by-word delay time, request input length, output length, and remaining length. The system state includes first delay constraint information determined based on the first-word delay time and second delay constraint information determined based on the word-by-word delay time. The strategy generation module is used to determine the scheduling result based on the request status, the system status, and the cost model. The scheduling result is used to determine the number and execution order of target requests in the execution queue in the i-th iteration, where i is an integer greater than or equal to 2.

9. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 7.

Images