Model input request scheduling method and device, storage medium and program product

Abstract

Embodiments of the invention provide a model input request scheduling method and device, a storage medium and a program product. The method comprises the steps of receiving an input request of a preset language model; determining prediction time consumption of the input request in a pre-filling stage in a preset language model processing process; determining the latest execution time of the pre-filling stage of the input request according to the receiving time of the input request, the predicted consumed time of the pre-filling stage and the target first lexical element delay; and according to the latest execution time of the pre-filling stage, sending the input request to a service node of a preset language model for processing. According to the embodiment of the invention, the latest execution time of the pre-filling stage is determined based on the predicted time consumption of the pre-filling stage of the input request to schedule the input request, and the system throughput and the calculation load balance are optimized while the service level target is met, so that the resource utilization efficiency and the service quality guarantee capability of the service node of the preset language model can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a model input request scheduling method, device, storage medium, and program product. Background Technology

[0002] In model systems such as Large Language Model (LLM), the AI ​​gateway, as a core component for request scheduling, directly impacts the overall system performance through its strategy selection.

[0003] Existing model systems typically use static scheduling strategies for request scheduling in their AI gateways, which limits the resource utilization efficiency and service quality assurance capabilities of these systems. Summary of the Invention

[0004] This disclosure provides a model input request scheduling method, device, storage medium, and program product to improve the resource utilization efficiency and service quality assurance capability of service nodes of a preset model.

[0005] In a first aspect, embodiments of this disclosure provide a model input request scheduling method, including:

[0006] Receive input requests from a preset model;

[0007] Determine the predicted time consumption of the input request during the pre-filling stage of the preset model processing;

[0008] Based on the receiving time of the input request, the predicted time of the pre-filling stage, and the delay of the target first word, the latest execution time of the pre-filling stage of the input request is determined.

[0009] Based on the latest execution time of the pre-filling stage, the input request is sent to the service node of the preset model for processing.

[0010] Secondly, embodiments of this disclosure provide a model input request scheduling device, comprising:

[0011] The receiving unit is used to receive input requests from a preset model;

[0012] A prediction unit is used to determine the prediction time of the input request in the pre-filling stage of the preset model processing process;

[0013] An execution time determination unit is used to determine the latest execution time of the pre-filling stage of the input request based on the receiving time of the input request, the predicted time of the pre-filling stage, and the delay of the target first word.

[0014] The scheduling unit is used to send the input request to the service node of the preset model for processing according to the latest execution time of the pre-filling stage.

[0015] Thirdly, embodiments of this disclosure provide an electronic device, including: a processor and a memory;

[0016] The memory stores computer-executed instructions;

[0017] The processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the model input request scheduling method as described in the first aspect and various possible designs of the first aspect.

[0018] Fourthly, embodiments of this disclosure provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the model input request scheduling method described in the first aspect and various possible designs of the first aspect.

[0019] Fifthly, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the model input request scheduling method as described in the first aspect and various possible designs of the first aspect.

[0020] The model input request scheduling method, device, storage medium, and program product provided in this embodiment receive input requests from a preset model; determine the predicted time consumption of the pre-filling phase in the processing of the input request in the preset model; determine the latest execution time of the pre-filling phase of the input request based on the reception time of the input request, the predicted time consumption of the pre-filling phase, and the target first word delay; and send the input request to the service node of the preset model for processing based on the latest execution time of the pre-filling phase. This embodiment determines the latest execution time of the pre-filling phase of the input request based on the predicted time consumption of the pre-filling phase, and then schedules the input request based on the latest execution time of the pre-filling phase. This optimizes system throughput and computational load balancing while meeting service level objectives, thereby improving the resource utilization efficiency and service quality assurance capabilities of the service node of the preset model. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 An example architecture diagram of a model input request scheduling method provided in an embodiment of this disclosure;

[0023] Figure 2 This is a schematic flowchart of a model input request scheduling method provided in an embodiment of the present disclosure;

[0024] Figure 3 This is a schematic flowchart of a model input request scheduling method provided in another embodiment of the present disclosure;

[0025] Figure 4 A structural block diagram of a model input request scheduling device provided in an embodiment of this disclosure;

[0026] Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0028] In model systems such as Large Language Model (LLM), the AI ​​gateway, as a core component for request scheduling, directly impacts the overall system performance through its strategy selection.

[0029] Existing AI gateways in model systems typically employ static scheduling strategies for request scheduling, such as First-In-First-Out (FIFO) or Round-Robin scheduling strategies, which have the following key limitations:

[0030] 1) Lack of resource awareness: Unable to dynamically identify and adapt to actual resource bottlenecks in the system, such as the imbalance between resource requirements in the prefill and decoding stages;

[0031] 2) Insufficient scheduling flexibility: Failed to comprehensively consider request characteristics (such as latency sensitivity and model size) and inference stage characteristics (prefill / decode) when adjusting priorities;

[0032] 3) Performance bottleneck: Under high load scenarios, it can easily lead to: idle and wasted computing resources during the decoding process, and increased risk of Service Level Objective (SLO) default.

[0033] Therefore, the static scheduling strategy adopted by the AI ​​gateway of the existing model system restricts the resource utilization efficiency and service quality assurance capability of the model system.

[0034] To address the aforementioned technical issues, considering that the Time To First Token (TTFT) essentially constitutes a service quality contract between the system and the user, meaning that delivering the first token within the agreed timeframe satisfies user experience requirements, but excessive optimization of TTFT (such as pursuing extremely low latency) not only fails to improve user experience but also leads to decreased resource utilization. This is because, in addition to outputting the first token, other tokens following the first token also need to be output, and the output speed of these other tokens also affects user experience. Therefore, this disclosure provides a model input request scheduling method that aims to optimize system throughput and computational load balancing while satisfying the service level objective (TTFT) by adaptively scheduling the model's input requests.

[0035] The application scenarios of the model input request scheduling method in this embodiment of the disclosure are as follows: Figure 1 As shown, this can be applied to an AI gateway, which can receive input requests from a preset model; determine the predicted time of the pre-filling stage in the processing of the input request by the preset model. One possible method is to query the predicted time of the pre-filling stage of the input request from a database, where the database stores the predicted time of the pre-filling stage corresponding to different metadata. The database can be an ETCD (Distributed Key-Value Store) database or any other feasible database. The AI ​​gateway can determine the latest execution time of the pre-filling stage of the input request based on the receiving time of the input request, the predicted time of the pre-filling stage, and the delay of the target first word; and send the input request to the service node of the preset model for processing based on the latest execution time of the pre-filling stage.

[0036] In addition, the NATS server can collect various statuses and indicators of service nodes of the preset model, thereby achieving closed-loop control of the AI ​​gateway.

[0037] The model input request scheduling method of this disclosure will be described in detail below with reference to specific embodiments.

[0038] refer to Figure 2 , Figure 2This is a schematic flowchart of a model input request scheduling method according to an embodiment of the present disclosure. The method of this embodiment can be applied to an AI gateway, where the AI ​​gateway can be a physical device or an application deployed in an electronic device such as a terminal device or server. The model input request scheduling method includes:

[0039] S201, Receive input request for preset model.

[0040] In this embodiment, when a user needs to use a preset model, they can send an input request for the preset model to the AI ​​gateway. The preset model can be any AI model that requires a prefilling stage. The prefilling stage is a crucial preparatory stage before text generation; its main function is to pre-calculate and cache key-value (KV) vectors for the model's initial generation step. The first token, or first word, can be output through the prefilling stage. Optionally, in scenarios such as large language models, the input request can be a prompt word.

[0041] S202. Determine the prediction time of the input request in the pre-filling stage of the preset model processing.

[0042] In this embodiment, considering that the pre-filling stage is the process from input request to preset model to output first word, and the pre-filling stage time is the time from input request to preset model to output first word, in order to meet the Service Level Objective (SLO), that is, to meet the Target First Term Delay (TTFT), where the first term delay is the delay time from receiving the input request to outputting the first word, and the target first term delay is the first term delay required by the user, it is necessary to predict the time of the pre-filling stage in the preset model processing of the input request, that is, the pre-filling stage prediction time, to provide a quantitative basis for subsequent scheduling.

[0043] In this embodiment, the input request can be parsed to obtain the metadata of the input request. The metadata of the input request includes, but is not limited to, the type of the preset model, the hardware information for running the preset model, the batch size of the input request, and one or more of the representative lengths of the buckets to which the input request belongs after being bucketed according to the length of the input request. Of course, other metadata such as the receiving time of the input request can also be extracted, but this embodiment does not impose any limitations.

[0044] In this embodiment, based on the metadata of the input request, any feasible method can be used to predict and determine the prediction time of the pre-filling stage of the input request in the preset model processing process. For example, the prediction time of the pre-filling stage of the current input request can be determined by referring to the time of the pre-filling stage of historical input requests with the same or similar metadata in the preset model processing process. Alternatively, some neural network models can be used for prediction, or some algorithms or formulas can be used for prediction, etc.

[0045] In one optional embodiment, based on the metadata of the input request, the pre-filling stage prediction time of the input request in the preset model processing process is queried from the database, wherein the database pre-stores the pre-filling stage prediction time corresponding to different metadata.

[0046] In this embodiment, some historical input requests can be obtained in advance to acquire metadata and the pre-filling stage time. Alternatively, some test input requests can be obtained to acquire metadata, and the pre-filling stage time can be determined through offline testing. Modeling can then be performed to establish a mathematical model for predicting the pre-filling stage time. For example:

[0047] Pre-filling stage predicted time = p1*L + p2*B + p3

[0048] Where L is the representative length of the bucket after binning according to the length of the input request. For example, binning according to the length of the input request, the length range of bin 1 is (0-64], the length range of bin 2 is (64-128], and so on. The representative length of the bin is to replace the length included in the bin with a fixed value. For example, it can be an endpoint or the median of the bin length range. For example, the representative length of bin 1 can be 32, and the representative length of bin 2 can be 96. B is the batch size; p1, p2, and p3 are fitting parameters.

[0049] Furthermore, the predicted time consumption of the pre-filling stage corresponding to different metadata can be determined based on the mathematical model. The predicted time consumption of the pre-filling stage corresponding to different metadata can be stored in the database. Specifically, it can be stored in key-value pair (KV) format, where the key is the metadata and the value is the predicted time consumption of the pre-filling stage.

[0050] Optionally, mathematical models for predicting the pre-filling stage time can be established for different types of preset models; mathematical models for predicting the pre-filling stage time can also be established for different hardware. This allows for the determination of mathematical models for predicting the pre-filling stage time for different types of preset models and different hardware. Furthermore, the predicted pre-filling stage time for different preset models, different hardware, different L, and different B can be stored in a database for later querying based on the metadata of the current input request. This allows for customized determination of the pre-filling stage time for each input request; significantly improves scheduling accuracy, avoids excessive delays for lightweight requests or ineffective preemption of heavy tasks; enables the system to have the basic capabilities for accurate scheduling and delay-constrained reasoning; and lays the foundation for future integration of more complex scheduling strategies (such as token-level scheduling and RL scheduling).

[0051] In another alternative embodiment, based on the metadata of the input request and / or the content of the input request, the predicted time of the pre-filling stage of the input request in the processing of the preset model is determined by a prediction model for the predicted time of the pre-filling stage.

[0052] In this embodiment, a prediction model for the predicted time consumption of the pre-filling stage can be established in advance. Optionally, the prediction model for the predicted time consumption of the pre-filling stage can be the mathematical model mentioned above, or it can be other possible models such as a neural network model. The input of the prediction model for the predicted time consumption of the pre-filling stage can be the metadata of the input request and / or the content of the input request. The predicted time consumption of the pre-filling stage of the input request is output by the prediction model for the predicted time consumption of the pre-filling stage.

[0053] Furthermore, considering that the performance of the service nodes of the preset model may change in real time, the predicted time of the pre-filling stage of the input request predicted by the above prediction methods may not be accurate and cannot adapt to the performance changes of the service nodes of the preset model. Therefore, the predicted time of the pre-filling stage of the input request can be corrected.

[0054] Optionally, a correction model for the predicted time of the pre-filling stage can be constructed, which can then be used to correct the predicted time of the input request's pre-filling stage. The input parameters of the correction model for the predicted time of the pre-filling stage can include, but are not limited to, the length of the input request, and can also include the predicted time of the input request's pre-filling stage. By using the correction model for the predicted time of the pre-filling stage, the predicted time of the pre-filling stage is corrected according to the length of the input request, ultimately yielding the corrected predicted time of the pre-filling stage. As an example, the correction model for the predicted time of the pre-filling stage can be the following mathematical model:

[0055] Corrected pre-filling stage prediction time = Pre-filling stage prediction time + α + β * (length of input request - L)

[0056] Where L is the representative length of the bucket after bucketing according to the length of the input request; α and β are adjustable fitting parameters that can reflect the performance changes of the service nodes of the preset model. That is, when the performance of the service nodes of the preset language model changes, α and β also need to be updated.

[0057] Optionally, a correction model for the predicted pre-filling stage time can be constructed based on the actual and predicted pre-filling stage times of historical input requests within a historical time window. For example, α and β can be determined by linear fitting based on the actual and predicted pre-filling stage times of historical input requests within a historical time window, thus obtaining the correction model for the predicted pre-filling stage time. The process of constructing the correction model for the predicted pre-filling stage time can be performed periodically or when a performance change in the service node of the preset model is detected. Optionally, in this embodiment, a feedback resonator can be used to update the correction model for the predicted pre-filling stage time; or the NATS server can collect the operating status and some indicators (such as the actual pre-filling stage time mentioned above) of the service node of the preset model to update the correction model for the predicted pre-filling stage time, achieving closed-loop control.

[0058] In other alternatives, the above parameters can be configured using a fixed configuration through an instance configuration file, or dynamically updated using an exponentially weighted moving average (EWMA) of the residuals from the pre-filling phase.

[0059] In another alternative embodiment, if the predicted time of the pre-filling stage of the input request cannot be determined by any of the above methods, such as when the database is unavailable in the above embodiments, or when the predicted time of the pre-filling stage of the input request cannot be found, or when the prediction model for the predicted time of the pre-filling stage cannot predict it, the predicted time of the pre-filling stage of the input request with the same or similar metadata can be selected from the predicted time of the pre-filling stage of the recent input requests in the local cache; or a static heuristic method can be used, for example, the predicted time of the pre-filling stage = 5ms + 0.1 * the length of the input request.

[0060] S203. Based on the receiving time of the input request, the prediction time of the pre-filling stage, and the delay of the target first word, determine the latest execution time of the pre-filling stage of the input request.

[0061] In this embodiment, after determining the prediction time of the pre-filling stage of the input request, the latest execution time (deadline) of the pre-filling stage of the input request can be determined by combining the input request reception time, the prediction time of the pre-filling stage, and the target first-term delay (TTFT). The specific formula for the latest execution time (deadline) of the pre-filling stage of the input request can be:

[0062] deadline = reception time + TTFT - pre-filling stage prediction time;

[0063] The above formula can be used to determine the time of the pre-filling stage for the latest execution of the input request that satisfies the target first-word delay (TTFT), ensuring that the first word is output before the target first-word delay (TTFT).

[0064] It should be noted that if the latest execution time of the input request pre-filling stage is earlier than the current time, it means that even if the input request pre-filling stage is executed immediately at the current time, the target first word delay cannot be met. In this case, the input request can not be scheduled further, and a prompt message that the input request cannot be responded to can be returned to the user. The service level target can be avoided through the filtering process.

[0065] S204. Based on the latest execution time of the pre-filling stage, the input request is sent to the service node of the preset model for processing.

[0066] In this embodiment, after determining the latest execution time of the pre-filling phase for input requests, input requests can be scheduled based on this latest execution time. Input requests are sent to the service nodes of the preset model for processing at an appropriate time before the latest execution time of the pre-filling phase, maintaining scheduling affinity and ensuring timely entry into the pre-filling phase. This allows for a reasonable allocation of resources between the pre-filling and decoding tasks of the service nodes of the preset model. While ensuring the minimum resource allocation required for the pre-filling task to achieve the target TTFT, as many resources as possible are allocated to the decoding task. Since the decoding task can continue to output tokens after the first word, allocating as many resources as possible to the decoding task can significantly improve the token throughput of the service nodes of the preset model, optimize system throughput and computational load balancing, and improve the resource utilization efficiency and service quality assurance capabilities of the service nodes of the preset model.

[0067] It should be noted that, in this embodiment, as long as the input request is sent to the service node of the preset model for processing before the latest execution time of the input request pre-filling stage, any scheduling strategy or any feasible priority order can be used to send the input request to the service node of the preset model. For example, any scheduling strategy such as Earliest Deadline First (EDF) or Shortest Job First (SJF) can be used.

[0068] The model input request scheduling method provided in this embodiment receives input requests from a preset model; determines the predicted time consumption of the pre-filling stage in the processing of the input request in the preset model; determines the latest execution time of the pre-filling stage of the input request based on the reception time of the input request, the predicted time consumption of the pre-filling stage, and the target first word delay; and sends the input request to the service node of the preset model for processing based on the latest execution time of the pre-filling stage. This embodiment determines the latest execution time of the pre-filling stage of the input request based on the predicted time consumption of the pre-filling stage, and then schedules the input request based on the latest execution time of the pre-filling stage. This optimizes system throughput and computational load balancing while meeting service level objectives, thereby improving the resource utilization efficiency and service quality assurance capabilities of the service node of the preset model.

[0069] Furthermore, this embodiment uses TTFT as the user experience target and quantifies it as the scheduling deadline for each input request, i.e., the latest execution time of the pre-filling phase. Combined with the driver's scheduling strategy, a closed-loop feedback mechanism from service level target to system behavior is constructed, improving the TTFT achievement rate. The system can find the optimal scheduling balance between "resource constraints" and "latency target", achieving tail latency optimization. Through priority scheduling of latency-sensitive requests (such as the EDF mechanism), high percentile latency (P99) is effectively reduced, which is suitable for real-time interactive scenarios. It can also support diversified service levels (multi-SLO) to meet mixed scenarios of high, medium and low priority requests. The SLO target can be dynamically adjusted according to the user, service level or request type, i.e., the target TTFT can be adjusted, realizing quality of service (QoS) control in multi-tenant scenarios and having good scalability.

[0070] Furthermore, to enhance the overall scalability and engineering adaptability of the system, this embodiment adopts a modular design approach, resulting in a system with excellent pluggability, monitorability, and evolvability. Key modules, including the prediction time estimation for the pre-filling stage, scheduling strategies, and dispatch logic, can be flexibly replaced and upgraded. This facilitates rapid adaptation to inference clusters of different sizes and model structures; it is easy to integrate and deploy with existing infrastructure (such as NATS and ETCD), reducing engineering modification costs; and it provides a solid foundation for the subsequent introduction of strategies such as RL (Reinforcement Learning) schedulers, dynamic batch processing, and KV cache awareness.

[0071] Based on any of the above embodiments, optionally, in S204, the input request is sent to the service node of the preset model for processing according to the latest execution time of the pre-filling stage, which may specifically include:

[0072] Based on the latest execution time of the pre-filling stage of the multiple input requests, the priority of the multiple input requests is determined by a preset priority rule, and the multiple input requests are added to the scheduling queue based on the priority; the input requests in the scheduling queue are sent to the service nodes of the preset model for processing in sequence.

[0073] In this embodiment, for multiple input requests, the priority of multiple input requests can be determined by using a preset priority rule based on the latest execution time of the pre-filling stage of the multiple input requests. The preset priority rule can be any feasible priority rule. Then, the multiple input requests are added to the scheduling queue based on their priority and sorted in the scheduling queue according to their priority order. Then, the input requests can be sent to the service nodes of the preset model for processing in sequence through the scheduling queue.

[0074] In one alternative embodiment, a three-tier priority rule can be used to determine the priority of each input request:

[0075] First-level priority: Input requests are bucketed based on the latest execution time of the pre-filling phase, with higher priority for input requests in buckets whose latest execution time is earlier. For example, for the first bucket with the latest execution time of the pre-filling phase being t1 to t2, and the second bucket with the latest execution time of the pre-filling phase being t3 to t4, since t1 to t2 is earlier than t3 to t4, all input requests in the first bucket have a higher priority than all input requests in the second bucket.

[0076] Second-level priority: Within the same bucket, the priority of input requests is determined according to the predicted time of the pre-filling stage, where the input request with the shorter the predicted time of the pre-filling stage has the higher priority.

[0077] The third priority level: Among multiple input requests with the same prediction time in the pre-filling stage, the priority of the input requests is determined according to the reception time, where the earlier the reception time, the higher the priority of the input request.

[0078] By combining the earliest deadline first (EDF) and shortest task first (SJF) scheduling strategies through the above three-layer priority rules, the urgency of input requests and computational costs can be dynamically judged, thereby flexibly determining the scheduling order of input requests. This improves resource utilization efficiency while satisfying different latency-sensitive requests; significantly reduces high percentile latency (P95 / P99), enhancing the experience of real-time interactive services; and improves overall service quality assurance capabilities without increasing resources.

[0079] In one alternative embodiment, multiple input requests are bucketed according to the latest execution time of the pre-filling phase based on multiple input requests, wherein the input requests in the bucket with the earlier latest execution time of the pre-filling phase have higher priority.

[0080] If the length difference of input requests in the same bucket is less than the preset difference, that is, if the length of the input requests is highly consistent, then the priority of the input requests is determined according to the reception time, with the input requests that are received earlier having higher priority.

[0081] In this embodiment, when the lengths of input requests are highly consistent, only two layers of priority rules can be used. The first layer determines the priority of the input request based on the latest execution time of the pre-filling stage and the receiving time. The second layer determines the priority of the input request based on the receiving time. This eliminates the need to determine the priority of the input request based on the predicted time of the pre-filling stage, thus skipping the sorting based on the predicted time of the pre-filling stage and reducing computational overhead.

[0082] Based on any of the above embodiments, such as Figure 3 As shown, when the input requests in the scheduling queue are sequentially sent to the service nodes of the preset model for processing, the specific process may include:

[0083] S301. Detect the current load status of the service nodes of the preset model;

[0084] If the current load status is idle, then execute S302; if the current load status is not idle, then execute S303.

[0085] S302. If the current load state of the service node of the preset model is idle, the input requests in the scheduling queue are sent directly to the service node of the preset model for processing.

[0086] S303. If the current load state of the service node of the preset model is not idle, the input requests in the scheduling queue are sequentially sent to the service node of the preset model for processing at a preset time point before the latest execution time of their respective pre-filling stages.

[0087] In this embodiment, when the current load state of the service node of the preset model is detected to be idle, it indicates that the service node of the preset model has a large amount of idle resources. At this time, the input requests in the scheduling queue can be sent directly to the service node of the preset model for processing based on priority. That is, an opportunistic scheduling path is adopted to improve resource utilization and avoid resource waste when the service node of the preset model is idle, thus achieving "computation on demand".

[0088] If the current load state of the service node of the preset model is not idle, it indicates that the idle resources of the service node of the preset model are limited. In order to ensure the progress of the decoding task and allocate as many resources as possible to the decoding task to improve the token throughput, the input requests in the scheduling queue can be sent to the service node of the preset model in turn at a preset time point before the latest execution time of their respective pre-filling stage. There is a preset time interval between the preset time point before the latest execution time of the pre-filling stage and the latest execution time of the pre-filling stage. That is, the sending time of the input request is the deadline minus the preset time interval. In this way, the input requests are sent to the service node of the preset model as completely as possible, satisfying the target TTFT while controlling resource contention.

[0089] When dispatching input requests according to the preset time point before the latest execution time of the pre-filling stage, there may be multiple input requests with the same preset time point before the latest execution time of the pre-filling stage. These input requests also need to be sent in a certain order. At this time, these input requests can be sent to the service nodes of the preset model according to the priority order to avoid conflicts.

[0090] In the above embodiments, by introducing a mechanism to perceive the current load status of service nodes in a preset model and combining it with opportunistic scheduling paths, lightweight requests can be dispatched in advance when resources are idle, thereby improving the resource utilization of the decoding end. In high-load scenarios, the throughput is increased by an average of 8% to 10% or more. Resource waste is avoided in the idle state, achieving "computation on demand". Queue backlog and scheduling delay are significantly reduced, effectively responding to sudden traffic surges.

[0091] It should be noted that in S302, if the current load state of the service node of the preset model is idle, the input requests in the scheduling queue will be sent directly to the service node of the preset model for processing. After sending several input requests, the current load state may no longer be idle. In this case, S303 needs to be executed to continue sending.

[0092] Based on any of the above embodiments, the current load status of the service node of the preset model can be detected in any feasible manner. One feasible lightweight detection method is as follows:

[0093] Obtain the cache utilization rate of the service nodes of the preset model and the number of pending input requests in the service nodes of the preset model; based on the cache utilization rate and the number of pending requests, determine whether the current load status of the service nodes of the preset model is idle.

[0094] In this embodiment, the cache utilization rate of the service node of the preset model, such as the GPU cache utilization rate (GPUcache usage%), can reflect the current occupancy of the model KV cache on the GPU, and indirectly reflect the number of decoding tasks currently running. The number of input requests to be processed in the service node of the preset model, that is, the request queue length, can represent the number of input requests currently waiting to be processed in the service node. The number of input requests to be processed in the service node of the preset model can be normalized, that is, the ratio of the number of input requests to be processed in the service node of the preset model to the maximum queue length can be obtained as the information of the number of input requests to be processed in the service node of the preset model.

[0095] The load score of the service nodes in the preset model can then be determined by weighted summation, as shown in the following formula:

[0096] The load score of the service node in the preset model = k1 * cache utilization rate + k2 * number of pending input requests;

[0097] Among them, k1 and k2 are adjustable parameters used to determine the weight of the two factors in the overall load judgment (e.g., k1 = 0.7, k2 = 0.3).

[0098] The final calculated load score of the service node in the preset model is compared with a preset idle state threshold. If the load score is lower than the preset idle state threshold, the service node in the preset model is determined to be in an idle state or an underutilized resource state; if the load score is not lower than the preset idle state threshold, the service node in the preset model is determined to be not idle, i.e., the resource is busy. This lightweight monitoring mechanism can accurately and quickly determine the current load status, thereby providing support for input request scheduling.

[0099] Optionally, the cache utilization rate of the service nodes of the preset model and the number of pending input requests in the service nodes of the preset model can be collected by the NATS server to determine the current load status of the service nodes of the preset model, thereby realizing closed-loop control.

[0100] It should be noted that if the current load status of the service nodes in the preset model cannot be determined, the scheduling can be performed by default according to the S303 method.

[0101] It should be noted that the various strategies and mathematical models involved in the above embodiments can be dynamically adjusted according to the status of the service nodes of the preset model. The feedback information of the service nodes of the preset model can be collected through the NATS server to achieve closed-loop control.

[0102] Corresponding to the model input request scheduling method in the above embodiment, Figure 4 This is a structural block diagram of a model input request scheduling device provided in an embodiment of this disclosure. For ease of explanation, only the parts relevant to the embodiments of this disclosure are shown. (Refer to...) Figure 4 The model input request scheduling device 400 includes: a receiving unit 401, a prediction unit 402, an execution time determination unit 403, and a scheduling unit 404.

[0103] The receiving unit 401 is used to receive input requests from a preset model.

[0104] Prediction unit 402 is used to determine the prediction time of the input request in the pre-filling stage of the preset model processing process;

[0105] The execution time determination unit 403 is used to determine the latest execution time of the pre-filling stage of the input request based on the receiving time of the input request, the predicted time of the pre-filling stage, and the delay of the target first word.

[0106] The scheduling unit 404 is used to send the input request to the service node of the preset model for processing according to the latest execution time of the pre-filling stage.

[0107] The model input request scheduling device provided in this embodiment receives input requests from a preset model; determines the predicted time of the pre-filling stage in the processing of the input request in the preset model; determines the latest execution time of the pre-filling stage of the input request based on the receiving time of the input request, the predicted time of the pre-filling stage, and the target first word delay; and sends the input request to the service node of the preset model for processing based on the latest execution time of the pre-filling stage. In this embodiment, by determining the latest execution time of the pre-filling stage of the input request based on the predicted time of the pre-filling stage, and then scheduling the input request based on the latest execution time of the pre-filling stage, the system throughput and computational load balancing are optimized while meeting service level objectives, thereby improving the resource utilization efficiency and service quality assurance capabilities of the service node of the preset model.

[0108] In one or more embodiments of this disclosure, the prediction unit 402, when determining the prediction time of the input request in the pre-filling stage of the preset model processing, is configured to:

[0109] Based on the metadata of the input request, query the database for the predicted time consumption of the pre-filling stage in the processing of the preset model, wherein the database stores the predicted time consumption of the pre-filling stage corresponding to different metadata; or

[0110] Based on the metadata of the input request and / or the content of the input request, the predicted time of the pre-filling stage of the input request in the processing of the preset model is determined by a prediction model for the predicted time of the pre-filling stage.

[0111] In one or more embodiments of this disclosure, the metadata of the input request includes at least one of the following: the type of the preset model, the hardware information for running the preset model, the batch size of the input request, and the representative length of the bucket to which the input request belongs after being bucketed according to the length of the input request.

[0112] In one or more embodiments of this disclosure, after determining the prediction time of the input request in the pre-filling stage of the preset model processing, the prediction unit 402 is further configured to:

[0113] The predicted time for the pre-filling stage is corrected based on the length of the input request using a modified model for the predicted time of the pre-filling stage.

[0114] In one or more embodiments of this disclosure, the prediction unit 402 is further configured to:

[0115] Based on the actual time taken for the pre-filling stage and the predicted time taken for the pre-filling stage of historical input requests within the historical time window, a correction model for the predicted time taken for the pre-filling stage is constructed.

[0116] In one or more embodiments of this disclosure, when the scheduling unit 404 sends the input request to the service node of the preset model for processing based on the latest execution time of the pre-filling stage, it is used to:

[0117] Based on the latest execution time of the pre-filling stage of the multiple input requests, the priority of the multiple input requests is determined by a preset priority rule, and the multiple input requests are added to the scheduling queue based on the priority.

[0118] The input requests in the scheduling queue are sequentially sent to the service nodes of the preset model for processing.

[0119] In one or more embodiments of this disclosure, when the scheduling unit 404 determines the priority of the multiple input requests based on the latest execution time of the pre-filling stage of the multiple input requests using a preset priority rule, it is used to:

[0120] The input requests are bucketed according to the latest execution time of the pre-filling phase based on the input requests, wherein the input requests in the bucket with the earlier latest execution time of the pre-filling phase have higher priority.

[0121] Within the same bucket, the priority of the input request is determined according to the predicted time of the pre-filling stage, wherein the input request with the shorter predicted time of the pre-filling stage has a higher priority.

[0122] In the pre-filling phase, among multiple input requests with the same prediction time, the priority of the input requests is determined according to the reception time, wherein the input request with the earlier reception time has a higher priority.

[0123] In one or more embodiments of this disclosure, when the scheduling unit 404 determines the priority of the multiple input requests based on the latest execution time of the pre-filling stage of the multiple input requests using a preset priority rule, it is used to:

[0124] The input requests are bucketed according to the latest execution time of the pre-filling phase based on the input requests, wherein the input requests in the bucket with the earlier latest execution time of the pre-filling phase have higher priority.

[0125] If the length difference of the input requests in the same bucket is less than a preset difference, the priority of the input requests is determined according to the reception time, wherein the input request with the earlier reception time has a higher priority.

[0126] In one or more embodiments of this disclosure, when the scheduling unit 404 sequentially sends the input requests in the scheduling queue to the service nodes of the preset model for processing, it is used to:

[0127] If the current load state of the service node of the preset model is idle, then the input requests in the scheduling queue are directly sent to the service node of the preset model for processing; or

[0128] If the current load state of the service node of the preset model is not idle, the input requests in the scheduling queue will be sent to the service node of the preset model for processing only after the preset time point before the latest execution time of the pre-filling stage.

[0129] In one or more embodiments of this disclosure, the scheduling unit 404 is further configured to:

[0130] For multiple input requests that have the same preset time point before the latest execution time of the pre-filling stage, they are sent to the service node of the preset model for processing in priority order.

[0131] In one or more embodiments of this disclosure, the scheduling unit 404 is further configured to:

[0132] Obtain the cache utilization rate of the service nodes of the preset model and the number of pending input requests in the service nodes of the preset model; based on the cache utilization rate and the number of pending requests, determine whether the current load status of the service nodes of the preset model is idle.

[0133] The device provided in this embodiment can be used to execute the technical solutions of the above method embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0134] To implement the above embodiments, this disclosure also provides an electronic device.

[0135] refer to Figure 5 The diagram illustrates a structural schematic of an electronic device 500 suitable for implementing embodiments of the present disclosure. The electronic device 500 can be a terminal device or a server. The terminal device can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablet computers, portable media players (PMPs), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 5The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0136] like Figure 5 As shown, the electronic device 500 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the electronic device 500. The processing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0137] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 An electronic device 500 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0138] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, it performs the functions defined in the methods of embodiments of this disclosure.

[0139] It should be noted that the computer-readable storage medium described in this disclosure can be a computer-readable signal medium, a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable storage medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0140] The aforementioned computer-readable storage medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0141] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the method shown in the above embodiments.

[0142] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0143] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0144] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0145] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0146] The electronic device, computer-readable storage medium, and computer program product provided in this disclosure receive an input request from a preset model; determine the predicted time of the pre-filling phase of the input request during the processing of the preset model; determine the latest execution time of the pre-filling phase of the input request based on the receiving time of the input request, the predicted time of the pre-filling phase, and the target first word delay; and send the input request to the service node of the preset model for processing based on the latest execution time of the pre-filling phase. In this embodiment, by determining the latest execution time of the pre-filling phase of the input request based on the predicted time of the pre-filling phase, and then scheduling the input request based on the latest execution time of the pre-filling phase, the system throughput and computational load balancing are optimized while meeting service level objectives, thereby improving the resource utilization efficiency and service quality assurance capabilities of the service node of the preset model.

[0147] In a first aspect, according to one or more embodiments of this disclosure, a model input request scheduling method is provided, comprising:

[0148] Receive input requests from a preset model;

[0149] Determine the predicted time consumption of the input request during the pre-filling stage of the preset model processing;

[0150] Based on the receiving time of the input request, the predicted time of the pre-filling stage, and the delay of the target first word, the latest execution time of the pre-filling stage of the input request is determined.

[0151] Based on the latest execution time of the pre-filling stage, the input request is sent to the service node of the preset model for processing.

[0152] According to one or more embodiments of this disclosure, determining the prediction time of the input request in the pre-filling stage of the preset model processing includes:

[0153] Based on the metadata of the input request, query the database for the predicted time consumption of the pre-filling stage in the processing of the preset model, wherein the database stores the predicted time consumption of the pre-filling stage corresponding to different metadata; or

[0154] Based on the metadata of the input request and / or the content of the input request, the predicted time of the pre-filling stage of the input request in the processing of the preset model is determined by a prediction model for the predicted time of the pre-filling stage.

[0155] According to one or more embodiments of this disclosure, the metadata of the input request includes at least one of the following: the type of the preset model, the hardware information for running the preset model, the batch size of the input request, and the representative length of the bucket to which the input request belongs after being bucketed according to the length of the input request.

[0156] According to one or more embodiments of this disclosure, after determining the prediction time of the input request in the pre-filling stage of the preset model processing, the method further includes:

[0157] The predicted time for the pre-filling stage is corrected based on the length of the input request using a modified model for the predicted time of the pre-filling stage.

[0158] According to one or more embodiments of this disclosure, the method further includes:

[0159] Based on the actual time taken for the pre-filling stage and the predicted time taken for the pre-filling stage of historical input requests within the historical time window, a correction model for the predicted time taken for the pre-filling stage is constructed.

[0160] According to one or more embodiments of this disclosure, sending the input request to the service node of the preset model for processing based on the latest execution time of the pre-filling stage includes:

[0161] Based on the latest execution time of the pre-filling stage of the multiple input requests, the priority of the multiple input requests is determined by a preset priority rule, and the multiple input requests are added to the scheduling queue based on the priority.

[0162] The input requests in the scheduling queue are sequentially sent to the service nodes of the preset model for processing.

[0163] According to one or more embodiments of this disclosure, determining the priority of the multiple input requests based on the latest execution time of the pre-filling stage according to a preset priority rule includes:

[0164] The input requests are bucketed according to the latest execution time of the pre-filling phase based on the input requests, wherein the input requests in the bucket with the earlier latest execution time of the pre-filling phase have higher priority.

[0165] Within the same bucket, the priority of the input request is determined according to the predicted time of the pre-filling stage, wherein the input request with the shorter predicted time of the pre-filling stage has a higher priority.

[0166] In the pre-filling phase, among multiple input requests with the same prediction time, the priority of the input requests is determined according to the reception time, wherein the input request with the earlier reception time has a higher priority.

[0167] According to one or more embodiments of this disclosure, determining the priority of the multiple input requests based on the latest execution time of the pre-filling stage according to a preset priority rule includes:

[0168] The input requests are bucketed according to the latest execution time of the pre-filling phase based on the input requests, wherein the input requests in the bucket with the earlier latest execution time of the pre-filling phase have higher priority.

[0169] If the length difference of the input requests in the same bucket is less than a preset difference, the priority of the input requests is determined according to the reception time, wherein the input request with the earlier reception time has a higher priority.

[0170] According to one or more embodiments of this disclosure, the step of sequentially sending the input requests in the scheduling queue to the service nodes of the preset model for processing includes:

[0171] If the current load state of the service node of the preset model is idle, then the input requests in the scheduling queue are directly sent to the service node of the preset model for processing; or

[0172] If the current load state of the service node of the preset model is not idle, the input requests in the scheduling queue will be sent to the service node of the preset model for processing only after the preset time point before the latest execution time of the pre-filling stage.

[0173] According to one or more embodiments of this disclosure, the method further includes:

[0174] For multiple input requests that have the same preset time point before the latest execution time of the pre-filling stage, they are sent to the service node of the preset model for processing in priority order.

[0175] According to one or more embodiments of this disclosure, the method further includes:

[0176] Obtain the cache utilization rate of the service nodes of the preset model and the number of pending input requests in the service nodes of the preset model; based on the cache utilization rate and the number of pending requests, determine whether the current load status of the service nodes of the preset model is idle.

[0177] Secondly, according to one or more embodiments of this disclosure, a model input request scheduling device is provided, comprising:

[0178] The receiving unit is used to receive input requests from a preset model;

[0179] A prediction unit is used to determine the prediction time of the input request in the pre-filling stage of the preset model processing process;

[0180] An execution time determination unit is used to determine the latest execution time of the pre-filling stage of the input request based on the receiving time of the input request, the predicted time of the pre-filling stage, and the delay of the target first word.

[0181] The scheduling unit is used to send the input request to the service node of the preset model for processing according to the latest execution time of the pre-filling stage.

[0182] According to one or more embodiments of this disclosure, the prediction unit, in determining the prediction time consumption of the input request during the pre-filling phase of the preset model processing, is configured to:

[0183] Based on the metadata of the input request, query the database for the predicted time consumption of the pre-filling stage in the processing of the preset model, wherein the database stores the predicted time consumption of the pre-filling stage corresponding to different metadata; or

[0184] Based on the metadata of the input request and / or the content of the input request, the predicted time of the pre-filling stage of the input request in the processing of the preset model is determined by a prediction model for the predicted time of the pre-filling stage.

[0185] According to one or more embodiments of this disclosure, the metadata of the input request includes at least one of the following: the type of the preset model, the hardware information for running the preset model, the batch size of the input request, and the representative length of the bucket to which the input request belongs after being bucketed according to the length of the input request.

[0186] According to one or more embodiments of this disclosure, after determining the prediction time of the input request in the pre-filling stage of the preset model processing, the prediction unit is further configured to:

[0187] The predicted time for the pre-filling stage is corrected based on the length of the input request using a modified model for the predicted time of the pre-filling stage.

[0188] According to one or more embodiments of this disclosure, the prediction unit is further configured to:

[0189] Based on the actual time taken for the pre-filling stage and the predicted time taken for the pre-filling stage of historical input requests within the historical time window, a correction model for the predicted time taken for the pre-filling stage is constructed.

[0190] According to one or more embodiments of this disclosure, when the scheduling unit sends the input request to the service node of the preset model for processing based on the latest execution time of the pre-filling stage, it is used to:

[0191] Based on the latest execution time of the pre-filling stage of the multiple input requests, the priority of the multiple input requests is determined by a preset priority rule, and the multiple input requests are added to the scheduling queue based on the priority.

[0192] The input requests in the scheduling queue are sequentially sent to the service nodes of the preset model for processing.

[0193] According to one or more embodiments of this disclosure, when the scheduling unit determines the priority of multiple input requests based on the latest execution time of the pre-filling phase of the multiple input requests using a preset priority rule, it is used to:

[0194] The input requests are bucketed according to the latest execution time of the pre-filling phase based on the input requests, wherein the input requests in the bucket with the earlier latest execution time of the pre-filling phase have higher priority.

[0195] Within the same bucket, the priority of the input request is determined according to the predicted time of the pre-filling stage, wherein the input request with the shorter predicted time of the pre-filling stage has a higher priority.

[0196] In the pre-filling phase, among multiple input requests with the same prediction time, the priority of the input requests is determined according to the reception time, wherein the input request with the earlier reception time has a higher priority.

[0197] According to one or more embodiments of this disclosure, when the scheduling unit determines the priority of multiple input requests based on the latest execution time of the pre-filling phase of the multiple input requests using a preset priority rule, it is used to:

[0198] The input requests are bucketed according to the latest execution time of the pre-filling phase based on the input requests, wherein the input requests in the bucket with the earlier latest execution time of the pre-filling phase have higher priority.

[0199] If the length difference of the input requests in the same bucket is less than a preset difference, the priority of the input requests is determined according to the reception time, wherein the input request with the earlier reception time has a higher priority.

[0200] According to one or more embodiments of this disclosure, when the scheduling unit sequentially sends the input requests in the scheduling queue to the service nodes of the preset model for processing, it is used to:

[0201] If the current load state of the service node of the preset model is idle, then the input requests in the scheduling queue are directly sent to the service node of the preset model for processing; or

[0202] If the current load state of the service node of the preset model is not idle, the input requests in the scheduling queue will be sent to the service node of the preset model for processing only after the preset time point before the latest execution time of the pre-filling stage.

[0203] According to one or more embodiments of this disclosure, the scheduling unit is further configured to:

[0204] For multiple input requests that have the same preset time point before the latest execution time of the pre-filling stage, they are sent to the service node of the preset model for processing in priority order.

[0205] According to one or more embodiments of this disclosure, when the scheduling unit detects the current load status of the service nodes of the preset model in real time, it is used to:

[0206] Obtain the cache utilization rate of the service nodes of the preset model and the number of pending input requests in the service nodes of the preset model; based on the cache utilization rate and the number of pending requests, determine whether the current load status of the service nodes of the preset model is idle.

[0207] Thirdly, according to one or more embodiments of the present disclosure, an electronic device is provided, comprising: at least one processor and a memory;

[0208] The memory stores computer-executed instructions;

[0209] The at least one processor executes the computer execution instructions stored in the memory, causing the at least one processor to perform the model input request scheduling method as described in the first aspect and various possible designs of the first aspect.

[0210] Fourthly, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein computer-executable instructions are stored therein, and when a processor executes the computer-executable instructions, the model input request scheduling method described in the first aspect and various possible designs of the first aspect is implemented.

[0211] Fifthly, according to one or more embodiments of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the model input request scheduling method as described in the first aspect and various possible designs of the first aspect.

[0212] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0213] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0214] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A model input request scheduling method, characterized by, The method comprises: receiving an input request of a preset model; determining a predicted time consumption of a pre-filling stage of the input request in a processing procedure of the preset model; determining a latest execution time of the pre-filling stage of the input request according to a receiving time of the input request, the predicted time consumption of the pre-filling stage and a target first word delay; sending the input request to a service node of the preset model for processing according to the latest execution time of the pre-filling stage.

2. The method of claim 1, wherein, The determining of the predicted time consumption of the pre-filling stage of the input request in the processing procedure of the preset model comprises: querying the predicted time consumption of the pre-filling stage of the input request in the processing procedure of the preset model from a database according to meta information of the input request, wherein the database pre-stores predicted time consumptions of pre-filling stages corresponding to different meta information; or determining the predicted time consumption of the pre-filling stage of the input request in the processing procedure of the preset model according to the meta information of the input request and / or content of the input request by using a prediction model for predicted time consumption of a pre-filling stage.

3. The method of claim 2, wherein, After the determining of the predicted time consumption of the pre-filling stage of the input request in the processing procedure of the preset model, the method further comprises: correcting the predicted time consumption of the pre-filling stage according to a length of the input request by using a correction model for predicted time consumption of a pre-filling stage.

4. The method according to any one of claims 1 to 3, characterized in that, The sending of the input request to the service node of the preset model for processing according to the latest execution time of the pre-filling stage comprises: determining priorities of multiple input requests according to preset priority rules based on the latest execution times of the pre-filling stages of the multiple input requests, and adding the multiple input requests into a scheduling queue based on the priorities; sending the input requests in the scheduling queue to the service node of the preset model for processing in sequence.

5. The method of claim 4, wherein, The determining of the priorities of the multiple input requests according to the preset priority rules based on the latest execution times of the pre-filling stages of the multiple input requests comprises: dividing the multiple input requests into different buckets according to the latest execution times of the pre-filling stages of the multiple input requests, wherein the priorities of the input requests in a bucket are higher if the input requests in the bucket have earlier latest execution times of the pre-filling stages; determining the priorities of the input requests in a same bucket according to the predicted time consumptions of the pre-filling stages, wherein the priorities of the input requests are higher if the input requests have shorter predicted time consumptions of the pre-filling stages; determining the priorities of the input requests in the multiple input requests with the same predicted time consumptions of the pre-filling stages according to receiving times, wherein the priorities of the input requests are higher if the input requests have earlier receiving times.

6. The method of claim 4, wherein, The determining of the priorities of the multiple input requests according to the preset priority rules based on the latest execution times of the pre-filling stages of the multiple input requests comprises: dividing the multiple input requests into different buckets according to the latest execution times of the pre-filling stages of the multiple input requests, wherein the priorities of the input requests in a bucket are higher if the input requests in the bucket have earlier latest execution times of the pre-filling stages; If the length difference of the input requests in the same sub-bucket is less than a preset difference, the priority of the input requests is determined according to the receiving time, and the input request with the earlier receiving time has a higher priority.

7. The method of claim 4, wherein, The method further includes: If the current load state of the service node of the preset model is an idle state, the input requests in the scheduling queue are sequentially sent to the service node of the preset model for processing; or If the current load state of the service node of the preset model is not an idle state, the input requests in the scheduling queue are sequentially sent to the service node of the preset model for processing after waiting until a preset time point before the latest execution time of the pre-filling stage.

8. The method of claim 7, wherein, Further comprising: For a plurality of input requests with the same preset time point before the latest execution time of the pre-filling stage, the input requests are sent to the service node of the preset model for processing in the order of priority.

9. The method of claim 7, wherein, Further comprising: Obtaining the cache usage rate of the service node of the preset model and the number of input requests to be processed in the service node of the preset model; Determining whether the current load state of the service node of the preset model is in an idle state according to the cache usage rate and the number of input requests to be processed.

10. A model input request scheduling apparatus characterized by comprising: The method further includes: A receiving unit configured to receive an input request of a preset model; A prediction unit configured to determine a pre-filling stage predicted time consumption of the input request in a processing process of the preset model; An execution time determination unit configured to determine a pre-filling stage latest execution time of the input request according to a receiving time of the input request, the pre-filling stage predicted time consumption, and a target first word element delay; A scheduling unit configured to send the input request to a service node of the preset model for processing according to the pre-filling stage latest execution time.

11. An electronic device, comprising: The method further includes: A processor and a memory; The memory stores computer execution instructions; The processor executes the computer execution instructions stored in the memory, so that the processor executes the method according to any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer execution instructions, and when the processor executes the computer execution instructions, the method according to any one of claims 1-9 is implemented.

13. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the method according to any one of claims 1-9.

Images