GPU (Graphics Processing Unit) server scheduling method and system and computer equipment

Abstract

The embodiment of the invention relates to the technical field of server scheduling, in particular to a GPU (Graphics Processing Unit) server scheduling method, which comprises the following steps of: acquiring a first scheduling time table and task time which is request time of a target task request; if the server type corresponding to the current interval time is not the first GPU server, acquiring execution data of all task requests executed in the historical interval time; calculating a first operation cost of the second GPU server in the current interval time based on the execution data; calculating a second operation cost of the first GPU server in the current interval time based on the rental cost of the first GPU server; and if the first operation cost is greater than the second operation cost, scheduling the first GPU server to execute the target task request. According to the embodiment of the invention, the estimated operation cost of different server types in the specific time interval is compared in real time, switching is carried out between a charging mode and GPU services with different performances by taking economic cost optimization as a target, and the utilization efficiency of a cluster in a hybrid resource environment is improved.

Description

Technical Field

[0001] This application relates to the field of server scheduling technology, and in particular to a GPU server scheduling method, system and computer device. Background Technology

[0002] With the widespread application of artificial intelligence technology, especially the surge in demand for large language models, the reliance on high-performance GPU computing power is deepening. However, the high deployment and maintenance costs of GPU servers have become a key factor restricting the widespread adoption of the technology. To balance efficiency and cost, there are currently two main service deployment models in the market.

[0003] One approach is the Serverless service model, which employs an on-demand allocation and per-second billing mechanism. Resources can be dynamically scaled based on load, achieving zero-cost idle time during off-peak periods. However, this model suffers from cold start latency; when resources are released and requests are made again, the model needs to be reloaded, often resulting in startup times of tens of seconds, severely impacting real-time tasks. Furthermore, resource contention is prone to occur in multi-user scenarios, leading to uncertainties in task waiting times.

[0004] Another option is the Cloud service model, which also supports on-demand use, but requires a fixed amount of GPU resources to be pre-configured. Its advantages include lower computational cost per unit time, faster task response, and higher stability, with no cold start or resource contention issues. The main disadvantage is that once enabled, billing continues, and even when there are no tasks, idle resources still result in waste.

[0005] However, although scheduling algorithms exist in existing server clusters, these algorithms are mostly designed for homogeneous cloud service clusters, focusing on load balancing and resource allocation efficiency, without considering the economic costs between servers with heterogeneous service models. Summary of the Invention

[0006] One objective of this application is to provide a GPU server scheduling method, system, and computer device to solve the technical problem of the lack of server scheduling methods for heterogeneous service modes with the goal of optimizing economic costs in the prior art.

[0007] In a first aspect, embodiments of this application provide a GPU server scheduling method, including: Obtain the first scheduling schedule and task time. The first scheduling schedule includes all server types with preset intervals for the day and are scheduled in advance. The task time is the request time of the target task request. In response to the fact that the server type corresponding to the current interval time is not the first GPU server, the execution data of all task requests executed in the historical interval time are obtained, wherein the current interval time is the preset interval time corresponding to the task time, and the historical interval time is the previous preset interval time of the current interval time. Calculate the first operating cost of the second GPU server during the current interval based on the execution data; Calculate the second operating cost of the first GPU server during the current interval based on the rental cost of the first GPU server; In response to the first operating cost being greater than the second operating cost, the first GPU server is scheduled to execute the target task request at the current interval.

[0008] Optionally, calculating the first operating cost of the second GPU server during the current interval based on the execution data includes: Based on the execution data, predict the number of first prediction tasks and the duration of the first prediction tasks at the current interval. The first operating cost of the second GPU server during the current interval is calculated based on the number of the first prediction tasks, the duration of the first prediction tasks, and the rental cost of the second GPU server.

[0009] Optionally, the step of predicting the number of first prediction tasks and the duration of the first prediction tasks based on the execution data at the current interval includes: Based on the execution data, the number of first prediction tasks and the duration of the first prediction task are predicted at the current interval using the Kalman filter algorithm.

[0010] Optionally, the first scheduling schedule includes the number of predicted tasks and the duration of predicted tasks in all preset intervals of the day. The step of calculating the first operating cost of the second GPU server in the current interval based on the first number of predicted tasks, the first duration of predicted tasks, and the rental cost of the second GPU server further includes: Based on the first scheduling schedule, obtain the number of second predicted tasks and the duration of the second predicted tasks at the current interval; The target number of prediction tasks is obtained by taking a weighted average of the first number of prediction tasks and the second number of prediction tasks. The target prediction task duration is obtained by taking a weighted average of the durations of the first prediction task and the second prediction task. The first operating cost of the second GPU server during the current interval is calculated based on the number of target prediction tasks, the duration of the target prediction tasks, and the rental cost of the second GPU server.

[0011] Optionally, before obtaining the number of second predicted tasks and the duration of the second predicted tasks based on the first scheduling schedule at the current interval, the following steps are included: Based on historical execution data, the LSTM algorithm is used to predict the number of predicted tasks and the duration of predicted tasks in all preset intervals on a given day. The historical execution data includes all execution data from the previous four weeks.

[0012] Optionally, scheduling the first GPU server to execute the target task request at the current interval includes: The first scheduling timetable is updated to obtain the target scheduling timetable; The first GPU server is scheduled to execute the target task request at the current interval based on the target scheduling schedule.

[0013] Optionally, updating the first scheduling schedule to obtain the target scheduling schedule includes: The preset interval time corresponding to the current interval time in the first scheduling schedule is marked as the scheduling of the first GPU server, and then updated to obtain the target scheduling schedule.

[0014] Optionally, the first GPU server is a Cloud GPU server, and the second GPU server is a Serverless GPU server.

[0015] Secondly, embodiments of this application provide a computer device including a memory and a processor. The memory is connected to the processor, and the processor is configured to execute one or more computer programs stored in the memory. When the processor executes the one or more computer programs, it causes the computer device to implement the method described above.

[0016] Thirdly, embodiments of this application provide a GPU server scheduling system, including: The computer equipment described above; The server cluster, consisting of the Cloud GPU server and Serverless GPU server described above, is configured with a large language model and communicates with the computer device.

[0017] The embodiments of this application can achieve the following technical effects: In the GPU server scheduling method provided in the embodiments of this application, the method includes: obtaining a first scheduling schedule and task time, wherein the first scheduling schedule includes all preset interval times and predicted server types for the current day, and the task time is the request time of the target task request; in response to the server type corresponding to the current interval time not being the first GPU server, obtaining the execution data of all task requests executed in the historical interval time, wherein the current interval time is the preset interval time corresponding to the task time, and the historical interval time is the previous preset interval time of the current interval time; calculating the first operating cost of the second GPU server in the current interval time based on the execution data; calculating the second operating cost of the first GPU server in the current interval time based on the rental cost of the first GPU server; in response to the first operating cost being greater than the second operating cost, scheduling the first GPU server to execute the target task request in the current interval time.

[0018] This application's embodiments dynamically select the most economically optimal scheduling scheme by comparing the estimated operating costs of different server types within a specific time interval in real time. When economic efficiency is superior, it can proactively schedule Cloud GPU servers without cold starts or resource contention, reducing the uncertainty of task latency and waiting time, and ensuring the real-time performance and reliability of task processing. It breaks through the traditional scheduling mode of homogeneous resource pools and can intelligently switch between GPU services with different billing modes and performance characteristics based on the goal of optimizing economic costs, thereby improving the utilization efficiency and overall economic benefits of the cluster in a mixed resource environment. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of the architecture of a GPU server scheduling system provided in an embodiment of this application; Figure 2 A flowchart illustrating a GPU server scheduling method provided in an embodiment of this application; Figure 3 A schematic diagram of a first scheduling timetable provided for an embodiment of this application; Figure 4 A schematic diagram illustrating the number of prediction tasks provided in this application embodiment; Figure 5 A schematic diagram illustrating the prediction task duration provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of a GPU server scheduling device provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0022] It should be noted that, unless there is a conflict, the various features in the embodiments of this application can be combined with each other, all of which are within the protection scope of this application. Furthermore, although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than the module division in the device or the order in the flowchart. Moreover, the terms "first," "second," and "third" used in this application do not limit the data or execution order, but only distinguish identical or similar items with essentially the same function and effect.

[0023] Please see Figure 1 , Figure 1 This is a schematic diagram of the architecture of a GPU server scheduling system provided in an embodiment of this application. The GPU server scheduling system 100 provided in this embodiment includes a server cluster 11 and a computer device 12, with the computer device 12 communicatively connected to the server cluster 11. The server cluster 11 includes multiple Serverless GPU servers 111 and Cloud GPU servers 112, with the computer device 12 communicatively connected to both the Serverless GPU servers 111 and the Cloud GPU servers 112.

[0024] The Serverless GPU Server 111 is a server deployed using a Serverless service model. Serverless services employ an on-demand, per-second billing model with dynamic scalability. Its core mechanism is non-persistent deployment; the service program is loaded only when a task request is received, and resources are automatically unloaded if no new requests are received within a set time (e.g., one minute) after the task is completed. In this mode, the billing period is strictly limited to the actual runtime from receiving the request to processing completion, making it suitable for the initial verification and trial-and-error phases of business operations, effectively avoiding the high costs of dedicated servers. When there are no task requests, the service can achieve zero billing. When the number of concurrent tasks increases, it can automatically allocate multiple GPU resources to meet computing demands.

[0025] However, serverless services have the following limitations: First, due to resource release mechanisms, after a period of no requests, the service needs to reload the algorithm model, resulting in a cold start delay of 40 to 90 seconds. Second, given that its idle waiting window is typically only 1 to 2 minutes, if there are intervals between task requests, each computation must undergo a complete cold start process, severely impacting response efficiency. Furthermore, serverless services also suffer from resource contention issues. When the service provider lacks sufficient available GPU resources, it cannot immediately allocate resources, causing tasks to enter a waiting queue, and the duration of this waiting is completely uncertain.

[0026] Cloud GPU Server 112 is a server deployed using a Cloud-based service model. Cloud-based services also support on-demand and per-second billing models, but their resource scale needs to be pre-configured and cannot be dynamically scaled at runtime. Its core working mechanism is persistent deployment; billing begins once deployed and continues until dedeployment, even if there are no task requests during deployment. With the same chip specifications, the unit-time billing cost of Cloud-based services is significantly lower than that of Serverless services. For example, currently, using the same Nvidia L4 chip from Serverless service provider Modal and Cloud service provider RunPod, Modal's price is $0.8 / hour, while RunPod's price is $0.43 / hour, making the unit cost of Cloud-based services only half that of Serverless services. Cloud-based services also have the following advantages: no cold start process, tasks can respond instantly and execute quickly; dedicated resource protection, completely avoiding resource contention issues in multi-user environments. However, Cloud-based services have an inherent resource idleness defect; once the instance starts, billing continues regardless of whether task requests exist. During periods of low load (such as nighttime or weekends), the allocated GPU resources will be idle, resulting in low actual resource utilization and wasted costs.

[0027] Computer device 12 acts as a scheduling and request distribution center, communicating and interacting with the remotely deployed Serverless GPU server 111 and Cloud GPU server 112. The large language model and its required complete computing environment are both deployed on the two types of GPU servers. Computer device 12 itself does not perform model computations; its core responsibility is to intelligently route task requests to Serverless GPU server 111 or Cloud GPU server 112 according to a preset scheduling method when end users submit tasks (such as text generation, question answering, etc.).

[0028] Please see below. Figure 2 , Figure 2 This is a flowchart illustrating a GPU server scheduling method provided in an embodiment of this application. The GPU server scheduling method provided in this embodiment includes the following steps S21 to S25.

[0029] S21: Obtain the first scheduling schedule and task time. The first scheduling schedule includes all server types with preset intervals and predicted scheduling for the day. The task time is the request time of the target task request.

[0030] In step S21, the preset interval time can be set according to the response accuracy requirements. In this embodiment, the preset interval time is set to 15 minutes. The first scheduling schedule includes 24×4 preset interval times for the predicted server types to be scheduled. The first scheduling schedule is an array that divides the entire day into 24x4 consecutive units. Each unit can be characterized by only two states to represent the type of server to be scheduled. In some embodiments, it can also be constructed in other ways, such as a state lookup table, a dictionary, etc.

[0031] Please see Figure 3 In the diagram, the blue areas represent multiple preset intervals for scheduling Cloud GPU servers, and the white areas represent multiple preset intervals for scheduling Serverless GPU servers. It's important to note that, by default or initially, the computer sends task requests to the Serverless GPU service cluster to leverage its pay-as-you-go pricing. The computer monitors execution data from interactions with the Serverless service in real time and estimates operating costs. Once scheduling conditions are met, the computer starts or switches to a Cloud GPU server and redirects subsequent task requests to it. The large language model loaded by the Cloud GPU server then performs continuous and efficient computation.

[0032] The computer equipment includes a task allocation module, which distributes task requests to corresponding GPU servers according to a first scheduling schedule. If the first scheduling schedule already meets the economic cost optimization requirements and no adjustment is needed, it will continue to be used on the second day. If the first scheduling schedule does not meet the economic cost optimization requirements within a preset interval, it will be updated until the last preset interval of the day. The task allocation module will then use the updated first scheduling schedule to allocate task requests to corresponding GPU servers for the second day.

[0033] S22: In response to the fact that the server type corresponding to the current interval time is not the first GPU server, obtain the execution data of all task requests executed in the historical interval time. The current interval time is the preset interval time corresponding to the task time, and the historical interval time is the previous preset interval time of the current interval time.

[0034] In step S22, the first GPU server is a Cloud GPU server. The computer device detects the server type scheduled for the current interval in the first scheduling schedule. If the server type corresponding to the current interval is not a Cloud GPU server, the computer device obtains the execution data of all task requests executed in historical intervals.

[0035] S23: Calculate the first running cost of the second GPU server in the current interval based on the execution data.

[0036] In step S23, the second GPU server is a Serverless GPU server. The first operating cost is the estimated operating cost of scheduling the Serverless GPU server to execute the task requests during the current interval. The computer device predicts the first predicted task number and the first predicted task duration in the current interval based on execution data in historical intervals. The first predicted task duration is the average duration of task request execution.

[0037] Specifically, the computer equipment is equipped with a next-moment scheduling prediction module. This module uses the Kalman filter algorithm based on the execution data of the previous moment (15 minutes) to predict the number and duration of the first predicted task in the next moment. This scheduling prediction is calculated in real time, meaning the prediction data for the next moment is calculated one minute before the end of each moment. In some embodiments, the least squares method or other commonly used real-time prediction algorithms can also be used for prediction. It should be noted that network task requests have significant randomness and uncertainty, often exhibiting dynamic changes such as sudden spikes or sharp drops. For such scenarios, a prediction algorithm capable of real-time response and optimized estimation is required. Kalman filtering, due to its ability to recursively update and estimate execution data in real time, can effectively track the instantaneous fluctuations in task requests, thus making it suitable for the rapid response requirements of short-term burst requests.

[0038] In some embodiments, the computer device calculates the estimated operating cost of the Serverless GPU server in the current interval using the number of first prediction tasks, the duration of the first prediction tasks, and the rental cost of the Serverless GPU server. Specifically, if the rental cost of the Serverless GPU server per hour is Ps, then the estimated operating cost of the Serverless GPU server in the current interval is equal to the product of the number of first prediction tasks, the duration of the first prediction tasks, and Ps / 3600.

[0039] It should be noted that if the estimated running cost of the Serverless GPU server in the current interval is greater than the running cost of the Cloud GPU server in the current interval for executing the task requests during this period, then the Cloud GPU server will be scheduled to execute the task requests in the current interval.

[0040] In some embodiments, the computer device is configured with a 24×4-hour scheduling prediction module for the next day. This module predicts the number and duration of tasks for each of the 24×4 time slots within the current day at the last moment of the previous day; that is, the number and duration of tasks for each of the 24×4 preset intervals within the current day. Specifically, the 24×4-hour scheduling prediction module performs periodic learning and calculation based on historical execution data from the past four weeks, aiming to predict the number and duration of tasks for the 96 scheduling time slots (24 hours × 4 quarters) of the next day. This prediction operation is performed once daily, starting at 23:50 each day, and generates the prediction data for the next day.

[0041] It should be noted that the task requests exhibit periodic characteristics, specifically intraday fluctuations with a 24-hour cycle and weekly fluctuations with a 7-day cycle (such as the significant difference between weekdays and weekends). For such time series data with strong periodicity, prediction algorithms with periodic characteristics can be used. This application's embodiments employ a Long Short-Term Memory (LSTM) network for prediction because it can effectively capture and model long-term dependencies and multiple periodic patterns in time series. In other embodiments, SARIMAX, SVR, LTM-SVR, or non-periodic prediction algorithms such as ARIMA and SARIMA can also be used.

[0042] The computer device will perform a weighted average of the first and second prediction task counts to obtain the target prediction task count, where the second prediction task count is the number of prediction tasks corresponding to the current time interval. The computer device will then perform a weighted average of the first and second prediction task durations to obtain the target prediction task duration, where the second prediction task duration is the prediction task duration corresponding to the current time interval.

[0043] The computer equipment calculates the estimated operating cost of the Serverless GPU server in the current interval based on the target prediction task number, target prediction task duration, and Serverless GPU server rental cost. Specifically, if the hourly rental cost of the Serverless GPU server is Ps, then the estimated operating cost of the Serverless GPU server in the current interval is equal to the product of the target prediction task number, target prediction task duration, and Ps / 3600.

[0044] S24: Calculate the second operating cost of the first GPU server during the current interval based on the rental cost of the first GPU server.

[0045] In step S24, the second operating cost is the operating cost of scheduling the Cloud GPU server to execute the task requests during the current interval. Specifically, the first GPU server is the Cloud GPU server. The hourly rental cost of the Cloud GPU server can be directly obtained. If the hourly rental cost of the Cloud GPU server is Pc, then the estimated operating cost of the Cloud GPU server during the current interval is Pc / 4.

[0046] S25: In response to the first operating cost being greater than the second operating cost, schedule the first GPU server to execute the target task request at the current interval.

[0047] In step S25, the first operating cost is the estimated operating cost of scheduling the Serverless GPU server to execute the task requests during the current interval. The second operating cost is the operating cost of scheduling the Cloud GPU server to execute the task requests during the current interval. The first GPU server is the Cloud GPU server.

[0048] Specifically, if the computer device detects that the estimated execution cost of scheduling the Serverless GPU server to execute the task request during the current interval is greater than the execution cost of scheduling the Cloud GPU server to execute the task request during the current interval, then the computer device will schedule the Cloud GPU server to execute the target task request during the current interval. Specifically, before scheduling the Cloud GPU server to execute the target task request during the current interval, the computer device will update the first scheduling schedule, marking the preset interval corresponding to the current interval in the first scheduling schedule as the time to schedule the Cloud GPU server, and then updating it to obtain the target scheduling schedule. Please continue reading. Figure 3 The red part in the diagram corresponds to the preset interval time, which is marked as the preset interval time for scheduling Cloud GPU servers. It can be understood that the red part in the first scheduling schedule should be the white part for scheduling Serverless GPU servers.

[0049] This application's embodiments dynamically select the most economically optimal scheduling scheme by comparing the estimated operating costs of different server types within a specific time interval in real time. When economic efficiency is superior, it can proactively schedule Cloud GPU servers without cold starts or resource contention, reducing the uncertainty of task latency and waiting time, and ensuring the real-time performance and reliability of task processing. It breaks through the traditional scheduling mode of homogeneous resource pools and can intelligently switch between GPU services with different billing modes and performance characteristics based on the goal of optimizing economic costs, thereby improving the utilization efficiency and overall economic benefits of the cluster in a mixed resource environment.

[0050] In some embodiments, when scheduling the first GPU server to execute the target task request at the current interval in response to a first operating cost being greater than a second operating cost, the economically optimal objective formula needs to be satisfied. The economically optimal objective formula is as follows:

[0051] Where N represents the number of cold start calculations per hour for the Serverless GPU server, Tsl represents the single cold start calculation time for the Serverless GPU server, M represents the number of fast calculations without cold start for the Serverless GPU server, Tsq represents the single fast calculation time for the Serverless GPU server, Ps represents the hourly rental cost for the Serverless GPU server, and Pc represents the hourly rental cost for the Cloud GPU server.

[0052] The computer equipment is equipped with an execution data logging module, which records execution data for the Serverless GPU server and Cloud GPU server for at least four weeks. This data includes: the timestamp of each task's receipt, Serverless startup timestamp, startup completion timestamp, computation start timestamp, and computation completion timestamp. From this timestamp information, the cold start time, computation time, intervals between cold starts, and intervals between fast computations can be calculated. The number of tasks processed per quarter hour and the average task duration can also be calculated. Tsl and Tsq can be calculated by averaging the execution data over a 24-hour period or the previous quarter hour. Ps and Pc can be obtained directly. It is difficult to accurately calculate the range of N and M using a single objective equation; only an optimal window range determined by the upper and lower limits of the number of tasks and the task duration can be established.

[0053] It should be noted that, to simplify the problem, the inventors determined the boundaries of N and M using both worst-case and general case scenarios. In the worst-case scenario, with only cold starts, the lower limit for the number of tasks (i.e., the minimum number of tasks per hour) is calculated as: N>= Pc x 3600 / (Tsl x Ps), and the upper limit for task duration (i.e., the average longest task computation time per hour) is Tsl. In the general case, the number of cold starts follows a Poisson distribution, i.e., 20% cold starts and 80% fast computation. The upper limit for the number of tasks (i.e., the maximum number of tasks per hour) is calculated as: N+M>= Pc x 3600 x 5 / [(Tsl + 4xTsq) x Ps], and the lower limit for task duration (i.e., the average shortest task computation time per hour) is calculated as: 3600 / (Tsl + 4xTsq). The values ​​of N and M fall between these two boundaries.

[0054] In some embodiments, calculating the first operating cost of the second GPU server during the current interval based on execution data includes the following steps: S231: Based on the execution data, predict the number of first prediction tasks and the duration of the first prediction task at the current interval.

[0055] S232: Calculate the first operating cost of the second GPU server in the current interval based on the number of first prediction tasks, the duration of the first prediction tasks, and the rental cost of the second GPU server.

[0056] In step S231, the first predicted task quantity is the number of predicted tasks in the current interval time predicted based on the execution data using the Kalman filter algorithm. The first predicted task duration is the predicted task duration in the current interval time predicted based on the execution data using the Kalman filter algorithm. The computer device predicts the first predicted task quantity and the first predicted task duration in the current interval time based on the execution data in historical interval times. The first predicted task duration is the average execution time of the task requests. Specifically, the computer device is equipped with a next-moment scheduling prediction module, which predicts the first predicted task quantity and the first predicted task duration in the next moment based on the execution data of the previous moment (15 minutes) using the Kalman filter algorithm. This scheduling prediction is calculated in real time, that is, the prediction data for the next moment is calculated one minute before the end of each moment.

[0057] In step S232, the first operating cost is the estimated operating cost of scheduling the Serverless GPU server to execute the task requests during the current interval. The computer device calculates the estimated operating cost of the Serverless GPU server during the current interval using the first predicted task number, the first predicted task duration, and the rental cost of the Serverless GPU server. Specifically, if the hourly rental cost of the Serverless GPU server is Ps, then the estimated operating cost of the Serverless GPU server during the current interval is equal to the product of the first predicted task number, the first predicted task duration, and Ps / 3600.

[0058] In some embodiments, the first scheduling schedule includes the number of predicted tasks and the duration of predicted tasks in all preset intervals of the day. The calculation of the first operating cost of the second GPU server in the current interval, based on the first number of predicted tasks, the first duration of predicted tasks, and the rental cost of the second GPU server, further includes the following steps: S2321: Obtain the number of second predicted tasks and the duration of the second predicted tasks at the current interval based on the first scheduling schedule.

[0059] S2322: The target number of prediction tasks is obtained by taking a weighted average of the number of the first prediction task and the number of the second prediction task.

[0060] S2323: The target prediction task duration is obtained by weighted averaging the durations of the first and second prediction tasks.

[0061] S2324: Calculate the first operating cost of the second GPU server in the current interval based on the number of target prediction tasks, the duration of target prediction tasks, and the rental cost of the second GPU server.

[0062] In step S2321, the first scheduling schedule includes the number of predicted tasks and the duration of each predicted task for each preset interval within all preset intervals of the day. Specifically, the computer equipment is equipped with a 24×4-hour scheduling prediction module for the next day. This module predicts the number of predicted tasks and the duration of each of the 24×4 time intervals of the current day at the last moment of the previous day, i.e., the number of predicted tasks and the duration of each of the 24×4 preset intervals of the current day. Specifically, the 24×4-hour scheduling prediction module for the next day uses a Long Short-Term Memory (LSTM) network for periodic learning and computation based on historical execution data from the past four weeks, aiming to predict the number of predicted tasks and the duration of each of the 96 scheduling time intervals (24 hours × 4 minutes) of the next day. This prediction operation is performed once daily, starting at 23:50 each day, and generates the prediction data for the next day, which is then stored in association with the first scheduling schedule.

[0063] In step S2322, the second number of prediction tasks is the number of prediction tasks corresponding to the current interval. The computer device will perform a weighted average of the first number of prediction tasks and the second number of prediction tasks to obtain the target number of prediction tasks.

[0064] In step S2323, the second prediction task duration is the prediction task duration corresponding to the current interval. The computer device will perform a weighted average of the first and second prediction task durations to obtain the target prediction task duration.

[0065] In step S2324, the computer device calculates the estimated operating cost of the Serverless GPU server in the current interval based on the target prediction task quantity, the target prediction task duration, and the serverless GPU server rental cost. Specifically, if the hourly rental cost of the Serverless GPU server is Ps, then the estimated operating cost of the Serverless GPU server in the current interval is equal to the product of the target prediction task quantity, the target prediction task duration, and Ps / 3600.

[0066] It should be noted that if the estimated running cost of the Serverless GPU server in the current interval is greater than the running cost of the Cloud GPU server executing the task requests during the current interval, then the Cloud GPU server will be scheduled to execute the task requests for the current interval. To more easily determine whether the Cloud GPU server needs to be scheduled, this application embodiment designs an optimal scheduling threshold formula, which is shown below: ((L_count + K_count) / 2) x ((L_time + K_time) / 2) x Ps>(3600 / 4) xPc.

[0067] Where L_count represents the number of predicted tasks at the current interval output by the 24x4 scheduling prediction module based on the LSTM algorithm on the next day, i.e., the number of second tasks; K_count represents the number of predicted tasks at the current interval output by the next-second scheduling prediction module based on the Kalman filter algorithm; L_time represents the duration of the predicted tasks at the current interval output by the 24x4 scheduling prediction module based on the LSTM algorithm on the next day; K_time represents the duration of the predicted tasks at the current interval output by the next-second scheduling prediction module based on the Kalman filter algorithm; Ps represents the hourly rental cost of the Serverless GPU server; and Pc represents the hourly rental cost of the Cloud GPU server. Based on the optimal scheduling threshold formula, the computer equipment can directly determine whether the Cloud GPU server needs to be scheduled at the current interval based on the prediction data output by the 24x4 scheduling prediction module on the next day and the next-second scheduling prediction module.

[0068] This application's embodiments achieve automated intelligent scheduling and cost optimization across service modes, enabling autonomous and precise switching and scheduling between Serverless and Cloud GPU services based on real-time load and economic objectives. It fully leverages the cost advantages of both service modes, reducing the overall deployment and operational costs of large-scale AI model computing. Furthermore, by innovatively combining Long Short-Term Memory (LSTM) networks and Kalman filtering algorithms, it can effectively learn and predict medium- to long-term task loads with periodic patterns, while also providing rapid responses and state estimations for random, short-term bursts of requests. This ensures the lowest possible economic cost for resource scheduling while also guaranteeing the real-time performance and stability of task processing, achieving an optimal balance between cost control and user experience.

[0069] In some embodiments, scheduling a first GPU server to execute a target task request at the current interval includes the following steps: S251: Update the first scheduling schedule to obtain the target scheduling schedule; S252: Schedule the first GPU server to execute the target task request at the current interval based on the target scheduling schedule.

[0070] In step S251, the computer device detects whether the operating costs of the Cloud GPU server and the Serverless GPU server meet the optimal scheduling threshold formula based on the prediction data output by the next day's 24x4 scheduling prediction module and the next-period scheduling prediction module.

[0071] Please see Figure 4 The next day's 24x4 scheduling prediction module outputs "LSTM predicts the number of tasks at each of the next day's 24x4 time slots," while the next-second scheduling prediction module outputs "Kalman filter predicts the number of tasks at this moment." Please refer to [link / reference]. Figure 5 The next day's 24x4 scheduling prediction module outputs "LSTM predicts the task duration at 24x4 the next day", and the next moment's scheduling prediction module outputs "Kalman filter predicts the task duration at this moment".

[0072] If the optimal scheduling threshold formula is satisfied, the preset interval time corresponding to the current interval time in the first scheduling timetable is marked as the time to schedule the first GPU server, and the timetable is updated to obtain the target scheduling timetable. Specifically, if the optimal scheduling threshold formula ((L_count + K_count) / 2) x ((L_time + K_time) / 2) x Ps>(3600 / 4) xPc is satisfied, the corresponding time in the first scheduling timetable is set and marked as a change status bit. When there is a change status bit in the first scheduling timetable, the time of the change status bit is updated to obtain the target scheduling timetable, and then sent to the task allocation module and the service scheduling configuration module.

[0073] It should be noted that the computer equipment is equipped with a service scheduling configuration module. This module is used to configure the Cloud GPU servers based on the target scheduling schedule at preset intervals when they need to be scheduled. Specifically, the service scheduling configuration module configures the start and stop times of the Cloud GPU servers through automated scripts. The service scheduling configuration module needs to compare the target scheduling schedule with the first scheduling schedule, changing only the necessary time portions and ensuring time continuity. For example, if the scheduling plan in the first scheduling schedule does not allow Cloud GPU servers to be scheduled from 13:15 to 13:30, but requires scheduling after the preset scheduling method, then the two time periods of 13:30 to 13:45 will be automatically merged into 13:15 to 13:45 to facilitate the configuration of Cloud GPU servers through command scripts. Conversely, if the original plan was to schedule Cloud GPU servers from 13:00 to 13:45, but after the preset scheduling method, Cloud GPU servers do not need to be scheduled from 13:15 to 13:30, then it is necessary to reconfigure that Cloud GPU servers are not scheduled from 13:15 to 13:30, and are scheduled from 13:30 to 13:45.

[0074] In step S252, the computer device is configured with a service scheduling configuration module. The service scheduling configuration module obtains the target scheduling schedule after the first scheduling schedule is updated. The service scheduling configuration module configures the start and stop times of the Cloud GPU server through an automatic script according to the target scheduling schedule, so that the Cloud GPU server executes the target task request at the current interval.

[0075] It should be noted that in the above embodiments, there is no necessarily a certain order between the steps. Those skilled in the art can understand from the description of the embodiments of this application that the above steps may have different execution orders in different embodiments, that is, they may be executed in parallel or in turn, etc.

[0076] As another aspect of the embodiments of this application, this application provides a GPU server scheduling device. The GPU server scheduling device can be a software module, which includes several instructions stored in a memory. A processor can access the memory, invoke the instructions for execution, and complete the GPU server scheduling method described in the various embodiments above.

[0077] In some implementations, the GPU server scheduling device can also be constructed from hardware components. For example, the GPU server scheduling device can be constructed from one or more chips, which can coordinate with each other to complete the GPU server scheduling method described in the various implementations above. As another example, the GPU server scheduling device can also be constructed from various logic devices, such as general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), microcontrollers, ARM (Acorn RISC Machine) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination of these components.

[0078] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of a GPU server scheduling device provided in an embodiment of this application. The GPU server scheduling device 600 provided in this embodiment includes a task acquisition module 601, a data acquisition module 602, a first cost calculation module 603, a second cost calculation module 604, and a server scheduling module 605.

[0079] The task acquisition module 601 is used to acquire a first scheduling schedule and task time. The first scheduling schedule includes all server types scheduled for the current day at preset intervals, and the task time is the request time of the target task request. The data acquisition module 602 is used to acquire execution data of all task requests executed in historical intervals in response to the current interval not being the first GPU server. The current interval is the preset interval corresponding to the task time, and the historical interval is the previous preset interval. The first cost calculation module 603 is used to calculate a first operating cost of the second GPU server in the current interval based on the execution data. The second cost calculation module 604 is used to calculate a second operating cost of the first GPU server in the current interval based on the rental cost of the first GPU server. The server scheduling module 605 is used to schedule the first GPU server to execute the target task request in the current interval in response to the first operating cost being greater than the second operating cost.

[0080] The first cost calculation module 603 is further specifically used for: predicting the number of first prediction tasks and the duration of the first prediction tasks based on the execution data for the current interval; and calculating the first operating cost of the second GPU server in the current interval based on the number of first prediction tasks, the duration of the first prediction tasks, and the rental cost of the second GPU server.

[0081] The first cost calculation module 603 is also specifically used to: predict the number of first prediction tasks and the duration of the first prediction tasks at the current interval based on the execution data and in combination with the Kalman filter algorithm.

[0082] The first scheduling schedule includes the number of predicted tasks and the duration of predicted tasks in all preset intervals of the day. The first cost calculation module 603 is further specifically used for: obtaining the number of second predicted tasks and the duration of second predicted tasks in the current interval based on the first scheduling schedule; taking a weighted average of the number of first predicted tasks and the number of second predicted tasks to obtain the target number of predicted tasks; taking a weighted average of the duration of first predicted tasks and the duration of second predicted tasks to obtain the target duration of predicted tasks; and calculating the first operating cost of the second GPU server in the current interval based on the target number of predicted tasks, the target duration of predicted tasks, and the rental cost of the second GPU server.

[0083] The first cost calculation module 603 is also specifically used to: predict the number of predicted tasks and the duration of predicted tasks in all preset intervals of the day based on historical execution data and combined with the LSTM algorithm, wherein the historical execution data includes all execution data of the previous four weeks.

[0084] The server scheduling module 605 is further configured to: update the first scheduling timetable to obtain a target scheduling timetable; and schedule the first GPU server to execute the target task request at the current interval based on the target scheduling timetable.

[0085] The server scheduling module 605 is also specifically used to: mark the preset interval time corresponding to the current interval time in the first scheduling schedule as the scheduling of the first GPU server, and update it to obtain the target scheduling schedule.

[0086] It should be noted that the aforementioned GPU server scheduling device can execute the GPU server scheduling method provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects of the method. Technical details not described in detail in the embodiments of the GPU server scheduling device can be found in the GPU server scheduling method provided in the embodiments of this application.

[0087] See Figure 7 , Figure 7 This is a schematic diagram of the structure of a computer device according to an embodiment of this application. The computer device includes one or more processors 71 and a memory 72. The memory 72 is connected to one or more processors 71, for example, via a bus.

[0088] Processor 71 is configured to support the computer device in performing the corresponding functions in the methods described in the above method embodiments. The processor may be a central processing unit (CPU), a network processor (NP), a hardware chip, or any combination thereof. The aforementioned hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The aforementioned PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0089] Memory 72 is used to store program code, etc. Memory 72 may include volatile memory (VM), such as random access memory (RAM); memory may also include non-volatile memory (NVM), such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); memory may also include combinations of the above types of memory.

[0090] The memory 72 can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the GPU server scheduling method in the embodiments of this application. The processor 71 executes various functional applications and data processing of the GPU server scheduling method and GPU server scheduling device by running the non-volatile software programs, instructions, and modules stored in the memory, that is, it implements the functions of each module or unit of the GPU server scheduling method and GPU server scheduling device provided in the above method embodiments.

[0091] The memory 72 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function. The data storage area may store data created based on the use of the GPU server scheduling device, etc. In some embodiments, the memory 72 may optionally include memory remotely located relative to the processor, which can be connected to the GPU server scheduling device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0092] The one or more modules are stored in the memory 72. When executed by the one or more processors 71, they execute the GPU server scheduling method in any of the above method embodiments. For example, they execute the method steps described in the above method embodiments to realize the functions of the modules described in the above device embodiments.

[0093] This application also provides a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a computer, cause the computer to perform the method described in the foregoing embodiments.

[0094] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0095] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A GPU server scheduling method, characterized in that, include: Obtain the first scheduling schedule and task time. The first scheduling schedule includes all server types with preset intervals for the day and are scheduled in advance. The task time is the request time of the target task request. In response to the fact that the server type corresponding to the current interval time is not the first GPU server, the execution data of all task requests executed in the historical interval time are obtained, wherein the current interval time is the preset interval time corresponding to the task time, and the historical interval time is the previous preset interval time of the current interval time. Calculate the first operating cost of the second GPU server during the current interval based on the execution data; Calculate the second operating cost of the first GPU server during the current interval based on the rental cost of the first GPU server; In response to the first operating cost being greater than the second operating cost, the first GPU server is scheduled to execute the target task request at the current interval.

2. The method according to claim 1, characterized in that, The calculation of the first operating cost of the second GPU server during the current interval based on the execution data includes: Based on the execution data, predict the number of first prediction tasks and the duration of the first prediction tasks at the current interval. The first operating cost of the second GPU server during the current interval is calculated based on the number of the first prediction tasks, the duration of the first prediction tasks, and the rental cost of the second GPU server.

3. The method according to claim 2, characterized in that, The first prediction task number and first prediction task duration based on the execution data to predict the current interval time include: Based on the execution data, the number of first prediction tasks and the duration of the first prediction task are predicted at the current interval using the Kalman filter algorithm.

4. The method according to claim 2, characterized in that, The first scheduling schedule includes the number of predicted tasks and the duration of predicted tasks in all preset intervals of the day. The calculation of the first operating cost of the second GPU server in the current interval, based on the first number of predicted tasks, the first duration of predicted tasks, and the rental cost of the second GPU server, further includes: Based on the first scheduling schedule, obtain the number of second predicted tasks and the duration of the second predicted tasks at the current interval; The target number of prediction tasks is obtained by taking a weighted average of the first number of prediction tasks and the second number of prediction tasks. The target prediction task duration is obtained by taking a weighted average of the durations of the first prediction task and the second prediction task. The first operating cost of the second GPU server during the current interval is calculated based on the number of target prediction tasks, the duration of the target prediction tasks, and the rental cost of the second GPU server.

5. The method according to claim 4, characterized in that, Before obtaining the number of second predicted tasks and the duration of the second predicted tasks at the current interval based on the first scheduling schedule, the process includes: Based on historical execution data, the LSTM algorithm is used to predict the number of predicted tasks and the duration of predicted tasks in all preset intervals on a given day. The historical execution data includes all execution data from the previous four weeks.

6. The method according to claim 1, characterized in that, The step of scheduling the first GPU server to execute the target task request at the current interval includes: The first scheduling timetable is updated to obtain the target scheduling timetable; The first GPU server is scheduled to execute the target task request at the current interval based on the target scheduling schedule.

7. The method according to claim 6, characterized in that, The step of updating the first scheduling schedule to obtain the target scheduling schedule includes: The preset interval time corresponding to the current interval time in the first scheduling schedule is marked as the scheduling of the first GPU server, and then updated to obtain the target scheduling schedule.

8. The method according to any one of claims 1-7, characterized in that, The first GPU server is a Cloud GPU server, and the second GPU server is a Serverless GPU server.

9. A computer device, characterized in that, The device includes a memory and a processor, the memory being connected to the processor, the processor being configured to execute one or more computer programs stored in the memory, the processor causing the computer device to perform the method as described in any one of claims 1-8 when executing the one or more computer programs.

10. A GPU server scheduling system, characterized in that, include: The computer device as described in claim 9; The server cluster consisting of the Cloud GPU server and the Serverless GPU server as described in claim 8 is configured with a large language model and is communicatively connected to the computer device.

Images