A GPU dynamic scheduling method and device based on real-time load and electronic equipment

Abstract

The application relates to the technical field of graphic processors, and discloses a GPU dynamic scheduling method based on real-time load, a GPU dynamic scheduling device and electronic equipment.The GPU dynamic scheduling method comprises the following steps: acquiring service alignment index data, calculating derivative index data reflecting service stability and reliability, and guiding the allocation and scheduling of resources and services based on the alignment index data and / or the derivative index data.The GPU dynamic scheduling method based on real-time load can be used for cloud GPU use scenarios, can plan scheduling through real-time GPU load information, can shield the differences between underlying GPU resources, can fully exploit the use efficiency of GPU resources, and can reduce the complexity of scheduling.

Description

Technical Field

[0001] This invention relates to the field of graphics processor technology, and more specifically to a GPU dynamic scheduling method, apparatus, and electronic device based on real-time load. Background Technology

[0002] In real-world scenarios, many GPU-based services cannot fully utilize a single GPU, resulting in a waste of GPU resources. There are many types of GPUs, each with different load capacities and memory specifications. The same service deployment architecture may not run well on a different GPU model.

[0003] Regarding the scheduling of GPU services, the following technical methods currently exist:

[0004] 1. Manual and hybrid deployment scenarios.

[0005] 2. In cloud scheduling scenarios, the GPU memory usage of a service is defined in the Kubernetes service orchestration. The scheduler checks whether the GPU memory is available and performs stacking or tiling processing.

[0006] Current problems with the technology:

[0007] 1. In the case of manual and hybrid deployment scenarios, not only is a lot of manpower and resources required, but it also often relies on the accumulation of experience, which makes it difficult to handle large-scale GPU service scenarios.

[0008] 2. Cloud scheduling scenarios do not take into account the actual load capacity of GPU services. Stacking processing cannot handle the service demands of small video memory and high load. Stacking high-load services will lead to a decrease in service quality. Using tiling processing will be subject to the order of service scheduling and will cause waste of graphics card resources.

[0009] In actual use cases, due to the diversity of service frameworks, code implementations, and card models, there is a significant waste of GPU computing power. The existing cloud environment lacks a mature real-time scheduling solution based on the actual usage of GPU resources.

[0010] In view of this, the present invention is hereby proposed. Summary of the Invention

[0011] The purpose of this invention is to provide a real-time scheduling scheme based on the actual resource usage of GPUs in a cloud environment.

[0012] To achieve the above-mentioned objectives, the present invention provides the following technical solution:

[0013] A GPU dynamic scheduling method based on real-time load includes:

[0014] Obtain service alignment index data and calculate derived index data reflecting service stability and reliability;

[0015] Based on the alignment index data and / or derived index data, the allocation and scheduling of resources and services are guided.

[0016] As an optional embodiment of the present invention, the acquisition of service alignment index data includes:

[0017] Obtain GPU service resource monitoring data, which includes at least the running data of the GPUs corresponding to the existing services;

[0018] Optionally, the operational data includes GPU load and GPU memory usage.

[0019] As an optional embodiment of the present invention, the acquisition of GPU service resource monitoring data includes:

[0020] The GPU driver is invoked to obtain the process ID and the corresponding GPU load of the process;

[0021] The deployment unit of the cloud control platform is obtained by reading the control group of the corresponding operating system through the process ID;

[0022] Based on the cloud control platform, information about the deployment unit is obtained, and the correspondence between the service to which the deployment unit belongs and the GPU load is further obtained.

[0023] Organize and store the runtime data in a preset format according to the corresponding relationship.

[0024] As an optional embodiment of the present invention, the derived index data for calculating the stability and reliability of the response service include:

[0025] By subscribing to GPU service logs through a unified log analysis service, parsing the logs, and calculating and obtaining traffic monitoring data according to the interface dimension, the obtained traffic monitoring data includes the service's QPS and latency.

[0026] As an optional implementation of the present invention, the GPU dynamic scheduling method based on real-time load includes: organizing and storing the service alignment index data and derived index data that conform to a preset format, and waiting to be called;

[0027] Optionally, a time-series database can be used to organize and store GPU service resource monitoring data and traffic monitoring data that conform to the time-series database format.

[0028] As an optional implementation of the present invention, the allocation and scheduling of resources and services are guided based on the alignment index data and / or derived index data:

[0029] Obtain service alignment index data and derived index data of services using GPUs within a certain time window;

[0030] Calculate the balance point of resource utilization and performance for a service under a certain GPU card type based on service alignment index data and derived index data;

[0031] The calculated resource utilization and / or performance metric balance points are aggregated and used for the allocation and scheduling of GPU resources for existing services.

[0032] As an optional implementation of the present invention, the aggregation and calculation of the comprehensive service alignment index data and derived index data includes:

[0033] Retrieve GPU service resource monitoring data and traffic monitoring data within a certain time window from a time-series database;

[0034] By summarizing the GPU service resource monitoring data and traffic monitoring data, including GPU utilization, GPU memory utilization, CPU utilization, memory utilization, service QPS, and latency metrics, the balance point between resource utilization and performance metrics during service operation is calculated.

[0035] As an optional implementation of the present invention, guiding the allocation and scheduling of resources and services based on the alignment index data and / or derived index data includes:

[0036] When a new service replica is created, the service memory usage and the relative amount of optimal load usage are obtained based on the service alignment index data. GPUs with more remaining resources in the cluster than the resources required to run the new service replica are then selected and allocated.

[0037] Optionally, from the set of available GPUs, select the GPUs with the remaining load closest to the amount of resources required to run the new service replica.

[0038] This embodiment also provides a GPU dynamic scheduling device based on real-time load, including:

[0039] The service monitoring module acquires service alignment index data and calculates derivative index data reflecting service stability and reliability.

[0040] The service scheduling module guides the allocation and scheduling of resources and services based on the alignment index data and / or derived index data.

[0041] The present invention also provides an electronic device, including a processor and a memory, wherein the memory is used to store a computer-executable program.

[0042] When the computer program is executed by the processor, the processor executes the GPU dynamic scheduling method based on real-time load.

[0043] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0044] The GPU dynamic scheduling method based on real-time load of the present invention, for cloud GPU usage scenarios, performs scheduling planning through real-time GPU load information, shields the differences in underlying GPU resources, fully explores the utilization efficiency of GPU resources, and reduces the complexity of scheduling.

[0045] The GPU dynamic scheduling method based on real-time load of the present invention, which guides the allocation and scheduling of resources and services based on the alignment index data and / or derived index data, mainly includes:

[0046] For existing services, historical monitoring data of services using GPUs within a certain time window is periodically retrieved. The resource utilization and performance balance point of the service under a specific GPU type is calculated and used for GPU resource allocation and scheduling of existing services. This invention implements GPU scheduling for existing services based on a real-time load dynamic GPU scheduling method, enabling the scheduling of the optimal GPU resources as the number of services increases, thereby achieving the best service computing performance.

[0047] For new service instances, GPU allocation is performed based on the alignment metric data and / or derived metric data. This invention's real-time load-based dynamic GPU scheduling method selects and schedules GPUs with remaining loads closest to the resource requirements of the new service instance during its creation. This ensures the computational performance of the new service while improving GPU resource utilization and avoiding performance overkill.

[0048] Compared to the two existing technologies described in the background section, this invention requires no manual processing and can adapt to large-scale GPU hybrid deployments; compared to brute-force stacking or tiling, it improves GPU service computing power and graphics card utilization.

[0049] This invention provides a GPU dynamic scheduling method based on real-time load analysis of GPU service resource requirements. This method guides subsequent scheduling behavior of the service, thereby improving GPU service computing power and graphics card utilization. Attached Figure Description

[0050] Figure 1 The flowchart of the GPU dynamic scheduling method based on real-time load of this invention;

[0051] Figure 2 The GPU scheduling strategy flow of the GPU dynamic scheduling method based on real-time load in this invention Figure 1 ;

[0052] Figure 3 The GPU scheduling strategy flow of the GPU dynamic scheduling method based on real-time load in this invention Figure 2 ;

[0053] Figure 4 A schematic diagram of the GPU dynamic scheduling device based on real-time load of the present invention. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention.

[0055] Therefore, the following detailed description of embodiments of the present invention is not intended to limit the scope of the claimed invention, but merely illustrates some embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0056] It should be noted that, unless otherwise specified, the embodiments and features and technical solutions in the present invention can be combined with each other.

[0057] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0058] In the description of this invention, it should be noted that the terms "upper," "lower," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product of this invention is in use, or the orientation or positional relationship commonly understood by those skilled in the art. These terms are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention. In addition, the terms "first," "second," etc., are only used to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0059] See Figure 1 As shown in the figure, this embodiment provides a GPU dynamic scheduling method based on real-time load, including:

[0060] Obtain service alignment index data and calculate derived index data reflecting service stability and reliability;

[0061] Based on the alignment index data and / or derived index data, the allocation and scheduling of resources and services are guided.

[0062] The GPU dynamic scheduling method based on real-time load in this embodiment collects service alignment index data, such as GPU load, GPU storage usage and other resource usage, as well as traffic information such as service latency, service QPS, success and failure rates, and stores it for use in computation.

[0063] The GPU dynamic scheduling method based on real-time load in this embodiment performs comprehensive calculations by calling the above-mentioned service alignment index data to obtain derived index data reflecting service stability and reliability, such as latency indices from P99 to P50.

[0064] This embodiment generates GPU scheduling strategies to improve GPU service computing power and GPU utilization based on service alignment index data and derived index data for different types of service call data.

[0065] The GPU dynamic scheduling method based on real-time load in this embodiment, which guides the allocation and scheduling of resources and services based on the alignment index data and / or derived index data, mainly includes:

[0066] For existing services, historical monitoring data of services using GPUs within a certain time window is periodically obtained, and the balance point of resource utilization and performance of the service under a certain card type is calculated for the allocation and scheduling of GPU resources for existing services.

[0067] And / or, for new service copies, graphics card allocation is performed based on the alignment metric data and / or derived metric data.

[0068] The GPU dynamic scheduling method based on real-time load in this embodiment is suitable for cloud GPU usage scenarios. It uses real-time GPU load information for scheduling planning, shields the differences in underlying GPU resources, fully explores the utilization efficiency of GPU resources, and reduces the complexity of scheduling.

[0069] Furthermore, the acquisition and storage of service alignment index data in this embodiment includes: acquiring GPU service resource monitoring data, wherein the GPU service resource monitoring data includes at least the running data of the GPU load corresponding to the existing services.

[0070] Optionally, the operational data includes GPU load and GPU memory usage.

[0071] This embodiment acquires and stores GPU service resource monitoring data to perform data analysis and regression of service GPU load, and calculates the current optimal process; it can also be used as a reference for GPU scheduling of new services.

[0072] As an optional implementation of this embodiment, the acquisition of GPU service resource monitoring data in this embodiment includes:

[0073] The GPU driver is invoked to obtain the process ID and the corresponding GPU load of the process;

[0074] The deployment unit of the cloud control platform is obtained by reading the control group of the corresponding operating system through the process ID;

[0075] Based on the cloud control platform, information about the deployment unit is obtained, and the correspondence between the service to which the deployment unit belongs and the GPU load is further obtained.

[0076] Organize and store the runtime data in a preset format according to the corresponding relationship.

[0077] As an optional implementation of this embodiment, the derived index data for calculating the stability and reliability of the computing response service includes: subscribing to GPU service logs through a unified log analysis service, parsing the logs, and calculating and obtaining traffic monitoring data according to the interface dimension. The obtained traffic monitoring data includes the service's QPS and latency.

[0078] Furthermore, this embodiment of a GPU dynamic scheduling method based on real-time load includes: organizing and storing the service alignment index data and derived index data that conform to a preset format, waiting to be invoked.

[0079] Optionally, a time-series database can be used to organize and store GPU service resource monitoring data and traffic monitoring data that conform to the time-series database format.

[0080] Specifically, organize metrics data that conforms to the Prometheus format and wait for Prometheus to collect it.

[0081] In this embodiment, the time-series database VictoriaMetrics is used to store monitoring metric data. Other TSDB or non-time-series databases, such as InfluxDB and TiDB, can also be used to achieve the same result.

[0082] See Figure 1 As shown, as an optional implementation of this embodiment, the GPU dynamic scheduling method based on real-time load in this embodiment includes:

[0083] The data collected includes the following alignment indexes:

[0084] Obtain GPU service resource monitoring data;

[0085] The GPU driver is invoked to obtain the process ID and the corresponding GPU load of the process;

[0086] The deployment unit of the cloud platform is obtained by reading the control group of the corresponding operating system through the process ID;

[0087] Obtain the deployment unit information from the cloud control system, and further obtain the corresponding relationship data between the services to which the deployment units belong and the GPU loads.

[0088] Obtain traffic monitoring data;

[0089] Through the unified log analysis service, subscribe to the GPU service logs and parse the logs to obtain the traffic data of the services.

[0090] The derivative metric data in this embodiment is obtained by calculating the obtained traffic monitoring data, such as service QPS, delay metrics from P99 to P55, etc.

[0091] In the virtualized cloud environment of K8s and containers, the main process for obtaining the GPU service resource monitoring data includes:

[0092] 1. Obtain the GPU service resource monitoring data:

[0093] 1.1 Obtain the process id and the GPU load of the corresponding process by calling the GPU driver.

[0094] 1.2 Read the corresponding operating system cgroup (control group, which provides a set of mechanisms for controlling the resource usage of a specific set of processes; cgroup binds a set of processes to one or more subsystems, and the main application of cgroup is resource tracking) through the process id to obtain the container and pod id (Pod is the smallest / simplest basic unit created or deployed by Kubernetes. A Pod represents a process running on the cluster; a Pod encapsulates an application container (it can also have multiple containers), storage resources, an independent network IP, and policy options for managing and controlling the running mode of the container; a Pod represents a unit of deployment: an instance of a single application in Kubernetes, which may be composed of a single container or multiple containers sharing resources).

[0095] 1.3 Based on 1.2, call the k8s API Server (the k8s API Server provides HTTP Rest interfaces such as create, delete, update, query, and watch for various resource objects <pod, RC, Service, etc.> in k8s, and is the data bus and data center of the entire system) to obtain the pod information, and further obtain the corresponding relationship between the service to which the pod belongs and the GPU load.

[0096] 1.4 Organize metrics data that conform to the Prometheus format (Prometheus stores time-series data, i.e., a collection of continuous data stored in the same time series <same name and label>, stored in the time dimension) (Metrics is a Java library that can provide you with unparalleled code execution insights), and wait for Prometheus to call and collect it.

[0097] 2. Obtain traffic monitoring data

[0098] 2.1 Subscribe to GPU service mesh logs through a unified log analysis service, parse the logs, and calculate service QPS and latency according to the interface dimension.

[0099] 2.2 Organize metrics data in Prometheus format and wait for Prometheus to collect it.

[0100] See Figure 2 As shown, the GPU dynamic scheduling method based on real-time load in this embodiment guides the allocation and scheduling of resources and services based on the alignment index data and / or derived index data. This includes summarizing and calculating the service alignment index data and derived index data based on the stored historical monitoring data of existing services, and generating a GPU scheduling strategy for service operation based on the calculated data. The specific method is as follows:

[0101] Obtain service alignment index data and derived index data of services using GPUs within a certain time window;

[0102] Calculate the balance point of resource utilization and performance for a service under a certain GPU card type based on service alignment index data and derived index data;

[0103] The calculated resource utilization and / or performance metric balance points are aggregated and used for the GPU scheduling strategy when the service is run again.

[0104] This embodiment implements GPU scheduling for existing services based on a real-time load dynamic GPU scheduling method. For example, it determines whether to expand the GPU capacity for existing services based on resource utilization and / or performance indicator balance points. This allows the optimal GPU resources to be scheduled as the number of services increases, thereby achieving the best service computing performance.

[0105] Specifically, the summary calculation of the comprehensive service alignment index data and derived index data in the GPU dynamic scheduling method based on real-time load in this embodiment includes:

[0106] Retrieve GPU service resource monitoring data and traffic monitoring data within a certain time window from a time-series database;

[0107] By summarizing the GPU service resource monitoring data and traffic monitoring data, including GPU utilization, GPU memory utilization, CPU utilization, memory utilization, service QPS, and latency metrics, the balance point between resource utilization and performance metrics during service operation is calculated.

[0108] See Figure 3 As shown, in this embodiment, the GPU dynamic scheduling method based on real-time load guides the allocation and scheduling of resources and services based on the alignment index data and / or derived index data. It can be used for GPU scheduling strategies for new service deployments, primarily by calling the service alignment index data. The specific process is as follows:

[0109] When a new service instance is created, the service memory usage and the relative amount of optimal load usage are obtained based on service alignment index data. GPUs with more remaining resources than the new service instance are selected from the existing GPUs in the cluster and then allocated.

[0110] This embodiment uses a real-time load-based GPU dynamic scheduling method to allocate GPU resources to new service instances. Based on the expected GPU resources required by the new service instance, it selects GPUs with more remaining resources than the resources required to run the new service instance for allocation. This is more conducive to improving the resource utilization of GPUs and avoids GPUs with large remaining resources being occupied by new services with low GPU resource requirements, thus avoiding the waste of GPU resources.

[0111] Optionally, in this embodiment, the GPU dynamic scheduling method based on real-time load selects the GPU graphics cards whose remaining load is closest to the amount of resources required to run the new service instance from the set of available GPU graphics cards.

[0112] This embodiment uses a real-time load-based GPU dynamic scheduling method to select and schedule GPUs whose remaining load is closest to the resource requirements for running the new service instance when a new service instance is created. This ensures the computing performance of the new service while improving the resource utilization of the GPUs and avoiding performance overkill.

[0113] The GPU dynamic scheduling method based on real-time load in this embodiment has the following characteristics:

[0114] 1. Perform data analysis and regression analysis on the GPU load to calculate the current optimal process.

[0115] 2. New service scheduling process.

[0116] 3. Additional GPU load data monitoring and collection capabilities.

[0117] See Figure 4 As shown, this embodiment also provides a GPU dynamic scheduling device based on real-time load, including:

[0118] The service monitoring module acquires service alignment index data and calculates derivative index data reflecting service stability and reliability.

[0119] The service scheduling module guides the allocation and scheduling of resources and services based on the alignment index data and / or derived index data.

[0120] The GPU dynamic scheduling device based on real-time load in this embodiment collects service alignment index data, such as GPU load, GPU storage usage and other resource usage, as well as traffic information such as service latency, service QPS, success and failure rate, and stores it for use in calling computation.

[0121] The GPU dynamic scheduling device based on real-time load in this embodiment performs comprehensive calculations by calling the above-mentioned service alignment index data to obtain derived index data reflecting service stability and reliability, such as latency indices from P99 to P50.

[0122] This embodiment integrates service alignment index data and derived index data to generate GPU scheduling strategies that improve GPU service computing power and GPU utilization for different types of service call data.

[0123] In this embodiment of the GPU dynamic scheduling device based on real-time load, the guidance for resource and service allocation and scheduling based on the alignment index data and / or derived index data mainly includes:

[0124] For existing services, historical monitoring data of services using GPUs within a certain time window is periodically obtained, and the balance point of resource utilization and performance of the service under a certain card type is calculated for the allocation and scheduling of GPU resources for existing services.

[0125] And / or, for new service copies, graphics card allocation is performed based on the alignment metric data and / or derived metric data.

[0126] The GPU dynamic scheduling device based on real-time load in this embodiment performs scheduling planning based on real-time GPU load information for cloud GPU usage scenarios, shields the differences in underlying GPU resources, fully explores the utilization efficiency of GPU resources, and reduces the complexity of scheduling.

[0127] The GPU dynamic scheduling device based on real-time load in this embodiment also includes a data analysis and regression module, which summarizes and calculates the comprehensive service alignment index data and derived index data.

[0128] Furthermore, in this embodiment, the service monitoring module acquires and stores service alignment indicator data; specifically, it acquires GPU service resource monitoring data, which includes at least the running data of the GPU load corresponding to the existing services.

[0129] Optionally, the operational data includes GPU load and GPU memory usage.

[0130] This embodiment acquires and stores GPU service resource monitoring data to perform data analysis and regression of service GPU load, and calculates the current optimal process; it can also be used as a reference for GPU scheduling of new services.

[0131] As an optional implementation of this embodiment, the service monitoring module in this embodiment obtains GPU service resource monitoring data including:

[0132] The GPU driver is invoked to obtain the process ID and the corresponding GPU load of the process;

[0133] The deployment unit of the cloud platform is obtained by reading the control group of the corresponding operating system through the process ID;

[0134] Based on the cloud control system, deployment unit information is obtained, and the correspondence between the services to which the deployment unit belongs and the GPU load is further obtained.

[0135] As an optional implementation of this embodiment, the service monitoring module acquires and calculates derived indicator data reflecting service stability and reliability, including: the service monitoring module acquires traffic monitoring data, subscribes to GPU service logs through a unified log analysis service, parses the logs, and calculates service QPS and latency according to the interface dimension.

[0136] Furthermore, this embodiment of a GPU dynamic scheduling device based on real-time load includes: a database module, which organizes and stores the service alignment index data and derived index data conforming to a preset format, waiting to be called.

[0137] Optionally, the database module uses a time-series database to organize and store GPU service resource monitoring data and traffic monitoring data that conform to the time-series database format.

[0138] Specifically, organize metrics data that conforms to the Prometheus format and wait for Prometheus to collect it.

[0139] In this embodiment, the time-series database VictoriaMetrics is used to store monitoring metric data. Other TSDB or non-time-series databases, such as InfluxDB and TiDB, can also be used to achieve the same result.

[0140] As an optional implementation of this embodiment, in the GPU dynamic scheduling device based on real-time load in this embodiment:

[0141] The service monitoring module collects service alignment indicator data including:

[0142] Obtain GPU service resource monitoring data;

[0143] The GPU driver is invoked to obtain the process ID and the corresponding GPU load of the process;

[0144] The deployment unit of the cloud platform is obtained by reading the control group of the corresponding operating system through the process ID;

[0145] Based on the cloud control system, deployment unit information is obtained, and further, the corresponding relationship data between the services to which the deployment unit belongs and the GPU load is obtained.

[0146] The service monitoring module acquires traffic monitoring data;

[0147] By subscribing to GPU service logs through a unified log analysis service, traffic data of the service can be obtained by parsing the logs.

[0148] In this embodiment, the derived indicator data are obtained by the service monitoring module through calculation of traffic monitoring data, such as service QPS, latency indicators from P99 to P55, etc.

[0149] This embodiment of the GPU dynamic scheduling device based on real-time load balancing generates a GPU scheduling strategy for service operation based on computational data through the service scheduling module, according to the storage of historical monitoring data of existing services. The specific method is as follows:

[0150] Obtain service alignment index data and derived index data of services using GPUs within a certain time window;

[0151] Calculate the balance point of resource utilization and performance for a service under a certain GPU card type based on service alignment index data and derived index data;

[0152] The calculated resource utilization and / or performance metric balance points are aggregated and used for the GPU scheduling strategy when the service is run again.

[0153] This embodiment implements GPU scheduling for existing services based on a real-time load dynamic GPU scheduling method, so that as the number of services increases, the best GPU resources can be scheduled to obtain the best service computing effect.

[0154] Specifically, the data analysis and regression module in this embodiment comprehensively calculates and aggregates alignment index data and derived index data, including:

[0155] Retrieve GPU service resource monitoring data and traffic monitoring data within a certain time window from a time-series database;

[0156] By summarizing the GPU service resource monitoring data and traffic monitoring data, including GPU utilization, GPU memory utilization, CPU utilization, memory utilization, service QPS, and latency metrics, the balance point between resource utilization and performance metrics during service operation is calculated.

[0157] In this embodiment, the GPU scheduling strategy for deploying new services based on the derived index data, which is invoked by the GPU dynamic scheduling device under real-time load, includes:

[0158] When a new service instance is created, the service scheduling module obtains the service memory usage and the relative amount of optimal load usage based on the service alignment index data, and filters out GPUs in the cluster whose remaining resources are greater than the resources required to run the new service instance.

[0159] The service scheduling module selects the GPUs from the available GPU set whose remaining load is closest to the resource requirements for running the new service instance.

[0160] In this embodiment, the real-time load GPU dynamic scheduling device selects and schedules GPUs whose remaining load is closest to the resource requirements for running the new service instance when a new service instance is created. This ensures the computing performance of the new service while improving the resource utilization of the GPUs and avoiding performance overkill.

[0161] For virtualized cloud environments using Kubernetes and containers, the GPU dynamic scheduling device based on real-time load in this embodiment includes:

[0162] Service monitoring module

[0163] Main responsibilities: To comprehensively collect and store service accuracy metrics, such as GPU load, GPU memory usage and other resource usage, service latency, QPS (queries per second), success and failure rates and other traffic data, and to calculate derived metrics reflecting service stability and reliability, such as P99 (average latency of the slowest 1% of requests in the past 10 seconds). This data will be provided to the data analysis and regression module to calculate the service performance balance point.

[0164] Main process:

[0165] 1. Obtain GPU service resource monitoring data;

[0166] 1.1 Obtain the process ID and the corresponding GPU load of the process by calling the GPU driver;

[0167] 1.2 Read the corresponding operating system cgroup (control group, which provides a mechanism for controlling the resource usage of a specific set of processes; cgroup binds a set of processes to one or more subsystems, and the main application of cgroup is resource tracking) through the process ID to obtain the container and Pod ID (A Pod is the smallest / simplest basic unit created or deployed by Kubernetes. A Pod represents a process running on the cluster; a Pod encapsulates an application container <it can also have multiple containers>, storage resources, an independent network IP, and policy options for managing and controlling the running mode of the container; A Pod represents a unit of deployment: an instance of a single application in Kubernetes, which may be composed of a single container or a shared resource composed of multiple containers).

[0168] 1.3 Based on 1.2, call the k8s API Server (the k8s API Server provides HTTP Rest interfaces such as create, delete, update, query, and watch for various k8s resource objects <Pods, RC, Services, etc.>, and is the data bus and data center of the entire system) to obtain Pod information, and further obtain the corresponding relationship between the service to which the Pod belongs and the GPU load.

[0169] 1.4 Organize metrics data (Metrics is a Java library that can provide you with unparalleled insights into code execution) in a format conforming to Prometheus (Prometheus stores time series data, that is, a collection of continuous data stored in the time dimension according to the same time series <same name and labels>), and wait for Prometheus to call for collection.

[0170] 2. Obtain traffic monitoring data

[0171] 2.1 Subscribe to the gpu service mesh logs through the unified log analysis service, parse the logs, and calculate the service QPS and latency according to the interface dimension.

[0172] 2.2 Organize metrics data in a format conforming to Prometheus, and wait for Prometheus to call for collection.

[0173] Data analysis and regression module

[0174] The deployment method is single-instance deployment.

[0175] Main responsibilities: Periodically retrieve historical monitoring data of services using GPUs within a certain time window from the time-series database, and calculate the balance point of resource utilization and performance of the service under a certain card type by summarizing and calculating the data within a certain time window. This data will be applied to the service scheduling module to guide the allocation and scheduling of resources and services.

[0176] Main process:

[0177] 1. Retrieve monitoring data within a certain time window from the time series database;

[0178] 2. For the monitoring data metrics within the previous time window, such as GPU utilization, GPU memory utilization, CPU utilization, memory utilization, service QPS, latency metrics from P99 to P50, etc., and perform summary calculations to obtain the balance point between resource utilization and performance metrics of the service in actual use.

[0179] 3. Report this service data to the service scheduling module as a factor for the next service scheduling.

[0180] 4. This module will remain dormant until the next cycle.

[0181] The GPU dynamic scheduling device based on real-time load in this embodiment includes:

[0182] The service scheduling module obtains service alignment index data and derived index data of services using GPU within a certain time window;

[0183] Calculate the balance point of resource utilization and performance for a service under a certain GPU card type based on service alignment index data and derived index data;

[0184] The calculated resource utilization and / or performance metric balance points are aggregated and used for the GPU scheduling strategy when the service is run again.

[0185] Service scheduling module

[0186] The deployment method is single-instance deployment.

[0187] Main responsibility: Allocate graphics cards to new service replicas based on data from the data analysis and regression module.

[0188] Main process:

[0189] 1. Allocation Pre-selection Phase: When a new service instance is created, data is obtained based on the service instance information (service memory usage, relative usage of optimal service load) to filter out the existing GPU graphics cards in the cluster whose remaining resources are greater than the resources required to run the service instance.

[0190] 2. Scoring and sorting phase: From the set of available GPUs selected in phase 1, select the GPUs whose remaining load is closest to the resource requirements of the current service.

[0191] This embodiment uses a real-time load-based GPU dynamic scheduling device to select and schedule GPUs whose remaining load is closest to the resource requirements for running the new service instance when a new service instance is created. This ensures the computing performance of the new service while improving the resource utilization of the GPUs and avoiding performance overkill.

[0192] The GPU dynamic scheduling device based on real-time load in this embodiment has the following characteristics:

[0193] 1. Perform data analysis and regression analysis on the GPU load to calculate the current optimal process.

[0194] 2. New service scheduling process.

[0195] 3. Additional GPU load data monitoring and collection capabilities.

[0196] In addition, regarding the collection and storage of monitoring data, this embodiment uses a custom probe + log analysis + Prometheus + time-series database VictoriaMetrics to generate, collect and store relevant monitoring indicators such as GPU utilization, QPS, and latency of the GPU service.

[0197] In this embodiment, the time-series database VictoriaMetrics is used to store monitoring metric data. Other TSDB or non-time-series databases, such as InfluxDB and TiDB, can also be used to achieve the same result.

[0198] In this embodiment, a custom exporter, a log processing service, and Prometheus are used to generate and collect monitoring metric data. Other tools such as Falcon and Zabbix can also be used to achieve the same result.

[0199] This embodiment also provides an electronic device, including a processor and a memory, wherein the memory is used to store computer-executable programs.

[0200] When the computer program is executed by the processor, the processor executes the GPU dynamic scheduling method based on real-time load.

[0201] This embodiment also provides a computer-readable medium storing a computer-executable program, which, when executed, implements the GPU dynamic scheduling method based on real-time load.

[0202] From the above description of the embodiments, those skilled in the art will readily understand that the present invention can be implemented by hardware capable of executing specific computer programs, such as the system of the present invention, and the electronic processing unit, server, client, mobile phone, control unit, processor, etc. included in the system. The present invention can also be implemented by computer software that executes the methods of the present invention, for example, by control software executed by a microprocessor, electronic control unit, client, server, etc. However, it should be noted that the computer software executing the methods of the present invention is not limited to execution in one or a specific set of hardware entities; it can also be implemented in a distributed manner by unspecified hardware. For computer software, the software product can be stored in a computer-readable storage medium (such as a CD-ROM, USB flash drive, portable hard drive, etc.) or distributed across a network, as long as it enables electronic devices to execute the methods according to the present invention.

[0203] The above embodiments are only used to illustrate the present invention and are not intended to limit the technical solutions described herein. Although the present invention has been described in detail with reference to the above embodiments, the present invention is not limited to the specific embodiments described above. Therefore, any modifications or equivalent substitutions to the present invention, as well as all technical solutions and improvements that do not depart from the spirit and scope of the invention, are covered within the scope of the claims of the present invention.

Claims

1. A real-time load based GPU dynamic scheduling method, characterized in that, The method comprises the following steps: obtaining service alignment index data and calculating derivative index data reflecting service stability and reliability; based on the alignment index data and / or derivative index data, guiding the allocation and scheduling of resources and services; based on the alignment index data and / or derivative index data, guiding the allocation and scheduling of resources and services: obtain the service alignment index data and derivative index data of the GPU in a certain time window; According to the service alignment index data and the derivative index data, the balance point of the resource utilization rate and the performance of the service under a certain GPU card type is calculated; The calculated resource utilization rate and / or performance index balance point is summarized and used for the allocation and scheduling of GPU resources of the inventory service; The summary calculation of the service alignment index data and the derivative index data comprises: obtain GPU service resource monitoring data and traffic monitoring data in a certain time window from the time series database; The resource utilization rate and performance index balance point in the service running process is calculated by summarizing the GPU usage rate, GPU memory usage rate, CPU usage rate, memory usage rate, service QPS and time delay index in the GPU service resource monitoring data and traffic monitoring data; The method for guiding the allocation and scheduling of resources and services based on the alignment index data and / or derivative index data comprises: When a new service replica is created, the service memory usage and the optimal load usage relative amount are obtained based on the service alignment index data, the remaining resources of the GPU video card in the cluster are screened which are greater than the required resources of the new service replica, and the video card allocation is performed; In the set of available GPU video cards screened out, the GPU video card with the closest remaining load amount to the required resource amount of the new service replica is screened out. The method for obtaining service alignment index data comprises: obtain GPU service resource monitoring data, the GPU service resource monitoring data at least including running data of the GPU corresponding to the inventory service; The running data includes GPU load and GPU memory usage; the method for obtaining GPU service resource monitoring data comprises: obtain process ID and corresponding process GPU load by calling GPU driver; read the control group of the corresponding operating system through the process ID to obtain the deployment unit of the cloud control platform; obtain the deployment unit information based on the cloud control platform, and further obtain the corresponding relationship between the deployment unit and the GPU load; organize the running data in a preset format according to the corresponding relationship and store it.

2. The GPU dynamic scheduling method based on real-time load according to claim 1, characterized in that, The method for calculating derivative index data reflecting service stability and reliability comprises: subscribe to GPU service logs through a unified log analysis service, analyze the logs and calculate traffic monitoring data according to the interface dimension, the traffic monitoring data including service QPS and time delay.

3. The GPU dynamic scheduling method based on real-time load according to any one of claims 1-2, characterized in that, organize and store the service alignment index data and derivative index data in a preset format, and wait to be called; The method comprises the following steps:

4. A real-time load based GPU dynamic scheduling apparatus, characterized in that, service monitoring module, obtaining service alignment index data and calculating derivative index data reflecting service stability and reliability; ​ a service scheduling module, based on the alignment index data and / or derived index data, guiding allocation and scheduling of resources and services; the service scheduling module, based on the alignment index data and / or derived index data, guiding allocation and scheduling of resources and services; obtaining service alignment index data and derived index data of using GPU within a certain time window; calculating a balance point of resource utilization and performance of a service under a certain GPU card type according to the service alignment index data and derived index data; summarizing the calculated resource utilization and / or performance index balance point for allocation and scheduling of GPU resources of the inventory service; the summary calculation of the service alignment index data and derived index data includes: obtaining GPU service resource monitoring data and traffic monitoring data within a certain time window from a time series database; summarizing the GPU utilization rate, GPU memory utilization rate, CPU utilization rate, memory utilization rate, service QPS and latency index in the GPU service resource monitoring data and traffic monitoring data to calculate the resource utilization and performance index balance point in the service running process; the service scheduling module, based on the alignment index data and / or derived index data, guiding allocation and scheduling of resources and services includes: when a new service replica is created, obtaining service memory usage and service optimal load usage relative amount based on the service alignment index data, screening GPU graphics cards with remaining resources greater than the required resources of the new service replica in the cluster, and allocating the graphics cards; in the set of available GPU graphics cards screened out, screening the GPU graphics card with the closest remaining load amount to the required resource amount of the new service replica; the service alignment index data includes: obtaining GPU service resource monitoring data, the GPU service resource monitoring data at least including running data of the GPU corresponding to the inventory service; the running data includes GPU load and GPU memory usage; the GPU service resource monitoring data includes: calling GPU driver to obtain process ID and GPU load of the corresponding process; reading the control group of the corresponding operating system through the process ID to obtain the deployment unit of the cloud control platform; obtaining deployment unit information based on the cloud control platform, and further obtaining the corresponding relationship between the deployment unit and the GPU load; organizing the running data in a preset format according to the corresponding relationship and storing it.

5. An electronic device comprising a processor and a memory for storing a computer executable program, characterized in that: when the computer program is executed by the processor, the processor executes the GPU dynamic scheduling method based on real-time load according to any one of claims 1-3.

Images