Deep learning job-oriented heterogeneous resource scheduling method and related device

Abstract

The invention discloses a heterogeneous resource scheduling method for deep learning jobs and a related device, and relates to the field of resource scheduling. Packaging the deep learning jobs needing to be packaged in the plurality of deep learning jobs into job pairs in pairs, and obtaining a to-be-scheduled job set composed of the job pairs and / or single deep learning jobs not needing to be packaged; according to the respective resource demand data of the plurality of deep learning jobs and the number of idle resources of the target server cluster, predicting respective running durations of each job object in the job set to be scheduled on the GPU and the NPU, and according to the respective running durations of each job object on the GPU and the NPU, scheduling the plurality of deep learning jobs in the job set to be scheduled. And carrying out resource scheduling on the to-be-scheduled job set by adopting a round scheduling mechanism on the target server cluster. According to the method and the device, the aim of heterogeneous resource scheduling of the deep learning task across the GPU and the NPU is fulfilled based on the preset adaptive class.

Description

Technical Field

[0001] The present application relates to the technical field of resource scheduling, and in particular to a heterogeneous resource scheduling method and related devices for deep learning operations. Background Art

[0002] A deep learning job refers to a training task for a deep learning model. In some current scenarios, multiple deep learning jobs need to use the same scheduler to schedule the resources used in the training process. The scheduling process includes: scheduling resources for multiple deep learning jobs based on their respective graphics processing unit (GPU) demand data.

[0003] As the application of Neural Processing Unit (NPU) in edge computing and mobile devices grows, its performance on specific jobs (such as convolutional reasoning and quantized reasoning) is often better than that of GPU. However, the current scheduler lacks the perception and scheduling support of NPU performance, making the existing scheduling methods unsuitable for NPU environment. Summary of the invention

[0004] In view of the above problems, the present application provides a heterogeneous resource scheduling method and related devices for deep learning operations to achieve the purpose of cross-GPU and NPU resource scheduling for deep learning tasks. The specific scheme is as follows:

[0005] The first aspect of the present application provides a heterogeneous resource scheduling method for deep learning operations, comprising:

[0006] Obtain resource requirement data of each of the multiple deep learning jobs, resource usage data of the target server cluster, and the number of idle resources, wherein the number of idle resources includes the number of idle graphics processors GPUs and the number of idle neural processing units NPUs of various models, and the resource requirement data includes GPU requirement data and NPU requirement data;

[0007] According to the resource requirement data and the resource usage data of each of the multiple deep learning jobs, the deep learning jobs that need to be packaged among the multiple deep learning jobs are packaged into job pairs in groups of two, to obtain a set of jobs to be scheduled consisting of the job pairs and / or single deep learning jobs that do not need to be packaged;

[0008] Predicting the respective running time of each job object in the set of jobs to be scheduled on the GPU and the NPU according to the respective resource requirement data of the multiple deep learning jobs and the number of idle resources, wherein the job object is the job pair or the single deep learning job;

[0009] According to the respective running time of each job object on the GPU and the NPU, a round-robin scheduling mechanism is used on the target server cluster to perform resource scheduling on the set of jobs to be scheduled, wherein when it is necessary to switch scheduling resources between the GPU and the NPU, a preset adaptive class is called to complete the switching of the dependency library between the GPU and the NPU, and the scheduling resource configuration parameters are modified.

[0010] In a possible implementation, packaging the deep learning jobs that need to be packaged among the multiple deep learning jobs into job pairs in groups of two according to the resource requirement data and the resource usage data of each of the multiple deep learning jobs includes:

[0011] Based on the resource usage data, a throughput prediction value of the target server cluster is obtained for a set time period in the future;

[0012] Determining whether the target server cluster supports job bundling according to the throughput prediction value;

[0013] If so, determining a plurality of deep learning jobs to be packaged from the plurality of deep learning jobs according to the GPU requirement data included in the resource requirement data of each of the plurality of deep learning jobs, and taking each of the deep learning jobs to be packaged as a first job to be packaged;

[0014] Determine, according to the resource requirement data of each of the plurality of first jobs to be packaged, the iteration resource ratio of each of the plurality of first jobs to be packaged and the resource interleaving efficiency between every two of the first jobs to be packaged;

[0015] The job pair is obtained according to the iteration resource proportions of each of the plurality of the first jobs to be packaged and the resource interleaving efficiency between every two of the first jobs to be packaged.

[0016] In a possible implementation, determining a plurality of deep learning jobs to be packaged from the plurality of deep learning jobs according to the GPU requirement data included in the resource requirement data of each of the plurality of deep learning jobs includes:

[0017] Determine the throughput of each of the multiple deep learning jobs on the GPU according to the GPU requirement data included in the resource requirement data of each of the multiple deep learning jobs;

[0018] According to the throughputs of the multiple deep learning jobs on the GPU respectively, the multiple deep learning jobs that need to be packaged are determined from the multiple deep learning jobs.

[0019] In a possible implementation, obtaining the job pair according to the iteration resource proportions of each of the plurality of first jobs to be packaged and the resource interleaving efficiency between every two of the first jobs to be packaged includes:

[0020] Determining at least one resource-complementary job set from the plurality of first jobs to be packaged according to respective iteration resource proportions of the plurality of first jobs to be packaged, wherein the job set includes two first jobs to be packaged, and resource complementarity means that the dependent resources are the same but the dependency order is different, and the sum of the iteration resource proportions on the same resource is less than 1;

[0021] Packing each of the job sets into one of the job pairs;

[0022] Determine each of the first jobs to be packaged, except the job set, among the plurality of the first jobs to be packaged as a second job to be packaged;

[0023] According to the resource interleaving efficiency between every two of the second jobs to be packaged, a plurality of the second jobs to be packaged are packaged into the job pair.

[0024] In a possible implementation, the performing resource scheduling on the set of to-be-scheduled jobs by using a round-robin scheduling mechanism on the target server cluster according to the respective running time of each job object on the GPU and the NPU includes:

[0025] For each round in the round scheduling mechanism:

[0026] Obtain the accumulated running time of each pending job object in the round, wherein the pending job object refers to the job object that has not been scheduled as of the round;

[0027] Determine a target job object according to the accumulated running time of each of the job objects to be processed and the running time of each of the job objects to be processed on the GPU and the NPU, wherein the target job object refers to the job object to be processed for resource scheduling in this round;

[0028] Determine, from among the resource scheduling nodes included in the target server cluster, a resource scheduling node having the highest similarity to a resource scheduling node in a previous round of this round, and use the determined resource scheduling node as the resource scheduling node in this round;

[0029] The resource scheduling node in this round performs resource scheduling on the target job object, and after the time slice of this round is exhausted, the scheduling progress data of the target job object is saved, and the scheduling progress data is used for the next round of scheduling of the target job object.

[0030] In a possible implementation, determining, from among the resource scheduling nodes included in the target server cluster, a resource scheduling node having the highest similarity to a resource scheduling node in a previous round of the current round includes:

[0031] Obtaining the transmission overhead from the resource scheduling node in the previous round of this round to each resource scheduling node included in the target server cluster;

[0032] If the current round and the previous round need to switch scheduling resources between the GPU and the NPU, then obtain the model conversion overhead when the GPU and the NPU are transferred to each other;

[0033] According to the model conversion overhead and / or the transmission overhead, a resource scheduling node with the highest similarity to a resource scheduling node in a previous round of this round is determined from among the resource scheduling nodes included in the target server cluster as the resource scheduling node in this round.

[0034] In a possible implementation, before performing resource scheduling on the to-be-scheduled job set by using a round-robin scheduling mechanism on the target server cluster according to the respective running time of each job object on the GPU and the NPU, the method further includes:

[0035] Determining whether the target server cluster is sufficient for resource scheduling according to the resource usage data of the target server cluster;

[0036] If not, then obtaining resource usage data of all server clusters at the current time step, wherein all server clusters do not include the target server cluster;

[0037] Determine a set of candidate server clusters sufficient for the resource scheduling from all the server clusters according to the resource usage data of all the server clusters at the current time step and the respective running time of each job object on the GPU and the NPU, wherein the set of candidate server clusters includes at least one candidate server cluster;

[0038] Determining a migration cost from the target server cluster to each of the candidate server clusters;

[0039] Determining a server cluster after migration from the set of candidate server clusters according to the migration cost;

[0040] According to the respective running time of each job object on the GPU and the NPU, a round-robin scheduling mechanism is used on the migrated server cluster to perform resource scheduling on the set of jobs to be scheduled.

[0041] In a possible implementation, determining the migration cost from the target server cluster to each of the candidate server clusters includes:

[0042] Obtaining the total job volume of all job objects in the set of jobs to be scheduled, the network bandwidth of each of the candidate server clusters, the network delay of migrating the set of jobs to be scheduled from the target server cluster to each of the candidate server clusters, and the remaining percentage of resources required by the set of jobs to be scheduled in each of the candidate server clusters;

[0043] The migration cost from the target server cluster to each of the candidate server clusters is determined based on the total job volume of all job objects in the set of jobs to be scheduled, the network bandwidth of each of the candidate server clusters, the network delay of migrating the set of jobs to be scheduled from the target server cluster to each of the candidate server clusters, and the remaining percentage of resources required by the set of jobs to be scheduled in each of the candidate server clusters.

[0044] A second aspect of the present application provides a heterogeneous resource scheduling device for deep learning operations, including:

[0045] A data acquisition module, used to obtain resource demand data of each of the multiple deep learning jobs, resource usage data of the target server cluster, and the number of idle resources, wherein the number of idle resources includes the number of idle graphics processors GPUs and the number of idle neural processing units NPUs of various models, and the resource demand data includes GPU demand data and NPU demand data;

[0046] A job packaging module, configured to package the deep learning jobs that need to be packaged among the multiple deep learning jobs into job pairs in groups of two according to the resource requirement data of each of the multiple deep learning jobs and the resource usage data, to obtain a set of jobs to be scheduled consisting of the job pairs and / or single deep learning jobs that do not need to be packaged;

[0047] A duration prediction module, configured to predict the respective running duration of each job object in the set of jobs to be scheduled on the GPU and the NPU according to the respective resource demand data of the multiple deep learning jobs and the number of idle resources, wherein the job object is the job pair or the single deep learning job;

[0048] A resource scheduling module is used to perform resource scheduling on the set of jobs to be scheduled on the target server cluster using a round-scheduling mechanism according to the respective running time of each job object on the GPU and the NPU, wherein when it is necessary to switch scheduling resources between the GPU and the NPU, a preset adaptive class is called to complete the switching of the dependency library between the GPU and the NPU, and the scheduling resource configuration parameters are modified.

[0049] The third aspect of the present application provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the heterogeneous resource scheduling method for deep learning jobs according to the first aspect or any implementation method of the first aspect.

[0050] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0051] The memory is used to store computer programs;

[0052] The processor is used to execute the computer program so that the electronic device can implement the heterogeneous resource scheduling method for deep learning operations according to the above-mentioned first aspect or any implementation method of the first aspect.

[0053] The fifth aspect of the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the heterogeneous resource scheduling method for deep learning operations according to the above-mentioned first aspect or any implementation method of the first aspect.

[0054] By means of the above technical scheme, the heterogeneous resource scheduling method for deep learning jobs provided by the present application obtains the resource demand data of each of the multiple deep learning jobs, the resource usage data of the target server cluster, and the number of idle resources. According to the resource demand data and the resource usage data, the deep learning jobs that need to be packaged in the multiple deep learning jobs are packaged into job pairs in pairs, and a set of jobs to be scheduled consisting of job pairs and / or single deep learning jobs that do not need to be packaged is obtained. According to the resource demand data and the number of idle resources, the respective running time of each job object in the set of jobs to be scheduled on the GPU and NPU is determined, and then according to the running time, the round scheduling mechanism is used on the target server cluster to perform resource scheduling on the set of jobs to be scheduled. It can be seen from this that the present application can reasonably package the jobs and schedule them to the most suitable GPU resources and NPU resources according to the resource demand data of each of the multiple deep learning jobs and the load of the target server cluster, so as to achieve the purpose of efficiently allocating deep learning jobs to the optimal nodes, improve the overall computing efficiency, and reduce resource waste. At the same time, in order to avoid the need to modify the original program when switching scheduling resources between GPU and NPU, the present application presets an adaptive class, so that the present application can automatically complete the switching of dependent libraries between GPU and NPU and the modification of scheduling resource configuration parameters, ensuring non-intrusive and automatic seamless switching between GPU and NPU, and improving the overall resource scheduling efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0056] Figure 1 A schematic diagram of a system architecture provided for this application;

[0057] Figure 2 A schematic diagram of an optional hardware structure of the terminal 100 provided in this application;

[0058] Figure 3 A schematic diagram of the structure of a server 200 provided in this application;

[0059] Figure 4 A flowchart of a heterogeneous resource scheduling method for deep learning operations provided in this application;

[0060] Figure 5 A schematic diagram of the structure of a heterogeneous resource scheduling device for deep learning operations provided in this application;

[0061] Figure 6A schematic diagram of the structure of an electronic device provided in this application. DETAILED DESCRIPTION

[0062] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0063] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0064] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0065] See also Figure 1 , Figure 1 A schematic diagram of a system architecture is shown. The system may include a terminal 100 and a server 200. The server 200 may include one or more servers ( Figure 1 In the example, a server is included, and the server 200 can provide the method provided in the embodiment of the present application for one or more terminals.

[0066] Among them, an application can be installed on the terminal 100, and the above application and web page can provide an interface. The terminal 100 can receive relevant parameters entered by the user on the interface and send the above parameters to the server 200. The server 200 can obtain processing results based on the received parameters and return the processing results to the terminal 100.

[0067] It should be understood that in some optional implementations, the terminal 100 can also complete the action of obtaining the processing result based on the received parameters by itself without the cooperation of the server, and the embodiments of the present application are not limited to this.

[0068] Next describe Figure 1 The product form of the mid-terminal 100;

[0069] The terminal 100 in the embodiment of the present application can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc., and the embodiment of the present application does not impose any restrictions on this.

[0070] Figure 2 An optional hardware structure diagram of the terminal 100 is shown.

[0071] refer to Figure 2 As shown, the terminal 100 may include a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160 (optional), a speaker 161 (optional), a microphone 162 (optional), an earphone jack 163 (optional), a processor 170, an external interface 180, a power supply 190 and other components. Those skilled in the art will appreciate that Figure 2 These are merely examples of terminals or multi-function devices and do not constitute limitations on the terminals or multi-function devices, which may include more or fewer components than those shown in the figures, or combinations of certain components, or different components.

[0072] The input unit 130 can be used to receive input digital or character information, and generate key signal input related to the user settings and function control of the portable multifunctional device. Specifically, the input unit 130 may include a touch screen 131 (optional) and / or other input devices 132. The touch screen 131 can collect the user's touch operations on or near it (such as the user's operation on or near the touch screen using any suitable object such as fingers, joints, stylus, etc.), and drive the corresponding connection device according to a pre-set program. The touch screen can detect the user's touch action on the touch screen, convert the touch action into a touch signal and send it to the processor 170, and can receive and execute the command sent by the processor 170; the touch signal at least includes the touch point coordinate information. The touch screen 131 can provide an input interface and an output interface between the terminal 100 and the user. In addition, the touch screen can be implemented using multiple types such as resistive, capacitive, infrared and surface acoustic wave. In addition to the touch screen 131, the input unit 130 can also include other input devices. Specifically, other input devices 132 may include, but are not limited to, one or more of a physical keyboard, function keys (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

[0073] Among them, the input device 132 can receive input data and the like.

[0074] The display unit 140 may be used to display information input by a user or provided to a user, various menus of the terminal 100, an interactive interface, file display, and / or playback of any multimedia file.

[0075] The memory 120 can be used to store instructions and data. The memory 120 can mainly include an instruction storage area and a data storage area. The data storage area can store various data, such as multimedia files, texts, etc.; the instruction storage area can store software units such as operating systems, applications, instructions required for at least one function, or their subsets and extensions. It can also include a non-volatile random access memory; provide the processor 170 with hardware, software and data resources including management of computing and processing equipment, and support control software and applications. It is also used for the storage of multimedia files, and the storage of running programs and applications.

[0076] The processor 170 is the control center of the terminal 100. It uses various interfaces and lines to connect various parts of the entire terminal 100. By running or executing instructions stored in the memory 120 and calling data stored in the memory 120, it executes various functions of the terminal 100 and processes data, thereby controlling the terminal device as a whole. Optionally, the processor 170 may include one or more processing units; preferably, the processor 170 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 170. In some embodiments, the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on separate chips. The processor 170 may also be used to generate corresponding operation control signals, send them to corresponding components of the computing and processing device, read and process data in the software, especially read and process data and programs in the memory 120, so that each functional module therein performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0077] Among them, the memory 120 can be used to store software codes related to the heterogeneous resource scheduling method for deep learning operations, the processor 170 can execute the steps of the heterogeneous resource scheduling method for deep learning operations, and can also schedule other units (such as the above-mentioned input unit 130 and the display unit 140) to achieve corresponding functions.

[0078] The radio frequency unit 110 (optional) can be used for receiving and sending information or receiving and sending signals during a call, for example, after receiving the downlink information of the base station, it is sent to the processor 170 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low Noise Amplifier, LNA), a duplexer, etc. In addition, the radio frequency unit 110 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile Communication (Global System of Mobile Communication, GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, Short Messaging Service (SMS), etc.

[0079] In this embodiment of the present application, the RF unit 110 can send data to the server 200 and receive processing results sent by the server 200.

[0080] It should be understood that the radio frequency unit 110 is optional and can be replaced by other communication interfaces, such as a network port.

[0081] The terminal 100 also includes a power supply 190 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 170 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption.

[0082] The terminal 100 further includes an external interface 180 , which may be a standard Micro USB interface or a multi-pin connector, and may be used to connect the terminal 100 to communicate with other devices, or to connect a charger to charge the terminal 100 .

[0083] Although not shown, the terminal 100 may also include a flashlight, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which are not described in detail here. Some or all of the methods described below may be applied in the following embodiments. Figure 2 In the terminal 100 shown.

[0084] Next describe Figure 1 The product form of the server 200;

[0085] Figure 3 A schematic diagram of the structure of a server 200 is provided, such as Figure 3 As shown, the server 200 includes a bus 201, a processor 202, a communication interface 203, and a memory 204. The processor 202, the memory 204, and the communication interface 203 communicate with each other via the bus 201.

[0086] The bus 201 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0087] The processor 202 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0088] The memory 204 may include a volatile memory, such as a random access memory (RAM). The memory 204 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0089] Among them, the memory 204 can be used to store software codes related to the heterogeneous resource scheduling method for deep learning operations, the processor 202 can execute the steps of the heterogeneous resource scheduling method for deep learning operations of the chip, and can also schedule other units to achieve corresponding functions.

[0090] It should be understood that the above-mentioned terminal 100 and server 200 can be centralized or distributed devices, and the processors in the above-mentioned terminal 100 and server 200 (such as processor 170 and processor 202) can be hardware circuits (such as application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), general-purpose processors, DSPs, microprocessors or microcontrollers, etc.), or a combination of these hardware circuits. For example, the processor can be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0091] The present application provides a heterogeneous resource scheduling method for deep learning operations. The heterogeneous resource scheduling method for deep learning operations of the embodiment of the present application is described in detail below in conjunction with the accompanying drawings.

[0092] Reference Figure 4 , Figure 4 A flow chart of a heterogeneous resource scheduling method for deep learning operations provided in an embodiment of the present application, the method may include:

[0093] Step S401: Obtain resource demand data of each of the multiple deep learning jobs, resource usage data of the target server cluster, and the amount of idle resources.

[0094] As mentioned above, a deep learning job refers to a training task of a deep learning model. Those skilled in the art should understand that the overall training process of a deep learning model usually requires multiple iterations, and each iteration includes multiple stages, such as data loading, preprocessing, forward and backward propagation, and gradient synchronization, etc.

[0095] Each of the above stages has its own characteristics, so each stage often has its own specific resource requirements. For example, the forward and backward propagation stages mainly rely on GPU and / or NPU resources, while gradient synchronization is more limited by network latency.

[0096] In order to achieve optimal resource scheduling of multiple deep learning jobs in the same scheduler, this embodiment can pre-acquire the resource requirement data of each of the multiple deep learning jobs. Different from the prior art, the resource requirement data in this embodiment includes GPU requirement data and NPU requirement data. Here, the GPU requirement data includes the requirement data of various models of GPUs.

[0097] Various models of GPUs may include, for example, A100GPU and V100GPU. For a deep learning job job1 included in multiple deep learning jobs, this embodiment may obtain the demand data of job1 on A100GPU, the demand data of job1 on V100GPU (i.e., the above-mentioned GPU demand data), and the demand data of job1 on NPU (i.e., the above-mentioned NPU demand data).

[0098] Of course, the model of the above-mentioned GPU can also be other models, and this application does not make specific limitations.

[0099] The above resource demand data can reflect the use demand of various resources by deep learning jobs. For example, the resource demand data includes: the overall predicted running time of deep learning jobs, the average running time of one iteration, and the average running time of each stage of one iteration. Here, the overall predicted running time = the average running time of one iteration × the preset total number of iterations; the average running time of one iteration = the sum of the average running time of each stage of one iteration.

[0100] Optionally, the process of "obtaining resource requirement data of each of the multiple deep learning jobs" in this embodiment may include: for each of the multiple deep learning jobs, if the deep learning job has historical operation data, determining the resource requirement data of each of the deep learning jobs according to the historical operation data; if the deep learning job has no historical operation data, obtaining the short-term operation time limit , making the deep learning job run within a short time limit Several iterations are performed on each resource (mainly GPU and NPU in this embodiment), the average value of the resource requirement data of the several iterations is taken, and the resource requirement data of the deep learning job is determined according to the average value.

[0101] This embodiment can also obtain resource usage data and the number of idle resources of the target server cluster. Here, the target server cluster refers to the server cluster used for current resource scheduling, and the number of idle resources includes the number of idle GPUs and the number of idle NPUs of various models. Here, "idle GPU" means that there is no load mounted on the GPU, and "idle NPU" means that there is no load mounted on the NPU.

[0102] Optionally, the resource usage data of the target server cluster includes the resource usage data of the target server cluster at a current time step and the resource usage data of the target server cluster at each historical time step before the current time step.

[0103] It should be understood that in actual scenarios, there may be multiple server clusters, so it is necessary to determine the target server cluster from the multiple server clusters. Optionally, the throughput prediction values ​​of the multiple server clusters at the future set time can be predicted based on the resource usage data of the multiple server clusters, and then the target server cluster can be determined from the multiple server clusters based on the throughput prediction values ​​of the multiple server clusters at the future set time and the maximum throughput supported by the multiple server clusters.

[0104] For example, for each server cluster among multiple server clusters, the throughput prediction value of the server cluster for a set period of time in the future is divided by the maximum throughput supported by the server cluster to obtain the corresponding throughput percentage of the server cluster; then, among the multiple server clusters, the server cluster with the smallest throughput percentage is determined as the target server cluster.

[0105] Step S402: According to the resource requirement data and resource usage data of each of the multiple deep learning jobs, the deep learning jobs that need to be packaged among the multiple deep learning jobs are packaged into job pairs in groups of two, to obtain a set of jobs to be scheduled consisting of job pairs and / or single deep learning jobs that do not need to be packaged.

[0106] Here, the resource requirement data of the deep learning jobs that need to be packaged is smaller than the same resource requirement data of the deep learning jobs that do not need to be packaged.

[0107] As mentioned above, each stage of a deep learning job has its own specific resource requirements, that is, deep learning jobs present staged resource requirements. Therefore, the resources that the entire training process of a deep learning job relies on have a certain order, so it is possible that two deep learning jobs rely on the same resources but in different order. For example, stage 1 of deep learning job 1 relies on the GPU, and stage 2 relies on the CPU (Central Processing Unit), while stage 1 of deep learning job 2 relies on the CPU, and stage 2 relies on the GPU.

[0108] In view of the fact that the above situation is not an isolated case, the present embodiment may adopt a resource interleaving method to perform job packaging processing. Specifically, the present embodiment may package the deep learning jobs that need to be packaged among the multiple deep learning jobs into job pairs in groups of two according to the resource requirement data and resource usage data of the multiple deep learning jobs.

[0109] For example, multiple deep learning jobs include job1 to job10, among which job1 to job6 are deep learning jobs that need to be packaged, and job7 to job10 are deep learning jobs that do not need to be packaged. In this embodiment, job1 to job6 can be packaged into job pairs in pairs based on their respective resource demand data and resource usage data. For example, job1 and job2 are the first job pair, job3 and job4 are the second job pair, and job5 and job6 are the third job pair.

[0110] It should be noted that in order to avoid affecting the job completion time by packaging 3 or more jobs, this embodiment will only package 2 deep learning jobs together. If the number of deep learning jobs that need to be packaged is odd, one of the deep learning jobs that need to be packaged will not be packaged. At this time, the unpackaged deep learning job is regarded as a single deep learning job that does not need to be packaged.

[0111] In this embodiment, if multiple deep learning jobs are packaged, then the set of jobs to be scheduled is composed of all packaged job pairs; if some of the multiple deep learning jobs are packaged, then the set of jobs to be scheduled is composed of all packaged job pairs and single deep learning jobs that do not need to be packaged; if multiple deep learning jobs are not packaged, then the set of jobs to be scheduled is composed of all single deep learning jobs that do not need to be packaged.

[0112] Step S403: predict the running time of each job object in the set of jobs to be scheduled on the GPU and the NPU according to the resource demand data and the amount of idle resources of each of the multiple deep learning jobs.

[0113] In this embodiment, if the set of jobs to be scheduled consists only of job pairs, the above-mentioned job object is a job pair; if the set of jobs to be scheduled consists only of a single deep learning job, the above-mentioned job object is a single deep learning job; if the set of jobs to be scheduled consists of a job pair and a single deep learning job, each job pair or each single deep learning job in the set of jobs to be scheduled can be used as a job object.

[0114] For each job object in the set of jobs to be scheduled, if the job object is a single deep learning job, this embodiment can predict the respective running time of the single deep learning job on the GPU and the NPU based on the resource requirement data and the number of idle resources of the single deep learning job; if the job object is a job pair, this embodiment can predict the respective running time of the job pair on the GPU and the NPU based on the resource requirement data and the number of idle resources of the two deep learning jobs included in the job pair.

[0115] In this embodiment, the predicted running time of the job object on the GPU and NPU respectively enables the job object to maximize the use of GPU and NPU resources and complete as quickly as possible, thereby improving resource utilization and reducing the job completion time (JCT) of the job object.

[0116] Optionally, this embodiment can use CVXPY to determine the time proportion of each job object in the set of jobs to be scheduled on the GPU and NPU according to the resource demand data and the number of idle resources of each of the multiple deep learning jobs, thereby obtaining the resource allocation matrix X. Here, the elements in the resource allocation matrix X are It represents the time proportion allocated to job object i on idle accelerator j. Then, according to the resource allocation matrix X and the overall predicted running time of each job object, the running time of each job object in the set of jobs to be scheduled on each model of GPU and NPU is obtained.

[0117] It should be noted that there may be a situation in this embodiment where a certain job object does not use a certain model of GPU or NPU. If a certain model of GPU is not used, the running time of the job object on the GPU of this model is 0. Similarly, if the NPU is not used, the running time of the job object on the NPU is 0.

[0118] Step S404: According to the respective running time of each job object on the GPU and the NPU, a round-robin scheduling mechanism is used on the target server cluster to perform resource scheduling on the set of jobs to be scheduled.

[0119] When it is necessary to switch the scheduling resources between the GPU and the NPU, the preset adaptive class is called to complete the switching of the dependency library between the GPU and the NPU, and the scheduling resource configuration parameters are modified. Here, the scheduling resource configuration parameters represent the currently scheduled resources.

[0120] Specifically, this embodiment can use a round-robin scheduling mechanism on the target server cluster to perform resource scheduling on each job object in the set of scheduled jobs. In the round-robin scheduling mechanism, each round corresponds to a fixed time slice, such as 10 seconds.

[0121] Then, taking the example that the running time of job object job1 on V100GPU is 15 seconds, the running time on A100GPU is 10 seconds, and the running time on NPU is 18 seconds, and the order of job1 dependent resources is V100GPU, A100GPU, and NPU, the resource scheduling process based on the round scheduling mechanism is as follows: assuming that job1 is scheduled in the first round, a server containing V100GPU can be selected from the target server cluster, and the server runs on the V100GPU of the server for 10 seconds, and the scheduling progress data 1 of job1 is recorded; assuming that job1 is scheduled again in the next fifth round, a server containing V100GPU can be selected from the target server cluster, and the server runs on the V100GPU of the server for 5 seconds based on the scheduling progress data 1, and the scheduling progress data 1 of job1 is recorded. The scheduling progress data 2 of job1 is recorded; assuming that job1 is scheduled again in the next 6th round, a server containing A100GPU can be selected from the target server cluster, and the A100GPU of the server is run for 10 seconds based on the scheduling progress data 2, and the scheduling progress data 3 of job1 is recorded; assuming that job1 is scheduled again in the next 9th round, a server containing NPU can be selected from the target server cluster, and the NPU of the server is run for 10 seconds based on the scheduling progress data 3, and the scheduling progress data 4 of job1 is recorded; assuming that job1 is scheduled again in the next 12th round, a server containing NPU can be selected from the target server cluster, and the NPU of the server is run for 8 seconds based on the scheduling progress data 4, and the scheduling progress data 5 of job1 is recorded. At this time, the scheduling progress data 5 indicates that job1 has been scheduled, and job1 will not be scheduled in the future.

[0122] In the above resource scheduling, the round in which job1 is scheduled can be determined according to the priorities of job1 and other jobs.

[0123] It should be noted that the resource dependency order, running time and time slice size mentioned in the above scheduling process are all examples and are not intended to limit the present application.

[0124] When the job objects in the set of jobs to be scheduled include both job objects that depend on the GPU and job objects that depend on the NPU, it is necessary to switch scheduling resources between the GPU and the NPU in the round scheduling mechanism. For example, in the first round, job1 is scheduled, and job1 first depends on the V100GPU, so the V100GPU is scheduled in the first round; if job2 is scheduled in the second round, and job2 first depends on the NPU, then it is necessary to switch from the V100GPU to the NPU in the second round; if job3 is scheduled in the third round, and job3 first depends on the A100GPU, then it is necessary to switch from the NPU to the A100GPU in the second round.

[0125] Those skilled in the art should understand that the dependency libraries of GPU and NPU are different. In order to avoid the need to manually adjust the dependency libraries when switching scheduling resources between GPU and NPU, this embodiment presets an adaptive class DeviceManager, which can automatically identify and adjust the computing device required by the job object to ensure seamless switching of the job object between GPU and NPU.

[0126] More specifically, when it is necessary to switch scheduling resources between GPU and NPU, this embodiment can call the preset adaptive class DeviceManager to complete the dependent library switching between GPU and NPU. At the same time, the adaptive class DeviceManager can also complete the switching of scheduling resource configuration parameters between GPU and NPU, that is, modify the resource scheduling configuration parameter device in the program code that represents the currently scheduled resources. For example, the resource scheduled in the previous round is V100GPU, and the resource scheduled in this round is NPU, then the adaptive class DeviceManager can modify the scheduling resource configuration parameter device in the program code from V100GPU to NPU.

[0127] In this embodiment, during the operation of the job object, the adaptive class DeviceManager can intelligently select the most suitable device according to the hardware resources, and automatically import device dependencies (such as torch_npu) and adjust the device settings in the program code. If the job object needs to switch devices or run between different hardware architectures, DeviceManager will also automatically adapt and convert the model to ensure that the model is compatible with the target hardware architecture and executes efficiently. The whole process is non-intrusive and does not require modification of the user's original code, achieving the purpose of automated resource scheduling.

[0128] In summary, the heterogeneous resource scheduling method for deep learning jobs provided by the present application obtains the resource demand data of each of the multiple deep learning jobs, the resource usage data of the target server cluster, and the number of idle resources. According to the resource demand data and resource usage data, the deep learning jobs that need to be packaged in the multiple deep learning jobs are packaged into job pairs in pairs, and a set of jobs to be scheduled consisting of job pairs and / or single deep learning jobs that do not need to be packaged is obtained. According to the resource demand data and the number of idle resources, the respective running time of each job object in the set of jobs to be scheduled on the GPU and NPU is predicted, and then according to the running time, the round scheduling mechanism is used on the target server cluster to perform resource scheduling on the set of jobs to be scheduled. It can be seen from this that the present application can reasonably package the jobs and schedule them to the most suitable GPU resources and NPU resources according to the resource demand data of each of the multiple deep learning jobs and the load of the target server cluster, so as to achieve the purpose of efficiently allocating deep learning jobs to the optimal nodes, improve the overall computing efficiency, and reduce resource waste. At the same time, in order to avoid the need to modify the original program when switching scheduling resources between GPU and NPU, the present application presets an adaptive class, so that the present application can automatically complete the switching of dependent libraries between GPU and NPU and the modification of scheduling resource configuration parameters, ensuring non-intrusive and automatic seamless switching between GPU and NPU, and improving the overall resource scheduling efficiency.

[0129] In some embodiments of the present application, the process of the previous step S402 "packaging the deep learning jobs that need to be packaged among the multiple deep learning jobs into job pairs of two according to the resource requirement data and resource usage data of each of the multiple deep learning jobs" is introduced.

[0130] Taking into account the performance of some server clusters which may make the server clusters not support job packaging, in order to avoid job packaging and scheduling the packaged jobs to the target server cluster when the target server cluster does not support job packaging, resulting in resource scheduling failure, this embodiment can determine in advance whether the target server cluster supports job packaging based on the resource usage data of the target server cluster.

[0131] Specifically, this embodiment can predict the throughput of the target server cluster for a set time in the future based on the resource usage data of the target server cluster, obtain the throughput prediction value of the target server cluster for the set time in the future, and then determine whether the target server cluster supports job packaging based on the throughput prediction value of the target server cluster for the set time in the future.

[0132] Optionally, the resource usage data of the target server cluster is ,in, Represents resource usage data at time step i, including but not limited to CPU usage, GPU usage, NPU usage, etc. at time step i.

[0133] Optionally, in this embodiment, the resource usage data Times of the target server cluster may be input into the GA2M model to obtain a throughput prediction value of the target server cluster for a set time period in the future output by the model.

[0134] Here, GA2M refers to Generalized Additive Models plus Interactions, i.e., a generalized additive model with interaction terms, which is a machine learning model that adds interaction terms on the basis of the generalized additive model (GAM), thereby improving the flexibility and prediction accuracy of the model while maintaining the interpretability of the model. In this embodiment, the GA2M model is trained using the training resource usage data labeled with the actual value label of the throughput of the training cluster at the first time step as the training data, and the training resource usage data includes the resource usage data of the training cluster at the second time step and before the second time step, and the second time step is the time step before the first time step.

[0135] The above throughput refers to the number of iterations of the deep learning job per unit time (optional, the unit time is 1 second).

[0136] Optionally, the process of "determining whether the target server cluster supports job bundling based on the throughput prediction value of the target server cluster at a set time in the future" may include: determining whether the target server cluster supports job bundling based on the throughput prediction value of the target server cluster at a set time in the future and the maximum throughput supported by the target server cluster.

[0137] For example, in this embodiment, the throughput prediction value of the target server cluster for a set period of time in the future can be divided by the maximum throughput, and the obtained quotient can be compared with the first preset packaging threshold. If the quotient is greater than the first preset packaging threshold, GSS=1 is determined, indicating an exclusive mode, that is, the target server cluster does not support job packaging; conversely, if the quotient is less than or equal to the first preset packaging threshold, GSS=2 is determined, indicating that the target server cluster supports job packaging.

[0138] When the target server cluster does not support job bundling, this embodiment may not perform job bundling, and directly form a set of jobs to be scheduled consisting of multiple deep learning jobs.

[0139] If the target server cluster supports job packaging, perform job packaging.

[0140] In a possible implementation, this embodiment may first determine multiple deep learning jobs that need to be packaged from multiple deep learning jobs according to the GPU requirement data included in the resource requirement data of each of the multiple deep learning jobs.

[0141] Optionally, the process of "determining multiple deep learning jobs that need to be packaged from multiple deep learning jobs according to the GPU requirement data included in the resource requirement data of each of the multiple deep learning jobs" may include: determining the throughput of the multiple deep learning jobs on the GPU according to the GPU requirement data included in the resource requirement data of each of the multiple deep learning jobs, and determining multiple deep learning jobs that need to be packaged from the multiple deep learning jobs according to the throughput of the multiple deep learning jobs on the GPU.

[0142] As introduced above, the resource requirement data includes the average running data of each stage of an iteration. In this embodiment, the GPU requirement data may be the average running data of the stage that depends on the GPU in an iteration. This embodiment may calculate the throughput of multiple deep learning jobs on the GPU based on the average running data of the stage that depends on the GPU. For example, taking the deep learning job job1 as an example, assuming that the average running time of job1 in the stage that depends on the GPU is t seconds, and the unit time is 1 second, the throughput of job1 on the GPU is 1 / t.

[0143] Here, the greater the throughput of the deep learning job on the GPU, the higher the resource demand of the deep learning job, and the easier it is to cause resource contention and reduce the job completion time JCT when packaging the deep learning job.

[0144] In this embodiment, a second packaging threshold may be set For each deep learning job among the multiple deep learning jobs, if the throughput of the deep learning job on the GPU is greater than the second packaging threshold, SS=1 is determined, indicating that the deep learning job is a deep learning job that does not need to be packaged; conversely, if the throughput of the deep learning job on the GPU is less than or equal to the second packaging threshold, SS=2 is determined, indicating that the deep learning job is a deep learning job that needs to be packaged.

[0145] For the convenience of subsequent description, each deep learning job that needs to be packaged is defined as the first job to be packaged, that is, the deep learning job with SS=2 is the first job to be packaged.

[0146] The present application may have a variety of implementation methods for the process of packaging multiple first jobs to be packaged, including but not limited to the following methods.

[0147] The first is to package according to a preset resource interleaving efficiency. Here, the resource interleaving efficiency between the two first jobs to be packaged represents the proportion of non-idle resources after the two first jobs to be packaged are packaged together. The larger the value, the higher the resource utilization rate.

[0148] Specifically, the resource interleaving efficiency between every two first jobs to be packaged may be determined according to the resource demand data of each of the plurality of first jobs to be packaged.

[0149] Optionally, this embodiment can determine the running time of each of the multiple first jobs to be packaged on each resource based on the resource demand data of each of the multiple first jobs to be packaged, and then calculate the overall running time after every two first jobs to be packaged are packaged together based on the running time of each of the multiple first jobs to be packaged on each resource, and then calculate the resource interleaving efficiency between every two first jobs to be packaged based on the overall running time after every two first jobs to be packaged are packaged together and the running time of each of the multiple first jobs to be packaged on each resource.

[0150] Optionally, taking the first job to be packaged 0 and the first job to be packaged 1 using two resources, CPU and GPU, as an example, the calculation formula for resource interleaving efficiency is as follows:

[0151] Formula (1);

[0152] in, represents the running time of the first job i to be packaged on resource j (j=0 means CPU, j=1 means GPU), Indicates the total running time after the first to-be-packaged job 0 and the first to-be-packaged job 1 are packaged together. , Indicates the resource interleaving efficiency between the first job to be packaged 0 and the first job to be packaged 1.

[0153] Then, for each of the multiple first jobs to be packaged, determine the first job to be packaged that has the highest resource interleaving efficiency with the first job to be packaged, and package it together with the first job to be packaged to obtain a job pair. Similarly, multiple first jobs to be packaged can be packaged together in pairs to obtain job pairs.

[0154] The second method is to package according to the preset resource interleaving efficiency and iterative resource ratio.

[0155] Specifically, this embodiment can determine the iteration resource ratio of each of the multiple first jobs to be packaged and the resource interleaving efficiency between every two first jobs to be packaged based on the resource demand data of each of the multiple first jobs to be packaged, and then obtain the job pairs based on the iteration resource ratio of each of the multiple first jobs to be packaged and the resource interleaving efficiency between every two first jobs to be packaged.

[0156] As described above, the resource demand data includes the average running time of each stage of an iteration. In this embodiment, the iteration resource proportion P of each of the plurality of first jobs to be packaged can be obtained according to the average running time.

[0157] For example, the iteration resource ratio P of a first job to be packaged is [{'Storage': 0.1}, {'CPU': 0.3}, {'GPU': 0.5}, {'network': 0.1}], which means that one iteration of the first job to be packaged includes four stages. The first stage depends on the storage space as a resource, and the iteration resource ratio (that is, the resource usage time ratio) is 0.1. The second stage depends on the CPU as a resource, and the iteration resource ratio is 0.3. The third stage depends on the GPU as a resource, and the iteration resource ratio is 0.5. The fourth stage depends on the network as a resource, and the iteration resource ratio is 0.1.

[0158] This embodiment can obtain a job pair based on the iterative resource ratio of each of the multiple first jobs to be packaged and the resource interleaving efficiency between every two first jobs to be packaged. Optionally, the process may include: first, according to the iterative resource ratio of each of the multiple first jobs to be packaged, determine at least one resource-complementary job set from the multiple first jobs to be packaged, and package each job set into a job pair, where a job set includes two first jobs to be packaged, and resource complementarity means that the dependent resources are the same but the dependency order is different, and the sum of the iterative resource ratios on the same resource is less than 1; then, the first jobs to be packaged that cannot be resource-complementary are packaged using the resource interleaving efficiency, that is, each first job to be packaged except the job set among the multiple first jobs to be packaged is determined as a second job to be packaged, and according to the resource interleaving efficiency between every two second jobs to be packaged, the multiple second jobs to be packaged are packaged into a job pair.

[0159] For example, the first job to be packaged, job1, uses GPU resources first and then CPU resources, and the first job to be packaged, job2, uses CPU resources first and then GPU resources, and the sum of the iteration resource proportion of GPU used by job1 and the iteration resource proportion of GPU used by job2 is less than 1, and the sum of the iteration resource proportion of CPU used by job1 and the iteration resource proportion of CPU used by job2 is less than 1, then job1 and job2 can be packaged into a job pair.

[0160] The process of using resource interleaving efficiency to package the second job to be packaged in this embodiment is the same as the process of using resource interleaving efficiency to package the first job to be packaged in the previous text. For details, please refer to the previous text introduction and will not be repeated here.

[0161] In summary, this embodiment packages each deep learning job through a resource interleaving strategy, so that the packaged job pairs can alternately use different computing resources (such as CPU, GPU, NPU, storage space, etc.), effectively avoiding excessive occupation or idleness of a single resource. In this way, the resource allocation of computing nodes is optimized, resource utilization is improved, and job execution time is reduced, thereby optimizing overall computing efficiency.

[0162] In other embodiments of the present application, the process of step S404 "using a round-robin scheduling mechanism on the target server cluster to perform resource scheduling on the set of scheduled jobs according to the respective running time of each job object on the GPU and the NPU" is introduced.

[0163] For ease of description, the following takes any round in the round scheduling mechanism as an example to introduce the process of performing resource scheduling on the target server cluster for the set of scheduled jobs under this round based on the respective running time of each job object on the GPU and NPU. The resource scheduling processes of other rounds are the same and will not be introduced one by one in this embodiment.

[0164] First, obtain the cumulative running time of each pending job object in this round, and then determine the target job object based on the cumulative running time of each pending job object and the respective running time of each pending job object on the GPU and NPU. Here, the pending job object refers to the job object that has not been scheduled as of this round, and the target job object refers to the pending job object for resource scheduling in this round.

[0165] Optionally, the following formula may be used to determine the priority of each to-be-processed job object in the to-be-scheduled job set.

[0166] Formula (2);

[0167] in, Indicates the priority of the pending job object i. The larger the value, the higher the priority of the job object to be processed; Indicates the cumulative running time of the pending job object i; It represents the sum of the running time of the object i to be processed on the GPU and NPU respectively.

[0168] In this embodiment, the to-be-processed job object with the highest priority may be determined as the target job object.

[0169] Next, this embodiment also needs to determine the resource scheduling node with the highest similarity to the resource scheduling node in the previous round of this round from among the resource scheduling nodes included in the target server cluster, and use the determined resource scheduling node as the resource scheduling node in this round. Here, the switching overhead between resource scheduling nodes with higher similarity is smaller.

[0170] Optionally, this embodiment can obtain the transmission overhead from the resource scheduling node in the previous round of this round to each resource scheduling node included in the target server cluster. If this round and the previous round need to switch scheduling resources between GPU and NPU (that is, the resource scheduling node in the previous round is GPU, and the resource scheduling node in this round needs to use NPU, or the resource scheduling node in the previous round is NPU, and the resource scheduling node in this round needs to use GPU), then the model conversion overhead when the GPU and NPU are transferred to each other is obtained. Finally, based on the obtained model conversion overhead and / or transmission overhead, the resource scheduling node with the highest similarity to the resource scheduling node in the previous round of this round is determined from the resource scheduling nodes included in the target server cluster as the resource scheduling node in this round.

[0171] The similarity calculation formula between resource scheduling node i (i.e., the resource scheduling node in the previous round of this round) and resource scheduling node j (i.e., a resource scheduling node included in the target server cluster) is as follows:

[0172] Formula (3);

[0173] in, represents the similarity between resource scheduling node i and resource scheduling node j; represents the transmission overhead from resource scheduling node i to resource scheduling node j, which can be determined according to the network delay between resource scheduling node i and resource scheduling node j and the job size of the target job object; It represents the model conversion cost when transferring between GPU and NPU. Since the model structure of deep learning jobs does not change once it is determined, simple linear regression can be used to predict the model conversion cost.

[0174] The above α and β are weight parameters, satisfying α+β=1, and the ratio of α and β can be determined based on the current situation of the target server cluster. For example, if the network delay between resource scheduling node i and resource scheduling node j is high, the α value is low to reduce the impact of network delay. Otherwise, the α value is high. If model conversion is not required between resource scheduling node i and resource scheduling node j, that is, there is no need to switch scheduling resources between GPU and NPU, then β=0, and .

[0175] Optionally, Bayesian optimization can be used to calculate the most appropriate α and β every preset time interval (such as six minutes).

[0176] In this embodiment, the target server cluster may select resource scheduling nodes that have a similarity to the resource scheduling nodes in the previous round of the current round. The highest resource scheduling node is the resource scheduling node for this round.

[0177] Finally, this embodiment can perform resource scheduling on the target job object based on the resource scheduling node under this round, and save the scheduling progress data of the target job object after the time slice of this round is exhausted, and the scheduling progress data is used for the next round of scheduling of the target job object.

[0178] Optionally, the scheduling progress data may include: the calculation progress of the target job object , model parameters (The model here can be, for example, a deep learning model such as Resnet18, VGG19) and the intermediate results of the current schedule .

[0179] Of course, the scheduling progress data may also be other data, which is not specifically limited in this application.

[0180] Optionally, this embodiment may store the above-mentioned scheduling progress data in a persistent storage space, so as to be used for restoring the target job object in the next round of scheduling of the target job object.

[0181] In summary, this embodiment adopts a scheduling strategy based on node similarity, which preferentially schedules job objects to computing nodes that are close to the previous round of execution nodes, which can reduce the overhead when the job switches nodes, especially when a large amount of data transmission or state recovery is required.

[0182] It should also be noted that in the case of round-based scheduling, especially in a cluster environment with heterogeneous computing resources such as GPUs and NPUs, job scheduling not only involves the allocation of tasks, but may also include complex operations such as the transmission of job data, progress recovery, and model conversion. In order to reduce this additional time overhead, the scheduling strategy should consider the similarity between computing nodes. Specifically, nodes with high hardware configuration, processing power, load status, and matching degree with the job type should be prioritized for scheduling, thereby reducing the performance loss caused by job migration, status recovery, and model conversion, and improving the overall operation efficiency and response speed of the system.

[0183] It should be understood that the target server cluster may have insufficient resources and be unable to meet job requirements in the short term. In order to avoid the inability to smoothly execute job scheduling in this situation, this embodiment can attempt to migrate the set of jobs to be scheduled to other server clusters with sufficient resources to ensure that the jobs can be executed smoothly.

[0184] Based on this, optionally, before the above step S404, this embodiment can also determine whether the target server cluster is sufficient for resource scheduling based on the resource usage data of the target server cluster, and if so, execute according to step S404.

[0185] Optionally, you can determine whether the target server cluster is sufficient for resource scheduling based on the resource usage data of the target server cluster at the current time step. For example, the resource utilization rate in the target server cluster reaches 90%. At this time, there may be other job objects in the queue. In this case, adding new job objects will face long queueing times and may even cause high contention for cluster resources. Therefore, this situation can be determined as insufficient resources in the target server cluster, which cannot meet job needs in the short term.

[0186] In the case where the target server cluster is insufficient for resource scheduling, this embodiment can obtain the resource usage data of all server clusters at the current time step, wherein the target server cluster is not included in all server clusters; determine a set of candidate server clusters sufficient for resource scheduling from all server clusters based on the resource usage data of all server clusters at the current time step, and the respective running time of each job object on the GPU and the NPU, wherein the set of candidate server clusters includes at least one candidate server cluster; determine the migration cost from the target server cluster to each candidate server cluster; determine a post-migration server cluster from the set of candidate server clusters based on the migration cost; and use a round-robin scheduling mechanism on the post-migration server cluster to perform resource scheduling on the set of jobs to be scheduled based on the respective running time of each job object on the GPU and the NPU.

[0187] Specifically, first request to obtain the cluster specification cluster_spec of all server clusters and the resource usage data cluster_resource of all server clusters at the current time step. Then, based on the running time of each job object on the GPU and NPU respectively (that is, the resource requirements of each job object) and the resource usage data of all server clusters at the current time step, find out from the cluster specification cluster_spec whether there is a server cluster that meets the resource requirements of each job object, and obtain a list of candidate server clusters that meet the resource requirements of each job object. At least one candidate server cluster in the candidate server cluster list constitutes a candidate server cluster set. Finally, based on the migration cost of the job set to be scheduled from the target server cluster to each candidate server cluster, determine the migrated server cluster from the candidate server cluster set. Here, the migrated server cluster refers to the server cluster that replaces the target server cluster to continue resource scheduling.

[0188] Optionally, the process of determining the migration cost from the target server cluster to each candidate server cluster may include: obtaining the total job volume of all job objects in the job set to be scheduled, the network bandwidth of each candidate server cluster, the network delay of migrating the job set to be scheduled from the target server cluster to each candidate server cluster, and the remaining percentage of resources required for the job set to be scheduled in each candidate server cluster; determining the migration cost from the target server cluster to each candidate server cluster based on the total job volume of all job objects in the job set to be scheduled, the network bandwidth of each candidate server cluster, the network delay of migrating the job set to be scheduled from the target server cluster to each candidate server cluster, and the remaining percentage of resources required for the job set to be scheduled in each candidate server cluster.

[0189] Optionally, the migration cost is calculated as:

[0190] Formula (4);

[0191] in, represents the migration cost of the set of scheduled jobs from the target server cluster to the candidate server cluster, Represents the total workload of all job objects in the set of jobs to be scheduled. represents the network bandwidth of the candidate server cluster, Indicates the network delay of migrating the set of scheduled jobs from the target server cluster to each candidate server cluster. Indicates the remaining percentage of resources required by the set of jobs to be scheduled in the candidate server cluster.

[0192] Above , and are three weight coefficients, satisfying .when When it is larger, it can be increased ; For real-time operation objects, it can be improved ; If you want to select a cluster with more remaining resources and less load to migrate the scheduled jobs, you can increase .

[0193] Optionally, Bayesian optimization can be used to calculate the most appropriate , and .

[0194] In summary, in this embodiment, when the internal resources of the target server cluster are tight and the resource requirements of the new job cannot be met in the short term, in order to avoid the long waiting time of the job and the decline of system efficiency, the job can be scheduled to other clusters for execution. Through the cross-cluster scheduling mechanism, the job can be dynamically assigned to the cluster with sufficient resources, so as to achieve load balancing, optimize resource utilization, and reduce the waiting time of the job. This mechanism not only helps to relieve the pressure of a single server cluster, but also ensures the efficient flow of jobs between different clusters. Especially when there is good bandwidth and low latency between multiple server clusters, cross-cluster scheduling can significantly improve the overall performance.

[0195] The above introduces a heterogeneous resource scheduling method for deep learning operations provided by an embodiment of the present application. The following will introduce a device for executing the above-mentioned heterogeneous resource scheduling method for deep learning operations.

[0196] See also Figure 5 , Figure 5 A schematic diagram of the structure of a heterogeneous resource scheduling device for deep learning operations provided in an embodiment of the present application. Figure 5 As shown, the device may include:

[0197] The data acquisition module 501 is used to obtain resource demand data of each of the multiple deep learning jobs, resource usage data of the target server cluster, and the number of idle resources, wherein the number of idle resources includes the number of idle graphics processors GPUs and the number of idle neural processing units NPUs of various models, and the resource demand data includes GPU demand data and NPU demand data;

[0198] The job packaging module 502 is used to package the deep learning jobs that need to be packaged among the multiple deep learning jobs into job pairs in groups of two according to the resource requirement data and resource usage data of the multiple deep learning jobs, and obtain a set of jobs to be scheduled consisting of job pairs and / or single deep learning jobs that do not need to be packaged;

[0199] The duration prediction module 503 is used to predict the running duration of each job object in the set of jobs to be scheduled on the GPU and the NPU according to the resource demand data and the number of idle resources of each of the multiple deep learning jobs, wherein the job object is a job pair or a single deep learning job;

[0200] The resource scheduling module 504 is used to perform resource scheduling on the set of scheduled jobs on the target server cluster using a round-robin scheduling mechanism according to the respective running time of each job object on the GPU and the NPU. When it is necessary to switch scheduling resources between the GPU and the NPU, the preset adaptive class is called to complete the switching of the dependency library between the GPU and the NPU, and the scheduling resource configuration parameters are modified.

[0201] In a possible implementation, the process of the job packaging module packaging the deep learning jobs to be packaged among the multiple deep learning jobs into job pairs in groups of two according to the resource requirement data and resource usage data of the multiple deep learning jobs may include:

[0202] Based on the resource usage data, the throughput forecast value of the target server cluster is obtained for a set period of time in the future;

[0203] Determine whether the target server cluster supports job bundling based on the throughput prediction value;

[0204] If so, determining multiple deep learning jobs to be packaged from the multiple deep learning jobs according to the GPU requirement data included in the resource requirement data of each of the multiple deep learning jobs, and taking each deep learning job to be packaged as a first job to be packaged;

[0205] Determine, according to the resource requirement data of each of the plurality of first jobs to be packaged, the iteration resource ratio of each of the plurality of first jobs to be packaged and the resource interleaving efficiency between every two first jobs to be packaged;

[0206] A job pair is obtained according to the iteration resource proportions of each of the plurality of first jobs to be packaged and the resource interleaving efficiency between every two first jobs to be packaged.

[0207] In a possible implementation, the process of the job packaging module determining a plurality of deep learning jobs to be packaged from the plurality of deep learning jobs according to the GPU requirement data included in the resource requirement data of the plurality of deep learning jobs may include:

[0208] Determine throughputs of the multiple deep learning jobs on the GPU, respectively, according to GPU requirement data included in the resource requirement data of the multiple deep learning jobs;

[0209] According to the throughputs of the multiple deep learning jobs on the GPU respectively, multiple deep learning jobs that need to be packaged are determined from the multiple deep learning jobs.

[0210] In a possible implementation, the process of obtaining the job pair by the job packaging module according to the iteration resource proportions of each of the plurality of first jobs to be packaged and the resource interleaving efficiency between every two first jobs to be packaged may include:

[0211] Determine at least one resource-complementary job set from the multiple first jobs to be packaged according to their respective iteration resource proportions, wherein the job set includes two first jobs to be packaged, and resource complementarity means that the dependent resources are the same but the dependency order is different, and the sum of the iteration resource proportions on the same resource is less than 1;

[0212] Package each job set into a job pair;

[0213] Determine each first job to be packaged except the job set among the plurality of first jobs to be packaged as a second job to be packaged;

[0214] According to the resource interleaving efficiency between every two second jobs to be packaged, multiple second jobs to be packaged are packaged into job pairs.

[0215] In a possible implementation, the resource scheduling module may schedule resources for the set of scheduled jobs on the target server cluster using a round-robin scheduling mechanism according to the respective running time of each job object on the GPU and the NPU, and the process may include:

[0216] For each round in the round scheduling mechanism:

[0217] Get the cumulative running time of each pending job object in this round, where the pending job object refers to the job object that has not been scheduled as of this round;

[0218] Determine the target job object according to the accumulated running time of each job object to be processed and the running time of each job object to be processed on the GPU and the NPU, wherein the target job object refers to the job object to be processed for resource scheduling in this round;

[0219] Determine, from among the resource scheduling nodes included in the target server cluster, a resource scheduling node having the highest similarity to a resource scheduling node in a previous round of this round, and use the determined resource scheduling node as the resource scheduling node in this round;

[0220] The resource scheduling node in this round is used to schedule the target job object. After the time slice in this round is used up, the scheduling progress data of the target job object is saved. The scheduling progress data is used for the next round of scheduling of the target job object.

[0221] In a possible implementation, the process of the resource scheduling module determining, from among the resource scheduling nodes included in the target server cluster, a resource scheduling node having the highest similarity to a resource scheduling node in a previous round of the current round may include:

[0222] Obtaining the transmission cost from the resource scheduling node in the previous round of this round to each resource scheduling node included in the target server cluster;

[0223] If the current round and the previous round need to switch scheduling resources between GPU and NPU, obtain the model conversion overhead when transferring between GPU and NPU;

[0224] According to the model conversion overhead and / or transmission overhead, a resource scheduling node with the highest similarity to a resource scheduling node in a previous round of this round is determined from among the resource scheduling nodes included in the target server cluster as the resource scheduling node in this round.

[0225] In a possible implementation, the heterogeneous resource scheduling device for deep learning jobs provided by the present application may also include: a cross-cluster migration module.

[0226] The cross-cluster migration module is used to determine whether the target server cluster is sufficient for resource scheduling based on the resource usage data of the target server cluster before using the round-robin scheduling mechanism to perform resource scheduling on the set of scheduled jobs on the target server cluster according to the respective running time of each job object on the GPU and NPU; if not, obtain the resource usage data of all server clusters at the current time step, wherein the target server cluster is not included in all server clusters; determine a set of candidate server clusters sufficient for resource scheduling from all server clusters according to the resource usage data of all server clusters at the current time step and the respective running time of each job object on the GPU and NPU, wherein the set of candidate server clusters includes at least one candidate server cluster; determine the migration cost from the target server cluster to each candidate server cluster; determine the post-migration server cluster from the set of candidate server clusters according to the migration cost; and use the round-robin scheduling mechanism on the post-migration server cluster to perform resource scheduling on the set of scheduled jobs according to the respective running time of each job object on the GPU and NPU.

[0227] In a possible implementation, the process in which the cross-cluster migration module determines the migration cost from the target server cluster to each candidate server cluster may include:

[0228] Obtain the total job volume of all job objects in the set of jobs to be scheduled, the network bandwidth of each candidate server cluster, the network delay of migrating the set of jobs to be scheduled from the target server cluster to each candidate server cluster, and the remaining percentage of resources required by the set of jobs to be scheduled in each candidate server cluster;

[0229] The migration cost from the target server cluster to each candidate server cluster is determined based on the total job volume of all job objects in the set of jobs to be scheduled, the network bandwidth of each candidate server cluster, the network delay of migrating the set of jobs to be scheduled from the target server cluster to each candidate server cluster, and the remaining percentage of resources required by the set of jobs to be scheduled in each candidate server cluster.

[0230] The heterogeneous resource scheduling device for deep learning operations provided in the embodiment of the present application corresponds to the heterogeneous resource scheduling method for deep learning operations provided in the previous text. For details, please refer to the previous introduction and will not be repeated here.

[0231] The present application also provides an electronic device in an embodiment. Figure 6 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 6 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0232] like Figure 6 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0233] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0234] An embodiment of the present application also provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any heterogeneous resource scheduling method for deep learning operations provided in the embodiments of the present application.

[0235] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When one or more computer programs are executed by an electronic device, the electronic device can implement any heterogeneous resource scheduling method for deep learning operations provided in an embodiment of the present application.

[0236] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0237] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0238] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0239] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A heterogeneous resource scheduling method for deep learning jobs, characterized in that: include: Obtain resource requirement data of each of the multiple deep learning jobs, resource usage data of the target server cluster, and the number of idle resources, wherein the number of idle resources includes the number of idle graphics processors GPUs and the number of idle neural processing units NPUs of various models, and the resource requirement data includes GPU requirement data and NPU requirement data; According to the resource requirement data and the resource usage data of each of the multiple deep learning jobs, the deep learning jobs that need to be packaged among the multiple deep learning jobs are packaged into job pairs in groups of two, to obtain a set of jobs to be scheduled consisting of the job pairs and / or single deep learning jobs that do not need to be packaged; Predicting the respective running time of each job object in the set of jobs to be scheduled on the GPU and the NPU according to the respective resource requirement data of the multiple deep learning jobs and the number of idle resources, wherein the job object is the job pair or the single deep learning job; According to the respective running time of each job object on the GPU and the NPU, a round-robin scheduling mechanism is used on the target server cluster to perform resource scheduling on the set of jobs to be scheduled, wherein when it is necessary to switch scheduling resources between the GPU and the NPU, a preset adaptive class is called to complete the switching of the dependency library between the GPU and the NPU, and the scheduling resource configuration parameters are modified.

2. The heterogeneous resource scheduling method for deep learning operations according to claim 1 is characterized in that: The step of packaging the deep learning jobs that need to be packaged among the multiple deep learning jobs into job pairs in groups of two according to the resource requirement data and the resource usage data of each of the multiple deep learning jobs comprises: Based on the resource usage data, a throughput prediction value of the target server cluster is obtained for a set time period in the future; Determining whether the target server cluster supports job bundling according to the throughput prediction value; If so, determining a plurality of deep learning jobs to be packaged from the plurality of deep learning jobs according to the GPU requirement data included in the resource requirement data of each of the plurality of deep learning jobs, and taking each of the deep learning jobs to be packaged as a first job to be packaged; Determine, according to the resource requirement data of each of the plurality of first jobs to be packaged, the iteration resource ratio of each of the plurality of first jobs to be packaged and the resource interleaving efficiency between every two of the first jobs to be packaged; The job pair is obtained according to the iteration resource proportions of each of the plurality of the first jobs to be packaged and the resource interleaving efficiency between every two of the first jobs to be packaged.

3. The heterogeneous resource scheduling method for deep learning operations according to claim 2 is characterized in that: The determining, from the multiple deep learning jobs, multiple deep learning jobs that need to be packaged according to the GPU requirement data included in the resource requirement data of each of the multiple deep learning jobs, includes: Determine the throughput of each of the multiple deep learning jobs on the GPU according to the GPU requirement data included in the resource requirement data of each of the multiple deep learning jobs; According to the throughputs of the multiple deep learning jobs on the GPU respectively, the multiple deep learning jobs that need to be packaged are determined from the multiple deep learning jobs.

4. The heterogeneous resource scheduling method for deep learning operations according to claim 2, characterized in that: The obtaining the job pair according to the respective iteration resource proportions of the plurality of the first jobs to be packaged and the resource interleaving efficiency between every two of the first jobs to be packaged comprises: Determining at least one resource-complementary job set from the plurality of first jobs to be packaged according to respective iteration resource proportions of the plurality of first jobs to be packaged, wherein the job set includes two first jobs to be packaged, and resource complementarity means that the dependent resources are the same but the dependency order is different, and the sum of the iteration resource proportions on the same resource is less than 1; Packing each of the job sets into one of the job pairs; Determine each of the first jobs to be packaged, except the job set, among the plurality of the first jobs to be packaged as a second job to be packaged; According to the resource interleaving efficiency between every two of the second jobs to be packaged, a plurality of the second jobs to be packaged are packaged into the job pair.

5. The heterogeneous resource scheduling method for deep learning operations according to claim 1, characterized in that: The method of performing resource scheduling on the set of to-be-scheduled jobs by using a round-robin scheduling mechanism on the target server cluster according to the respective running time of each job object on the GPU and the NPU includes: For each round in the round scheduling mechanism: Obtain the accumulated running time of each pending job object in the round, wherein the pending job object refers to the job object that has not been scheduled as of the round; Determine a target job object according to the accumulated running time of each of the job objects to be processed and the running time of each of the job objects to be processed on the GPU and the NPU, wherein the target job object refers to the job object to be processed for resource scheduling in this round; Determine, from among the resource scheduling nodes included in the target server cluster, a resource scheduling node having the highest similarity to a resource scheduling node in a previous round of this round, and use the determined resource scheduling node as the resource scheduling node in this round; The resource scheduling node in this round performs resource scheduling on the target job object, and after the time slice of this round is exhausted, the scheduling progress data of the target job object is saved, and the scheduling progress data is used for the next round of scheduling of the target job object.

6. The heterogeneous resource scheduling method for deep learning operations according to claim 5, characterized in that: The determining, from among the resource scheduling nodes included in the target server cluster, a resource scheduling node having the highest similarity to a resource scheduling node in a previous round of this round, comprises: Obtaining the transmission overhead from the resource scheduling node in the previous round of this round to each resource scheduling node included in the target server cluster; If the current round and the previous round need to switch scheduling resources between the GPU and the NPU, then obtain the model conversion overhead when the GPU and the NPU are transferred to each other; According to the model conversion overhead and / or the transmission overhead, a resource scheduling node with the highest similarity to a resource scheduling node in a previous round of this round is determined from among the resource scheduling nodes included in the target server cluster as the resource scheduling node in this round.

7. The heterogeneous resource scheduling method for deep learning operations according to claim 1, characterized in that: Before performing resource scheduling on the to-be-scheduled job set by using a round-robin scheduling mechanism on the target server cluster according to the respective running time of each job object on the GPU and the NPU, the method further includes: Determining whether the target server cluster is sufficient for resource scheduling according to the resource usage data of the target server cluster; If not, then obtaining resource usage data of all server clusters at the current time step, wherein all server clusters do not include the target server cluster; Determine a set of candidate server clusters sufficient for the resource scheduling from all the server clusters according to the resource usage data of all the server clusters at the current time step and the respective running time of each job object on the GPU and the NPU, wherein the set of candidate server clusters includes at least one candidate server cluster; Determining a migration cost from the target server cluster to each of the candidate server clusters; Determining a server cluster after migration from the set of candidate server clusters according to the migration cost; According to the respective running time of each job object on the GPU and the NPU, a round-robin scheduling mechanism is used on the migrated server cluster to perform resource scheduling on the set of jobs to be scheduled.

8. The heterogeneous resource scheduling method for deep learning operations according to claim 7, characterized in that: The determining of the migration cost from the target server cluster to each of the candidate server clusters includes: Obtaining the total job volume of all job objects in the set of jobs to be scheduled, the network bandwidth of each of the candidate server clusters, the network delay of migrating the set of jobs to be scheduled from the target server cluster to each of the candidate server clusters, and the remaining percentage of resources required by the set of jobs to be scheduled in each of the candidate server clusters; The migration cost from the target server cluster to each of the candidate server clusters is determined based on the total job volume of all job objects in the set of jobs to be scheduled, the network bandwidth of each of the candidate server clusters, the network delay of migrating the set of jobs to be scheduled from the target server cluster to each of the candidate server clusters, and the remaining percentage of resources required by the set of jobs to be scheduled in each of the candidate server clusters.

9. A scheduler, characterized in that: include: A data acquisition module, used to obtain resource demand data of each of the multiple deep learning jobs, resource usage data of the target server cluster, and the number of idle resources, wherein the number of idle resources includes the number of idle graphics processors GPUs and the number of idle neural processing units NPUs of various models, and the resource demand data includes GPU demand data and NPU demand data; A job packaging module, configured to package the deep learning jobs that need to be packaged among the multiple deep learning jobs into job pairs in groups of two according to the resource requirement data of each of the multiple deep learning jobs and the resource usage data, to obtain a set of jobs to be scheduled consisting of the job pairs and / or single deep learning jobs that do not need to be packaged; A duration prediction module, configured to predict the respective running duration of each job object in the set of jobs to be scheduled on the GPU and the NPU according to the respective resource demand data of the multiple deep learning jobs and the number of idle resources, wherein the job object is the job pair or the single deep learning job; A resource scheduling module is used to perform resource scheduling on the set of jobs to be scheduled on the target server cluster using a round-scheduling mechanism according to the respective running time of each job object on the GPU and the NPU, wherein when it is necessary to switch scheduling resources between the GPU and the NPU, a preset adaptive class is called to complete the switching of the dependency library between the GPU and the NPU, and the scheduling resource configuration parameters are modified.

10. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the heterogeneous resource scheduling method for deep learning operations as described in any one of claims 1 to 8.

Images