Task scheduling method and task scheduling device

Abstract

The invention provides a task scheduling method, and the method comprises the steps: obtaining a first configuration file which at least comprises node identifiers of hosts and data processing units, and a relation between the hosts, between the data processing units or between the hosts and the data processing units; obtaining a second configuration file, wherein the second configuration file at least comprises an identifier of the first load task and a type identifier of a host or a data processing unit executing the first load task; and when the first load task is deployed, scheduling is carried out based on the first configuration file and the second configuration file, so that the host or the data processing unit executes the first load task. The invention further provides a task scheduling device.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a task scheduling method and a task scheduling device. Background Art

[0002] With the development of cloud computing technology, DPU hardware acceleration has become a common solution to the problem of insufficient computing power in edge cloud scenarios. The ability of DPU hardware to run OS systems makes it possible for DPU systems to be added to the Kubernetes cloud platform cluster. DPU systems no longer rely on the physical server for business control and can be used as independent working nodes to carry Kubernetes business orchestration. However, the characteristics of DPU hardware are that the total amount of hardware resources is small and it is accelerated for specific application scenarios. Therefore, as a working node of Kubernetes, it is managed by the default scheduling mechanism and cannot fully meet the deployment requirements of special businesses. For example, DPU specifically carries the application data plane, the host carries the application control plane, and applications between multiple DPU cards hope to be deployed on the same host. Such orchestration requirements specific to hardware acceleration require a lot of additional configuration management work through the existing scheduling mechanism, and dynamic configuration is also impossible. Therefore, it is very necessary to propose a mechanism based on host plus DPU hybrid scheduling. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a task scheduling method and a task scheduling device.

[0004] The present application provides a task scheduling method, the method comprising:

[0005] Obtaining a first configuration file, wherein the first configuration file at least includes node identifiers of a host and a data processing unit, and a relationship between the hosts, between the data processing units, or between the hosts and the data processing units;

[0006] Acquire a second configuration file, where the second configuration file at least includes an identifier of the first load task and an identifier of a host or a data processing unit type that executes the first load task;

[0007] When deploying the first load task, scheduling is performed based on the first configuration file and the second configuration file so that the host or the data processing unit executes the first load task.

[0008] In some embodiments, when deploying the first load task, scheduling is performed based on the first configuration file and the second configuration file so that the host or the data processing unit executes the first load task, including:

[0009] Based on the data processing unit type identifier that executes the first load task, an optimal data processing unit node is selected as a node that executes the first load task.

[0010] In some embodiments, the first configuration file further includes a parent-child node relationship between the data processing unit and the host, and the second configuration file further includes an association tag of the first load task, the association tag is used to identify a second load task associated with the first load task;

[0011] When deploying the load task, scheduling is performed based on the first configuration file and the second configuration file so that the host or the data processing unit executes the first load task, including:

[0012] The node executing the second load task is determined by the associated tag, and the parent node of the node executing the second load task is used as the node executing the first load task.

[0013] In some embodiments, the first configuration file includes a parent-child node relationship between a data processing unit and a host, and the second configuration file further includes an association tag of the first load task, the association tag being used to identify a second load task associated with the first load task;

[0014] When deploying the load task, scheduling is performed based on the first configuration file and the second configuration file so that the host or the data processing unit executes the first load task, including:

[0015] In the event that the parent node of the node that schedules the second load task fails to be used as the node that executes the first load task, other available host nodes are used as nodes that execute the first load task.

[0016] In some embodiments, the first configuration file includes a sibling node relationship between data processing units, and the second configuration file further includes an association tag of the first load task, the association tag being used to identify a second load task associated with the first load task;

[0017] When deploying the load task, scheduling is performed based on the first configuration file and the second configuration file so that the host or data processing unit executes the first load task, including: determining the node that executes the second load task through the associated label, and using the brother node of the node that executes the second load task as the node that executes the first load task.

[0018] In some embodiments, the first configuration file includes a sibling node relationship between data processing units, and the second configuration file further includes an association tag of the first load task, the association tag being used to identify a second load task associated with the first load task;

[0019] When deploying the load task, scheduling is performed based on the first configuration file and the second configuration file so that the host or the data processing unit executes the first load task, including:

[0020] In the event that a sibling node of the node for scheduling the second load task as a node for executing the first load task fails, other available host nodes are used as nodes for executing the first load task.

[0021] In some embodiments, the method further includes: calculating health indicators of each of the data processing units, and determining an optimal data processing unit as the data processing unit that executes the first load task according to the health indicators;

[0022] Wherein, when calculating the health indicator of each of the data processing units, it is calculated based on at least one hardware resource of the CPU, memory, and disk capacity of the data processing unit.

[0023] In some embodiments, the method further comprises: calculating a health indicator of each of the data processing units based on the power consumption or temperature of the data processing units.

[0024] In some embodiments, the method further includes: calculating health indicators of each data processing unit based on the IO status of the data processing unit node.

[0025] The present application provides a task scheduling device, the device at least comprising:

[0026] A first configuration file acquisition module, used to acquire a first configuration file, wherein the first configuration file at least includes node identifiers of hosts and data processing units, and relationships between the hosts, between the data processing units, or between the hosts and the data processing units;

[0027] A second configuration file acquisition module, configured to acquire a second configuration file, wherein the second configuration file at least includes an identifier of a first load task and an identifier of a type of a host or a data processing unit that executes the first load task;

[0028] The scheduling module performs scheduling based on the first configuration file and the second configuration file when deploying the first load task, so that the host or the data processing unit executes the first load task.

[0029] The present application determines the scheduling and execution of tasks by configuring the relationship between the host and the data processing unit in the configuration file, as well as the execution of each load task and the required host and data processing unit type, so that tasks are scheduled on different hosts or data processing units. On the one hand, mixed scheduling of hosts and data processing units is achieved, and on the other hand, flexibility in scheduling load tasks of physical devices is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A flowchart of a task scheduling method according to an embodiment of the present application;

[0031] Figure 2 This is a schematic diagram of the structure of a task scheduling device according to an embodiment of the present application;

[0032] Figure 3 This is a schematic diagram of a first configuration file according to an embodiment of the present application;

[0033] Figure 4 This is a schematic diagram of a second configuration file according to an embodiment of the present application.

[0034] Reference numerals:

[0035] 1000, task scheduling device; 1001, first configuration file acquisition module; 1002, second configuration file acquisition module; 1003, scheduling module; 1100, bus; 1200, processor; 1300, memory; 1400, other circuits. DETAILED DESCRIPTION

[0036] Various aspects and features of the present application are described herein with reference to the accompanying drawings.

[0037] It should be understood that various modifications may be made to the embodiments of the present application. Therefore, the above description should not be considered as limiting, but only as an example of an embodiment. Other modifications within the scope and spirit of the present application will occur to those skilled in the art.

[0038] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present application and, together with the general description of the present application given above and the detailed description of the embodiments given below, serve to explain the principles of the present application.

[0039] These and other characteristics of the present application will become apparent from the following description of a preferred form of embodiment given as a non-limiting example with reference to the accompanying drawings.

[0040] It should also be understood that although the present application has been described with reference to some specific examples, those skilled in the art will be able to readily implement many other equivalent forms of the present application.

[0041] Figure 1 A task scheduling method flow chart is provided according to the present application. Figure 1 As shown, the method includes:

[0042] S101, obtaining a first configuration file, where the first configuration file at least includes node identifiers of a host and a data processing unit, and a relationship between hosts, between data processing units, or between a host and a data processing unit;

[0043] S102, obtaining a second configuration file, where the second configuration file at least includes an identifier of the first load task and an identifier of a type of a host or a data processing unit that executes the first load task;

[0044] S103: When deploying the first load task, scheduling is performed based on the first configuration file and the second configuration file so that the host or the data processing unit executes the first load task.

[0045] The present application determines the scheduling and execution of tasks by configuring the relationship between the host and the data processing unit in the configuration file, as well as the execution of each load task and the required host and data processing unit type, so that tasks are scheduled on different hosts or data processing units. On the one hand, mixed scheduling of hosts and data processing units is achieved, and on the other hand, flexibility in scheduling load tasks of physical devices is achieved.

[0046] Regarding the above step S101, for ease of understanding, Figure 3 Provide explanation. Figure 3 A schematic diagram of a first configuration file of an embodiment of the present application is shown in FIG. Figure 3 As shown, the first configuration file can describe the detailed information of each host node and data processing unit through metadata in YAML format, which can specifically include name, label and annotations, including host node and data processing unit identification, the host node is Worker NodeDPUHost1 and Worker Node DPUHost2, respectively identified as parent-node, the data processing unit is WorkerNode DPU1, Worker Node DPU2, respectively identified as dpu-node-1, dpu-node-2, dpu-scheduler.kubernetes.io / children-nodes:

[0047] "dpu-node-1, dpu-node-2" indicates that the child nodes of all data processing units DPU of the physical host parent node parent-node are dpu-node-1, dpu-node-2. node-role.kubernetes.io / dpu: "true" indicates that it is identified as a DPU child node, dpu-scheduler.kubernetes.io / parent-node: "parent-node", records the DPU's parent physical node, and dpu-scheduler.kubernetes.io / sibling-nodes: "dpu-node-2" indicates that other sibling DPU nodes under the DPU's parent node are recorded.

[0048] It should be noted that the scheduling method of the present application can be used in a Kubernetes cluster, but is not limited to a Kubernetes cluster.

[0049] Regarding the above step S102, combined Figure 4 Provide explanation. Figure 4 An example of a second configuration file is shown. The second configuration file is expressed in YAML format. The second configuration file includes multiple tasks, such as five tasks POD 1 to POD 5. Each task may include annotations and labels. The meaning of annotations and labels is explained below using POD 1 as an example.

[0050] YAML

[0051] POD 1

[0052] annotations:

[0053] dpu-scheduler.kubernetes.io / host-type:dpu

[0054] labels:

[0055] app:demo-base

[0056] Among them, POD 1 represents the first task, which is identified as demo-base. Demo-base can be used as the basis for scheduling and binding other subsequent load POD tasks. dpu-scheduler.kubernetes.io / host-type:dpu means that the first task POD 1 is specified to run on the data processing unit (dpu) node, and the dpu-scheduler scheduler automatically selects the optimal dpu node.

[0057] The first configuration file and the second configuration file provided in this application enhance the node description information: more comprehensive labels and annotation configurations are added to the node description information to distinguish the physical host from the DPU card, and the perception capability of the parent-child relationship and the sibling relationship is provided.

[0058] In the above step S103, when deploying the first load task such as POD 1, scheduling can be performed based on the first configuration file and the second configuration file so that the host Worker Node DPUHost1 / Worker Node DPUHost1 or the data processing unit Worker Node DPU1 / Worker Node DPU2 executes the first load task.

[0059] In some embodiments, when deploying the first load task, scheduling is performed based on the first configuration file and the second configuration file so that the host or data processing unit executes the first load task, including: based on the data processing unit type identifier that executes the first load task, selecting the optimal data processing unit node as the node that executes the first load task.

[0060] Since the configurations and operating environments of the data processing units Worker Node DPU1 / Worker Node DPU2 are different, and their execution efficiencies are also different, in order to obtain better execution efficiency, based on the data processing unit type identifier that executes the first load task POD 1, an optimal data processing unit node is selected as the node that executes the first load task POD 1.

[0061] In some embodiments, the first configuration file also includes a parent-child node relationship between the data processing unit and the host, and the second configuration file also includes an association tag of the first load task, and the association tag is used to identify a second load task associated with the first load task; when deploying the load task, scheduling is performed based on the first configuration file and the second configuration file so that the host or the data processing unit executes the first load task, including: determining the node that executes the second load task through the association tag, and using the parent node of the node that executes the second load task as the node that executes the first load task.

[0062] For ease of understanding, continue to combine Figure 3The first configuration file includes the host node and data processing unit identifiers, the host node is Worker Node DPUHost1 / Worker Node DPUHost2, the data processing unit is Worker Node DPU1 / Worker Node DPU2, and the host node Worker Node DPUHost1 is the parent node and has two child nodes, namely, data processing unit nodes dpu-node-1 and dpu-node-2. Figure 4 The configuration file is described in detail. The configuration file deploys a new POD load POD2 associated with the above POD1, and it must be scheduled to run on the parent node of the dpu where the associated POD1 load is located. The key configuration information is as follows: dpu-scheduler.kubernetes.io / host-type:dpuhost, which indicates that it is specified to run on the dpu parent node, that is, on the physical server, dpu-scheduler.kubernetes.io / require-parent-pod-affinity:"app=demo-base" indicates that the load with the associated label "app=demo-base" is mandatory to be scheduled to the parent node of the POD1 load, and the scheduling fails if it fails. labels:app:demo-parent, which indicates that the application label is added, and the application label identifies the second task, which is used as the basis for scheduling and binding other load PODs. The second configuration file also includes the association label "demo-parent" of the first load task POD 1, which is used to identify the second load task POD 2 associated with the first load task POD 1. When deploying a load task, the node executing the second load task POD 2 is determined by the association tag demo-parent, and the parent node of the node executing the second load task POD 2 is used as the node executing the first load task POD 1. In some embodiments, the first configuration file includes a parent-child node relationship between the data processing unit and the host, and the second configuration file also includes an association tag of the first load task, and the association tag is used to identify the second load task associated with the first load task; when deploying a load task, scheduling is performed based on the first configuration file and the second configuration file so that the host or the data processing unit executes the first load task, including: in the case of a failure to schedule the parent node of the node executing the second load task as the node executing the first load task, other available host nodes are used as nodes executing the first load task.

[0063] For ease of understanding, continue to combine Figure 3The first configuration file includes the host node and data processing unit identifiers, the host node is Worker Node DPUHost1 / Worker Node DPUHost2, the data processing unit is Worker Node DPU1 / Worker Node DPU2, and the host node Worker Node DPUHost1 is the parent node and has two child nodes, namely, data processing unit nodes dpu-node-1 and dpu-node-2. Figure 4 , Figure 4 The second configuration file includes the second task POD3, and also includes the association tag of the first load task POD 1 as "demo-parent", which is used to identify the second load task POD 3 associated with the first load task POD 1. Among them, dpu-scheduler.kubernetes.io / prefer-parent-pod-affinity:

[0064] "app=demo-base" means that it is scheduled to run on the parent node of the dpu where the associated POD load is located as much as possible. In the case of scheduling failure, it may also be scheduled to other available physical service nodes. When deploying load tasks, the node that executes the second load task POD 3 is determined by the associated label demo-parent. In the case that the parent node of the node that schedules the second load task POD 3 as the node that executes the first load task POD 1 fails, the parent node of the node that executes the second load task POD 3 will be used as the node that executes the first load task POD 1.

[0065] In some embodiments, the first configuration file includes a sibling node relationship between data processing units, and the second configuration file also includes an associated tag of the first load task, and the associated tag is used to identify a second load task associated with the first load task; when deploying the load task, scheduling is performed based on the first configuration file and the second configuration file so that the host or data processing unit executes the first load task, including: determining the node that executes the second load task through the associated tag, and using the sibling node of the node that executes the second load task as the node that executes the first load task.

[0066] Continue to combine Figure 3 Description, such as Figure 3As shown, the first configuration file may also include the sibling node relationship between the data processing units, such as the sibling-nodes (sibling nodes) "dpu-node-2" of the data processing unit Worker Node DPU1, and the sibling-nodes (sibling nodes) "dpu-node-1" of the data processing unit Worker Node DPU2. Figure 4 , Figure 4 The second configuration file also includes the second task POD4, dpu-scheduler.kubernetes.io / require-sibling-pod-affinity:

[0067] "app=demo-base" means that the load with the associated label "app=demo-base" must be scheduled to the brother dpu node of the POD1 load, and the scheduling fails if it is not successful. When describing the second task POD 4 through the above second configuration file, app=demo-parent is used as the associated label to identify the second load task POD4 associated with the first load task POD 1. The node executing the second load task POD 4 is determined by the associated label, and the brother node of the node executing the second load task POD 4 is used as the node executing the first load task POD 1.

[0068] In some embodiments, the first configuration file includes a sibling node relationship between data processing units, and the second configuration file also includes an associated tag of the first load task, and the associated tag is used to identify a second load task associated with the first load task; when deploying the load task, scheduling is performed based on the first configuration file and the second configuration file so that the host or data processing unit executes the first load task, including: in the event that the sibling node of the node scheduling the second load task fails to be used as the node executing the first load task, other available host nodes are used as nodes executing the first load task.

[0069] Continue to combine Figure 3 Description, such as Figure 3 As shown, the first configuration file may also include the sibling node relationship between the data processing units, such as the sibling-nodes (sibling nodes) "dpu-node-2" of the data processing unit Worker Node DPU1, and the sibling-nodes (sibling nodes) "dpu-node-1" of the data processing unit Worker Node DPU2. Figure 4 , Figure 4The second configuration file may also include a second task POD5, dpu-scheduler.kubernetes.io / prefer-sibling-pod-affinity:

[0070] "app=demo-base" means that it is scheduled to run on the brother node of the dpu where the associated POD load is located as much as possible. If the scheduling fails, it is scheduled to other available physical service nodes. When describing the second task POD 5 through the above second configuration file, demo-parent is used as an associated label to identify the second load task POD 5 associated with the first load task POD 1. The node that executes the second load task POD 5 is determined by the associated label demo-parent. If the brother node of the node that schedules the second load task POD 5 as the node that executes the first load task fails, other available host nodes are used as nodes that execute the first load task, such as the host node Worker Node DPUHost2.

[0071] This application can better meet the needs of associated loads by introducing an affinity scheduling mechanism for the parent host and the brother DPU. Deploying the associated loads on nodes related to the DPU can minimize data transmission and communication overhead and improve the performance and efficiency of the associated loads.

[0072] In some embodiments, any scheduling method may also include: calculating the health indicators of each data processing unit, and determining the optimal data processing unit as the data processing unit to execute the first load task based on the health indicators; wherein, when calculating the health indicators of each data processing unit, the health indicators are calculated based on at least one hardware resource of the data processing unit's CPU, memory, and disk capacity.

[0073] For ease of understanding, the following is an example of a method for calculating the health index of a data processing unit node:

[0074] TotalNodeScore=(NodeScoreStorageIO^p1*NodeScoreNetworkIO^p2*NodeScoreCPU^p3*NodeScoreMemory^p4*NodeScoreDisk^p5*

[0075] HealthScore^p6)^(1 / ∑pi)

[0076] The parameters are described as follows: NodeScoreStorageIO represents the storage IO health index, NodeScoreNetworkIO represents the network IO health index, NodeScoreCPU represents the CUP health index, NodeScoreMemory represents the memory health index, NodeScoreDisk represents the disk health index, p1, p2, …, p6 are the corresponding weighted parameters of each indicator, and ∑pj is the sum of all weighted parameters.

[0077] This application can evaluate the remaining resources of basic hardware resources such as CPU, memory, disk capacity, etc. according to the resource shortage characteristics of DPU cards, and evaluate the health indicators of temperature and power consumption of DPU cards, and try to select DPU nodes with better operation stability. In addition, DPU cards usually handle the needs of IO acceleration scenarios, and will also evaluate the busyness of IO and select DPU nodes with better IO bandwidth. Through a refined resource scoring mechanism and optimized scheduling strategy, it can better balance load and resource allocation, avoid resource waste and over-allocation, and thus improve resource utilization efficiency.

[0078] In some embodiments, the task scheduling method may further include: calculating a health indicator of each data processing unit based on the power consumption or temperature of the data processing unit.

[0079] For ease of understanding, the DPU data processing unit is used as an example to illustrate the calculation method of the DPU health score. To more accurately measure the health of the DPU, the temperature and power consumption are considered in their respective health ranges, as well as the deviation of the current values ​​from these ranges. Each item is given a score, which is then combined into an overall health score:

[0080] Temperature health index: TemperatureScore = e^(-k1*(CurrentTemperature-OptimalTemperature)^2)

[0081] Power consumption health indicator: PowerUsageScore = e^(-k2*(CurrentPowerUsage-OptimalPowerUsage)^2)

[0082] UDP health index determined based on temperature health index and power consumption health index: HealthScore = WeightTemperature*TemperatureScore+WeightPower*PowerUsageScore

[0083] The relevant parameters in the above expression are explained as follows: k1, k2 are parameters in the exponential function, which determine the steepness of the curve to reflect the sensitivity of each indicator to the overall health. WeightTemperature, WeightPower are the weights of each sub-score. CurrentTemperature represents the current temperature value, and OptimalTemperature represents the optimal temperature value. CurrentPowerUsage represents the current power consumption, and OptimalPowerUsage represents the optimal power consumption.

[0084] For ease of understanding, taking CPU, memory, and disk as components of the data processing unit as an example, the calculation method of the health indicators of each component is as follows:

[0085] Health indicator of CPU component: NodeScoreCpu = (AvailableCpu / TotalCpu)*WeightCpu

[0086] Health indicator of memory component: NodeScoreMemory = (AvailableMemory / TotalMemory)*WeightMemory

[0087] Disk component health indicator: NodeScoreDisk = (AvailableDisk / TotalDisk)*WeightDisk

[0088] The relevant parameters of the above expression are described as follows: AvailableX is the amount of resources available on the data processing unit node, TotalX is the total amount of resources on the data processing unit node, and WeightX refers to the weight of the resources of the data processing unit node.

[0089] In some embodiments, the task scheduling method may further include: calculating health indicators of each data processing unit based on the IO status of the data processing unit node.

[0090] According to the IO usage and upper limit, a scoring method that can reflect the node IO load is designed. A nonlinear function is used to reflect the IO load. With this setting, when the IO load is close to the upper limit, the application score drops faster. The calculation method of the IO load health indicator is as follows:

[0091] NodeScoreStorageIO=(MaxStorageIOCapacity-UsedStorageIOCapacity) / MaxStorageIOCapacity-λ*(CurrentStorageIOThroughput / MaxStorageIOThroughput)

[0092] NodeScoreNetworkIO=(MaxNetworkIOCapacity-UsedNetworkIOCapacity) / MaxNetworkIOCapacity-μ*(CurrentNetworkIOThroughput / MaxNetworkIOThroughput)

[0093] The relevant parameters in the above expression are explained as follows: λ and μ parameters are used to control the impact of throughput on the score. MaxStorageIOCapacity indicates the maximum storage / memory IO capacity, UsedStorageIOCapacity indicates the used storage / memory capacity, CurrentStorageIOThroughput indicates the current IO throughput of storage / memory, MaxStorageIOThroughput indicates the maximum IO throughput of storage / memory, MaxNetworkIOCapacity indicates the maximum network IO capacity, UsedNetworkIOCapacity indicates the used network IO capacity, CurrentNetworkIOThroughput indicates the current network IO throughput, and MaxNetworkIOThroughput indicates the maximum network IO throughput.

[0094] This application provides an integrated DPU resource scoring mechanism specifically targeting the characteristics of the data processing unit (DPU), such as temperature, power consumption, I / O bandwidth, and current processing rate. This scoring mechanism will be based on the existing CPU and memory resource allocation mechanism, and will additionally consider the resource consumption and performance indicators of the DPU, so as to perform more refined scheduling optimization of the load Pod. Through a more accurate resource scoring mechanism and optimized scheduling strategy, the system can better utilize DPU resources, thereby improving overall performance. Efficient use of the DPU can accelerate processing rates and reduce latency, making applications more responsive.

[0095] This application can be used in a kubernetes cluster to implement host plus DPU hybrid scheduling. By adding a DPU family relationship to the role design of the physical machine and the DPU card, a binding relationship exists between the physical machine and the DPU card, and then a health check and resource calculation rule with DPU acceleration characteristics is proposed for the DPU node scoring mechanism based on the characteristics of the DPU hardware, so that the DPU nodes matched by scheduling are more stable, and DPU nodes with more efficient hardware resources are obtained for applications with hardware acceleration requirements.

[0096] Figure 2 The structure diagram of the task scheduling device 1000 provided by the present application is shown as follows: Figure 2 As shown, the task scheduling device 1000 at least includes: a first configuration file acquisition module 1001, a second configuration file acquisition module 1002 and a scheduling module 1003. The first configuration file acquisition module 1001 is used to acquire a first configuration file, the first configuration file at least includes the node identification of the host and the data processing unit, and the relationship between the hosts, between the data processing units, or between the host and the data processing unit; the second configuration file acquisition module 1002 is used to acquire a second configuration file, the second configuration file at least includes the identification of the first load task, the host or data processing unit type identification that executes the first load task; the scheduling module 1003 is used to perform scheduling based on the first configuration file and the second configuration file when deploying the first load task, so that the host or the data processing unit executes the first load task.

[0097] It should be noted that the various modules of the task scheduling device 1000 of the present application correspond to the character scheduling method provided by the present application, and will not be described one by one here.

[0098] Figure 2 An example diagram of a task scheduling device using a hardware implementation of a processing system is shown. The device may include a corresponding module for executing each or several steps in the above flowchart. Therefore, each step or several steps in the above flowchart can be performed by a corresponding module, and the device may include one or more modules in these modules. The module may be one or more hardware modules specially configured to perform the corresponding steps, or implemented by a processor configured to perform the corresponding steps, or stored in a computer-readable medium for implementation by a processor, or implemented by some combination.

[0099] The hardware structure can be implemented using a bus architecture. The bus architecture can include any number of interconnecting buses and bridges, depending on the specific application and overall design constraints of the hardware. The bus 1100 connects various circuits including one or more processors 1200, memory 1300, and / or hardware modules together. The bus 1100 can also connect various other circuits 1400 such as peripherals, voltage regulators, power management circuits, external antennas, etc.

[0100] The bus 1100 may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the figure only uses one connecting line, but does not mean that there is only one bus or one type of bus.

[0101] The processor 1200 may be a general purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof. The general purpose processor may be a microprocessor or any conventional processor.

[0102] Any process or method description in the flowchart or otherwise described herein can be understood as a module, fragment or portion of a code representing one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes other implementations, in which the functions may not be performed in the order shown or discussed, including performing the functions in a substantially simultaneous manner or in a reverse order according to the functions involved, which should be understood by a person skilled in the art of the technical field to which the embodiments of the present application belong. The processor performs the various methods and processes described above. For example, the method implementation in the present application can be implemented as a software program, which is tangibly contained in a machine-readable medium, such as a memory. In some embodiments, part or all of the software program can be loaded and / or installed via a memory and / or a communication interface. When the software program is loaded into the memory and executed by the processor, one or more steps in the method described above can be performed. Alternatively, in other embodiments, the processor can be configured to perform one of the above methods in any other appropriate manner (e.g., by means of firmware).

[0103] The logic and / or steps represented in the flowchart or otherwise described herein may be embodied in any readable storage medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can fetch instructions from and execute instructions on an instruction execution system, apparatus, or device), or in combination with such instruction execution systems, apparatuses, or devices.

[0104] For the purposes of this specification, a "readable storage medium" may be any device that can contain, store, communicate, propagate or transmit a program for use with or in conjunction with an instruction execution system, device or apparatus. More specific examples of readable storage media (a non-exhaustive list) include the following: an electrical connection with one or more wirings (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), a fiber optic device, and a portable read-only memory (CDROM). In addition, the readable storage medium may even be paper or other suitable medium on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a memory.

[0105] It should be understood that the various parts of the present application can be implemented by hardware, software or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0106] A person skilled in the art may understand that all or part of the steps of implementing the above-mentioned implementation method may be completed by instructing related hardware through a program, and the program may be stored in a readable storage medium, which, when executed, includes one or a combination of the steps of the implementation method.

[0107] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a readable storage medium. The storage medium may be a read-only memory, a disk or an optical disk, etc.

[0108] In the description of this specification, the description with reference to the terms "one embodiment / method", "some embodiments / methods", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment / method or example are included in at least one embodiment / method or example of the present application. In this specification, the schematic representations of the above terms are not necessarily the same embodiment / method or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments / methods or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments / methods or examples described in this specification and the features of the different embodiments / methods or examples, unless they are contradictory.

[0109] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0110] Those skilled in the art should understand that the above embodiments are only for the purpose of clearly illustrating the present application, and are not intended to limit the scope of the present application. For those skilled in the art, other changes or modifications may be made based on the above disclosure, and these changes or modifications are still within the scope of the present application.

Claims

1. A task scheduling method, the method comprising: Obtaining a first configuration file, wherein the first configuration file at least includes node identifiers of a host and a data processing unit, and a relationship between the hosts, between the data processing units, or between the hosts and the data processing units; Acquire a second configuration file, where the second configuration file at least includes an identifier of the first load task and an identifier of a type of a host or a data processing unit that executes the first load task; When deploying the first load task, scheduling is performed based on the first configuration file and the second configuration file so that the host or the data processing unit executes the first load task.

2. The task scheduling method according to claim 1, wherein when the first load task is deployed, scheduling is performed based on the first configuration file and the second configuration file so that the host or the data processing unit executes the first load task, comprising: Based on the data processing unit type identifier that executes the first load task, an optimal data processing unit node is selected as a node that executes the first load task.

3. The task scheduling method according to claim 1, wherein the first configuration file further includes a parent-child node relationship between the data processing unit and the host, and the second configuration file further includes an association tag of the first load task, wherein the association tag is used to identify a second load task associated with the first load task; When deploying the load task, scheduling is performed based on the first configuration file and the second configuration file so that the host or the data processing unit executes the first load task, including: The node executing the second load task is determined by the associated tag, and the parent node of the node executing the second load task is used as the node executing the first load task.

4. The task scheduling method according to claim 1, wherein the first configuration file includes a parent-child node relationship between the data processing unit and the host, and the second configuration file further includes an association tag of the first load task, wherein the association tag is used to identify a second load task associated with the first load task; When deploying the load task, scheduling is performed based on the first configuration file and the second configuration file so that the host or the data processing unit executes the first load task, including: In the event that the parent node of the node that schedules the second load task fails to be used as the node that executes the first load task, other available host nodes are used as nodes that execute the first load task.

5. The task scheduling method according to claim 1, wherein the first configuration file includes a brother node relationship between data processing units, and the second configuration file further includes an association tag of the first load task, wherein the association tag is used to identify a second load task associated with the first load task; When deploying the load task, scheduling is performed based on the first configuration file and the second configuration file so that the host or the data processing unit executes the first load task, including: The node executing the second load task is determined by the associated label, and the brother node of the node executing the second load task is used as the node executing the first load task.

6. The task scheduling method according to claim 1, wherein the first configuration file includes a brother node relationship between data processing units, and the second configuration file further includes an association tag of the first load task, wherein the association tag is used to identify a second load task associated with the first load task; When deploying the load task, scheduling is performed based on the first configuration file and the second configuration file so that the host or the data processing unit executes the first load task, including: In the event that a sibling node of the node for scheduling the second load task as a node for executing the first load task fails, other available host nodes are used as nodes for executing the first load task.

7. The task scheduling method according to claim 1, further comprising: Calculate the health indicator of each of the data processing units, and determine the optimal data processing unit as the data processing unit that executes the first load task based on the health indicator; wherein, when calculating the health indicator of each of the data processing units, the health indicator is calculated based on at least one hardware resource of the data processing unit's CPU, memory, and disk capacity.

8. The task scheduling method according to claim 7, further comprising: The health indicator of each data processing unit is calculated based on the power consumption or temperature of the data processing unit.

9. The task scheduling method according to claim 7 or 8, further comprising: The health index of each data processing unit is calculated based on the IO status of the data processing unit node.

10. A task scheduling device, the device comprising at least: A first configuration file acquisition module, used to acquire a first configuration file, wherein the first configuration file at least includes node identifiers of hosts and data processing units, and relationships between the hosts, between the data processing units, or between the hosts and the data processing units; A second configuration file acquisition module, configured to acquire a second configuration file, wherein the second configuration file at least includes an identifier of a first load task and an identifier of a type of a host or a data processing unit that executes the first load task; The scheduling module performs scheduling based on the first configuration file and the second configuration file when deploying the first load task, so that the host or the data processing unit executes the first load task.

Images