A task hybrid scheduling method, system and computer readable storage medium

Abstract

The application discloses a task hybrid scheduling method, system and computer readable storage medium, and the method comprises the steps of obtaining a task to be scheduled; creating a Kubernetes controller and a Yarn container group resource, processing the Yarn container group resource by using the Kubernetes controller to obtain a container group resource, and making the task to be scheduled correspond to the Yarn container group resource; and scheduling the container group resource by using the Kubernetes controller to schedule the container group resource to a preset physical machine node, wherein the preset physical machine node is deployed with a data node container group, and the data node container group is a container group served by a data node in an HDFS cluster. Through the above manner, the application can improve the balance of task scheduling.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of task scheduling, in particular to a task hybrid scheduling method, system and computer readable storage medium. BACKGROUND

[0002] Distributed scheduling framework is an important link in big data system, responsible for the distribution and scheduling of various tasks, for example: using Yet Another Resource Negotiator (Yarn) or Kubernetes, etc. In the Hadoop system, the on Yarn scheme can be used, that is, all tasks are scheduled by Yarn, and the task process runs on the NodeManager node. This scheme makes the central processing unit (CPU) resource isolation not thorough, resulting in multiple processes running on a physical machine. Some schemes use the on Kubernetes scheme, in which tasks run in the form of a Pod, which can achieve resource isolation, but not all distributed frameworks support the on Kubernetes scheme. Some schemes use a hybrid scheduling architecture of Yarn and Kubernetes, but this scheme will lead to uneven scheduling. SUMMARY

[0003] The present application provides a task hybrid scheduling method, system and computer readable storage medium, which can improve the balance of task scheduling.

[0004] To solve the above technical problems, the technical scheme adopted by the present application is to provide a task hybrid scheduling method, which comprises: obtaining a task to be scheduled; creating a Kubernetes controller and a Yarn container group resource, processing the Yarn container group resource using the Kubernetes controller to obtain a container group resource, the task to be scheduled corresponding to the Yarn container group resource; and scheduling the container group resource using the Kubernetes controller to schedule the container group resource to a preset physical machine node, the preset physical machine node being deployed with a data node container group, the data node container group being a container group served by a data node in an HDFS cluster.

[0005] To solve the above technical problems, another technical scheme adopted by the present application is to provide a task hybrid scheduling system, which comprises a memory and a processor connected to each other, wherein the memory is used to store a computer program, and the computer program is used to implement the task hybrid scheduling method in the above technical scheme when executed by the processor.

[0006] To solve the above technical problems, another technical solution adopted by the present application is to provide a computer readable storage medium for storing a computer program, which is used to implement the task mixed scheduling method in the above technical solution when executed by a processor.

[0007] Through the above scheme, the beneficial effects of the present application are: first, the to-be-scheduled task is obtained; then the Kubernetes controller and the Yarn container group resource corresponding to the to-be-scheduled task are created, the Yarn container group resource is processed by using the Kubernetes controller, and the container group resource is obtained; then the generated container group resource is scheduled by using the Kubernetes controller to schedule the container group resource to the physical machine node where the data node container group is deployed, and the deployment and scheduling of the to-be-scheduled task are realized; since the Kubernetes controller is used for scheduling the task, the logic of scheduling is controlled by the Kubernetes cluster, which can improve the balance of scheduling and make the scheduling balanced; moreover, since the container group resource and the data node container group are deployed on the same physical machine node, it is convenient to directly use the data node on the same physical machine node for reading and writing data in the subsequent process, which helps to improve the performance of data reading and writing. BRIEF DESCRIPTION OF DRAWINGS

[0008] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor. Among them:

[0009] Figure 1 is a schematic diagram of the mixed scheduling architecture of Yarn and Kubernetes provided by the present application;

[0010] Figure 2 is a flowchart of an embodiment of the task mixed scheduling method provided by the present application;

[0011] Figure 3 is a flowchart of another embodiment of the task mixed scheduling method provided by the present application;

[0012] Figure 4 is a structural schematic diagram of an embodiment of the task mixed scheduling system provided by the present application;

[0013] Figure 5 is a structural schematic diagram of an embodiment of the computer readable storage medium provided by the present application. DETAILED DESCRIPTION

[0014] The application will be described in further detail below with reference to the drawings and embodiments. It is particularly pointed out that the following embodiments are only for illustration of the application and do not limit the scope of the application. Similarly, the following embodiments are only part of the embodiments of the application, and all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of protection of the application.

[0015] In the present application, the term "embodiment" means that the specific features, structures or properties described in connection with the embodiment can be included in at least one embodiment of the present application. The phrase appears in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment to other embodiments. Those skilled in the art explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.

[0016] It should be noted that the terms "first", "second", "third" in the present application are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second", "third" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise specifically limited. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units not listed, or can optionally include other steps or units inherent to the process, method, product or device.

[0017] First, the professional terms involved in the present application are explained:

[0018] Yarn: a distributed task scheduling framework in the field of big data, which includes a node manager and a resource manager (ResourceManager). The resource manager is the master (Master) role in the Yarn framework, and the node manager is the slave (Slave) role in the Yarn framework.

[0019] Kubernetes: a task orchestration and scheduling framework in the field of cloud native, which includes a Master node and a physical machine node (Node).

[0020] Pod: the basic unit of Kubernetes framework scheduling, which is composed of multiple containers (Containers), and Containers is a resource isolation technology.

[0021] Persistent Volume (PV): a storage abstraction resource in Kubernetes environment.

[0022] Hadoop Distributed File System (HDFS): a distributed storage framework, including NameNode and DataNode, the DataNode is a child node of the distributed storage framework HDFS, and the NameNode is a Master node of the distributed storage framework HDFS.

[0023] The hybrid scheduling architecture based on Yarn and Kubernetes is as shown in Figure 1 Task is a scheduled task, and Task1 and Task2 are tasks executed on a physical machine node. The Yarn and Kubernetes solutions in the related art have the following problems:

[0024] ① In the on Yarn solution, multiple task processes run on the physical machine node where the node manager is located, and multiple task processes on the same physical machine node can conveniently share dependent resources. However, in the hybrid deployment solution of Yarn and Kubernetes, the HostPath mode cannot be directly used to read and write the disk, because HostPath is not secure and has no isolation, and needs to be read and written through the Pod mounting PV mode. This mode needs the node manager to create PV on each physical machine node in advance, and then mount the Pod to the PV. The node manager needs to perceive the relationship between the underlying physical machine node and the PV, and does not completely give the scheduling logic to Kubernetes, which will cause uneven scheduling.

[0025] ② In the on Yarn solution, the data node and the node manager are deployed on the same physical machine node, and in this scenario, the task process can read the data of the data node on the same physical machine node in proximity, improving the task running performance. However, after HDFS and Yarn are arranged on Kubernetes, this data locality advantage is lost, because multiple processes are run in the form of a Pod, and each Pod has independent Internet Protocol (IP) information, and the task process cannot perceive whether a certain Pod is on the same physical machine node as itself. This problem can be divided into two specific problems, a) how does Kubernetes schedule the task Pod to the corresponding physical machine node with the DataNode container group (i.e. DataNode Pod) when scheduling; b) how does the task Pod perceive which physical machine node it is on.

[0026] To solve the above problems, the application provides a task hybrid scheduling method, and the technical solutions adopted by the application are described in detail below.

[0027] Please refer to Figure 2 , Figure 2 is a flowchart of an embodiment of the task hybrid scheduling method provided by the application, and the method comprises the following steps:

[0028] Step 11: obtaining a task to be scheduled, creating a Kubernetes controller and a Yarn container group resource, and processing the Yarn container group resource by using the Kubernetes controller to obtain a container group resource.

[0029] The task to be scheduled is a task that needs to be scheduled at present, and the task to be scheduled corresponds to the Yarn container group resource. In actual task scheduling, in order to schedule the task to be scheduled, a custom Kubernetes controller (denoted as Yarn deployment) and a Yarn container group resource corresponding to the task to be scheduled are first created, the Yarn container group (denoted as YarnPod) resource in the Kubernetes cluster is monitored by using the Kubernetes controller, and the YarnPod resource is processed (such as parsing or conversion), for example, the YarnPod resource can be parsed into a corresponding resource format to obtain a correct container group resource (denoted as Pod resource), and the Pod resource is scheduled by the Kubernetes cluster subsequently, and the Pod resource comprises at least one Pod.

[0030] Step 12: scheduling the container group resource by using the Kubernetes controller to schedule the container group resource to a preset physical machine node.

[0031] The DataNode Pod is deployed on the preset physical machine node, and the DataNode Pod is a container group served by a data node in the HDFS cluster. In order to schedule the Pod resource corresponding to the task to be scheduled, the Kubernetes controller in the Kubernetes cluster can be used to schedule the Pod resource, and the Pod resource is deployed to the physical machine node that already has the DataNode Pod, and the physical machine node is the preset physical machine node, so that the preset physical machine node is used to execute the task to be scheduled.

[0032] Furthermore, the task to be scheduled may include multiple subtasks, which can be deployed to a certain physical machine node for execution, or distributed to different physical machine nodes in the Kubernetes cluster for execution; for example, assuming that the task to be scheduled includes 5 subtasks P1-P5, and there are two physical machine nodes: H1 and H2, subtasks P1-P3 are deployed to physical machine node H1 for execution, and subtasks P4-P5 are deployed to physical machine node H2 for execution.

[0033] This embodiment provides a Yarn task scheduling optimization solution based on Kubernetes, involving scheduling optimization technology in a hybrid deployment scenario of Yarn and Kubernetes. This solution first creates a Kubernetes controller and container group resources corresponding to the task to be scheduled; then uses the Kubernetes controller to process the YarnPod resources to obtain Pod resources; then uses the Kubernetes controller to schedule the Pod resources to schedule the Pod resources to the physical machine node where the DataNodePod is deployed, thereby realizing the deployment scheduling of the task to be scheduled; since the Kubernetes controller is used to schedule the task, that is, the scheduling logic is controlled by the Kubernetes cluster, the scheduling balance can be improved; in addition, since the Pod resources and the DataNode Pod are deployed on the same physical machine node, it is convenient to directly use the data node on the same physical machine node as the Pod resource to read and write data, which helps to improve the performance of processing distributed tasks.

[0034] See also Figure 3 , Figure 3 : is a flow chart of another embodiment of the task hybrid scheduling method provided by the present application, the method comprising:

[0035] Step 21: Get the tasks to be scheduled and create Kubernetes controller and YARN container group resources.

[0036] A new resource type is defined in the Kubernetes cluster: YarnPod resource. Specifically, the node manager in the Yarn cluster is used to create the YarnPod resource. The YarnPod resource includes an annotation field, and the parsed data includes a first prefix field and a second prefix field. The first prefix field is the name prefix of the container group served by the data node in the current environment, and the second prefix field is the prefix of the PV mounted by the Pod resource in the current environment.

[0037] Further, the first prefix field is denoted as a DataNodePodPrefix field, and the second prefix field is denoted as a MountPVPrefix field. The format of the YarnPod resource is basically the same as that of the Pod, except that the YarnPod resource adds the DataNodePodPrefix field and the MountPVPrefix field in the annotation field, so as to be used by the Kubernetes controller in subsequent generation of the Pod resource from the YarnPod resource.

[0038] Step 22: Monitor whether there is a Yarn container group resource by using the Kubernetes controller.

[0039] After the Kubernetes controller is created, the Kubernetes controller is used to monitor whether there is a new YarnPod resource in the Kubernetes cluster.

[0040] Step 23: If there is a Yarn container group resource, the Yarn container group resource is parsed to obtain parsed data.

[0041] When the custom Kubernetes controller detects that a new YarnPod resource is generated, the annotation field in the YarnPod resource is parsed to obtain a first prefix field and a second prefix field. Based on the value of the first prefix field, the address information (denoted as DataNode IP information) of the DataNode Pod in the current environment can be obtained. The DataNode IP information is injected into the Pod resource in the form of an environment variable. The second prefix field is parsed to obtain a PV prefix, which is the prefix of the PV mounted by the Pod resource. In combination with a load balancing strategy, a PV with the same PV prefix is selected from the PV resource, so that the Pod resources running on the same preset physical machine node are mounted to the same PV, so as to realize resource sharing. Further, it can be judged whether there is a PV. If there is no PV, the PV is created, and the Pod resource is mounted to the PV.

[0042] By declaring the PV prefix to be mounted in the description of the YarnPod resource, the Kubernetes controller selects a PV with the same prefix, and divides the PV to the Pod for mounting (if there is no PV, a PV is created), so that multiple Pods on the same physical machine node can conveniently share storage, thereby realizing data sharing.

[0043] Step 24: Create a container group resource based on the parsed data.

[0044] Based on the address information and the PV, a Pod resource is created, and the generated Pod resource has several special places: 1) the mount point (mountPath) mounted PV information is selected by the Kubernetes controller; 2) two environment variables are injected: DN_LIST (representing the correspondence between the DataNode IP information in the current environment and the physical machine node) and NODE (which physical machine node the current Pod belongs to), and the environment variable DN_LIST and the environment variable NODE can help the client in the HDFS cluster running in the Pod to select the most suitable data node to read and write data, thereby improving the performance of data read and write.

[0045] Step 25: The Kubernetes controller is used to schedule the container group resource to schedule the container group resource to the preset physical machine node.

[0046] The Kubernetes controller is created to schedule the YarnPod resource created by the node manager, and the generated Pod resource is scheduled to the physical machine node on which the DataNode Pod has been deployed, under the premise of ensuring balance.

[0047] In a specific embodiment, a node is selected as a preset physical machine node from all physical machine nodes in combination with a load balancing strategy, the physical machine node has a DataNode Pod deployed thereon, and the Pod resource corresponding to the to-be-scheduled task currently requiring scheduling is deployed to the preset physical machine node.

[0048] Further, the load balancing strategy is a strategy for achieving load balancing in the related art, and will not be described herein; the embodiment can find the physical machine node having the DataNode Pod deployed thereon on the basis of load balancing; for example, assuming that there are 3 physical machine nodes, and the Pod resource corresponding to the to-be-processed task includes 10 Pods, the 10 Pods can be preferentially deployed to the physical machine node having the DataNode Pod, and the number of Pods on the 3 physical machine nodes is substantially the same, so as to improve the overall processing performance.

[0049] In another specific embodiment, it is judged whether the ratio of the number of physical machine nodes in the Kubernetes cluster to the number of container groups corresponding to the to-be-scheduled task is greater than a preset value, and the specific value of the preset value can be set according to specific application scenarios or experience; if the ratio of the number of physical machine nodes to the total number of container groups is greater than the preset value, it indicates that the number of physical machine nodes in the current Kubernetes cluster is relatively sufficient, and at this time, all Pods corresponding to the to-be-scheduled task can be preferentially scheduled to the same preset physical machine node. For example, assuming that the number of physical machine nodes is 100 and the number of Pods corresponding to the to-be-scheduled task is 10, the 10 Pods can be preferentially deployed to a physical machine node having a DataNode Pod.

[0050] Step 26: A client selects a data node from all data nodes based on an environment variable to read and / or write to-be-processed data.

[0051] The HDFS cluster further includes a client, and a Pod resource is used to run the client, and the client perceives through an environment variable whether the data node corresponding to the obtained DataNode IP information is on the same physical machine node as itself; if it is judged that the data node is on the same physical machine node as itself, the local (i.e., the physical machine node) data node is preferentially selected to read to-be-processed data, which can be HDFS data. Specifically, the network segment of each physical machine node is different, and the IP information of the components (such as Pods, clients, or data nodes) arranged on the physical machine node is related to the network segment, for example, assuming that the network segment of physical machine A is 128.a1.b1.c1, the IP information of the data node D1 arranged on the physical machine node A is 128.a1.b1.c1.e1, the IP information of the client U arranged on the physical machine node A is 128.a1.b1.c1.e2, the network segment of physical machine B is 192.a2.b2.c2, and the IP information of the data node D2 arranged on the physical machine node B is 192.a2.b2.c2.e3. By comparing the IP information, it can be known that the client U and the data node D1 are located on the same physical machine node, and the client U and the data node D1 are not located on the same physical machine node.

[0052] In a specific embodiment, a data node can be selected from all data nodes as a candidate data node first; then the client judges whether the obtained candidate data node is deployed on the same physical machine node as the client through an environment variable; if the candidate data node is deployed on the same physical machine node as the client, the candidate data node is used to read and / or write the data to be processed; if the candidate data node is not deployed on the same physical machine node as the client, the step of selecting a data node from all data nodes as a candidate data node is returned until a preset selection end condition is met, and the preset selection end condition includes that all data nodes in the Yarn cluster are traversed or it is determined that the candidate data node is deployed on the same physical machine node as the client. It can be understood that if all data nodes are traversed and no data node is found to be located on the same physical machine node as the client, other strategies can be used to select a data node, such as: a data node can be randomly selected to read and write the data to be processed, or a data node on the physical machine node closest to the client can be selected to read and write the data to be processed.

[0053] The technical solution provided in the embodiment introduces a new application programming interface (API) resource (that is, a YarnPod resource) and a new controller (that is, a Yarn deployment) to complete corresponding PV mounting and environment variable injection, avoids the node manager from relating to the underlying PV, and reduces the workload of the node manager; moreover, the introduction of the Kubernetes controller can complete the mounting of multiple Pods on the same physical machine node, and the use of the HostPath mode is avoided. , The sharing of Java archive (Jar) packages and other resources avoids the use of the HostPath mode, helps to improve the security and isolation of data reading and writing, and improves the performance of the program; in addition, when reading and writing HDFS data, the resources of the Pod on the same physical machine node can be selected for reading according to the environment variable, the performance of the program is improved, the performance of processing distributed tasks can be improved in the hybrid deployment scheme of Yarn and Kubernetes, and the use of local data is preferred, the data locality in the Kubernetes environment is realized, and the bandwidth cost can be saved in the use scenario of a private cloud.

[0054] Please refer to Figure 4 , Figure 4 is a structural schematic diagram of an embodiment of the task hybrid scheduling system provided in the application. The task hybrid scheduling system 40 includes a memory 41 and a processor 42 connected to each other. The memory 41 is used to store a computer program. When the computer program is executed by the processor 42, the computer program is used to implement the task hybrid scheduling method in the above embodiments.

[0055] Please refer to Figure 5, Figure 5 FIG. 1 is a structural schematic diagram of an embodiment of a computer readable storage medium provided in the present application, the computer readable storage medium 50 is used to store a computer program 51, the computer program 51 is used to implement the task mixed scheduling method in the above embodiment when executed by a processor.

[0056] The computer readable storage medium 50 can be a server, a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and the like various media that can store program codes.

[0057] In several embodiments provided in the present application, it should be understood that the disclosed method and device can be implemented by other ways. For example, the above described device embodiment is only schematic, for example, the division of the module or unit is only a logical function division, and actual implementation can have another division way, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0058] The unit described as a separate component can be or can not be physically separated, and the component displayed as a unit can be or can not be a physical unit, that is, it can be located in one place, or can be distributed to a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the present embodiment scheme.

[0059] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0060] The above is only an embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent process transformation using the content of the present application specification and drawings, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A mixed task scheduling method, characterized in that: include: Get the tasks to be scheduled; Create a Kubernetes controller and a YARN container group resource, use the Kubernetes controller to process the YARN container group resource to obtain a container group resource, and the task to be scheduled corresponds to the YARN container group resource; wherein the YARN container group resource includes a comment field; use the Kubernetes controller to parse the comment field in the YARN container group resource to obtain a first prefix field and a second prefix field; obtain the address information of the data node container in the current environment based on the value of the first prefix field, parse the second prefix field to obtain the PV prefix mounted by the container group resource; create the container group resource based on the address information and the PV prefix; The Kubernetes controller is used to schedule the container group resources to schedule the container group resources to a preset physical machine node, on which a data node container group is deployed. The data node container group is a container group served by a data node in an HDFS cluster.

2. The task hybrid scheduling method according to claim 1, characterized in that: The step of using the Kubernetes controller to process the Yarn container group resources to obtain the container group resources includes: The Kubernetes controller is used to parse the monitored Yarn container group resources to obtain parsed data; The container group resource is created based on the parsed data.

3. The task hybrid scheduling method according to claim 2, characterized in that: The YARN container group resource includes a comment field, the parsed data includes a first prefix field and a second prefix field, the first prefix field is a name prefix of the container group served by the data node, and the second prefix field is a prefix of a persistent storage volume mounted by the container group resource, and the method further includes: Parsing the comment field to obtain the first prefix field and the second prefix field; Acquire address information of the data node container group based on the first prefix field; Injecting the address information into the container group resource through environment variables; Parse the second prefix field to obtain a persistent storage volume prefix, where the persistent storage volume prefix is ​​the prefix of the persistent storage volume mounted by the container group resource; In combination with the load balancing strategy, a persistent storage volume with the same prefix as the persistent storage volume is selected from the persistent storage volume resources, so that the container group resources running on the preset physical machine node are mounted to the same persistent storage volume.

4. The task hybrid scheduling method according to claim 1, characterized in that: The container group resource includes at least one container group, and the method further includes: Determine whether the ratio of the number of physical machine nodes in the Kubernetes cluster to the number of the container groups is greater than a preset value; If yes, all container groups corresponding to the tasks to be scheduled are scheduled to the preset physical machine nodes.

5. The task hybrid scheduling method according to claim 1, characterized in that: The HDFS cluster further includes a client, and the container group resources are used to run the client. The method further includes: The client is used to select a data node from all the data nodes based on the environment variables to read and / or write the data to be processed.

6. The task hybrid scheduling method according to claim 5, characterized in that: The step of selecting a data node from all the data nodes to read and / or write the data to be processed includes: Selecting a data node from all the data nodes as a candidate data node; Using the client to determine, through the environment variable, whether the acquired candidate data node is deployed on the same physical machine node as the client; If yes, then using the candidate data node to read and / or write the data to be processed; If not, return to the step of selecting a data node from all the data nodes as a candidate data node until a preset selection end condition is met.

7. The task hybrid scheduling method according to claim 3, characterized in that: The method further comprises: Determine whether the persistent storage volume exists; If not, create the persistent storage volume; The container group resource is created based on the address information and the persistent storage volume.

8. The task hybrid scheduling method according to claim 1, characterized in that: The steps for creating the Kubernetes controller and YARN container group resources include: The Yarn container group resource is created using the node manager in the Yarn cluster.

9. A mixed task scheduling system, characterized in that: The system comprises a memory and a processor connected to each other, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, it is used to implement the task mixing scheduling method according to any one of claims 1 to 8.

10. A computer-readable storage medium for storing a computer program, characterized in that: When the computer program is executed by a processor, it is used to implement the task mixing scheduling method according to any one of claims 1 to 8.

Images