Job scheduling method, system and apparatus, and communication device and storage medium

By receiving the preset job programs by users and determining the running location based on the execution step type and resource consumption, combining the scheduling strategies of independent task manager nodes and Yarn clusters, the problem of resource competition in the Yarn mode of Flink jobs is solved, and heterogeneity is achieved Deployment and efficient use of resources.

WO2024234638A1PCT designated stage expired Publication Date: 2024-11-21INSPUR SUZHOU INTELLIGENT TECH CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2023/139241
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-18
Filing Date
2023-12-15
Publication Date
2024-11-21

AI Technical Summary

Technical Problem

In Yarn mode, Flink jobs are inconsistent in operation location of TaskManager, resulting in serious resource competition, affecting latency and stability, and cannot effectively schedule steps that rely on specific resources.

Method used

By receiving preset job programs from users, generating execution plans, and determining their operating location based on the type of execution steps and resource consumption, using independent task manager nodes or Yarn clusters for scheduling, introducing node status trackers and execution step allocators to Dynamically adjust nodes and scheduling policies.

Benefits of technology

The Flink jobs fully utilize heterogeneous cluster resources, reduce the risk of performance bottlenecks, and improve the flexibility and stability of the job.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2023139241_21112024_PF_FP_ABST
    Figure CN2023139241_21112024_PF_FP_ABST
Patent Text Reader

Abstract

The embodiments of the present application provide a job scheduling method, system and apparatus, and a communication device and a non-volatile readable storage medium. The method comprises: receiving a preset job program sent by a user, wherein the preset job program comprises a preset independent task manager mark; generating an execution plan according to the preset job program, wherein the execution plan comprises execution steps; determining operation positions of the execution steps according to the types of the execution steps, wherein the types of the execution steps are generated according to the independent task manager mark; acquiring resource consumption corresponding to the execution steps; and determining target nodes corresponding to the execution steps according to the resource consumption and the operation positions of the execution steps, and starting the execution steps. In the present application, by combining a Yarn cluster and an independent task manager node, and determining an operation position of an execution step according to the type of the execution step, a flexible resource sharing characteristic of Yarn cluster scheduling is retained, and a resource-sensitive-type calculation step can use sufficient resources of the independent task manager node.
Need to check novelty before this filing date? Find Prior Art

Description

Job scheduling method, system, device, communication equipment and storage medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese patent application filed with the China Patent Office on May 18, 2023, with application number 202310563072.6 and application name “Job Scheduling Method, System, Device, Communication Equipment and Storage Medium”, all contents of which are incorporated by reference into this application. Technical Field

[0003] The present application relates to the field of data processing technology, and in particular to a job scheduling method, system, apparatus, communication equipment, and non-volatile readable storage medium. Background Art

[0004] Flink is a distributed streaming data computing engine, while Yarn is a scheduling framework for the big data Hadoop (distributed computing) system, responsible for unified management of cluster resources. Traditionally, Flink supports either standalone mode or Yarn mode. Standalone mode starts resident JobManager and TaskManager services on fixed nodes. These services and their nodes are fixed, resulting in limited flexibility.

[0005] Therefore, in order to better perform Flink jobs, the relevant technology generally uses the Yarn mode for jobs. However, the running location of the TaskManager in this mode is uncertain, and its lifecycle is closely related to the Flink job. Although it is flexible enough, the location of job scheduling is unpredictable. Therefore, it cannot solve the problem of job execution steps relying on specific resources. Even if the Yarn cluster allocates a dedicated queue for Flink jobs, some resources on the node, such as disk I / O and network bandwidth, will still be subject to fierce competition, seriously affecting the latency and stability of Flink jobs.

[0006] Summary of the Invention

[0007] The purpose of the embodiments of the present application is to provide a job scheduling method, system, apparatus, communication device, and non-volatile readable storage medium to solve the technical problem in the prior art of being unable to quickly obtain the actual server interface for the function required by the user. The specific technical solution is as follows:

[0008] In a first aspect of the present application, a job scheduling method is provided, which is applied to Yarn. The method includes:

[0009] receiving a preset operation program sent by a user, wherein the preset operation program includes a preset independent task manager mark;

[0010] Generate an execution plan according to a preset operation procedure, wherein the execution plan includes multiple execution steps;

[0011] Determining a running position of each execution step according to a type of each execution step, wherein the type of the execution step is generated according to a tag of an independent task manager;

[0012] Get the resource consumption corresponding to each execution step;

[0013] The target node corresponding to each execution step is determined based on resource consumption and the running location of each execution step, and the execution step is started.

[0014] In some embodiments, according to the job scheduling method of claim 1, the types of execution steps include: unbound mark execution steps and bound mark execution steps, and the bound mark execution steps are executed on the node corresponding to the independent task manager.

[0015] In some embodiments, determining the running location of each execution step according to the type of each execution step includes:

[0016] When it is detected that the execution step type is a binding tag execution step, the running location corresponding to the execution step is set to an independent task manager;

[0017] When it is detected that the execution step type is an unbound tag execution step, the running location corresponding to the execution step is set to the Yarn cluster.

[0018] In some embodiments, the resource consumption includes resource consumption corresponding to the unbound tag execution step and resource consumption corresponding to the bound tag execution step.

[0019] In some embodiments, determining the target node corresponding to each execution step based on resource consumption and the running location of each execution step includes:

[0020] When it is detected that the resource consumption meets the preset operating conditions, the target node corresponding to each execution step is determined according to the running position of each execution step.

[0021] In some embodiments, determining the target node corresponding to each execution step according to the running location of each execution step includes:

[0022] When it is detected that the running location corresponding to the execution step is set to the independent task manager, the target node corresponding to the execution step is the independent task manager node;

[0023] When it is detected that the running location corresponding to the execution step is set to a Yarn cluster, the target node corresponding to the execution step is any node in the Yarn cluster.

[0024] In some embodiments, Yarn includes a node status tracker and an execution step distributor. The node status tracker is used to maintain node information corresponding to the independent task manager, and the execution step distributor is used to schedule the independent task manager according to the node information corresponding to the independent task manager.

[0025] In some embodiments, after determining the execution location of each execution step according to the type of each execution step, the method includes:

[0026] The node status tracker receives heartbeats sent by independent task managers;

[0027] The node status tracker determines whether the independent task manager is in a failed state based on the heartbeat and sends node change information to the execution step distributor.

[0028] In some embodiments, after the step of determining, by the node status tracker, whether the independent task manager is in a failed state based on the heartbeat and sending node change information to the execution step distributor, the method includes:

[0029] When the node status tracker detects a change in the independent task manager node, the execution step distributor sends a node change notification to the work manager corresponding to the current job, so that the work manager changes the independent task manager node corresponding to the current job.

[0030] In some embodiments, determining the running location of each execution step according to the type of each execution step includes:

[0031] When it is detected that the execution step type is a binding tag execution step, the execution step allocator obtains the independent task manager node information corresponding to the current job from the node status tracker, matches the label corresponding to the binding tag execution step in the independent task manager node information, and generates the running location of the binding tag execution step.

[0032] In some embodiments, after determining the target node corresponding to each execution step based on resource consumption and the running location of each execution step and starting the execution step, the method includes:

[0033] When a target node is detected to be in a faulty state, if the target node is an independent task manager node, the execution steps corresponding to the target node are scheduled to a second target node with the same binding tag, or, based on a preset node scheduling policy, the execution steps corresponding to the target node are scheduled to any node in the Yarn cluster;

[0034] If the target node is a Yarn node, the target node will be removed from the Yarn cluster, and the corresponding execution steps on the target node will be scheduled to other nodes in the Yarn cluster.

[0035] In some embodiments, after the step of scheduling the execution step corresponding to the target node to any node in the YARN cluster based on a preset node scheduling policy, the method includes:

[0036] Stop executing the execution steps corresponding to the target node, and send the fault information corresponding to the target node to the user so that the user can perform maintenance on the target node.

[0037] In some embodiments, after determining the target node corresponding to each execution step based on resource consumption and the running location of each execution step and starting the execution step, the method includes:

[0038] Receive independent task manager node expansion instructions sent by users;

[0039] Create the target independent task manager node according to the independent task manager node expansion instruction;

[0040] Get the average number of execution steps corresponding to all independent task manager nodes;

[0041] The target independent task manager node is scheduled according to the average number of execution steps until the number of execution steps corresponding to the target independent task manager node reaches the average number of execution steps.

[0042] In some embodiments, after determining the target node corresponding to each execution step based on resource consumption and the running location of each execution step and starting the execution step, the method includes:

[0043] receiving an upgrade instruction for executing step scheduling sent by a user, wherein the upgrade instruction includes a target execution step;

[0044] According to the upgrade instructions and the preset node scheduling strategy, any node in the Yarn cluster where the target execution step is located is scheduled to an independent task manager node.

[0045] In some embodiments, after determining the target node corresponding to each execution step based on resource consumption and the running location of each execution step and starting the execution step, the method includes:

[0046] receiving a downgrade instruction for executing a step schedule sent by a user, wherein the downgrade instruction includes a target execution step;

[0047] According to the downgrade instructions and the preset node scheduling strategy, the independent task manager node where the target execution step is located is scheduled to any node in the Yarn cluster.

[0048] In another aspect of the present application, a job scheduling system is provided, the job scheduling system including an independent task manager node, a Yarn cluster node, and Yarn;

[0049] Communication connection between independent task manager nodes and Yarn cluster nodes;

[0050] The independent task manager node includes a pre-set independent task manager tag, and the independent task manager node is used to execute the binding tag execution step;

[0051] Yarn cluster nodes are used to execute unbound tag execution steps;

[0052] Yarn is used to receive a preset job program sent by a user, wherein the preset job program includes a pre-set independent task manager tag; generate an execution plan based on the preset job program, wherein the execution plan includes multiple execution steps; determine the running location of each execution step based on the type of each execution step, wherein the type of the execution step is generated based on the independent task manager tag; obtain the resource consumption corresponding to each execution step; determine the target node corresponding to each execution step based on the resource consumption and the running location of each execution step, and start the execution step.

[0053] In some embodiments, Yarn includes a node status tracker and an execution step dispatcher;

[0054] The node status tracker is used to maintain the node information corresponding to the independent task manager;

[0055] The execution step allocator is used to schedule the independent task manager according to the node information corresponding to the independent task manager.

[0056] In another aspect of the present application, a job scheduling device is provided, which is applied to a server management platform. The device includes:

[0057] A receiving module, configured to receive a preset operation program sent by a user, wherein the preset operation program includes a preset independent task manager mark;

[0058] A generation module, configured to generate an execution plan according to a preset operation procedure, wherein the execution plan includes a plurality of execution steps;

[0059] A first determining module is configured to determine a running position of each execution step according to a type of each execution step, wherein the type of the execution step is generated according to a tag of an independent task manager;

[0060] The acquisition module is used to obtain the resource consumption corresponding to each execution step;

[0061] The second determining module is used to determine the target node corresponding to each execution step according to resource consumption and the running location of each execution step, and start the execution step.

[0062] In another aspect of the present application, a communication device is provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0063] Memory for storing computer programs;

[0064] The processor is configured to implement any of the above-mentioned job scheduling methods when executing a program stored in the memory.

[0065] In another aspect of the present application, a computer non-volatile readable storage medium is provided, in which instructions are stored. When the computer non-volatile readable storage medium is run on a computer, the computer executes any of the above-mentioned job scheduling methods.

[0066] In another aspect of the implementation of the present application, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute any of the above-mentioned job scheduling methods.

[0067] The job scheduling method provided by the embodiment of the present application receives a preset job program sent by a user, wherein the preset job program includes a pre-set independent task manager tag; generates an execution plan according to the preset job program, wherein the execution plan includes multiple execution steps; determines the running location of each execution step according to the type of each execution step, wherein the type of the execution step is generated according to the independent task manager tag; obtains the resource consumption corresponding to each execution step; determines the target node corresponding to each execution step according to the resource consumption and the running location of each execution step, and starts the execution step. The embodiment of the present application combines the advantages of the Yarn cluster and the independent task manager node deployment form, determines the running location of each execution step according to the type of each execution step, that is, retains the flexibility and resource sharing characteristics of the Yarn cluster scheduling, and realizes that resource-sensitive computing steps can use sufficient resources of the independent task manager node, realizes the heterogeneous deployment capability of the Flink job cluster, and enables Flink jobs to fully utilize the resources of the heterogeneous cluster. By scheduling steps that rely on special resources to matching nodes for execution, resource utilization is maximized, and the risk of performance bottlenecks encountered in the stream computing process is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art.

[0069] FIG1 shows a flowchart of the steps of the job scheduling method provided in an embodiment of the present application;

[0070] FIG2 shows a second flowchart of the steps of the job scheduling method provided in an embodiment of the present application;

[0071] FIG3 shows a flowchart of the steps of the job scheduling method provided in an embodiment of the present application;

[0072] FIG4 shows a fourth step flow chart of the job scheduling method provided in an embodiment of the present application;

[0073] FIG5 shows a flowchart of the fifth step of the job scheduling method provided in an embodiment of the present application;

[0074] FIG6 shows a flowchart of the steps of the job scheduling method provided in an embodiment of the present application;

[0075] FIG7 shows a schematic diagram of a job scheduling system provided by an embodiment of the present application;

[0076] FIG8 shows a device block diagram of a job scheduling device provided in an embodiment of the present application;

[0077] FIG9 shows a structural block diagram of a communication device provided in an embodiment of the present application;

[0078] FIG10 shows a schematic diagram of a job scheduling provided by an embodiment of the present application;

[0079] FIG11 shows a flowchart of a job scheduling fault handling method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0080] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0081] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, each embodiment of the present application will be described in detail below with reference to the accompanying drawings. However, it will be understood by those skilled in the art that in each embodiment of the present application, many technical details are proposed in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solutions claimed in the present application can be implemented. The division of the following embodiments is for convenience of description and should not constitute any limitation on the specific implementation of the present application. The various embodiments can be combined with each other and referenced to each other under the premise of no contradiction.

[0082] 1 , a flowchart of the first step of a job scheduling method provided by an embodiment of the present application is shown. The method may include:

[0083] Step 101: receiving a preset operation program sent by a user, wherein the preset operation program includes a preset independent task manager mark;

[0084] It should be noted that in the embodiment of the present application, the entire job scheduling only requires binding the scheduling node mark to the place that requires special scheduling during programming. On the basis of realizing the function, the changes to the Flink programming API are very small.

[0085] Therefore, when writing a job program, in the execution steps that need to be scheduled to an independent TaskManager node, the TaskManager tag corresponding to this job stage is specified. That is, the user sends a pre-written job program to Yarn, and the preset job program includes the independent TaskManager tag pre-set by the user. Therefore, the scheduling position binding in the job planning stage is completed in step 101.

[0086] It should be noted that in the embodiment of the present application, the independent task manager mark is pre-set. In the operation program set by the user, the independent task manager mark can be set for the execution steps that may be included in the operation program, so that the mark can be used as the basis for dividing the execution step types.

[0087] Step 102: Generate an execution plan according to a preset operation procedure, wherein the execution plan includes a plurality of execution steps;

[0088] It should be noted that in the embodiment of the present application, during the stage of job submission and execution plan generation, an execution plan is generated according to a preset job program. The various execution steps in the preset job program can be divided into two categories: one category is execution steps that are not bound to a TaskManager tag. Execution steps that are not bound to a TaskManager tag can directly participate in the scheduling of the Yarn cluster and obey the corresponding scheduling rules according to the scheduler configured within the traditional Yarn cluster. The other category is execution steps that are bound to a TaskManager tag. Execution steps that are not bound to a TaskManager tag do not participate in the scheduling of the Yarn cluster, but are fixedly scheduled to the node where the independent TaskManager with the matching tag is located.

[0089] It should be noted that an independent task manager mark is pre-set in this application, that is, the running position of each execution step is determined according to whether it is bound to an independent task manager mark, wherein an unbound mark execution step represents that the execution step is not bound to an independent task manager mark, and a bound mark execution step represents that the execution step is bound to an independent task manager mark.

[0090] Furthermore, in order to implement an improved scheduling method, the present application introduces two modules, a node status tracker and an execution step distributor, into the Yarn cluster. For details, please refer to the subsequent description.

[0091] Step 103: determining the running position of each execution step according to the type of each execution step, wherein the type of the execution step is generated according to the mark of the independent task manager;

[0092] It should be noted that the execution plan generated in step 102 includes multiple execution steps, each of which can be assigned to different nodes for processing. In this application, the execution steps are divided into two types. Specifically, the types of execution steps include: unbound marked execution steps and bound marked execution steps.

[0093] The execution location of the execution step can be determined based on the type of the execution step. Specifically, the execution step with a binding tag is executed on the node corresponding to the independent task manager, and the execution step with an unbound tag can be executed in the Yarn cluster.

[0094] Furthermore, determining the running location of each execution step according to the type of each execution step includes: when it is detected that the type of the execution step is a bound mark execution step, setting the running location corresponding to the execution step to an independent task manager; when it is detected that the type of the execution step is an unbound mark execution step, setting the running location corresponding to the execution step to a Yarn cluster.

[0095] Step 104: Obtain resource consumption corresponding to each execution step.

[0096] Furthermore, the resource consumption includes resource consumption corresponding to the unbound tag execution step and resource consumption corresponding to the bound tag execution step.

[0097] It should be noted that resource consumption must be calculated for each execution step. Specifically, this calculation is divided into two types of job steps: Execution steps scheduled to Yarn cluster nodes are subject to the resource limits configured by the Yarn scheduler. The upper limit of each step's resources (number of CPU cores, memory capacity) must not exceed the limits of the Yarn queue in which it is located. The resources consumed by all execution steps running on Yarn, along with other jobs in the cluster, are centrally managed and uniformly allocated by the Yarn scheduler. Therefore, this portion of resource consumption is considered shared resource consumption. The other type of execution step is scheduled to an independent TaskManager node. Its resource limits are subject to the TaskManager's resource configuration. TaskManager resources are exclusively used by Flink jobs and are not shared with other jobs. Therefore, this portion of resource consumption is considered exclusive resource consumption.

[0098] Step 105 : determining the target node corresponding to each execution step according to resource consumption and the running location of each execution step, and starting the execution step.

[0099] It should be noted that in an embodiment of the present application, when the computing resource consumption can meet the operating conditions in step 104, the target node of the final deployment schedule is determined according to the bound mark execution steps and unbound mark execution steps allocated in the preset job program, and all execution steps are started.

[0100] It should be noted that the above steps 101-105 implement the entire Flink job scheduling and execution process.

[0101] In addition, referring to Figure 10, Figure 10 shows a job scheduling schematic diagram provided by an embodiment of the present application. For a specific example of an affinity scheduling process, there are two job processes, where the job process of job A is: step 1 in job A is bound to label A, and steps 2 and 3 are not bound to labels; the job process of job B is: step 1 in job B is bound to label A, step 2 is bound to label B, and steps 3 and 4 are not bound to labels; the scheduling results corresponding to the above two job processes are shown in Figure 10, label A is pre-set on independent task manager node 1, and label B is pre-set on independent task manager node 2. Therefore, step 1 in job A and step 1 in job B match independent task manager node 1, and step 2 in job B matches independent task manager node 2. The remaining steps are based on Yarn's resource allocation and match the nodes in the Yarn cluster on demand.

[0102] The job scheduling method provided by the embodiment of the present application receives a preset job program sent by a user, wherein the preset job program includes a pre-set independent task manager tag; generates an execution plan according to the preset job program, wherein the execution plan includes multiple execution steps; determines the running location of each execution step according to the type of each execution step, wherein the type of the execution step is generated according to the independent task manager tag; obtains the resource consumption corresponding to each execution step; determines the target node corresponding to each execution step according to the resource consumption and the running location of each execution step, and starts the execution step. The embodiment of the present application combines the advantages of the Yarn cluster and the independent task manager node deployment form, determines the running location of each execution step according to the type of each execution step, that is, retains the flexibility and resource sharing characteristics of the Yarn cluster scheduling, and realizes that resource-sensitive computing steps can use sufficient resources of the independent task manager node, realizes the heterogeneous deployment capability of the Flink job cluster, and enables Flink jobs to fully utilize the resources of the heterogeneous cluster. By scheduling steps that rely on special resources to matching nodes for execution, resource utilization is maximized, and the risk of performance bottlenecks encountered in the stream computing process is reduced.

[0103] 2 , a second flowchart of the steps of the job scheduling method provided in an embodiment of the present application is shown. The method may include:

[0104] Step 201: receiving a preset operation program sent by a user, wherein the preset operation program includes a preset independent task manager mark;

[0105] Step 202: Generate an execution plan according to a preset operation procedure, wherein the execution plan includes a plurality of execution steps;

[0106] Step 203: determining the running position of each execution step according to the type of each execution step, wherein the type of the execution step is generated according to the mark of the independent task manager;

[0107] Step 204: Obtain resource consumption corresponding to each execution step;

[0108] It should be noted that the above steps 201-204 are described with reference to the previous discussion and will not be repeated here.

[0109] Step 205 : When it is detected that the resource consumption meets the preset operating conditions, the target node corresponding to each execution step is determined according to the running position of each execution step.

[0110] It should be noted that in the embodiments of the present application, it is necessary to calculate the resource consumption of each execution step. Specifically, the resource consumption is calculated for two types of job steps: the execution steps scheduled to the Yarn cluster nodes are subject to the resource restrictions configured by the Yarn scheduler. The upper limit of each step's resources (number of CPU cores, memory capacity) must not exceed the limit of the Yarn queue. The resources consumed by all execution steps running on Yarn and other jobs in the cluster participate in the centralized management and unified allocation of the Yarn scheduler. Therefore, this part belongs to the calculation of shared resource consumption. The other type of execution steps are scheduled to independent TaskManager nodes. Its resource restrictions are subject to the resource configuration of TaskManager. TaskManager resources are exclusively used by Flink jobs and are not shared with other jobs. Therefore, this part belongs to the calculation of exclusive resource consumption.

[0111] Furthermore, in step 205, the target node corresponding to each execution step is determined according to the running location of each execution step, including: when it is detected that the running location corresponding to the execution step is set to an independent task manager, the target node corresponding to the execution step is the independent task manager node; when it is detected that the running location corresponding to the execution step is set to a Yarn cluster, the target node corresponding to the execution step is any node in the Yarn cluster.

[0112] It should be noted that, when computing resource consumption can meet the operating conditions, the target node for final deployment scheduling is determined according to the bound mark execution steps and unbound mark execution steps allocated in the preset job program, and all execution steps are started.

[0113] Specifically, the target node can be an independent task manager node or any node in the Yarn cluster.

[0114] The job scheduling method provided by the embodiment of the present application receives a preset job program sent by a user, wherein the preset job program includes a pre-set independent task manager tag; generates an execution plan according to the preset job program, wherein the execution plan includes multiple execution steps; determines the running location of each execution step according to the type of each execution step, wherein the type of the execution step is generated according to the independent task manager tag; obtains the resource consumption corresponding to each execution step; determines the target node corresponding to each execution step according to the resource consumption and the running location of each execution step, and starts the execution step. The embodiment of the present application combines the advantages of the Yarn cluster and the independent task manager node deployment form, determines the running location of each execution step according to the type of each execution step, that is, retains the flexibility and resource sharing characteristics of the Yarn cluster scheduling, and realizes that resource-sensitive computing steps can use sufficient resources of the independent task manager node, realizes the heterogeneous deployment capability of the Flink job cluster, and enables Flink jobs to fully utilize the resources of the heterogeneous cluster. By scheduling steps that rely on special resources to matching nodes for execution, resource utilization is maximized, and the risk of performance bottlenecks encountered in the stream computing process is reduced.

[0115] 3 , a flowchart of the third step of the job scheduling method provided in an embodiment of the present application is shown. The method may include:

[0116] Step 301: receiving a preset operation program sent by a user, wherein the preset operation program includes a preset independent task manager mark;

[0117] Step 302: Generate an execution plan according to a preset operation procedure, wherein the execution plan includes a plurality of execution steps;

[0118] Step 303: determining the running position of each execution step according to the type of each execution step, wherein the type of the execution step is generated according to the mark of the independent task manager;

[0119] Step 304: Obtain resource consumption corresponding to each execution step;

[0120] Step 305 : Determine the target node corresponding to each execution step according to resource consumption and the running location of each execution step, and start the execution step.

[0121] Yarn includes a node status tracker and an execution step distributor. The node status tracker is used to maintain the node information corresponding to the independent task manager, and the execution step distributor is used to schedule the independent task manager according to the node information corresponding to the independent task manager.

[0122] Step 306: The node status tracker receives the heartbeat sent by the independent task manager;

[0123] Step 307: The node status tracker determines whether the independent task manager is in a fault state based on the heartbeat, and sends node change information to the execution step distributor.

[0124] Step 308: When the node status tracker detects a change in the independent task manager node, the node state tracker executes the step distributor to send a node change notification to the work manager corresponding to the current job, so that the work manager changes the independent task manager node corresponding to the current job.

[0125] It should be noted that in the above steps 306-307, Yarn in this application introduces two modules: a node status tracker and an execution step distributor.

[0126] As for the node status tracker, the node status tracker is used to maintain the information of all independent TaskManager nodes and classify all independent TaskManager nodes according to their corresponding tags.

[0127] Specifically, the relationship between tags and nodes can be stored and classified, for example (TM is the abbreviation of TaskManager): {"Tag 1": ["TM1", "TM2"]}, {"Tag 2": ["TM3"]}.

[0128] In addition, the node status tracker receives the heartbeat sent by the independent TaskManager to detect whether it is alive. For example, if the heartbeat sent by an independent TaskManager is not received for more than the heartbeat interval * 30 time, it can be considered that the TaskManager node has failed.

[0129] In addition, the node status tracker is also responsible for responding to node changes, such as the addition and removal of nodes. When the node status tracker detects a node change, it can immediately notify the execution step distributor.

[0130] The execution step dispatcher is responsible for executing scheduling and responding to node changes. During scheduling, it retrieves the latest independent TaskManager node information from the node status tracker and queries the matching independent TaskManager node based on the label bound to the execution step. When scheduling an execution step bound to an independent TaskManager tag, the data output target of the upstream execution step and the input source of the downstream execution step data are set to this independent TaskManager node. In addition, other unbound nodes participate in Flink's original Yarn scheduling process. Furthermore, the execution step dispatcher saves the scheduled job information after scheduling is completed.

[0131] Specifically, the node status tracker and execution step dispatcher implement the following operations: If the node status tracker detects a change in the TaskManager, the execution step dispatcher notifies the job's JobManager and changes the node where the execution step running on that TaskManager is located. These changes include situations such as the target TaskManager node not existing (no TaskManager node with a matching tag can be found), there are multiple target TaskManager nodes, and a standalone TaskManager node fails while the job is running.

[0132] The embodiments of the present application combine the advantages of the Yarn cluster and independent task manager node deployment modes, and determine the running location of each execution step according to the type of each execution step. That is, the flexibility and resource sharing characteristics of the Yarn cluster scheduling are retained, and the sufficient resources of the independent task manager node are used for resource-sensitive computing steps. The heterogeneous deployment capability of the Flink job cluster is realized, so that Flink jobs can fully utilize the resources of the heterogeneous cluster. By scheduling steps that rely on specialized resources to matching nodes for execution, resource utilization is maximized, and the risk of performance bottlenecks in the stream computing process is reduced.

[0133] 4 , a fourth flowchart of the steps of the job scheduling method provided in an embodiment of the present application is shown. The method may include:

[0134] Step 401: receiving a preset operation program sent by a user, wherein the preset operation program includes a preset independent task manager mark;

[0135] Step 402: Generate an execution plan according to a preset operation procedure, wherein the execution plan includes a plurality of execution steps;

[0136] Step 403: When it is detected that the execution step type is a binding tag execution step, the execution step allocator obtains the independent task manager node information corresponding to the current job from the node status tracker, matches the tag corresponding to the binding tag execution step in the independent task manager node information, and generates the running location of the binding tag execution step, wherein the execution step type is generated according to the independent task manager tag.

[0137] It should be noted that in the embodiment of the present application, the execution step allocator is responsible for executing scheduling and responding to node changes. When executing scheduling, the latest independent TaskManager node information is retrieved from the node status tracker, and the matching independent TaskManager node is queried based on the label bound to the execution step. When scheduling an execution step marked with an independent TaskManager, its upstream execution step data output target and downstream execution step data input source are set to this independent TaskManager node. In addition, other unbound nodes participate in Flink's original Yarn scheduling process. In addition, the execution step allocator saves the scheduled job information after the scheduling is completed.

[0138] Step 404: Obtain resource consumption corresponding to each execution step;

[0139] Step 405 : Determine the target node corresponding to each execution step according to resource consumption and the running location of each execution step, and start the execution step.

[0140] Step 406: If the target node is detected to be in a faulty state, if the target node is an independent task manager node, the execution step corresponding to the target node is scheduled to a second target node with the same binding tag, or, based on a preset node scheduling policy, the execution step corresponding to the target node is scheduled to any node in the YARN cluster.

[0141] Step 407: If the target node is a Yarn node, the target node is removed from the Yarn cluster, and the corresponding execution steps on the target node are scheduled to other nodes in the Yarn cluster.

[0142] Furthermore, after step 406, if the target node is an independent task manager node, the execution steps corresponding to the target node may be stopped, and the fault information corresponding to the target node may be sent to the user so that the user can perform maintenance on the target node.

[0143] It should be noted that in the embodiment of the present application, steps 406-407, during the execution of the step startup operation, a node failure may occur, resulting in job failure. Specifically, referring to Figure 11, Figure 11 shows a job scheduling fault handling flow chart provided in the embodiment of the present application. It can be seen from the figure that when it is detected that the target node is in a faulty state, it is divided into two situations.

[0144] One is a node failure in the Yarn cluster. In this case, the failed node needs to be removed from the cluster, and then the job steps originally running on the failed node need to be transferred to other healthy nodes according to the scheduling rules. Finally, the job execution is resumed from the most recent successful checkpoint.

[0145] The other is a failure of an independent TaskManager node. Specifically, there are two situations:

[0146] Case 1: There are multiple independent TaskManager nodes matching the tag, and besides the failed node, there are other nodes matching the tag. In this case, the execution step scheduled to this independent TaskManager is rescheduled to another independent TaskManager node with the same tag. The execution step dispatcher modifies the upstream execution step data output target and downstream execution step output input source of all execution steps running on this independent TaskManager to another node matching the tag.

[0147] Case 2: The standalone TaskManager node matching the tag is unique and cannot be replaced after a failure. In this case, the scheduling mitigation strategy described in the previous section is used. With the node scheduling degradation strategy, the job steps on the failed node are rescheduled to the Yarn cluster. With the rejection strategy, the entire job fails and stops running.

[0148] It should be noted that in the embodiment of the present application, the preset node scheduling strategy is a node scheduling mitigation strategy, which specifically means that according to the user's needs or the current operation scenario requirements, the target node where the execution steps of the required operation are located can be replaced, for example, node scheduling downgrade or node scheduling upgrade.

[0149] In addition, it should be noted that when the target node where the execution steps of the required operation are located is replaced according to the preset node scheduling strategy, at this time, if the original independent manager node is replaced with a node in the Yarn cluster, it is necessary to determine any node in the Yarn cluster as the new target node according to the pre-set rules in the Yarn cluster, such as resource allocation on demand, or the user specifies any node in the Yarn cluster as the new target node.

[0150] The embodiments of the present application combine the advantages of the Yarn cluster and independent task manager node deployment modes, and determine the running location of each execution step according to the type of each execution step. That is, the flexibility and resource sharing characteristics of the Yarn cluster scheduling are retained, and the resource-sensitive computing steps can use the sufficient resources of the independent TaskManager node. The heterogeneous deployment capability of the Flink job cluster is realized, so that the Flink job can fully utilize the resources of the heterogeneous cluster. By scheduling the steps that rely on specialized resources to the matching nodes for execution, the resource utilization is maximized and the risk of performance bottlenecks in the stream computing process is reduced.

[0151] In addition, by setting a load balancing fault recovery strategy for hybrid deployment and scheduling methods, independent TaskManager nodes have both load balancing and fault backup functions, flexibly utilizing the resources of independent TaskManager nodes while reducing the risk of job interruption when independent TaskManager nodes fail. Independent TaskManager nodes have both load balancing and fault backup functions, improving the availability and fault tolerance of the cluster.

[0152] 5 , a flowchart of the fifth step of the job scheduling method provided by an embodiment of the present application is shown. The method may include:

[0153] Step 501: receiving a preset operation program sent by a user, wherein the preset operation program includes a preset independent task manager mark;

[0154] Step 502: Generate an execution plan according to a preset operation procedure, wherein the execution plan includes a plurality of execution steps;

[0155] Step 503: determining the running position of each execution step according to the type of each execution step, wherein the type of the execution step is generated according to the mark of the independent task manager;

[0156] Step 504: Obtain resource consumption corresponding to each execution step;

[0157] Step 505 : Determine the target node corresponding to each execution step according to resource consumption and the running location of each execution step, and start the execution step.

[0158] Step 506: Receive the independent task manager node expansion instruction sent by the user;

[0159] Step 507: Create a target independent task manager node according to the independent task manager node expansion instruction;

[0160] Step 508: Obtain the average number of execution steps corresponding to all independent task manager nodes;

[0161] Step 509 : Schedule the target independent task manager node according to the average number of execution steps until the number of execution steps corresponding to the target independent task manager node reaches the average number of execution steps.

[0162] It should be noted that in steps 506-509, regarding the expansion of independent TaskManager nodes, for example, during the job process, the user finds that memory resources and CPU resources need to be expanded in order to better complete the job. Therefore, the independent TaskManager node can be expanded, that is, a target independent task manager node (that is, the newly expanded node) is created. For the newly expanded node, there is no task running, and the original node may already have some task execution steps running on it. Therefore, at this time, it is necessary to calculate the average number of execution steps running on the original independent TaskManager node, and schedule the target independent task manager node according to the average value.

[0163] Specifically, the execution steps bound to the scheduling tag in the newly submitted task (new job) are scheduled to the newly expanded node first, until the number of execution steps running on the newly expanded node reaches the previously calculated average value, and then the polling scheduling of the independent task manager is restored.

[0164] It's important to note that the purpose of evenly distributing jobs by the number of execution steps rather than by resource usage is to minimize the impact of node failures on jobs. Flink is a stream computing engine, and stream computing jobs are unique in that if a single step is interrupted, the entire process will cease to function properly. If evenly distributing jobs by resource usage were used, a large number of execution steps with low resource usage might be assigned to the same node, while other nodes would only have a small number of execution steps with high resource usage. In this case, if one node crashes, even with fault response measures, a significant portion of the jobs would be affected.

[0165] In addition to scaling out nodes, you can also scale down nodes. Specifically, scaling down independent TaskManager nodes can adopt the same handling strategy as the independent TaskManager node failure described above, namely, downgrading or stopping execution. However, when scaling down nodes, if only one independent TaskManager node with a certain tag remains, scaling down further will result in the rescheduling of these execution steps to the Yarn cluster or job failure feedback.

[0166] The embodiments of the present application combine the advantages of the Yarn cluster and independent task manager node deployment modes, and determine the running location of each execution step according to the type of each execution step. That is, the flexibility and resource sharing characteristics of the Yarn cluster scheduling are retained, and the resource-sensitive computing steps can use the sufficient resources of the independent TaskManager node. The heterogeneous deployment capability of the Flink job cluster is realized, so that the Flink job can fully utilize the resources of the heterogeneous cluster. By scheduling the steps that rely on specialized resources to the matching nodes for execution, the resource utilization is maximized and the risk of performance bottlenecks in the stream computing process is reduced.

[0167] 6 , a flowchart of the fifth step of the job scheduling method provided in an embodiment of the present application is shown. The method may include:

[0168] Step 601: receiving a preset operation program sent by a user, wherein the preset operation program includes a preset independent task manager mark;

[0169] Step 602: Generate an execution plan according to a preset operation procedure, wherein the execution plan includes a plurality of execution steps;

[0170] Step 603: determining the running location of each execution step according to the type of each execution step, wherein the type of the execution step is generated according to the mark of the independent task manager;

[0171] Step 604: Obtain resource consumption corresponding to each execution step;

[0172] Step 605 : Determine the target node corresponding to each execution step according to resource consumption and the running location of each execution step, and start the execution step.

[0173] Step 606: Receive an upgrade instruction for executing step scheduling sent by the user, wherein the upgrade instruction includes a target execution step;

[0174] Step 607: According to the upgrade instruction and the preset node scheduling policy, any node in the Yarn cluster where the target execution step is located is scheduled to an independent task manager node.

[0175] It should be noted that in this embodiment of the present application, the scheduling location can be changed based on the upgrade command sent by the user, that is, whether the execution step is located in the Yarn cluster or on the independent TaskManager node. The process of upgrading from the Yarn cluster to the independent TaskManager node is called upgrading, and the scheduling process of downgrading from the independent TaskManager node to the Yarn cluster is called downgrading. The upgrade and downgrading logic reuses the previous fault recovery logic, stopping the execution step that needs to be upgraded and then resuming the execution step on the qualified node according to the new scheduling rules.

[0176] Furthermore, after step 605, a downgrade instruction for execution step scheduling sent by the user can also be received, including: receiving a downgrade instruction for execution step scheduling sent by the user, wherein the downgrade instruction includes the target execution step; according to the downgrade instruction and the preset node scheduling strategy, scheduling the independent task manager node where the target execution step is located to any node in the Yarn cluster.

[0177] Similarly, for downgrade, the execution steps that need to be downgraded are stopped, and then the execution steps are resumed on eligible nodes according to the new scheduling rules.

[0178] The embodiments of the present application combine the advantages of the Yarn cluster and independent task manager node deployment modes, and determine the running location of each execution step according to the type of each execution step. That is, the flexibility and resource sharing characteristics of the Yarn cluster scheduling are retained, and the resource-sensitive computing steps can use the sufficient resources of the independent TaskManager node. The heterogeneous deployment capability of the Flink job cluster is realized, so that the Flink job can fully utilize the resources of the heterogeneous cluster. By scheduling the steps that rely on specialized resources to the matching nodes for execution, the resource utilization is maximized and the risk of performance bottlenecks in the stream computing process is reduced.

[0179] In addition, load balancing failure recovery strategies and scheduling upgrades and downgrades are set for hybrid deployment and scheduling methods. This allows for flexible utilization of independent TaskManager node resources while reducing the risk of job interruption when an independent TaskManager node fails.

[0180] 7 , a system diagram of a job scheduling system according to an embodiment of the present application is shown. The job scheduling system includes an independent task manager node, a Yarn cluster node, and Yarn;

[0181] Communication connection between independent task manager nodes and Yarn cluster nodes;

[0182] The independent task manager node includes a pre-set independent task manager tag, and the independent task manager node is used to execute the binding tag execution step;

[0183] Yarn cluster nodes are used to execute unbound tag execution steps;

[0184] Yarn is used to receive a preset job program sent by a user, wherein the preset job program includes a pre-set independent task manager tag; generate an execution plan based on the preset job program, wherein the execution plan includes multiple execution steps; determine the running location of each execution step based on the type of each execution step, wherein the type of the execution step is generated based on the independent task manager tag; obtain the resource consumption corresponding to each execution step; determine the target node corresponding to each execution step based on the resource consumption and the running location of each execution step, and start the execution step.

[0185] Furthermore, Yarn includes a node status tracker and an execution step distributor; the node status tracker is used to maintain node information corresponding to the independent task manager; the execution step distributor is used to schedule the independent task manager according to the node information corresponding to the independent task manager.

[0186] It should be noted that in an embodiment of the present application, the job scheduling system includes a hybrid deployment architecture that uses both an independent TaskManager and a floating TaskManager in Yarn. The independent TaskManager runs on a specific resource-intensive node and runs as a resident service. The floating TaskManager accepts the scheduling of Yarn and is started on demand when the job runs. In a Flink job, steps that need to be specified to run on a specific node will be scheduled to the node where the independent TaskManager is located during runtime. Other steps are run using Yarn's floating TaskManager. After the job is completed, the floating TaskManager in Yarn automatically stops running and returns resources to the cluster. The independent mode TaskManager on a specific node always keeps running. It accepts the scheduling of other jobs.

[0187] The job scheduling system in the embodiment of the present application divides schedulable nodes into two types. Nodes with bound tags and unbound nodes. Among them, the nodes with bound tags need to specify the tags of the specific nodes bound, which can be used for Flink job execution steps to specify tags. Job steps with specified tags will be scheduled on nodes with binding tags that match the tags. Unbound steps are resource-insensitive nodes and accept Yarn's unified scheduling. When writing Flink jobs, specific operators (also known as operators, or job steps) can be bound to node tags. When the job scheduling is running, these two types of job steps are distinguished and scheduled separately. In this way, independent resource configuration can be achieved. The cluster resources that can be used by the TaskManager in independent mode are specially configured in the TaskManager, and the TaskManager in floating mode is still managed by Yarn to manage resources. It can achieve classified management and control of different types of node resources, avoid performance bottlenecks in job steps with large resource requirements and waste of resources in other steps, and even avoid the impact of monitoring of abnormal resource consumption.

[0188] In the hybrid architecture of this application, the resources of the independent TaskManager node are exclusive to the Flink job, but not exclusive to the execution steps of a certain Flink job. Other Flink jobs can also be scheduled to run on this node if they are configured with the same tag. There can be multiple independent TaskManager nodes with the same tag. In this way, when multiple steps bound to the same tag need to be scheduled, the execution step distributor will distribute these execution steps to all independent TaskManager nodes that match the tag in a round-robin manner. Make full use of the resources of these nodes. At the same time, risk sharing is achieved when a node fails (nodes that match other tags are equivalent to backup nodes).

[0189] Referring to FIG8 , FIG8 shows a job scheduling device provided by an embodiment of the present application, which may include:

[0190] The receiving module 801 is configured to receive a preset operation program sent by a user, wherein the preset operation program includes a preset independent task manager mark;

[0191] A generating module 802 is used to generate an execution plan according to a preset operation procedure, wherein the execution plan includes a plurality of execution steps;

[0192] A first determining module 803 is configured to determine a running position of each execution step according to a type of each execution step, wherein the type of the execution step is generated according to a tag of the independent task manager;

[0193] An acquisition module 804 is used to acquire resource consumption corresponding to each execution step;

[0194] The second determining module 805 is configured to determine the target node corresponding to each execution step according to resource consumption and the running location of each execution step, and start the execution step.

[0195] The embodiments of the present application combine the advantages of the Yarn cluster and independent task manager node deployment modes, and determine the running location of each execution step according to the type of each execution step. That is, the flexibility and resource sharing characteristics of the Yarn cluster scheduling are retained, and the resource-sensitive computing steps can use the sufficient resources of the independent TaskManager node. The heterogeneous deployment capability of the Flink job cluster is realized, so that the Flink job can fully utilize the resources of the heterogeneous cluster. By scheduling the steps that rely on specialized resources to the matching nodes for execution, the resource utilization is maximized and the risk of performance bottlenecks in the stream computing process is reduced.

[0196] The embodiment of the present application further provides a communication device, as shown in FIG9 , including a processor 901 , a communication interface 902 , a memory 903 and a communication bus 904 , wherein the processor 901 , the communication interface 902 , and the memory 903 communicate with each other via the communication bus 904 .

[0197] Memory 903, used for storing computer programs;

[0198] When the processor 901 is used to execute the program stored in the memory 903, it can implement the following steps:

[0199] receiving a preset operation program sent by a user, wherein the preset operation program includes a preset independent task manager mark;

[0200] Generate an execution plan according to a preset operation procedure, wherein the execution plan includes multiple execution steps;

[0201] Determining a running position of each execution step according to a type of each execution step, wherein the type of the execution step is generated according to a tag of an independent task manager;

[0202] Get the resource consumption corresponding to each execution step;

[0203] The target node corresponding to each execution step is determined based on resource consumption and the running location of each execution step, and the execution step is started.

[0204] The communication bus mentioned in the terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not mean that there is only one bus or only one type of bus.

[0205] The communication interface is used for communication between the above terminal and other devices.

[0206] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage. In some embodiments, the memory may also be at least one storage device located away from the processor.

[0207] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0208] In another embodiment provided by the present application, a computer non-volatile readable storage medium is also provided, in which instructions are stored. When the instructions are executed on a computer, the computer executes the job scheduling of any one of the above embodiments.

[0209] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any one of the job scheduling in the above embodiments.

[0210] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable devices. The computer instructions can be stored in a non-volatile computer readable storage medium, or transmitted from a non-volatile computer readable storage medium to another non-volatile computer readable storage medium. For example, the computer instructions can be transmitted from a website, a computer, a server or a third database by wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode to another website, computer, server or third database. The non-volatile computer readable storage medium can be any available medium that a computer can access or a data storage device such as a server or a third database that includes one or more available media integrations. Available media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, DVDs), or semiconductor media (eg, solid state disks (SSDs)).

[0211] It should be noted that, in this document, relational terms such as first and first are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0212] The various embodiments in this specification are described in a related manner. Similar portions between the various embodiments can be referenced to each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so their description is relatively simple. For related portions, reference can be made to the description of the method embodiments.

[0213] The above are only preferred embodiments of the present application and are not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application are included in the scope of protection of the present application.

Claims

1. A job scheduling method, characterized in that: Applied to Yarn, the method includes: Receiving a preset operation program sent by a user, wherein the preset operation program includes a preset independent task manager mark; Generate an execution plan according to the preset operation procedure, wherein the execution plan includes a plurality of execution steps; Determining the running position of each of the execution steps according to the type of each of the execution steps, wherein the type of the execution step is generated according to the mark of the independent task manager; Obtaining resource consumption corresponding to each of the execution steps; The target node corresponding to each of the execution steps is determined according to the resource consumption and the running position of each of the execution steps, and the execution step is started.

2. The job scheduling method according to claim 1, characterized in that: The types of the execution steps include: unbound mark execution steps and bound mark execution steps, and the bound mark execution steps are executed on the node corresponding to the independent task manager.

3. The job scheduling method according to claim 2, characterized in that: Determining the running position of each of the execution steps according to the type of each of the execution steps includes: In the case where it is detected that the type of the execution step is a binding mark execution step, setting the running position corresponding to the execution step to the independent task manager; When it is detected that the type of the execution step is an unbound tag execution step, the running location corresponding to the execution step is set to a Yarn cluster.

4. The job scheduling method according to claim 1, characterized in that: The resource consumption includes resource consumption corresponding to the unbound tag execution step and resource consumption corresponding to the bound tag execution step.

5. The job scheduling method according to claim 4, characterized in that: Determining the target node corresponding to each of the execution steps according to the resource consumption and the running position of each of the execution steps includes: When it is detected that the resource consumption meets the preset operating condition, the target node corresponding to each of the execution steps is determined according to the operating position of each of the execution steps.

6. The job scheduling method according to claim 5, characterized in that: Determining the target node corresponding to each of the execution steps according to the running position of each of the execution steps includes: In the case where it is detected that the running location corresponding to the execution step is set to the independent task manager, the target node corresponding to the execution step is the independent task manager node; When it is detected that the running location corresponding to the execution step is set to a Yarn cluster, the target node corresponding to the execution step is any node in the Yarn cluster.

7. The job scheduling method according to claim 1, characterized in that: The Yarn includes a node status tracker and an execution step distributor. The node status tracker is used to maintain node information corresponding to the independent task manager, and the execution step distributor is used to schedule the independent task manager according to the node information corresponding to the independent task manager.

8. The job scheduling method according to claim 7, characterized in that: After the step of determining the target node corresponding to each of the execution steps according to the resource consumption and the running position of each of the execution steps, and starting the execution step, the method includes: The node status tracker receives the heartbeat sent by the independent task manager; The node status tracker determines whether the independent task manager is in a fault state according to the heartbeat, and sends node change information to the execution step distributor.

9. The job scheduling method according to claim 8, characterized in that: After the step of the node status tracker determining whether the independent task manager is in a fault state according to the heartbeat and sending node change information to the execution step distributor, the method includes: When the node status tracker detects a change in the independent task manager node, the execution step distributor sends a node change notification to the work manager corresponding to the current job, so that the work manager changes the independent task manager node corresponding to the current job.

10. The job scheduling method according to claim 1, characterized in that: Determining the running position of each of the execution steps according to the type of each of the execution steps includes: When it is detected that the execution step type is a binding mark execution step, the execution step allocator obtains the independent task manager node information corresponding to the current job from the node status tracker, matches the label corresponding to the binding mark execution step in the independent task manager node information, and generates the running location of the binding mark execution step.

11. The job scheduling method according to claim 1, characterized in that: After the step of determining the target node corresponding to each of the execution steps according to the resource consumption and the running position of each of the execution steps, and starting the execution step, the method includes: In the case where it is detected that the target node is in a faulty state, if the target node is an independent task manager node, the execution step corresponding to the target node is scheduled to the second target node with the same binding tag, or, based on a preset node scheduling strategy, the execution step corresponding to the target node is scheduled to any node in the Yarn cluster; If the target node is a Yarn node, the target node is removed from the Yarn cluster, and the corresponding execution steps on the target node are scheduled to other nodes in the Yarn cluster.

12. The job scheduling method according to claim 11, characterized in that: After the step of scheduling the execution step corresponding to the target node to any node in the Yarn cluster based on the preset node scheduling strategy, the method includes: Stop executing the execution step corresponding to the target node, and send the fault information corresponding to the target node to the user, so that the user can maintain the target node.

13. The job scheduling method according to claim 1, characterized in that: After the step of determining the target node corresponding to each of the execution steps according to the resource consumption and the running position of each of the execution steps, and starting the execution step, the method includes: Receive independent task manager node expansion instructions sent by users; Creating a target independent task manager node according to the independent task manager node expansion instruction; Obtain an average number of execution steps corresponding to all the independent task manager nodes; The target independent task manager node is scheduled according to the average number of execution steps until the number of execution steps corresponding to the target independent task manager node reaches the average number of execution steps.

14. The job scheduling method according to claim 1, characterized in that: After the step of determining the target node corresponding to each of the execution steps according to the resource consumption and the running position of each of the execution steps, and starting the execution step, the method includes: Receiving an upgrade instruction for executing step scheduling sent by a user, wherein the upgrade instruction includes a target execution step; According to the upgrade instruction and the preset node scheduling strategy, any node in the Yarn cluster where the target execution step is located is scheduled to an independent task manager node.

15. The job scheduling method according to claim 1, characterized in that: After the step of determining the target node corresponding to each of the execution steps according to the resource consumption and the running position of each of the execution steps, and starting the execution step, the method includes: receiving a downgrade instruction for scheduling an execution step sent by a user, wherein the downgrade instruction includes a target execution step; According to the downgrade instruction and the preset node scheduling strategy, the independent task manager node where the target execution step is located is scheduled to any node in the Yarn cluster.

16. A job scheduling system, characterized in that: The job scheduling system includes an independent task manager node, a Yarn cluster node, and Yarn; The independent task manager node is connected to the Yarn cluster node in a communication manner; The independent task manager node includes a pre-set independent task manager tag, and the independent task manager node is used to perform the binding tag execution step; The Yarn cluster node is used to execute the unbound tag execution step; The Yarn is used to receive a preset job program sent by a user, wherein the preset job program includes a preset independent task manager tag; generate an execution plan according to the preset job program, wherein the execution plan includes multiple execution steps; determine the running location of each execution step according to the type of each execution step, wherein the type of the execution step is generated according to the independent task manager tag; obtain the resource consumption corresponding to each execution step; determine the target node corresponding to each execution step according to the resource consumption and the running location of each execution step, and start the execution step.

17. The job scheduling system according to claim 16, characterized in that: The Yarn includes a node status tracker and an execution step distributor; The node status tracker is used to maintain the node information corresponding to the independent task manager; The execution step distributor is used to schedule the independent task manager according to the node information corresponding to the independent task manager.

18. A job scheduling device, characterized in that: The device comprises: A receiving module, used for receiving a preset operation program sent by a user, wherein the preset operation program includes a preset independent task manager mark; A generating module, used for generating an execution plan according to the preset operation procedure, wherein the execution plan includes a plurality of execution steps; A first determining module is used to determine the running position of each of the execution steps according to the type of each of the execution steps, Wherein, the type of the execution step is generated according to the independent task manager mark; An acquisition module, used to acquire resource consumption corresponding to each of the execution steps; The second determination module is used to determine the target node corresponding to each of the execution steps according to the resource consumption and the running position of each of the execution steps, and start the execution step.

19. A communication device, characterized in that: include: A transceiver, a memory, a processor, and a program stored on the memory and executable on the processor; The processor is used to read the program in the memory to implement the job scheduling method as described in any one of claims 1-15.

20. A non-volatile readable storage medium for storing a program, characterized in that: When the program is executed by a processor, the job scheduling method as described in any one of claims 1 to 15 is implemented.

Citation Information

Patent Citations

  • Data processing method and device

    CN108255619A

  • Task scheduling method, platform, apparatus, and computer-readable storage medium

    CN109117259A

  • Task execution method and computing device

    CN115729709A

  • Job scheduling method and device, electronic equipment and storage medium

    CN116126937A

  • Job scheduling method, system and device, communication equipment and storage medium

    CN116302452A