A distributed task scheduling method of a big data engine supporting stream batch integration

Abstract

The application discloses a kind of support flow batch integrated big data engine distributed task scheduling method, specifically related to computer field, including S1, multiple cluster access management;S2, construct workflow DAG process;S3, workflow instance backtracking and control and S4, job instance data processing feedback result audit.The application enables data developer to access multiple clusters by visual operation, realizes task distributed scheduling, reasonably and fully uses big data resources;By dragging and connecting multiple job components to build DAG workflow, merge data processing tasks, execute unified ETL period scheduling;Retain the DAG process instance of workflow execution, based on workflow instance realizes the dynamic control of re-deployment such as the re-run, pause, stop, resume failure, resume pause, resume stop of DAG process;Retain the instance of job, intuitively show the feedback result of big data task, facilitate data developer to troubleshoot abnormal program or task.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, more particularly, the present application relates to a kind of distributed task scheduling method of supporting stream batch integrated big data engine (stream batch integrated refers to using a set of logic to describe real-time and offline job, same job presents two characteristics of stream batch in different stages when running). BACKGROUND

[0002] Data lake is that enterprises use big data technology to collect, calculate and store internal and external massive data uniformly, and manage using unified data specifications, including data specifications, data models, metadata specifications, reference data standards, master data standards and business rules.Further, the broad sense of data middle platform also includes some technical components that have strong relevance to business, such as business tags, algorithm models and data products.The main role of data middle platform is to uniformly process all internal data to form standardized data, mine the most valuable data for the enterprise, build an enterprise data asset library, and provide consistent and highly available big data services internally and externally.

[0003] The big data engine task scheduling system is a core component in the data platform.In daily data processing, it is a common business requirement to run some jobs regularly, such as importing new data from the database to the data lake, exporting processed data from the data lake to the database or file system for use by business systems.

[0004] However, the existing ETL process is to write programs in corresponding languages by personnel (ETL: is the process of loading data from business systems to data warehouses after extraction, cleaning and conversion, the purpose is to integrate scattered, disorganized and non-uniform data in enterprises to provide analysis basis for enterprise decision-making and upper-layer application), and then execute the program in the cluster to realize the corresponding ETL logic, which produces a lot of inconvenience: 1. The way of personnel writing code cannot meet the high-speed iteration of business requirements in the era of big data, which is data-intensive, and pure manpower cannot meet the total demand due to excessive business requirements;2. When the requirements change, the code needs to be updated, which is tedious to operate;3. The closed nature of the code determines that the ETL information flow process will not be intuitive, and cannot be easily converted into a graphical relationship that people can quickly understand;4. The degree of platformization is low, the cluster management and business management are separated, which increases the operation difficulty and error rate;5. Different components need to be provided to serve different task modes, and the platform job integration degree is low;6. Different data sources cannot be processed with a unified architecture. SUMMARY

[0005] In order to overcome the above-mentioned defects of the prior art, embodiments of the present application provide a big data engine distributed task scheduling method supporting stream batch integration, and the technical problem to be solved by the present application is that existing ETL processes are all in the form of programs written by personnel in corresponding languages, and then the programs are executed in clusters to realize corresponding ETL logic, resulting in a large number of inconveniences.

[0006] In order to achieve the above-mentioned purpose, the present application provides the following technical scheme: a big data engine distributed task scheduling method supporting stream batch integration, including the following specific scheduling steps:

[0007] S1, multi-cluster access management:

[0008] Multi-cluster resource access is realized by configuring the parameters of hdfs-site, yarn-site or flink-site, and the cluster is specified for execution during job scheduling;

[0009] The user can divide multiple clusters into development environment, test environment, trial operation environment and production environment according to the user's own situation, and specify which environment to run when deploying the program;

[0010] When the cluster resources are insufficient, other clusters with more sufficient resources can be specified to run when deploying the program;

[0011] S2, constructing a workflow DAG process: a workflow is used to merge multiple data processing jobs / tasks in a visual drag-and-drop component and connection manner, and the construction of the workflow DAG process is realized;

[0012] S3, workflow instance backtracking and control: the DAG process instance of ETL data processing based on the workflow instance is permanently retained, the workflow instance is generated from the scheduling and execution of the workflow, and the DAG process instance of the workflow running at a certain time is recorded;

[0013] S3.1, workflow instance editing: workflow instance editing is applied to optimization and correction when the DAG process fails, stops, pauses or ends, and the editing object is the terminated workflow instance, and saving is divided into two cases of synchronizing the modified job to the original workflow and not synchronizing to the original workflow;

[0014] S3.2, workflow instance re-run: workflow instance re-run is applied to the terminated workflow instance of the DAG process, and the re-run object is the complete DAG process job node, even the successfully executed job node is also re-executed, and the workflow instance is unchanged but a new job instance is generated when re-running, and the running workflow instance cannot perform the re-run operation;

[0015] S3.3, Workflow instance stop: The workflow instance stop is applied to the workflow instance running in the DAG process, and a stop operation is performed on the running workflow instance;

[0016] S3.4, Workflow instance pause: The workflow instance pause is applied to the workflow instance running in the DAG process, and a pause operation is performed on the running workflow instance;

[0017] S3.5, Workflow instance recovery failure: The workflow instance recovery failure is applied to the workflow instance executing failure in the DAG process, and is the optimized, corrected and re-deployed of the executing failure workflow instance;

[0018] S3.6, Workflow instance recovery stop: The workflow instance recovery stop is applied to the stopped workflow instance in the DAG process, and the stopped workflow instance is re-deployed;

[0019] S3.7, Workflow instance recovery pause: The workflow instance recovery pause is applied to the paused workflow instance in the DAG process, and the paused workflow instance is re-deployed;

[0020] S4, Job instance data processing feedback result audit: The job instance is generated by the job scheduling in the workflow, and the instance is permanently retained for data processing. It is a direct manifestation of the feedback result of big data task scheduling, and data developers can troubleshoot abnormal tasks through this function.

[0021] In a preferred embodiment, in addition to the distributed scheduling management of the cluster, the function of multi-cluster access management in step S1 can also monitor the HDFS cluster and YARN cluster resources of the access platform in real time, including the node information of NameNode, DataNode, ResourceManager and NodeManager.

[0022] In a preferred embodiment, the job (data processing task) in step S2 is mainly based on YARN cluster resource computing, including but not limited to Flink program, Spark program, Hive program, HBase program and Kylin mainstream big data technology.

[0023] In a preferred embodiment, in step S3.1:

[0024] Synchronization to the original workflow: The current workflow DAG process instance is consistent with the workflow DAG process, and the subsequent workflow period scheduling continues to use the edited DAG process;

[0025] Synchronization to the original workflow: the workflow DAG process instance is inconsistent with the workflow DAG process, and the subsequent workflow cycle scheduling continues to use the edited DAG process.

[0026] In a preferred embodiment, the workflow instance re-running scenario in step S3.2 is that the first DAG process processes full data, and the subsequent DAG process processes incremental data.

[0027] In a preferred embodiment, the execution stop logic in step S3.3 is that the current job node continues to execute, and after the execution is completed, the subsequent job node cancels the execution.

[0028] The application scenario of stopping the workflow instance is that when the data developer finds that the program has an exception, the DAG process in which the program is located is blocked in time to avoid wasting resources and time. After the workflow instance is stopped, the data developer can edit, optimize and correct the DAG process.

[0029] In a preferred embodiment, the recovery logic in step S3.5 is to start execution from the failed job node, and when the recovery fails, the workflow instance does not change but a new job instance is generated. The successful offline job and the running real-time job do not generate a new workflow instance.

[0030] The application scenario of recovering the failed workflow instance is that the DAG process program exception causes the workflow execution to fail, and after the data developer edits, optimizes and corrects the DAG process, the data developer redeployments it.

[0031] In a preferred embodiment, the recovery logic in S3.7 is to start execution from the suspended job node, and the canceled real-time job and the unrun job generate a new job instance.

[0032] The application scenario of recovering the suspended workflow instance is similar to that of recovering the stopped workflow instance, that is, when the data developer finds that the program has an exception, the DAG process in which the program is located is blocked in time, and after the data developer edits, optimizes and corrects the DAG process, the data developer redeployments it.

[0033] Technical effects and advantages of the present application:

[0034] 1. The present application realizes distributed scheduling and execution of tasks in multiple clusters through a multi-cluster access platform, fully utilizes resources, and isolates production, testing and development environments; realizes execution of real-time computing tasks and batch computing tasks on a unified platform; and based on a workflow, supports task merging and unified scheduling of multiple job components such as Flink, Spark, Hive, HBase and Kylin.

[0035] 2、The application records the history instance of the workflow execution based on the workflow instance, realizes the backtracking audit of the workflow instance and the ability of task modification, correction and optimization, reduces the operation error rate of data processing; records the history instance of the execution of multiple jobs in the workflow based on the job instance, realizes the backtracking audit of the job instance and the risk investigation of the task, and reduces the abnormal rate of data processing. DETAILED DESCRIPTION

[0036] The technical solutions in the embodiments of the application will be described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative work belong to the protection scope of the application.

[0037] The application provides a distributed task scheduling method of a stream-batch integrated big data engine, including the following specific scheduling steps:

[0038] S1, multi-cluster access management:

[0039] With the continuous development of informatization, many enterprises pay more and more attention to the data level, which further creates the vigorous development of the big data industry. In most enterprises, the mainstream big data technology is the technology based on Hadoop ecology, including but not limited to HDFS, YARN, HBase, Kafka, Flink and Spark.

[0040] In the process of informatization business from demand to design, from design to research and development, from research and development to test, from test to trial operation, from trial operation to formal online, there are development environment, test environment, trial operation environment and production environment in different stages.

[0041] Similarly, the stage mode of data development also conforms to the mode of informatization business, so there will inevitably be development environment, test environment, trial operation environment and production environment.

[0042] Furthermore, big data is processed by improving resource occupancy and utilization to achieve the most efficient and maximum data processing. In general, the possibility of insufficient resources is relatively large when performing big data processing. In the case of business peak and program partial online in a hurry, it is unwise to increase the computing nodes of the cluster, which greatly affects other data business.

[0043] The application designs the following solutions to solve the above problems:

[0044] Access to resources of multiple clusters is achieved by configuring parameters such as hdfs-site, yarn-site or flink-site, and a cluster is specified for execution when a job (a big data processing task in a state of running, preparing to run, stopping running, pausing running, etc.) is scheduled;

[0045] A user can divide multiple clusters into development environments, test environments, trial running environments and production environments according to his / her own situation, and specify which environment to run when deploying a program;

[0046] Secondly, when the cluster resources are insufficient, another cluster with more sufficient resources can be specified to run when deploying a program;

[0047] In addition to distributed scheduling management of clusters, the HDFS cluster and YARN cluster resources accessed to the platform can also be monitored in real time, including node information of NameNode, DataNode, ResourceManager and NodeManager;

[0048] S2, build a workflow DAG process (workflow: a task flow composed of big data jobs, a plurality of minimum work granularity jobs are built into a workflow meeting business requirements, but no actual job is performed):

[0049] In the ETL data processing DAG process definition process, there is usually a data dependency problem from upstream data processing to downstream data processing, that is, the upstream data processing must be successfully executed before the downstream data processing can be executed; for example, there is a data dependency problem from data collection layer to ODS layer and from ODS layer to DWD layer, and the data of the data collection layer to the ODS layer must be successfully executed before the data of the ODS layer to the DWD layer can be executed; therefore, merging data processing tasks is the most important in data processing, and merging multiple tasks by visualizing the dragging and connecting of components is the core of the present application;

[0050] Multiple data processing jobs / tasks are merged by visualizing the dragging and connecting of components in the workflow to build a workflow DAG process;

[0051] Jobs (data processing tasks) are mainly based on YARN cluster resource computing big data tasks, including but not limited to Flink programs, Spark programs, Hive programs, HBase programs and Kylin mainstream big data technologies;

[0052] S3, workflow instance backtracking and control (workflow instance: when the workflow is confirmed, the data update frequency is re-deployed, the workflow is generated, and the actual business processing is performed):

[0053] In the process of ETL data processing task running, various program problems are always encountered at the beginning of deployment, such as Jar package packaging dependency conflict, code logic exception and the like; although the above problems can be solved by editing workflow, the operation logic is increased in difficulty and complexity; therefore, the optimization and correction of DAG process in ETL data processing are particularly important;

[0054] Based on the permanent retention of DAG process instance of ETL data processing based on workflow instance, the workflow instance is generated from the scheduling execution of workflow, and the DAG process instance of workflow running at a certain time is recorded; the workflow instance of workflow DAG process provides a variety of dynamic control means for data developers, which is convenient for agile optimization and correction of DAG process re-deployment, including workflow instance editing, re-running, pausing, stopping, recovering failure, recovering pause, recovering stop and the like; under normal circumstances, the DAG process of workflow instance and workflow is highly unified, only when the workflow instance is re-deployed, the DAG process of workflow instance and workflow will be inconsistent, of course, this is the result of continuous optimization and correction of the workflow instance with problems;

[0055] Based on the history instance of workflow execution recorded by workflow instance, the backtracking audit of workflow instance and the ability of task modification, correction and optimization are realized, and the operation error rate of data processing is reduced; based on the history instance of multiple job execution recorded by job instance, the backtracking audit of job instance and the risk investigation of task are realized, and the abnormal rate of data processing is reduced;

[0056] Dynamic control of workflow instance:

[0057] S3.1, workflow instance editing: workflow instance editing is applied to the optimization and correction of DAG process execution failure, stop, pause and end, the editing object is the terminated workflow instance, and the saving is divided into two cases of synchronizing the modified job to the original workflow and unsynchronizing to the original workflow;

[0058] Synchronization to the original workflow: the DAG process instance of this workflow is consistent with the DAG process of the workflow, and the subsequent workflow period scheduling continues to use the edited DAG process;

[0059] Different synchronization to the original workflow: the DAG process of this workflow is inconsistent with the DAG process of the workflow, and the subsequent workflow period scheduling continues to use the edited DAG process;

[0060] S3.2, Workflow instance rerun: Workflow instance rerun is applied to the workflow instance whose DAG process has terminated, the object of rerun is the complete DAG process job node, even the successfully executed job node is re-executed, when rerun, the workflow instance does not change but a new job instance is generated (job instance: generated by running the job node in the workflow, which is the direct manifestation of the feedback result of big data task scheduling), the running workflow instance cannot perform the rerun operation;

[0061] The common scenario of workflow instance rerun is that the first DAG process processes the full data, and the subsequent DAG processes process the incremental data;

[0062] S3.3, Workflow instance stop: Workflow instance stop is applied to the workflow instance whose DAG process is running, and the stop operation is performed on the running workflow instance;

[0063] The execution stop logic is that the current job node continues to execute, and after the execution is completed, the subsequent job node cancels the execution;

[0064] The application scenario of workflow instance stop is that when the data developer finds that the program has an exception, the DAG process in which the program is located is blocked in time to avoid wasting resources and time, and after the workflow instance is stopped, the data developer can edit, optimize and correct the DAG process;

[0065] S3.4, Workflow instance pause:

[0066] Workflow instance pause is applied to the workflow instance whose DAG process is running, and the pause operation is performed on the running workflow;

[0067] The execution logic is that the current job node and the subsequent job node cancel the execution;

[0068] The application scenario of workflow instance pause is similar to that of workflow instance stop, which is usually that when the data developer finds that the program has an exception, the DAG process in which the program is located is blocked in time to avoid wasting resources and time, and after the workflow instance is stopped, the data developer can edit, optimize and correct the DAG process;

[0069] S3.5, Workflow instance recovery failure: Workflow instance recovery failure is applied to the workflow instance whose DAG process execution fails, which is the re-deployment after optimization and correction of the failed workflow instance;

[0070] The recovery logic is to start execution from the failed job node, and when recovery fails, the workflow instance does not change but a new job instance is generated; the successfully executed offline job and the running real-time job do not generate a new workflow instance;

[0071] The application scenario of the workflow instance recovery failure is that the DAG flow program is abnormally caused the workflow execution failure, and the data developer edits and optimizes and corrects the DAG flow and then redeployments the same;

[0072] S3.6, the workflow instance recovery stop:

[0073] The workflow instance recovery stop is applied to the stopped workflow instance of the DAG flow, and the stopped workflow instance is redeployed;

[0074] The recovery logic is to start executing from the next node of the stopped job node, and the canceled real-time job and the unrun job generate a new job instance;

[0075] The application scenario of the workflow instance recovery stop is that the data developer blocks the DAG flow in which the program exists abnormally in time, edits and optimizes and corrects the DAG flow, and then redeployments the same;

[0076] S3.7, the workflow instance recovery pause: the workflow instance recovery pause is applied to the paused workflow instance of the DAG flow, and the paused workflow instance is redeployed;

[0077] The recovery logic is to start executing from the paused job node, and the canceled real-time job and the unrun job generate a new job instance;

[0078] The application scenario of the workflow instance recovery pause is similar to the workflow instance recovery stop, that is, the data developer blocks the DAG flow in which the program exists abnormally in time, edits and optimizes and corrects the DAG flow, and then redeployments the same;

[0079] S4, job instance data processing feedback result audit: the job instance is generated by the job scheduling execution in the workflow, and the permanent instance of data processing is the direct manifestation of the feedback result of the big data task scheduling, and the data developer can troubleshoot the abnormal task through the function.

[0080] Finally: the above only describes the preferred embodiments of the present application and does not limit the present application, and any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for distributed task scheduling of a big data engine supporting stream-batch integration, characterized in that: The specific scheduling steps are as follows: S1, multi-cluster access management: Access to resources of multiple clusters is achieved by configuring parameters of hdfs-site, yarn-site or flink-site, and the cluster is specified for execution during job scheduling; Users can divide multiple clusters into development environment, test environment, trial operation environment and production environment according to their own circumstances, and specify which environment to run during program deployment; When the cluster resources are insufficient, other clusters with more sufficient resources can be specified to run during program deployment; S2, build workflow DAG process: multiple data processing jobs / tasks are combined through visual drag-and-drop components and connection lines to build a workflow DAG process; S3, workflow instance backtracking and control: the DAG process instance of ETL data processing is permanently retained based on the workflow instance, which is generated from the scheduling and execution of the workflow, and records the DAG process instance of the workflow running at a certain time; S3.1, workflow instance editing: workflow instance editing is applied to optimization and correction when the DAG process fails, stops, pauses or ends, and the editing object is the terminated workflow instance, which is saved in two cases: synchronizing the modified job to the original workflow and not synchronizing to the original workflow; S3.2, workflow instance rerun: workflow instance rerun is applied to the terminated workflow instance of the DAG process, and the rerun object is the complete DAG process job node, even if the job node is successfully executed, it will be re-executed, and the workflow instance is not changed but a new job instance is generated during rerun, and the running workflow instance cannot perform rerun operation; S3.3, workflow instance stop: workflow instance stop is applied to the running workflow instance of the DAG process, and the stop operation is performed on the running workflow instance; S3.4, workflow instance pause: workflow instance pause is applied to the running workflow instance of the DAG process, and the pause operation is performed on the running workflow instance; S3.5, workflow instance recovery failure: workflow instance recovery failure is applied to the workflow instance of the failed DAG process, which is the optimized and corrected re-deployment of the failed workflow instance; S3.6, workflow instance recovery stop: workflow instance recovery stop is applied to the stopped workflow instance of the DAG process, and the stopped workflow instance is re-deployed; S3.7, workflow instance recovery pause: workflow instance recovery pause is applied to the paused workflow instance of the DAG process, and the paused workflow instance is re-deployed; S4, job instance data processing feedback result audit: job instance is generated by job scheduling and execution in the workflow, and permanently retains the data processing instance, which is a direct manifestation of the feedback result of big data task scheduling, and data developers can troubleshoot abnormal tasks through this function. 2.The method of claim 1, wherein: The function of multi-cluster access management in step S1 can monitor the HDFS cluster and YARN cluster resources of the access platform in real time, including the node information of NameNode, DataNode, ResourceManager and NodeManager, in addition to the distributed scheduling management of the cluster. 3.The method of claim 1, wherein: The job (task of data processing) in step S2 is mainly based on the big data task calculated by the YARN cluster resources, including but not limited to the mainstream big data technologies of Flink program, Spark program, Hive program, HBase program and Kylin. 4.The method of claim 1, wherein: In step S3.1: Synchronization to the original workflow: the current workflow DAG process instance is consistent with the workflow DAG process, and the subsequent workflow periodic scheduling continues to use the edited DAG process; Different from the original workflow: the current workflow DAG process is inconsistent with the workflow DAG process, and the subsequent workflow periodic scheduling continues to use the unedited DAG process.

5. The distributed task scheduling method of a big data engine supporting stream and batch integration according to claim 1, characterized in that: In step S3.2, the common scenario of workflow instance re-running is that the first DAG process processes full data, and the subsequent DAG process processes incremental data.

6. The distributed task scheduling method of a big data engine supporting stream-batch integration according to claim 1, characterized in that: In step S3.3, the execution stop logic is that the current job node continues to execute, and after the execution is completed, the subsequent job node cancels the execution; The application scenario of workflow instance stop is that the data developer blocks the DAG process in time when the program has an exception, so as to avoid wasting resources and time. After the workflow instance is stopped, the data developer can edit, optimize and correct the DAG process.

7. The distributed task scheduling method of a big data engine supporting stream and batch integration according to claim 1, characterized in that: In step S3.5, the recovery logic is to start execution from the failed job node, and when the recovery fails, the workflow instance remains unchanged but a new job instance is generated; the successful offline job and the running real-time job do not generate a new workflow instance; The application scenario of workflow instance recovery failure is that the DAG process program exception causes the workflow execution to fail, and after the data developer edits, optimizes and corrects the DAG process, the data developer redeployments it. 8.The method of claim 1, wherein the method further comprises: In step S3.7, the recovery logic is to start execution from the suspended job node, and the canceled real-time job and the unrun job generate a new job instance; The application scenario of workflow instance recovery suspension is similar to that of workflow instance recovery stop, which is the dynamic control operation of the data developer blocking the DAG process in time when the program has an exception. After the data developer edits, optimizes and corrects the DAG process, the data developer redeployments it.