Flume task scheduling configuration method, device, equipment and storage medium

Abstract

The present invention discloses a Flume task scheduling configuration method, device, equipment and storage medium, the method comprising: adding a Flume data source to the data source of DolphinScheduler, so that DolphinScheduler can run Flume tasks on the host corresponding to the Flume data source; configuring Flume nodes in DolphinScheduler, so that DolphinScheduler can use Flume nodes to run Flume tasks; defining the scheduling mode of streaming tasks in DolphinScheduler, so that Flume tasks run in the mode of streaming tasks according to the scheduling mode. It can be seen that the present application greatly reduces the cumbersomeness of the management and control of Flume Agent, thereby effectively reducing the operation and maintenance costs.

Description

Technical Field

[0001] The present invention relates to the field of Flume technology, and more specifically, to a Flume task scheduling configuration method, device, equipment and storage medium. Background Art

[0002] In the era of big data, Flume is an excellent data collection tool used in many scenarios. Figure 1 As shown in the figure, it is the architecture of Flume data collection. Agent builds a basic data collection unit. An Agent includes Source, Sink and Channel. Source can collect data from multiple data sources such as log files, network ports, Kafka, etc., and encapsulate the collected data into Event. Event can be processed by a series of interceptors and written to Channel. After the data is successfully written to Channel, Sink will actively pull data from Channel and write it to multiple big data components such as HDFS, HBase, Hive, ES, etc. Then in actual scenarios, the logs that need to be collected usually exist in multiple hosts, and Flume's Agent will also be deployed and run on multiple hosts. Starting and stopping Agent, monitoring Agent status, and viewing Agent logs also require logging in to the corresponding host, and at startup, the startup order of each Agent needs to be manually managed; if there are too many hosts, the control of Flume Agent will be very cumbersome, resulting in a large operation and maintenance cost. Summary of the invention

[0003] The purpose of the present invention is to provide a Flume task scheduling configuration method, device, equipment and storage medium, which greatly reduces the cumbersomeness of Flume Agent management and control, thereby effectively reducing operation and maintenance costs.

[0004] In order to achieve the above object, the present invention provides the following technical solutions:

[0005] A Flume task scheduling configuration method, including:

[0006] Add a Flume data source to the data source of DolphinScheduler, so that the DolphinScheduler can run Flume tasks on the host corresponding to the Flume data source;

[0007] Configuring a Flume node in the DolphinScheduler so that the DolphinScheduler can use the Flume node to run a Flume task;

[0008] The scheduling mode of the streaming task is defined in the DolphinScheduler, so that the Flume task runs as a streaming task according to the scheduling mode.

[0009] Preferably, after adding the Flume data source to the data source of DolphinScheduler, the following is further included:

[0010] Connect to the corresponding host according to the host information configured in the Flume data source, and access the installation file in the connected host according to the installation path configured in the Flume data source. If the installation file is a Flume installation file, it is determined that the configuration of the Flume data source is correct; otherwise, it is determined that the configuration of the Flume data source is incorrect.

[0011] Preferably, configuring a Flume node in the DolphinScheduler includes:

[0012] Add a Flume type workflow in the DolphinScheduler, wherein the workflow includes a Flume node and an operation sequence between each Flume node;

[0013] Configure the Source, Sink, and Channel of the Flume node in the workflow respectively.

[0014] Preferably, the scheduling method of the streaming task is defined in the DolphinScheduler, including:

[0015] Parse the workflow, determine that the first Flume node in the workflow obtained by parsing is the current node, and issue a process PID with a value of 0 for the Flume task to be executed by the current node;

[0016] Start the Flume Agent corresponding to the current node on the host corresponding to the Flume data source. If the startup is successful, update the process PID of the Flume task executed by the corresponding Flume Agent to the running process PID, store the successful startup status and the corresponding process PID in the database, determine whether there is a Flume node after the current node, and determine the Flume node after the current node as the current node when the judgment result is yes, and execute the step of sending the process PID with a value of 0 for the Flume task to be executed by the current node. When the judgment result is no, determine that the startup is completed; if the startup fails, set the startup status to startup failure.

[0017] Preferably, the scheduling method of the streaming task is defined in the DolphinScheduler, including:

[0018] Parse the workflow, determine that the first Flume node in the workflow obtained by parsing is the current node, and issue a process PID with a value of 0 for the Flume task to be executed by the current node;

[0019] Stop the Flume Agent corresponding to the current node on the host corresponding to the Flume data source. If the stop is successful, store the status of the successful stop of the Flume task executed by the corresponding Flume Agent and the corresponding process PID in the database, determine whether there is a Flume node after the current node, and when the judgment result is yes, determine that the Flume node after the current node is the current node, execute the step of sending the process PID with a value of 0 for the Flume task to be executed by the current node, and when the judgment result is no, determine that the stop is completed; if the stop fails, set the stop status to stop failure.

[0020] Preferably, it also includes:

[0021] If the startup status is startup failure, the Flume node that failed to start is obtained, and the node that failed to start is used as the current node. The steps of sending the process PID with a value of 0 to the Flume task to be executed by the current node are executed;

[0022] If the stop status is stop failure, the Flume node that failed to stop is obtained, and the node that failed to stop is used as the current node, and the step of sending the process PID with a value of 0 to the Flume task to be executed by the current node is executed.

[0023] Preferably, configuring a Flume node in the DolphinScheduler includes:

[0024] Configure the Flume node in the front-end interface provided by the DolphinScheduler.

[0025] A Flume task scheduling configuration device, comprising:

[0026] The data source configuration module is used to: add a Flume data source to the data source of DolphinScheduler, so that the DolphinScheduler can run the Flume task on the host corresponding to the Flume data source;

[0027] A node configuration module, used to: configure a Flume node in the DolphinScheduler so that the DolphinScheduler can use the Flume node to run a Flume task;

[0028] The scheduling definition module is used to define the scheduling mode of the streaming task in the DolphinScheduler so that the Flume task runs in the streaming task mode according to the scheduling mode.

[0029] A Flume task scheduling configuration device includes:

[0030] Memory for storing computer programs;

[0031] A processor is used to implement the steps of the Flume task scheduling configuration method as described in any one of the above items when executing the computer program.

[0032] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the Flume task scheduling configuration method as described in any one of the above items.

[0033] The present invention provides a Flume task scheduling configuration method, device, equipment and storage medium. The method comprises: adding a Flume data source to a data source of a DolphinScheduler, so that the DolphinScheduler can run a Flume task on a host corresponding to the Flume data source; configuring a Flume node in the DolphinScheduler, so that the DolphinScheduler can use the Flume node to run a Flume task; defining a scheduling mode of a streaming task in the DolphinScheduler, so that the Flume task runs in a streaming task mode according to the scheduling mode. This application adds a Flume data source of a host for running Flume tasks to the data source of DolphinScheduler, configures a Flume node for running Flume tasks in DolphinScheduler, and defines the scheduling method of Flume tasks as the scheduling method of streaming tasks in DolphinScheduler, and then uses the Flume node on the host corresponding to the Flume data source to implement the scheduling and operation of Flume tasks according to the scheduling method of streaming tasks, thereby using DolphinScheduler to implement the scheduling and operation of Flume tasks and realize the management and control of Flume Agent, which greatly reduces the complexity of the management and control of Flume Agent, thereby effectively reducing the operation and maintenance costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0035] Figure 1 Architecture diagram for collecting data for Flume;

[0036] Figure 2 A flowchart of a Flume task scheduling configuration method provided by an embodiment of the present invention;

[0037] Figure 3 This is the architecture diagram of DolphinScheduler;

[0038] Figure 4 A schematic diagram of configuration information of a Flume data source for a Flume task scheduling configuration method provided by an embodiment of the present invention;

[0039] Figure 5 A style diagram of a Flume node constructed in a Flume task scheduling configuration method provided in an embodiment of the present invention;

[0040] Figure 6 A common configuration diagram of a Flume node in a Flume task scheduling configuration method provided in an embodiment of the present invention;

[0041] Figure 7 A Source configuration diagram in a Flume task scheduling configuration method provided in an embodiment of the present invention;

[0042] Figure 8 A structural diagram of a Flume task scheduling configuration device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] See also Figure 2 , which shows a flow chart of a Flume task scheduling configuration method provided by an embodiment of the present invention, which may specifically include:

[0045] S11: Add the Flume data source to the data source of DolphinScheduler so that DolphinScheduler can run Flume tasks on the host corresponding to the Flume data source.

[0046] DolphinScheduler is a distributed and scalable visual DAG workflow task scheduling system, which is a top-level project of Apache. Figure 3 The diagram shows the architecture of DolphinScheduler. The main modules of DolphinScheduler include: UI and API (interaction module, configuration management of directed acyclic graph DAG, resources, data source monitoring), Master cluster (parse and split DAG, distribute tasks in DAG to Worker), Worker cluster (execute specific tasks distributed by Master cluster, monitor the status of tasks and report to Master cluster), task (support running Shell, MR, Spark and other tasks in Worker cluster). Through the combination of the above modules, DolphinScheduler has two major functions: 1. Schedule monitoring and management of various types of tasks such as Shell and MR in the order of DAG, reducing the operation and maintenance cost of big data development; 2. Configure DAG graph and task information therein by visual drag and drop, which is convenient and easy to use. In addition, Flume is a distributed, reliable, and highly available system for massive log collection, aggregation and transmission. In order to reduce the cumbersomeness of the management and control of Flume Agent, the embodiment of the present application utilizes the above functions of DolphinScheduler, and then utilizes DolphinScheduler to realize the management and control of Flume Agent.

[0047] Flume tasks are tasks that Flume Agent uses to implement functions such as data collection, processing, and output. In order to enable DolphinScheduler to remotely control the corresponding host to implement the running of Flume tasks, the embodiment of the present application can add a Flume data source to the data source of DolphinScheduler. The Flume data source includes the configuration of the corresponding information of the host that implements the running of Flume tasks, so that DolphinScheduler can manage and control the Flume Agent by running Flume tasks on the configured host. That is, the present application adds a Flume data source to the data source to configure the host on which the Flume Agent runs so that it can schedule Flume tasks outside of DolphinScheduler. Specifically, in order to schedule a Flume Agent that is not in DolphinScheduler, combined with the characteristics of the Flume Agent itself, a Flume data source is added to the data source to execute FLume tasks on the host configured with the Flume data source. In a specific implementation method, the configuration information of the Flume data source can be as follows: Figure 4 As shown, specifically including:

[0048] Data source name: used to identify the Flume data source, required field;

[0049] Username: The user who logs in to the host where Flume is installed. This is a required field.

[0050] Password: The user who logs in to the host where Flume is installed. This is a required field.

[0051] Service host: the host where Flume is installed, required;

[0052] Flume installation path: Flume installation path of the remote host, required;

[0053] Flume log path: You can specify the Flume log path, which is optional.

[0054] Process PID path: You can specify the Flume process PID storage path, which is optional.

[0055] Flume configuration path: You can specify the Flume configuration storage path, which is optional;

[0056] Flume evn: environment variable when Flume is running, optional;

[0057] Description: Description of the Flume data source. This is optional.

[0058] According to the above configuration, the Worker in DolphinScheduler can execute the corresponding Flume task on the host specified in the configuration; the above configuration is concentrated in the data source of DolphinScheduler, which can be shared by multiple Agents when configuring the scheduling DAG, reducing the number of repeated configurations, thereby improving ease of use. And the default values ​​are automatically filled in the environment variables and installation paths.

[0059] S12: Configure Flume nodes in DolphinScheduler so that DolphinScheduler can use Flume nodes to run Flume tasks.

[0060] Among them, the Flume node is the Agent in Flume. The embodiment of the present application configures the Agent in DolphinScheduler, so that DolphinScheduler can run the corresponding Flume task by scheduling the configured Agent.

[0061] S13: Define the scheduling method of the streaming task in DolphinScheduler so that the Flume task runs as a streaming task according to the scheduling method.

[0062] Since Flume tasks are streaming tasks, that is, after completing and starting any Flume task, there is no need to wait for the task to be completed before starting the next Flume task. However, when DolphinScheduler executes tasks, it starts the next task only after a task is started and completed. The two are different. In order to enable DolphinScheduler to schedule and execute Flume tasks, the implementation example of the present application can define the scheduling method of Flume as the scheduling method of streaming tasks in DolphinScheduler, and then implement the scheduling method of streaming tasks when scheduling and executing Flume tasks. Specifically, the implementation example of the present application can borrow the scheduler of DolphinScheduler, perform modifications when scheduling Flume tasks, and modify the logic for judging the success of task execution, thereby solving the problem that the Flume Agent will not stop running, resulting in the inability to run subsequent tasks.

[0063] The present invention provides a Flume task scheduling configuration method, device, equipment and storage medium. The method comprises: adding a Flume data source to a data source of DolphinScheduler, so that DolphinScheduler can run Flume tasks on a host corresponding to the Flume data source; configuring a Flume node in DolphinScheduler, so that DolphinScheduler can use the Flume node to run Flume tasks; defining a scheduling mode of a streaming task in DolphinScheduler, so that the Flume task runs in a streaming task mode according to the scheduling mode.

[0064] This application adds a Flume data source of a host for running Flume tasks to the data source of DolphinScheduler, configures a Flume node for running Flume tasks in DolphinScheduler, and defines the scheduling method of Flume tasks as the scheduling method of streaming tasks in DolphinScheduler, and then uses the Flume node on the host corresponding to the Flume data source to implement the scheduling and operation of Flume tasks according to the scheduling method of streaming tasks, thereby using DolphinScheduler to implement the scheduling and operation of Flume tasks and realize the management and control of Flume Agent, which greatly reduces the complexity of the management and control of Flume Agent, thereby effectively reducing the operation and maintenance costs.

[0065] A Flume task scheduling configuration method provided by an embodiment of the present invention may further include, after adding a Flume data source to a DolphinScheduler data source:

[0066] Connect to the corresponding host according to the host information configured in the Flume data source, and access the installation file in the connected host according to the installation path configured in the Flume data source. If the installation file is the Flume installation file, it is determined that the configuration of the Flume data source is correct. Otherwise, it is determined that the configuration of the Flume data source is incorrect.

[0067] It should be noted that after configuring the various parameters in the Flume data source, you can click Test Connection to test whether the host corresponding to the host information (i.e., service host) configured in the Flume data source can be successfully connected, check whether the configured installation path (Flume installation path) can be accessed, and verify whether the installation file obtained after accessing the installation path is the Flume installation file. If the above judgments are all yes, it is determined that the configuration of the Flume data source is correct. Otherwise, it is determined that the configuration of the Flume data source is incorrect, thereby avoiding configuration errors of the Flume data source and failing to successfully run the Flume task.

[0068] An embodiment of the present invention provides a Flume task scheduling configuration method, which configures a Flume node in DolphinScheduler and may include:

[0069] Configure Flume nodes in the front-end interface provided by DolphinScheduler;

[0070] Configuring Flume nodes in DolphinScheduler can include:

[0071] Add Flume-type workflows in DolphinScheduler. The workflows include Flume nodes and the operation sequence between Flume nodes.

[0072] Configure the Source, Sink, and Channel of the Flume node in the workflow respectively.

[0073] The embodiment of the present application can add Flume visualization configuration to the Web UI of DolphinScheduler, thereby reducing the cost of configuring Flume, and after the configuration is completed, the file obtained by the visualization configuration is converted into a Flume configuration file; thus, the remote configuration of Flume Agent can be realized in combination with the configuration items in the DolphinScheduler data source. Specifically, with the help of the front-end interface configuration of DolphinScheduler, a Flume-type workflow is added. Because Flume-type tasks are streaming tasks, there are only Flume nodes in the workflow that have been built, such as Figure 5 When performing stop and start related operations, they will be executed one by one in the order of connection, where each Flume node is a FlumeAgent process. In the Flume node configuration, the common configuration of each Flume node is as follows Figure 6As shown in the figure, the public configuration of DolphinScheduler is mainly used to schedule the Flume task of Flume Agent. The host where the Flume task runs is the host corresponding to the Flume data source configured in the data source. Here, you only need to make the corresponding selection instead of filling in all the information, thereby reducing the number of repeated configurations of the host. At the same time, you need to fill in the name of the Agent to build the Flume configuration file.

[0074] Since the configuration of Flume's native Source, Sink, and Channel is relatively complicated, the embodiment of this application builds a set of simplified Flume configuration processes in configuring the Flume node, and uses the configuration of Source as an example to introduce the configuration conversion solution. Figure 7 As shown, since a Flume Agent can have multiple Sources, you can add a Source by clicking the "+" at the bottom and delete the Source configuration by clicking the "×" at the top. The filled configuration can be converted to an Agent configuration file at runtime. Taking Netcat Source as an example, after filling in the name of the Source and selecting the type in the configuration, you can configure its bind and port configuration. This configuration corresponds to the bind and port properties of the Flume component Netcat Source. After filling in, it will be spliced ​​into a Flume Source configuration according to the Agent name. Assuming that the Agent name filled in is agent, the Source name is Source, bind is filled in as localhost, and the port parameter is filled in as 44444, the corresponding background generated configuration file can be as follows:

[0075] agent.sources.Source.type=netcat

[0076] agent.sources.Source.bind=localhost

[0077] agent.sources.Source.port=44444

[0078] Other configurations of the Source can be added through custom configuration; when the Source custom configuration is converted to a Flume Agent configuration, the prefix of the Flume Agent configuration file will be automatically completed according to the filled Source name and Agent name. For example, if a Source with the Agent name of agent and the Source name of Source adds a configuration with the key of max-line-length and the value of 512, the generated Flume Agent configuration can be: agent.source.Source.max-line-length=512.

[0079] A Flume task scheduling configuration method provided by an embodiment of the present invention defines a scheduling method of a streaming task in DolphinScheduler, which may include:

[0080] Parse the workflow, determine that the first Flume node in the parsed workflow is the current node, and issue a process PID with a value of 0 for the Flume task to be executed by the current node;

[0081] Start the Flume Agent corresponding to the current node on the host corresponding to the Flume data source. If the startup is successful, update the process PID of the Flume task executed by the corresponding Flume Agent to the running process PID, store the successful startup status and the corresponding process PID in the database, determine whether there is a Flume node after the current node, and determine the Flume node after the current node as the current node when the judgment result is yes, and execute the step of sending the process PID with a value of 0 for the Flume task to be executed by the current node. When the judgment result is no, determine that the startup is completed; if the startup fails, set the startup status to startup failure;

[0082] In DolphinScheduler, define the scheduling method of streaming tasks, which can include:

[0083] Parse the workflow, determine that the first Flume node in the parsed workflow is the current node, and issue a process PID with a value of 0 for the Flume task to be executed by the current node;

[0084] Stop the Flume Agent corresponding to the current node on the host corresponding to the Flume data source. If the stop is successful, store the status of the Flume task executed by the corresponding Flume Agent and the corresponding process PID in the database, determine whether there is a Flume node after the current node, and if the result is yes, determine that the Flume node after the current node is the current node, and execute the step of sending the process PID with a value of 0 for the Flume task to be executed by the current node. If the result is no, determine that the stop is completed; if the stop fails, set the stop status to stop failure;

[0085] Flume task scheduling configuration methods can also include:

[0086] If the startup status is startup failure, the Flume node that failed to start is obtained, and the node that failed to start is used as the current node. The steps of sending the process PID with a value of 0 to the Flume task to be executed by the current node are executed;

[0087] If the stop status is stop failure, the Flume node that failed to stop is obtained, and the node that failed to stop is used as the current node, and the step of sending the process PID with a value of 0 to the Flume task to be executed by the current node is executed.

[0088] Since Flume collects streaming tasks, the processes will run consistently, and the DolphinScheduler scheduling method based on the task status reported after the task runs is no longer applicable. In the process of scheduling Flume, the start of the Flume Agent process is defined as a successful task start, and the process PID (unique identifier) ​​does not exist in the host, which means it is stopped. Therefore, the corresponding DAG includes the following states:

[0089] Start: Start all Flume Agent processes (i.e. Flume tasks) in the DAG in the order of the DAG, and obtain the corresponding process PID;

[0090] Startup failure: Nodes that failed when starting the Flume Agent process in the order of the DAG;

[0091] Stop: Stop all Flume Agent processes in the DAG in the order of the DAG, and record the process PID of the task as 0;

[0092] Stop failure: Nodes where a failure occurs when stopping the Flume Agent process in the order of the DAG.

[0093] For the above four states, there will be three corresponding operation logics, namely start, stop and recovery failure. The following is a detailed description of these three operation logics (all nodes are Flume nodes):

[0094] 1. Startup

[0095] The start operation can be performed only when the DAG status is stopped or failed to stop. The specific process after the start command is issued is as follows:

[0096] 1. The Master cluster performs DAG parsing and parses out the first node it contains;

[0097] 2. The Master cluster sends the node information and the data source information of the Flume data source to the Worker, and sends a process PID of 0 to the task of the node if it does not have a process PID.

[0098] 3. The Worker receives the startup command sent by the Master for the node and starts the startup process;

[0099] 4. The Worker determines whether the process PID in the startup command is 0. If it is 0, it executes step 6. If it is not 0, it executes step 5.

[0100] 5. The Worker logs in to the host corresponding to the Flume data source and tries to kill the process corresponding to the process PID according to the process PID. If the killing is successful, execute step 6. If the killing fails, determine that the task execution status of the node is failed, and execute step 7.

[0101] 6. Generate Flume configuration according to the configuration information, and copy the generated Flume configuration to the host corresponding to the Flume data source. Start the Flume Agent of the node on the host. If the Agent is started successfully, the status is recorded as success, and the process PID is recorded as the running process PID of the task; if the Agent fails to start, the status is recorded as failure, the process PID is 0, and step 7 is executed;

[0102] 7. Report the task status of the node to the Master;

[0103] 8. The Master receives the task status and stores the received task status and process PID in the database. If the task status is successful, it determines whether the node has a subsequent node based on the task status. If so, execute step 2. Otherwise, the status of the DAG is set to startup success and the process ends. If the task status is failed, the status of the DAG is set to startup failure and the process ends.

[0104] 2. Stop

[0105] The stop operation can be performed only when the DAG status is enabled or failed to be enabled. The specific process after the stop command is issued is as follows:

[0106] 1. The Master cluster performs DAG parsing and parses out the first node it contains;

[0107] 2. The Master cluster sends the node information and the data source information of the Flume data source to the Worker, and sends a process PID of 0 to the task of the node if it does not have a process PID.

[0108] 3. The Worker receives the stop command sent by the Master for the node and starts the stop process;

[0109] 4. The Worker determines whether the process PID in the stop command is 0. If it is 0, it executes step 6. If it is not 0, it executes step 5.

[0110] 5. The Worker logs in to the host corresponding to the Flume data source and tries to kill the process corresponding to the process PID according to the process PID. If the killing is successful, execute step 6. If the killing fails, determine that the task execution status of the node is failed, and execute step 7.

[0111] 6. Set the task status to stop and record the process PID as 0;

[0112] 7. Report the status of the task to the Master;

[0113] 8. If the task status is stopped successfully, go to step 9, otherwise go to step 12;

[0114] 9. Record the process PID and status of the task to the database;

[0115] 10. Determine whether the task has a subsequent node based on the task status. If yes, execute step 2. If no, execute step 11.

[0116] 11. Set the DAG status to stop and the process ends;

[0117] 12. Set the DAG status to stop failure and the process ends.

[0118] 3. Recovery Failure

[0119] Failed recovery includes two processes: failed recovery start and failed recovery stop. The process after issuing the failed recovery command is as follows:

[0120] 1. Determine the status of the task. If the start fails, execute step 2; if the stop fails, execute step 3;

[0121] 2. Get the node that failed to start, and run the DAG from that node according to the startup process;

[0122] 3. Get the node that failed to stop, and start running the DAG from this node according to the startup process.

[0123] This application includes three parts: Flume data source management, Flume visual configuration generation, and Flume type task scheduling. It reduces the configuration cost of Flume type tasks, can start and stop management of Flume through DAG sequence, and supports failure recovery function, configures Flume running host, greatly reduces the cost of management, operation and maintenance, and configuration of Flume Agent, improves the usability of Flume components, and expands the functions of DolphinScheduler components.

[0124] The embodiment of the present invention also provides a Flume task scheduling configuration device, such as Figure 8 As shown, it may specifically include:

[0125] The data source configuration module 11 is used to: add a Flume data source to the data source of DolphinScheduler, so that DolphinScheduler can run Flume tasks on the host corresponding to the Flume data source;

[0126] The node configuration module 12 is used to: configure the Flume node in DolphinScheduler so that DolphinScheduler can use the Flume node to run the Flume task;

[0127] The scheduling definition module 13 is used to define the scheduling method of the streaming task in DolphinScheduler, so that the Flume task runs as a streaming task according to the scheduling method.

[0128] The embodiment of the present invention further provides a Flume task scheduling configuration device, which may include:

[0129] Memory for storing computer programs;

[0130] A processor is used to implement any of the steps of the above Flume task scheduling configuration method when executing a computer program.

[0131] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above Flume task scheduling configuration methods can be implemented.

[0132] It should be noted that for the description of the relevant parts of a Flume task scheduling configuration device, equipment and storage medium provided in the embodiment of the present invention, please refer to the detailed description of the corresponding parts in a Flume task scheduling configuration device method provided in the embodiment of the present invention, which will not be repeated here. In addition, the parts of the above technical solutions provided in the embodiment of the present invention that are consistent with the implementation principles of the corresponding technical solutions in the prior art are not described in detail to avoid excessive elaboration.

[0133] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A Flume task scheduling configuration method, characterized in that: include: Add a Flume data source to the data source of DolphinScheduler, so that the DolphinScheduler can run Flume tasks on the host corresponding to the Flume data source; Configuring a Flume node in the DolphinScheduler so that the DolphinScheduler can use the Flume node to run a Flume task; The scheduling mode of the streaming task is defined in the DolphinScheduler, so that the Flume task runs as a streaming task according to the scheduling mode.

2. The method according to claim 1, characterized in that After adding Flume data source to the data source of DolphinScheduler, it also includes: Connect to the corresponding host according to the host information configured in the Flume data source, and access the installation file in the connected host according to the installation path configured in the Flume data source. If the installation file is a Flume installation file, it is determined that the configuration of the Flume data source is correct; otherwise, it is determined that the configuration of the Flume data source is incorrect.

3. The method according to claim 2, characterized in that Configure the Flume node in the DolphinScheduler, including: Add a Flume type workflow in the DolphinScheduler, wherein the workflow includes a Flume node and an operation sequence between each Flume node; Configure the Source, Sink, and Channel of the Flume node in the workflow respectively.

4. The method according to claim 3, characterized in that: The scheduling method of the streaming task is defined in the DolphinScheduler, including: Parse the workflow, determine that the first Flume node in the parsed workflow is the current node, and issue a process PID with a value of 0 for the Flume task to be executed by the current node; Start the Flume Agent corresponding to the current node on the host corresponding to the Flume data source. If the startup is successful, update the process PID of the Flume task executed by the corresponding Flume Agent to the running process PID, store the successful startup status and the corresponding process PID in the database, determine whether there is a Flume node after the current node, and determine the Flume node after the current node as the current node when the judgment result is yes, and execute the step of sending the process PID with a value of 0 for the Flume task to be executed by the current node. When the judgment result is no, determine that the startup is completed; if the startup fails, set the startup status to startup failure.

5. The method according to claim 4, characterized in that The scheduling method of the streaming task is defined in the DolphinScheduler, including: Parse the workflow, determine that the first Flume node in the parsed workflow is the current node, and issue a process PID with a value of 0 for the Flume task to be executed by the current node; Stop the Flume Agent corresponding to the current node on the host corresponding to the Flume data source. If the stop is successful, store the status of the successful stop of the Flume task executed by the corresponding Flume Agent and the corresponding process PID in the database, determine whether there is a Flume node after the current node, and when the judgment result is yes, determine that the Flume node after the current node is the current node, execute the step of sending the process PID with a value of 0 for the Flume task to be executed by the current node, and when the judgment result is no, determine that the stop is completed; if the stop fails, set the stop status to stop failure.

6. The method according to claim 5, characterized in that Also includes: If the startup status is startup failure, the Flume node that failed to start is obtained, and the node that failed to start is used as the current node. The steps of sending the process PID with a value of 0 to the Flume task to be executed by the current node are executed; If the stop status is stop failure, the Flume node that failed to stop is obtained, and the node that failed to stop is used as the current node, and the step of sending the process PID with a value of 0 to the Flume task to be executed by the current node is executed.

7. The method according to claim 6, characterized in that Configure the Flume node in the DolphinScheduler, including: Configure the Flume node in the front-end interface provided by the DolphinScheduler.

8. A Flume task scheduling configuration device, characterized in that: include: The data source configuration module is used to: add a Flume data source to the data source of DolphinScheduler, so that the DolphinScheduler can run the Flume task on the host corresponding to the Flume data source; A node configuration module, used to: configure a Flume node in the DolphinScheduler so that the DolphinScheduler can use the Flume node to run a Flume task; The scheduling definition module is used to define the scheduling mode of the streaming task in the DolphinScheduler so that the Flume task runs in the streaming task mode according to the scheduling mode.

9. A Flume task scheduling configuration device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the Flume task scheduling configuration method as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the Flume task scheduling configuration method according to any one of claims 1 to 7 are implemented.

Images