Flink real-time task management system method based on yarn operation

CN118916229APending Publication Date: 2024-11-08FOCUS TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410729543.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-06
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

[0004]但实时任务的管理是一个很大的困难,正常提交方式下,需要开发者上机器执行shell命令提交任务、查看日志等等操作是不方便且不安全的

Benefits of technology

[0020] Beneficial effects: This invention uses real-time machines, front-end users, and data 1 The interaction between database, HDFS, and YARN implements a real-time task management system. One of the most obvious advantages of the Yarn operating mode is that it will realize dynamic allocation of TaskManager resources. Deploy Flink tasks to the Yarn cluster. This invention manages real-time tasks. When publishing real-time tasks, you can pull and publish real-time tasks by inputting the coordinates of the real-time task jar package into the private maven warehouse. When executing tasks, you can submit the tasks to the yarn cluster by inputting the parameters for executing the tasks. Run in the platform, and monitor the task status of submitted tasks and view task logs through the platform. Running tasks can be stopped on the platform. And the whole process does not require developers to go to the machine to execute shell commands, ensuring the safety and efficiency of the overall process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118916229A_ABST
    Figure CN118916229A_ABST
Patent Text Reader

Abstract

The invention discloses an flink real-time task management method based on yarn operation, which comprises the following steps: 1) releasing a real-time task: a user pulls and releases the real-time task by inputting coordinates of a real-time task jar packet input into a private maven bin, and a program splices the coordinates input by the user into a shell of an mvn execution command; 2) task execution: a user selects a jar package in a list through a front end, inputs relevant parameters of real-time task execution, submits a task to a yarn through an ssh mode, monitors submitted relevant contents in a submission log through a new thread, returns a task submission condition, returns a successful state and records task information into mysql if submission succeeds, and executes the task in the mysql; and the front end returns the corresponding log content for the user to view. And (3) the mode of monitoring the state of the submitted task is that after the operation task is submitted, the program submits the task record to an operation task table in the mysql library.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to application fields such as big data real-time tasks and submitting real-time tasks to YARN, and in particular to a Flink real-time task management system method based on YARN operation. Background Art

[0002] In the current era of rapid development, a large amount of information needs to be processed in real time. In the field of big data, the importance of real-time streaming tasks is becoming increasingly important, but the release and submission of flink cluster tasks traditionally require command line submission through shell commands, which is extremely inconvenient for users or data development, and puts high demands on operation and maintenance, and cannot conveniently view logs, start and stop tasks, etc. Therefore, when developing in the field of big data real-time, a set of fully functional real-time task management systems is needed.

[0003] At present, real-time tasks are mainly used in the field of big data, and the mainstream architecture in the field of big data is the Hadoop+Yarn technology system. There are two main operating environments for Flink tasks, one is pure Flink cluster deployment, and the other is Flink on Yarn mode deployment. In the Flink on Yarn deployment mode, Flink tasks can achieve resource control through the Yarn cluster, which is more conducive to the operation of Flink tasks and ensures that resources are not wasted.

[0004] However, the management of real-time tasks is a big difficulty. Under normal submission mode, developers need to execute shell commands on the machine to submit tasks and view logs, which is inconvenient and unsafe. A fully functional and highly available system is urgently needed to manage real-time tasks.

[0005] Yarn refers to the resource management framework Yarn; its deployment process is that the client submits the Flink application to Yarn's ResourceManager, and the Flink cluster task is implemented by deploying the corresponding JobManager and TaskManager on the container applied for by the ResourceManager. Summary of the invention

[0006] With respect to the above-mentioned real-time tasks, the present invention provides a systematic method for managing the real-time tasks.

[0007] The technical solution of the present invention is a method for managing real-time tasks. 1) A method for publishing real-time tasks: users can pull and publish real-time tasks by inputting the coordinates of the real-time task jar package entered into a private Maven warehouse. The program will splice the coordinates input by the user into a shell for executing the mvn command.

[0008] Through the java ssh related API, jump to the corresponding machine to execute the shell script and pull the corresponding task jar package. If it cannot be pulled, an error message is returned;

[0009] 2) Task execution method: Users can select the jar package in the list through the front-end and enter the relevant parameters for real-time task execution, such as memory, number of cores, number of slots, task name and other basic parameters. They can also customize additional parameters. The program will splice it with the jar package information and generate a shell command to submit the task to YARN.

[0010] Submit the task to YARN through SSH, and monitor the submission-related content in the submission log through a new thread, and return the submission status of the task. If the submission is successful, it returns a success status and records its task information in MySQL. The front end returns the corresponding log content for users to view.

[0011] 3) How to monitor the status of submitted tasks: After submitting a task, the program will record the successfully submitted task in the running task table in the MySQL database;

[0012] Stores various parameters of the task, log address, application ID on yarn, etc.

[0013] The system starts a separate check thread to regularly check the tasks in the running task table and calls `getYarnApplicationState()` of the `ApplicationReport` class in the YARN API to obtain the running status of the corresponding application. If the application stops abnormally in a non-system way, it will be recorded by the system and an abnormal task stop alarm will be sent.

[0014] 4) How to view task logs: During the task running process or after the task fails, the log storage path of the real-time task will be continuously recorded in the MySQL database. When the user selects the corresponding task to be viewed on the front end, the program interface will be called to obtain the log path from the corresponding table in MySQL. The log content is obtained through the program and returned to the front end for viewing.

[0015] 5) Task stop method: When a task is running, the front-end user can choose between two task stop methods, namely, normal task stop and savepoint pause;

[0016] The common method of stopping a task is to splice the running task information selected by the user and stop the corresponding task through the yarn command;

[0017] The savepoint pause method will try to use Flink's savepoint pause command to stop the task. If the task has a checkpoint enabled, the savepoint can be used to stop the task normally and the savepoint path will be recorded in the task information. Otherwise, the task will be stopped normally and a related error will be returned.

[0018] After stopping a task, the corresponding task will be moved to the "Stopped Tasks" table and displayed in the "Historical Tasks" list. Users can restart ordinary paused tasks or checkpoint tasks by restarting tasks or savepoint restart tasks in the historical task list. The startup data of the task will be obtained from the corresponding table to restart the task.

[0019] The present invention includes functions such as pulling real-time task packages, running flink tasks, viewing flink task logs, stopping and restarting flink tasks, etc. The active steps are to obtain the real-time task jar package through ssh plus mvn command; run the flink task through ssh command and submit it to the yarn cluster; obtain the flink task logs on hdfs and linux through java; and execute shell commands through ssh to stop or restart the flink task to integrate them into a full-featured flink real-time task management system.

[0020] Beneficial effects: The present invention uses real-time machines, front-end users, and digital 1 The interaction between the database, HDFS, and Yarn realizes the real-time task management system. One of the most obvious advantages of the Yarn operating mode is that it can realize a dynamic allocation of TaskManager resources. Deploy Flink tasks to the Yarn cluster. The present invention manages real-time tasks. The real-time tasks can be released by inputting the coordinates of the real-time task jar package entered into the private Maven warehouse to pull and publish the real-time tasks. The execution tasks can be submitted to the Yarn cluster for running by inputting the parameters of the execution tasks, and the task status of the submitted tasks can be monitored and the task logs can be viewed through the platform, and the running tasks can be stopped on the platform. And the developer does not need to execute shell commands on the machine throughout the whole process to ensure the security and efficiency of the overall process. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A flowchart of a method for publishing real-time tasks in an embodiment of a Flink real-time task management system method based on YARN in the present invention;

[0022] Figure 2It is a flowchart of a task execution method of an embodiment of a Flink real-time task management system method based on YARN operation in the present invention;

[0023] Figure 3 A flowchart of a method for monitoring the status of submitted tasks in an embodiment of a Flink real-time task management system method based on YARN operation in the present invention;

[0024] Figure 4 A flowchart of a method for viewing task logs in an embodiment of a Flink real-time task management system method based on YARN in the present invention;

[0025] Figure 5 The present invention is a flowchart of a method for stopping a task in an embodiment of a Flink real-time task management system based on YARN operation.

[0026] Specific implementation mode The present invention is further described below in conjunction with the accompanying drawings and exemplary embodiments:

[0027] like Figure 1 The method of publishing real-time tasks shown in the figure is implemented as follows:

[0028] S11: In the front-end page setting form, you need to enter the Maven coordinates: groupId, artifactId, version, etc. to determine the jar package and its version.

[0029] S12: The backend Java program assembles the shell command into a complete Maven pull script according to the input parameters, and adds the parameters: -DgroupId=$groupid-DartifactId=$artifact-Dversion=$version to locate the jar that needs to be pulled.

[0030] S13: The Java program creates a new session connected to the machine that executes the real-time task. Using the StreamGobbler class, the real-time task machine executes the shell command spliced ​​by S12 and returns the output result to the program.

[0031] S14: After receiving the result returned by S13, the Java program checks whether the corresponding jar package exists in the task list and determines whether the jar package is pulled successfully.

[0032] S15: If yes, a response is returned to the front end indicating success of the pull, and the front end interface is refreshed. If no, the corresponding failure reason is returned, such as the corresponding jar cannot be found.

[0033] like Figure 2The way to execute real-time tasks is shown in the figure. The specific implementation method is:

[0034] S21: The front-end user clicks the name in the jar package list, jumps to the parameter table, and submits parameter information including: memory-ytm, number of cores-ys, class name-c, task name-ynm, alert email, etc.

[0035] S22: The backend Java program splices the shell commands of the submitted tasks according to the input parameters. For example, if the testjob task with 1c cores, 2g memory, and class name com.test.job is passed in, it will be spliced ​​into the shell of flink run-m yarn-cluster-ytm 2g-ys 1c-c com.test.job-ynm testjob for execution.

[0036] S23: The Java program creates a new session connected to the machine that executes the real-time task, uses the ch.ethz.ssh2.StreamGobbler class to make the real-time task machine execute the shell command spliced ​​by S12, submit the corresponding real-time task, and return the task log location submitted by the program.

[0037] S24: The newly opened thread of the Java program performs a timed check based on the task log location and uses keywords to determine whether the task has been successfully submitted to the yarn cluster.

[0038] S25: If the task is judged to be submitted successfully, the task startup parameters, task applicationId, jobid and other information are stored in the corresponding running task table of MySQL. A response indicating successful submission is returned to the front end, and the front end jumps to the task log information.

[0039] S26: If it is determined that the task submission fails, the task submission failure is returned, and the front end jumps to the task submission log information.

[0040] like Figure 3 The method of monitoring the status of submitted tasks is shown in the following example.

[0041] S31: When the system starts, a listen process will be started, which will run independently and execute the run method once every certain period of time.

[0042] S32: In the scheduled query run method, the task applicationId information of the running task table in the MySQL library is obtained through mybatis.

[0043] S33: Get the task status through the applicationId obtained in S32 through the `getYarnApplicationState()` method of the `ApplicationReport` class in the API provided by yarn.

[0044] S34: Determine the task running status.

[0045] S35: If it is running, then complete this check. If it is detected that the task fails, move the corresponding task from the running task table to the historical task table, and send an email alert via email.

[0046] like Figure 4 The method of viewing the task log shown is as follows:

[0047] S41: The user clicks a log button of a corresponding task in “Running Tasks” or “Historical Tasks”.

[0048] S42: The background program uses mybatis to obtain the task status and task log path through the application id.

[0049] S43: Determine whether the task is in a running state or a historical task. If it is a running task, obtain the log content through the Linux API. If it is a historical task, obtain the log content through the HDFS API.

[0050] S44: Return the log content to the front-end for display.

[0051] like Figure 5 The method of viewing the task log shown is as follows:

[0052] S51: The user clicks to stop the task on the front end, and can choose to stop by savepoint or stop normally.

[0053] S52: Assemble a shell stop command according to the stop mode and task information.

[0054] S53: Submit a shell command through S13 to stop the real-time task.

[0055] S54: Determine whether the task is stopped successfully. If successful, the front end returns task success and moves the task information into the historical task table. If failed, the stop failure log is returned to the front end.

[0056] The above is an embodiment of a Flink real-time task management system method based on YARN provided by the present invention. From the above embodiment, we can understand an implementation method of real-time task management, which is mainly implemented through the interaction between real-time machines, front-end users, databases, HDFS, and YARN.

[0057] It should be noted that the above functions can be freely changed in implementation mode, and the above is only a preferred implementation mode of the present invention. For relevant personnel in this technical field, various modifications and equivalent conversions can be carried out without departing from the idea provided by the present invention, which all belong to the protection scope of the present invention.

Claims

1. A method for managing Flink real-time tasks based on YARN, characterized in that: 1) Release real-time tasks: Users input the coordinates of the real-time task jar package inserted into the private Maven repository to pull and release the real-time tasks. The program will splice the coordinates entered by the user into the shell of the mvn execution command; Through the java ssh related API, jump to the corresponding machine to execute the shell script and pull the corresponding task jar package. If it cannot be pulled, an error message is returned; 2) Task execution: The user selects the jar package in the list through the front-end and enters the relevant parameters for real-time task execution, such as basic parameters such as memory, number of cores, number of slots, task name, etc., and can customize additional parameters. The program will splice it with the jar package information and generate a shell command to submit the task to YARN; Submit the task to YARN through SSH, and monitor the submission-related content in the submission log through a new thread, and return the submission status of the task. If the submission is successful, it returns a success status and records its task information in MySQL. The front end returns the corresponding log content for users to view; 3) How to monitor the status of submitted tasks: After submitting a task, the program will record the successfully submitted task in the running task table in the MySQL database; Stores various parameters of the task, log address, application ID on yarn, etc. The system starts a separate check thread to regularly check the tasks in the running task table and call the YARN API to obtain the running status of the corresponding application. If the application stops abnormally in a non-system way, the system will record it and send an abnormal task stop alarm. 4) View task logs: During the task running process or after a task fails, the log storage path of the real-time task will be continuously recorded in the MySQL database. When the user selects the corresponding task to be viewed on the front end, the program interface will be called to obtain the log path from the corresponding table in MySQL. The log content is obtained through the program and returned to the front end for viewing; 5) Stop task: When a task is running, the front-end user can choose two ways to stop the task, namely, normal stop task and savepoint pause; The common method of stopping a task is to splice the running task information selected by the user and stop the corresponding task through the yarn command; The savepoint pause method will try to use Flink's savepoint pause command to stop the task. If the task has a checkpoint enabled, the savepoint can be used to stop the task normally and the savepoint path will be recorded in the task information. Otherwise, the task will be stopped normally and a related error will be returned.

2. According to the method for managing Flink real-time tasks based on YARN operation according to claim 1, it is characterized in that: After stopping a task, the corresponding task will be moved to the "Stopped Tasks" table and displayed in the "Historical Tasks" list. Users can restart ordinary paused tasks or checkpoint tasks by restarting tasks or savepoint restart tasks in the historical task list. The startup data of the task will be obtained from the corresponding table to restart the task.