Platform system, method and device for realizing one-stop task development, deployment and operation and maintenance functions based on Flink, memory and storage medium

Abstract

The invention relates to a platform system for realizing one-stop task development, deployment and operation and maintenance functions based on Flink, and the system comprises: an authority management control module, which is used for carrying out task authority control on task developers; a task development processing module, which is used for performing project creation to generate a development task; a Flink cluster processing module, which is used for performing multi-process data processing on the registration data source; a task debugging processing module, which is used for performing single-step debugging processing on the development task; a task issuing processing module, which is used for issuing the development task to the Flink cluster; and a task maintenance processing module, which is used for operation and maintenance personnel to execute background maintenance of the development task. The invention further relates to a corresponding method and device, a processor and a storage medium thereof. By adopting the system, the method, the device, the processor and the storage medium of the invention, the tasks from release to production are operated and maintained on the operation and maintenance page, and the operation and maintenance page is displayed according to the levels of the items and the tasks, is clear and organized, and has a better application effect.

Description

technical field [0001] The present invention relates to the field of data processing technology, in particular to the field of data processing technology for task development, and specifically refers to a platform system, method, device, memory and computer based on Flink to realize one-stop task development, deployment, operation and maintenance functions storage medium. Background technique [0002] Flink is a common stream computing engine in the industry, but it has many shortcomings when used in a production environment. [0003] First of all, the development of flink tasks is extremely cumbersome. It is necessary to configure the development environment in the IDE first, and then use scala, java, python, or SQL to develop debugging and tasks. During development, developers need to perform data source registration, data cleaning, and aggregation. , Data calculation and output the whole process. This requires each developer to study various APIs and background principl...

AI Technical Summary

Problems solved by technology

At the same time, the code developed is also difficult to be reused by others directly, and the efficiency is very low

[0004] Secondly, after a large number of developed flink tasks are submitted to the cluster, it is difficult for developers to match tasks with specific development projects, and daily task maintenance is very troublesome

[0005] And because flink itself has almost no permission control function, any user who has access to the flink cluster can modify, start and stop other people's tasks, and even directly shut down and destroy the entire flink cluster, which has great potential risks

Images