A task processing system, method, electronic device and readable medium

Abstract

Embodiments of the present application provide a task processing system, method, electronic device and readable medium. The system comprises a task management module, a Flink task processing module and a cloud computing module; the task management module comprises a task creation unit, a task timing unit, a task state monitoring unit, a task life cycle management unit and a task performance monitoring unit; the Flink task processing module comprises a Flink UDF management unit, a data source configuration unit, a resource allocation unit and an automatic deployment unit connected in sequence; the cloud computing module comprises a K8S container orchestration management unit, a Docker containerization unit and a cloud monitoring unit; after receiving data processing logic, the task processing system performs resource automatic allocation through the data source configuration unit and the resource allocation unit based on the state monitoring data of the task state monitoring unit to execute the Flink task.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a task processing system, a task processing method, an electronic device and a computer readable medium. BACKGROUND

[0002] Flink is a distributed stream processing framework, mainly used for real-time data processing and large-scale data analysis. It can quickly and stably process continuous data streams, and can also process batch data, so it is suitable for various scenarios such as real-time data analysis, monitoring and large-scale batch processing tasks. Containerization is a technology that packages an application and its dependencies into a "container" so that it can run in any container-enabled environment. This technology provides application isolation, portability and consistency.

[0003] Automated deployment of tasks can significantly improve system efficiency, reduce the error rate of human intervention, and greatly shorten the time to put applications online. Automated deployment can dynamically adjust resources and configurations according to actual needs to meet real-time changing workloads, which helps to improve resource utilization, ensure application stability and rapid recovery, and support continuous integration and delivery processes of the system.

[0004] However, the existing task processing process requires a lot of manual intervention for setting, management and scheduling, which greatly reduces the efficiency of task processing and increases subjective errors caused by excessive human intervention. SUMMARY

[0005] The embodiments of the present application provide a task processing system, method, electronic device and computer readable storage medium to solve the problem that the existing task processing process requires a lot of manual intervention for setting, management and scheduling, which greatly reduces the efficiency of task processing and increases subjective errors caused by excessive human intervention.

[0006] The embodiments of the present application disclose a task processing system, characterized in that the system comprises a task management module, a Flink task processing module and a cloud computing module;

[0007] The task management module comprises a task creation unit, a task timing unit, a task state monitoring unit, a task life cycle management unit and a task performance monitoring unit;

[0008] The Flink task processing module comprises a Flink UDF management unit, a data source configuration unit, a resource allocation unit and an automated deployment unit connected in sequence, wherein the Flink UDF management unit comprises a custom UDF, and the custom UDF is used to receive the data processing logic of the Flink task;

[0009] The task state monitoring unit is connected with the data source configuration unit;

[0010] The cloud computing module comprises a K8S container arrangement management unit, a Docker containerization unit and a cloud monitoring unit, wherein the K8S container arrangement management unit and the Docker containerization unit are used for containerizing the Flink task;

[0011] The task performance monitoring unit is connected with the cloud monitoring unit, and the automatic deployment unit is connected with the Docker containerization unit;

[0012] After receiving the data processing logic, the task processing system performs resource automatic allocation through the data source configuration unit and the resource allocation unit based on the state monitoring data of the task state monitoring unit, so as to execute the Flink task.

[0013] The embodiment of the application further discloses a task processing method, which is applied to the task processing system of the embodiment of the application, and the method comprises the following steps:

[0014] S1: obtaining task data of a target task through a task management module, wherein the task data comprises task performance data obtained based on a task performance monitoring unit;

[0015] S2: setting data processing logic of the task data as Flink task data through a Flink UDF management unit of a Flink task processing module;

[0016] S3: containerizing the Flink task by using a K8S container arrangement management unit and a Docker containerization unit in a cloud computing module;

[0017] S4: allocating resources for the Flink task by using the K8S container arrangement management unit according to the task performance data;

[0018] S5: returning to step S4 until the target task is executed completely.

[0019] The embodiment of the application further discloses an electronic device comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus;

[0020] The memory is used for storing a computer program;

[0021] The processor is used for executing the program stored on the memory, and realizes the method of the embodiment of the application.

[0022] The embodiment of the application further discloses one or more computer readable media, which store instructions, and when executed by one or more processors, make the processor execute the method of the embodiment of the application.

[0023] Embodiments of the present application include the following advantages:

[0024] In the present application, the task management module is used to obtain the necessary task data, and then the Flink task processing module is used to convert the task into the Flink task flow processing framework which has real-time performance, fault tolerance and can meet the demand of large-scale data processing, and a custom UDF is set to provide users with quick and simple task logic setting, greatly improving the task processing efficiency. And based on the automatic fault tolerance mechanism of the Flink task flow processing framework, the efficiency and reliability of task processing are improved. Then the cloud computing module is used to containerize the Flink task using the K8S container orchestration management unit and the Docker container unit, and the K8S container orchestration management unit is used to complete the automatic intelligent resource allocation and optimization of the whole process of the Flink task. Reduce the manual intervention of task processing, reduce the complexity and manual management cost of task data processing, and improve the automation degree and efficiency of task processing. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 is a structural block diagram of a task processing system provided in an embodiment of the present application;

[0026] Figure 2 is a step flow chart of a task processing method provided in an embodiment of the present application;

[0027] Figure 3 is a block diagram of an electronic device provided in an embodiment of the present application;

[0028] Figure 4 is a schematic diagram of a computer readable medium provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0030] Referring to Figure 1 , a structural block diagram of a task processing system provided in an embodiment of the present application is shown.

[0031] The system includes a task management module, a Flink task processing module and a cloud computing module.

[0032] The task management module includes a task creation unit, a task timing unit, a task state monitoring unit, a task life cycle management unit and a task performance monitoring unit.

[0033] The task creation unit is specifically configured to provide an intuitive user interface through which users can easily create and configure data processing tasks. The interface includes fields such as task name, task type, task description, input data source, output target, etc., to allow users to clearly define the basic attributes of the task. Users can choose different task types, such as Flink tasks, data cleaning tasks, etc., to configure tasks according to specific needs. This makes the engine suitable for a variety of data processing scenarios. Users can set specific parameters for tasks, such as the parallelism of Flink tasks, window size, etc. These parameters vary depending on the task type to meet the individual needs of different tasks. Users can clearly define the input data source and output target of the task, including connecting to a database, reading a file, obtaining API data, etc. This simplifies the input and output configuration process of the task.

[0034] The task timing unit is specifically configured to support users in defining the execution frequency and time of tasks, enabling the scheduling of timed tasks. Users can choose different scheduling strategies such as one-time execution, daily execution, weekly execution, etc., to adapt to different data processing needs. Users can set the execution schedule of the task to ensure that the task is executed at the specified time. This helps to implement periodic data processing tasks, such as generating daily reports, weekly data aggregation, etc. Users can view the scheduling plan of the task through the visual interface to understand when the task is executed and the execution frequency, improving user control over task scheduling.

[0035] The task status monitoring unit is specifically configured to provide real-time task status monitoring functionality, allowing users to view the current status of the task at any time, including executing, pausing, completing, failing, etc. Users can view the execution progress of the task to understand the current processing stage. This helps users to understand the execution of the task in a timely manner and make necessary adjustments and interventions.

[0036] The task lifecycle management unit provides users with the ability to create tasks through the interface and manually start them, or set up timed tasks for automatic startup. Once a task is created, the engine will start the corresponding data processing task according to the user's configuration. Users can manually pause and stop the execution of the task. The pause operation can be used to temporarily interrupt the execution of the task, while the stop operation will terminate the entire lifecycle of the task. Users can delete tasks that are no longer needed to release system resources. This ensures the controllability and efficient management of the task lifecycle.

[0037] Among them, the task performance monitoring unit is specifically used for recording important indicators in the task process when the flink task is performed, and can analyze the conditions in the running process from the operation and maintenance point of view. Provide intuitive real-time performance monitoring interface, show the key performance indicators of the task, such as processing speed, delay, throughput, etc. Through the graphical way, the user can quickly understand the execution status of the task. Realize the fine-grained log recording of the task execution process, including the execution of each operator, data input and output, etc. This detailed log recording helps users to deeply understand the execution details of the task. Using machine learning and statistical analysis technology, the engine realizes intelligent anomaly detection and alarm system. The system can automatically identify abnormal conditions in task execution and timely alarm to the user, improving the reliability of task execution. Provide task history tracking function, users can backtracking to view the performance data and log records of the past tasks. This helps users to carry out long-term performance analysis, problem troubleshooting and system optimization. Support integration with common monitoring tools, so that users can import performance monitoring and log recording data into corresponding third-party tools to realize more comprehensive monitoring and analysis, for example, users can use prometheus and pushgateway to collect time series data in flink task.

[0038] The Flink task processing module includes a Flink UDF management unit, a data source configuration unit, a resource allocation unit and an automatic deployment unit connected in sequence, wherein the Flink UDF management unit includes a custom UDF, wherein the custom UDF is used to receive the data processing logic of the Flink task.

[0039] Among them, the Flink UDF management unit is a module specially managing and processing Flink UDF (User Defined Function). It is responsible for receiving, managing and executing user-defined data processing logic, ensuring that the user's custom UDF can be called by the Flink task and take effect in task execution. Through this management unit, users can apply specific business logic to data processing tasks, allowing the system to flexibly adjust processing methods according to specific needs. Custom UDF (User Defined Function) is a function written by the user to meet specific data processing needs. UDF is one of the important features of Flink, which allows users to write complex processing logic such as data conversion, filtering, aggregation, etc. in addition to standard data processing logic. Custom UDF can handle various data operations, such as filtering invalid data, converting data formats, calculating specific field values, etc. Flink will call these UDFs at runtime, thus realizing personalized and customized data processing. Flink tasks are data processing processes executed by Flink runtime, which complete a series of real-time processing or batch processing work through operations on data streams. Flink tasks are generally composed of multiple operators (Operators), such as filtering, aggregation, joining, etc., which form a complete task by defining different data stream operation logic. Flink tasks can be real-time stream processing tasks (such as processing sensor data streams) or batch processing tasks (such as data cleaning, batch report generation, etc.). In this system, Flink tasks receive processing logic defined by users (through custom UDF), and then execute in a containerized environment, thus realizing automatic real-time analysis and processing of large-scale data.

[0040] Specifically, the Flink UDF management unit is specifically used to utilize Kubernetes technology to automatically containerize Flink tasks, realizing dynamic deployment and management of containers. This improves the scalability and reliability of tasks. Through Kubernetes, the resources required by the task are dynamically calculated, including CPU, memory, etc. The resource requirements of the task are automatically submitted to Kubernetes, thus realizing intelligent allocation and optimization of resources. Users can set the parallelism of Flink tasks to adapt to tasks of different scales and performance requirements. This helps to fully utilize computing resources and improve the efficiency of task execution. Users can dynamically adjust task parameters such as parallelism, window size, etc. during task execution. This allows users to make real-time optimization and adjustment according to task execution. Through the automatic fault tolerance mechanism of Flink, it ensures that the task can be automatically restarted when a fault occurs, ensuring the continuity and reliability of data processing. Support for version control of Flink tasks allows users to easily roll back to previous task versions. This helps to quickly fall back when tasks are updated or problems occur.

[0041] The task state monitoring unit is connected with the data source configuration unit.

[0042] The cloud computing module includes a K8S container orchestration management unit, a Docker containerization unit, and a cloud monitoring unit. The K8S container orchestration management unit and the Docker containerization unit are used to containerize the Flink tasks.

[0043] The Docker containerization unit is a component in the system that encapsulates Flink tasks and their dependencies into Docker containers. The K8S container orchestration management unit (K8S refers to Kubernetes) is a component used to manage, schedule, and orchestrate these Docker containers. Kubernetes is responsible for monitoring and managing the life cycle of containers, and automatically expanding, reducing, restarting, or reallocating container resources when needed to ensure the high availability and load balancing of applications. In this system, the Docker containerization unit is responsible for encapsulating Flink tasks into independent Docker containers, while the K8S container orchestration management unit is responsible for managing the life cycle, scheduling, and expansion of these containers. This combination ensures that Flink tasks can efficiently and reliably run in a distributed cloud environment, with good scalability and fault tolerance.

[0044] The task performance monitoring unit is connected with the cloud monitoring unit, and the automated deployment unit is connected with the Docker containerization unit.

[0045] After receiving the data processing logic, the task processing system performs resource automatic allocation through the data source configuration unit and the resource allocation unit based on the state monitoring data of the task state monitoring unit to execute the Flink task.

[0046] It is noted that the system realizes efficient execution of Flink tasks and resource optimization through automatic management and containerized deployment. It utilizes Kubernetes and Docker containerization to ensure that tasks have elastic expansion and fault tolerance in a distributed cloud environment, while supporting user-defined personalized data processing logic to adapt to complex and diverse data processing needs. This system structure significantly improves the flexibility of task management, resource utilization, and real-time processing, providing reliable support for large-scale data processing.

[0047] In one possible implementation, the data source configuration unit is specifically configured to configure the parallelism and window size of the Flink task.

[0048] It is noted that the data source configuration unit flexibly adjusts the parallelism and window size of the Flink task, enabling the task to dynamically adapt to processing capacity according to data traffic changes, optimize resource allocation, and improve the overall efficiency of the system and the accuracy of data processing, thereby meeting the needs of different business scenarios.

[0049] In a possible implementation, the Flink task processing module further includes an automatic fault tolerance unit, wherein the automatic fault tolerance unit is configured to automatically restart the Flink task in the case of Flink task failure.

[0050] It should be noted that the automatic fault tolerance unit can automatically detect and trigger restart when the Flink task fails, ensuring continuous operation of the task without interruption. In this way, even if a failure occurs, the system can quickly recover, improving the reliability of data processing tasks and the stability of the system, avoiding the risk of delay and data loss caused by human intervention.

[0051] In a possible implementation, the Flink task processing module further includes a version control unit, wherein the version control unit is configured to roll back the version of the Flink task.

[0052] It should be noted that the version control unit allows the Flink task to roll back to a previous stable version when errors or unstable conditions occur, ensuring the continuity of the task and the accuracy of data processing. This version control mechanism improves the flexibility of the system, enabling users to quickly respond to and recover from abnormal conditions, thereby significantly reducing downtime and data loss risks caused by task failures.

[0053] In a possible implementation, the cloud monitoring unit includes a prometheus tool and a pushgateway tool.

[0054] It should be noted that the cloud monitoring unit integrates Prometheus and Pushgateway tools for collecting and storing real-time performance metrics of Flink tasks, such as processing speed, latency, and resource usage. Prometheus provides powerful monitoring and alerting functions, while Pushgateway allows short-term data to be relayed and stored, ensuring that the system can continuously monitor and quickly respond to abnormalities, helping users effectively manage and optimize task execution.

[0055] For example, assume a telecom operator needs to process a large amount of real-time generated communication data, including user call records, SMS records, data traffic, etc. The operator wants to implement real-time cleaning, integration, and division of these data through a task processing system to improve operational efficiency and data quality. This step specifically includes: first, in the task management module, the operator uses the user interface of the engine to create a real-time communication data processing task. In task configuration, define the task type as real-time data processing, input data source as communication data stream, and output target as analysis result storage database. In addition, since communication data needs to be processed in real time, the operator sets the task to run all the time to realize continuous processing of data stream. The operator can monitor the status of the task at any time, including the amount of data in processing, task progress, etc. After the operator creates the task, the task will continue to run throughout the data processing life cycle to ensure real-time data. Second, in the Flink task processing module, the Flink task is automatically containerized and dynamically deployed through Kubernetes. The resource requirements of the task are automatically calculated by the engine and submitted to Kubernetes to ensure the scalability and reliability of the task. In actual operation, if the operator finds that the data processing volume suddenly increases, they can dynamically adjust the parallelism of the task through the engine interface to adapt to the change in data volume. Through user-defined Flink UDF, real-time cleaning of communication data is performed, invalid records are filtered, and data is integrated into the analysis database. During the cleaning process, the operator writes a custom Flink UDF to convert communication data format to standard format and remove abnormal communication records. Apply Flink UDF to connect multiple data sources, including user call records, SMS records, and data traffic records, and integrate them into a data stream. Through the real-time processing capability of Flink, the operator realizes real-time cleaning and integration of communication data, ensuring the timeliness and accuracy of data. During the entire data processing process, the operator uses the real-time performance monitoring interface provided by the system to understand the processing speed, delay, and other key performance indicators of the task at any time. Detailed records of the logs during task execution, including the execution of each operator, data input and output, etc., facilitate the operator to deeply understand the execution details of the task. Through machine learning and statistical analysis techniques, abnormal situations in task execution are automatically detected and timely alarms are sent to the operator, improving the reliability of the system. Moreover, the operator can review the performance data and log records of past tasks to conduct long-term performance analysis, problem troubleshooting, and system optimization.

[0056] In practical application, the complexity of data processing tasks and the cost of manual management are effectively reduced through the Flink task processing module. Users do not need to manually create, start and stop tasks, thereby improving the automation level of tasks and reducing the operation burden of users. Containerized deployment based on Kubernetes is adopted, which allows dynamic allocation of resources to meet the performance requirements of different tasks. This helps to optimize resource utilization and reduce waste of cloud computing resources, saving costs. It can be used for data cleaning, enabling users to improve data quality during data processing. Users can define complex cleaning and conversion logic to ensure the accuracy and reliability of data. A timing task management function is provided, and users can execute tasks according to a schedule. This is very important for periodic data processing tasks, enabling users to flexibly respond to different data processing requirements. Users can dynamically adjust the parameters of tasks such as parallelism, window size, etc. during task execution. This enables users to make real-time optimization and adjustment according to task execution, improving the flexibility of the system. The engine provides comprehensive and intuitive performance monitoring through real-time performance monitoring interfaces, fine-grained task log records, intelligent anomaly detection and alarm systems. Users can understand the execution status and performance indicators of tasks in a timely manner, strengthening the monitoring of system running conditions. Users can customize Flink UDF according to specific requirements and apply it to tasks. This enables the processing logic of tasks to be highly flexible and customizable, meeting the needs of various data processing scenarios.

[0057] Embodiments of the present application include the following advantages:

[0058] In the present application, the task management module is used to obtain the necessary task data, and then the Flink task processing module is used to convert the task into a Flink task stream processing framework with real-time, fault tolerance and the ability to meet the needs of large-scale data processing, and a custom UDF is set up to provide users with a quick and simple task logic setting, greatly improving task processing efficiency. And also based on the automatic fault tolerance mechanism of the Flink task stream processing framework to improve the efficiency and reliability of task processing. Then use the cloud computing module to containerize the Flink task using the K8S container orchestration management unit and the Docker containerization unit, and use the K8S container orchestration management unit to complete the automatic intelligent resource allocation and optimization of the whole process of the Flink task. Reduce manual intervention in task processing, reduce the complexity of task data processing and the cost of manual management, and improve the automation level and efficiency of task processing.

[0059] Referring to Figure 2 , a step flowchart of a task processing method provided in an embodiment of the present application is shown.

[0060] Specifically, the following steps can be included:

[0061] It should be noted that for the method embodiments, the methods can be described as a series of acts combined to achieve the stated purpose, however, those skilled in the art should know that the present embodiments are not limited to the order of the acts described, since some steps can be performed in other orders or simultaneously in accordance with the present embodiments. In addition, those skilled in the art should know that the embodiments described in the specification are all preferred embodiments, and the acts involved are not necessarily essential to the present embodiments.

[0062] The method applied to the task processing system such as the system embodiment comprises:

[0063] S1: obtaining task data of a target task through a task management module.

[0064] Among them, the task data comprises task performance data obtained based on a task performance monitoring unit.

[0065] In a possible implementation, the task performance data comprises task processing speed, task processing delay, task processing throughput and fine-grained task log.

[0066] It should be noted that the task performance data covers key indicators of task processing, including processing speed, delay, throughput and fine-grained task log. Processing speed and throughput help monitor the efficiency of the task, delay reflects the real-time response capability of the task, and fine-grained log record provides detailed information of task execution, which facilitates problem troubleshooting and performance optimization. These data enable the system to accurately monitor task performance, adjust resource allocation in a timely manner, and ensure the stability and efficiency of data processing.

[0067] In a possible implementation, the task data further comprises task type, task parameter, task input target and task output target created based on a task creation unit, task execution frequency and visual task execution schedule set based on a task timing unit, task execution status and task execution progress obtained based on a task state monitoring unit, and task creation instruction, task start instruction, task pause instruction, task stop instruction and task deletion instruction determined based on a task life cycle management unit.

[0068] It should be noted that the task management module obtains and manages detailed task data through various units, including performance data (such as processing speed, delay, throughput) and information such as task type, parameter, execution frequency and status. Through such detailed task data collection, the system can accurately control the real-time status and execution progress of the task, and support flexible task life cycle management, so that the creation, execution, pause, stop and deletion of the task can be efficiently and smoothly performed. This data structure provides a basis for the whole process monitoring and scheduling of the task, ensuring that the task is stably and efficiently executed in the system.

[0069] S2: The Flink UDF management unit of the Flink task processing module sets the data processing logic of the task data as the Flink task data.

[0070] It should be noted that the Flink UDF management unit applies the user-defined data processing logic (i.e., UDF) to the task data, converting it into the processing logic required by the Flink task. This process ensures that the task data can be flexibly processed based on specific business needs, enabling the Flink task to perform personalized data operations such as filtering, transformation, and aggregation, thereby improving the customization level and processing efficiency of the task.

[0071] S3: In the cloud computing module, the Flink task is containerized using the K8S container orchestration management unit and the Docker containerization unit.

[0072] It should be noted that the cloud computing module packages the Flink task as a container through the K8S container orchestration management unit and the Docker containerization unit, facilitating cross-platform deployment and management. Docker encapsulates the Flink task and its dependent environment into a standardized container, while K8S is responsible for managing the scheduling, expansion, and fault tolerance of these containers, ensuring that the task can efficiently and flexibly run in a distributed environment. This containerization process improves the portability of the task and the resource utilization of the system.

[0073] S4: According to the task performance data, the K8S container orchestration management unit allocates resources to the Flink task.

[0074] Specifically, the system dynamically adjusts the resource allocation of the Flink task based on real-time performance data (such as processing speed and latency) of the task through the K8S container orchestration management unit. K8S automatically allocates or recycles resources such as CPU and memory based on the current load, ensuring that the Flink task can obtain sufficient computing power during peak loads, while saving resources during low loads. This dynamic resource allocation mechanism improves the response speed and resource utilization efficiency of the system, ensuring stable operation of the task and optimizing costs.

[0075] S5: Return to step S4 until the target task is completed.

[0076] For example, in the task processing process of data cleaning, the powerful features of Flink are fully utilized, and the user-defined function (UDF) of Flink is combined to achieve efficient data processing,

[0077] Specifically, in data cleaning and conversion using Flink UDF, first, custom data format conversion, i.e., users can write Flink UDF to convert data from one format to another format, to adapt to the format requirements of different data sources. Second, in complex data filtering logic setting, users can define complex filtering logic using Flink UDF, or use the pre-provided template logic in the system to filter invalid or abnormal data according to business requirements. In addition, Flink program application in data integration and inheritance, in the process of data integration and inheritance, users can flexibly apply Flink UDF. The following functions can be implemented: connecting multiple data sources: users can write Flink UDF to handle the connection logic of multiple data sources, and integrate data into a view. Complex logic in data inheritance: users can define Flink UDF to be applied to the data inheritance process to perform complex logic to improve data quality. Real-time data cleaning and integration: the engine combines the real-time processing capabilities of Flink to support real-time data cleaning and integration: real-time data cleaning: using Flink's window operation, users can clean streaming data in real time to ensure the timeliness and accuracy of data. Real-time data integration: users can define Flink UDF to integrate data from different sources into a data stream in real time, making data processing more real-time. Dynamic adjustment of data processing flow: the engine provides the ability to dynamically adjust the data processing flow, combined with the flexibility of Flink: process monitoring and performance analysis: the engine monitors the data processing flow in real time, collects performance indicators, throughput, and other information during task execution. Automated process adjustment suggestions: based on monitoring results, the engine can generate automated process adjustment suggestions to guide users on how to optimize task flow, such as changing operator order, adjusting window size, etc. Manual process adjustment: users can manually adjust the process based on monitoring information, such as adding or deleting a certain operator, adjusting the parallelism of the task, etc., to optimize the performance and efficiency of the task.

[0078] In practical application, this task processing method acquires detailed task data through the task management module and flexibly defines data processing logic using Flink's UDF, thereby realizing personalized cleaning, conversion, and integration of data. Through K8S and Docker containerization, this method enables Flink tasks to have cross-platform deployment capabilities and dynamically allocates resources based on real-time performance data to ensure efficient and stable operation of tasks. In addition, the system automatically generates process optimization suggestions based on monitoring data, supports users to manually adjust the task flow, thereby improving the accuracy, real-time performance, and resource utilization efficiency of data processing, making task management more flexible and intelligent.

[0079] Embodiments of the present application include the following advantages:

[0080] In the present application, the task management module is used to obtain the necessary task data, and then the Flink task processing module is used to convert the task into the Flink task flow processing framework which has real-time performance, fault tolerance and can meet the demand of large-scale data processing, and a custom UDF is set to provide users with quick and simple task logic setting, greatly improving the task processing efficiency. And based on the automatic fault tolerance mechanism of the Flink task flow processing framework, the efficiency and reliability of task processing are improved. Then the cloud computing module is used to containerize the Flink task by using the K8S container orchestration management unit and the Docker containerization unit, and the K8S container orchestration management unit is used to complete the automatic intelligent resource allocation and optimization of the whole process of the Flink task. Reduce the manual intervention of task processing, reduce the complexity and manual management cost of task data processing, and improve the automation degree and efficiency of task processing.

[0081] For the method embodiment, it is basically similar to the system embodiment, so the description is relatively simple, and the relevant part can be referred to the part of the method embodiment.

[0082] In addition, the embodiment of the present application also provides an electronic device, such as Figure 3 As shown in the figure, it includes a processor 1301, a communication interface 1302, a memory 1303 and a communication bus 1304, wherein the processor 1301, the communication interface 1302 and the memory 1303 complete the communication among each other through the communication bus 1304,

[0083] The memory 1303 is used to store computer programs;

[0084] The processor 1301 is used to execute the programs stored in the memory 1303, and realize the task processing method of the method embodiment.

[0085] The communication bus mentioned in the above terminal can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The communication bus can be divided into address bus, data bus, control bus, etc. For the convenience of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0086] The communication interface is used for the communication between the above terminal and other devices.

[0087] The memory can include a random access memory (RAM) and can also include a non-volatile memory such as at least one disk memory. Optionally, the memory can also be at least one storage device located remotely from the aforementioned processor.

[0088] The aforementioned processor can be a general-purpose processor including a central processing unit (CPU), a network processor (NP), etc., and can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component.

[0089] As shown in the above Figure 4 In yet another embodiment provided by the present application, a computer readable storage medium 1401 is provided, and the computer readable storage medium 1401 stores instructions, which, when executed on a computer, cause the computer to perform the task processing method described in the above embodiments.

[0090] In yet another embodiment provided by the present application, a computer program product containing instructions is provided, which, when executed on a computer, cause the computer to perform the task processing method described in the above embodiments.

[0091] In the embodiments described above, all or some of the steps can be implemented by software, hardware, firmware or any combination thereof. When implemented in software, all or some of the steps can be implemented in the form of one or more computer programs which are stored in a computer readable storage medium. The computer readable storage medium can be located in a computing device which is in operation. These computer programs (which may

[0092] It should be noted that, in the present document, the terms such as first and second are used only to differentiate one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Also, the terms "comprising", "containing" or any other similar term are intended to encompass non-exclusive inclusion such that a process, method, article or apparatus that comprises a list of elements does not necessarily include only those elements in the list, but can include other elements not expressly listed or inherent to such process, method, article or apparatus.

[0093] Each of the embodiments in the present document is described in a related manner, and the same or similar parts among the embodiments can be referred to each other. Each of the embodiments focuses on the difference from other embodiments. In particular, for the system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments.

[0094] The above merely provides the preferred embodiments of the application, and not intended to limit the protection scope of the application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the application shall fall within the protection scope of the application.

Claims

1. A task processing system, characterized by, The system comprises a task management module, a Flink task processing module and a cloud computing module. The task management module comprises a task creation unit, a task timing unit, a task state monitoring unit, a task life cycle management unit and a task performance monitoring unit. The Flink task processing module comprises a Flink UDF management unit, a data source configuration unit, a resource allocation unit and an automatic deployment unit connected in sequence, wherein the Flink UDF management unit comprises a custom UDF, wherein the custom UDF is used to receive data processing logic of a Flink task. The task state monitoring unit is connected with the data source configuration unit. The cloud computing module comprises a K8S container orchestration management unit, a Docker containerization unit and a cloud monitoring unit, wherein the K8S container orchestration management unit and the Docker containerization unit are used to containerize the Flink task. The task performance monitoring unit is connected with the cloud monitoring unit, and the automatic deployment unit is connected with the Docker containerization unit. After receiving the data processing logic, the task processing system performs resource automatic allocation through the data source configuration unit and the resource allocation unit based on the state monitoring data of the task state monitoring unit to execute the Flink task.

2. The task processing system according to claim 1, characterized by The data source configuration unit is specifically configured to configure the parallelism and window size of the Flink task.

3. The task processing system of claim 1, wherein, The Flink task processing module further comprises an automatic fault tolerance unit, wherein the automatic fault tolerance unit is used to automatically restart the Flink task in the case of failure of the Flink task.

4. The task processing system of claim 1, wherein, The Flink task processing module further comprises a version control unit, wherein the version control unit is used to roll back the version of the Flink task.

5. The task processing system of claim 1, wherein, The cloud monitoring unit comprises a prometheus tool and a pushgateway tool.

6. A task processing method characterized by, The method applied to the task processing system of any one of claims 1-5, comprising: S1: obtaining task data of a target task through the task management module, wherein the task data comprises task performance data obtained based on the task performance monitoring unit; S2: setting data processing logic of the task data as Flink task data through the Flink UDF management unit of the Flink task processing module; S3: containerizing the Flink task by using the K8S container orchestration management unit and the Docker containerization unit in the cloud computing module; S4: allocating resources for the Flink task by using the K8S container orchestration management unit according to the task performance data; S5: returning to step S4 until the target task is executed.

7. The task processing method according to claim 6, characterized by, The task performance data comprises task processing speed, task processing delay, task processing throughput and fine-grained task logs.

8. The task processing method according to claim 6, characterized by, The task data further comprises a task type, a task parameter, a task input target and a task output target created based on the task creating unit, a task execution frequency and a visual task execution schedule set based on the task timing unit, a task execution state and a task execution progress acquired based on the task state monitoring unit, and a task creation instruction, a task start instruction, a task pause instruction, a task stop instruction and a task deletion instruction determined based on the task life cycle management unit.

9. An electronic device, comprising: comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete communication with each other through the communication bus; The memory is used for storing a computer program. The processor is used for executing the program stored on the memory, and realizes the task processing method in any one of claims 6-8. 10.A computer readable medium having instructions stored thereon that, when executed by one or more processors, cause the processors to perform the task processing method in any one of claims 6-8.

Images