Task processing system and method, electronic equipment and readable medium

Abstract

The embodiment of the invention provides a task processing system and method, electronic equipment and a readable medium. The system comprises a task management module, an Flink task processing module and a cloud computing module. The task management module comprises a task creating unit, a task timing unit, a task state monitoring unit, a task life cycle management unit and a task performance monitoring unit; the Flink task processing module comprises an Flink UDF management unit, a data source configuration unit, a resource allocation unit and an automatic deployment unit which are connected in sequence; the cloud computing module comprises a K8S container arrangement management unit, a Docker containerization unit and a cloud monitoring unit; and after receiving the data processing logic, the task processing system automatically allocates resources through the data source configuration unit and the resource allocation unit based on the state monitoring data of the task state monitoring unit so as to execute the Flink task.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a task processing system, a task processing method, an electronic device and a computer-readable medium. Background Art

[0002] Flink is a distributed stream processing framework, mainly used for real-time data processing and large-scale data analysis. It can process continuously incoming data streams quickly and stably, and can also process batch data, so it is suitable for a variety of scenarios, such as real-time data analysis, monitoring, and large-scale batch processing tasks. Containerization is a technology that packages an application and its dependencies into a "container" so that it can run in any environment that supports containers. This technology provides isolation, portability, and consistency of applications.

[0003] Automated deployment of tasks can significantly improve system operation efficiency, reduce the error rate of human intervention, and significantly shorten the time it takes to launch an application. Automated deployment can dynamically adjust resources and configurations based on actual needs to meet real-time changing workloads, help improve resource utilization, ensure application stability and rapid recovery, and support the system's continuous integration and delivery process.

[0004] However, the setup, management and scheduling of existing task processing processes all require a lot of manual intervention, which greatly reduces the efficiency of task processing and increases subjective errors caused by excessive human intervention. Summary of the invention

[0005] The embodiments of the present invention provide a task processing system, method, electronic device and computer-readable storage medium to solve the problem that the setting, management and scheduling of the existing task processing process all require a large amount of manual intervention, which greatly reduces the task processing efficiency and increases the subjective errors caused by excessive human intervention.

[0006] The embodiment of the present invention discloses a task processing system, characterized in that the system includes a task management module, a Flink task processing module and a cloud computing module;

[0007] The task management module includes a task creation unit, a task timing unit, a task status monitoring unit, a task life cycle management unit and a task performance monitoring unit;

[0008] The Flink task processing module includes a Flink UDF management unit, a data source configuration unit, a resource allocation unit, and an automatic deployment unit connected in sequence, wherein the Flink UDF management unit includes a custom UDF, wherein the custom UDF is used to receive the data processing logic of the Flink task;

[0009] The task status monitoring unit is connected to the data source configuration unit;

[0010] The cloud computing module includes the K8S container orchestration management unit, the Docker containerization unit, and the cloud monitoring unit. The K8S container orchestration management unit and the Docker containerization unit are used to containerize Flink tasks.

[0011] The task performance monitoring unit is connected to the cloud monitoring unit, and the automated deployment unit is connected to the Docker containerization unit;

[0012] After receiving the data processing logic, the task processing system automatically allocates resources through the data source configuration unit and the resource allocation unit based on the status monitoring data of the task status monitoring unit to execute the Flink task.

[0013] The embodiment of the present invention further discloses a task processing method, which is characterized in that it is applied to the task processing system of the embodiment of the present invention, and the method includes:

[0014] S1: Acquire task data of a target task through a task management module, wherein the task data includes task performance data acquired based on a task performance monitoring unit;

[0015] S2: Set the data processing logic of the task data to Flink task data through the Flink UDF management unit of the Flink task processing module;

[0016] S3: In the cloud computing module, the Flink task is containerized using the K8S container orchestration management unit and the Docker containerization unit;

[0017] S4: Based on the task performance data, use the K8S container orchestration management unit to allocate resources for the Flink task;

[0018] S5: Return to step S4 until the target task is completed.

[0019] The embodiment of the present invention further discloses an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0020] Memory, used to store computer programs;

[0021] The processor is used to implement the method of the embodiment of the present invention when executing the program stored in the memory.

[0022] The embodiment of the present invention further discloses one or more computer-readable media on which instructions are stored. When executed by one or more processors, the processors execute the method according to the embodiment of the present invention.

[0023] The embodiments of the present invention include the following advantages:

[0024] In the present invention, the task management module is used to obtain the necessary task data, and then the Flink task processing module is used to convert the task into the Flink task flow processing framework that is real-time, fault-tolerant and can meet the needs of large-scale data processing, and a custom UDF is set to provide users with fast and concise task logic settings, greatly improving the efficiency of task processing. In addition, the automatic fault-tolerant mechanism based on the Flink task flow processing framework improves the efficiency and reliability of task processing. After that, the cloud computing module is used to containerize the Flink task using the K8S container orchestration management unit and the Docker containerization unit, and the K8S container orchestration management unit is used to complete the automatic intelligent resource allocation and optimization of the entire Flink task process. Reduce the manual intervention of task processing, reduce the complexity of task data processing and the manual management cost, and improve the automation degree and efficiency of task processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a structural block diagram of a task processing system provided in an embodiment of the present invention;

[0026] Figure 2 is a step flow chart of a task processing method provided in an embodiment of the present invention;

[0027] Figure 3 is a block diagram of an electronic device provided in an embodiment of the present invention;

[0028] Figure 4 is a schematic diagram of a computer-readable medium provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0030] Reference Figure 1 , shows a structural block diagram of a task processing system provided in an embodiment of the present invention.

[0031] The system includes a task management module, a Flink task processing module and a cloud computing module.

[0032] The task management module includes a task creation unit, a task timing unit, a task status monitoring unit, a task life cycle management unit and a task performance monitoring unit.

[0033] Among them, the task creation unit is specifically used to provide an intuitive user interface through which users can easily create and configure data processing tasks. The interface includes fields such as task name, task type, task description, input data source, and output target, so that users can clearly define the basic properties of the task. Users can choose different task types, such as Flink tasks, data cleaning tasks, etc., to configure tasks according to specific needs. This makes the engine suitable for a variety of data processing scenarios. Users can set specific parameters for tasks, such as the parallelism and window size of Flink tasks. These parameters vary depending on the task type to meet the personalized needs of different tasks. Users can clearly define the input data source and output target of the task, including connecting to a database, reading files, and obtaining API data. This simplifies the input and output configuration process of the task.

[0034] Among them, the task timing unit is specifically used to support users to define the execution frequency and time of tasks, and realize the scheduling of scheduled tasks. Users can choose different scheduling strategies such as one-time execution, daily execution, and weekly execution to adapt to different data processing needs. Users can set the execution schedule of tasks to ensure that tasks are executed at the specified time. This helps to implement periodic data processing tasks, such as daily report generation and weekly data aggregation. Users can view the scheduling plan of tasks through the visual interface to understand information such as when tasks are executed and the execution frequency, which improves the user's control over task scheduling.

[0035] The task status monitoring unit is specifically used to provide real-time task status monitoring functions. Users can view the current status of tasks at any time, including being executed, paused, completed, failed, etc. Users can view the execution progress of tasks and understand the current processing stage of the task. This helps users to understand the execution status of tasks in a timely manner and make necessary adjustments and interventions.

[0036] The task lifecycle management unit specifically provides users with the ability to create tasks and start them manually through an interface, or to set scheduled tasks for automatic start. Once a task is created, the engine will start the corresponding data processing task according to the user's configuration. Users can manually pause and stop the execution of tasks. The pause operation can be used to temporarily interrupt the execution of a task, while the stop operation terminates the entire lifecycle of the task. Users can delete tasks that are no longer needed to free up system resources. This ensures the controllability and efficient management of the task lifecycle.

[0037] Among them, the task performance monitoring unit is specifically used to record important indicators in the task process when performing flink tasks, and can analyze the status of its operation process from the perspective of operation and maintenance. Provide an intuitive real-time performance monitoring interface to display key performance indicators of tasks, such as processing speed, latency, throughput, etc. Through graphical methods, users can quickly understand the execution status of tasks. Implement fine-grained logging of the task execution process, including the execution status of each operator, data input and output, etc. This meticulous logging helps users to gain an in-depth understanding of the execution details of tasks. Using machine learning and statistical analysis techniques, the engine implements an intelligent anomaly detection and alarm system. The system can automatically identify abnormal situations in task execution and issue alarms to users in a timely manner, improving the reliability of task execution. Provide a task history tracing function, and users can look back to view the performance data and log records of previous tasks. This helps users to conduct long-term performance analysis, troubleshooting, and system optimization. It supports integration with common monitoring tools, allowing users to import performance monitoring and logging data into corresponding third-party tools for more comprehensive monitoring and analysis. For example, users can use Prometheus and Pushgateway to collect time series data in Flink tasks.

[0038] The Flink task processing module includes a Flink UDF management unit, a data source configuration unit, a resource allocation unit, and an automatic deployment unit connected in sequence, wherein the Flink UDF management unit includes a custom UDF, wherein the custom UDF is used to receive the data processing logic of the Flink task.

[0039] Among them, the Flink UDF management unit is a module dedicated to managing and processing Flink UDF (user defined function). It is responsible for receiving, managing and executing user-defined data processing logic, ensuring that the user's customized UDF can be called by Flink tasks and take effect in task execution. Through this management unit, users can apply specific business logic to data processing tasks, so that the system can flexibly adjust the processing method according to specific needs. Custom UDF (User Defined Function) is a function written by users to meet specific data processing needs. UDF is one of the important features of Flink. It allows users to write complex processing logic such as data conversion, filtering, aggregation, etc. in addition to standard data processing logic. Custom UDF can handle various data operations, such as filtering invalid data, converting data formats, calculating the value of specific fields, etc. Flink will call these UDFs at runtime to achieve personalized and customized data processing. Flink task is the data processing flow executed by Flink at runtime. It completes a series of real-time processing or batch processing tasks by operating on data streams. Flink tasks are generally composed of multiple operators, such as filtering, aggregation, connection, etc., and a complete task is formed by defining different data stream operation logic. Flink tasks can be real-time stream processing tasks (such as processing sensor data streams) or batch processing tasks (such as data cleaning, batch report generation, etc.). In this system, Flink tasks receive processing logic defined by the user (through custom UDFs) and then execute them in a containerized environment, thereby achieving automatic real-time analysis and processing of large-scale data.

[0040] Specifically, the Flink UDF management unit is specifically used to use Kubernetes technology to automate Flink tasks into containers and realize dynamic deployment and management of containers. This improves the scalability and reliability of tasks. The resources required for tasks, including CPU, memory, etc., are dynamically calculated through Kubernetes. The resource requirements of tasks are automatically submitted to Kubernetes, thereby realizing intelligent allocation and optimization of resources. Users are allowed to set the parallelism of Flink tasks to adapt to tasks of different scales and performance requirements. This helps to make full use of computing resources and improve the execution efficiency of tasks. Users can dynamically adjust the parameters of tasks, such as parallelism, window size, etc., while tasks are running. This enables users to optimize and adjust in real time according to the execution of tasks. Through Flink's automatic fault-tolerance mechanism, it is ensured that tasks can be automatically restarted in the event of a failure, ensuring the continuity and reliability of data processing. Version control of Flink tasks is supported, and users can easily roll back to the previous task version. This helps to quickly roll back when tasks are updated or problems occur.

[0041] The task status monitoring unit is connected to the data source configuration unit.

[0042] The cloud computing module includes a K8S container orchestration management unit, a Docker containerization unit, and a cloud monitoring unit. The K8S container orchestration management unit and the Docker containerization unit are used to containerize Flink tasks.

[0043] Among them, the Docker containerization unit is a component in the system that encapsulates Flink tasks and their dependencies as Docker containers. The K8S container orchestration management unit (K8S refers to Kubernetes) is a component used to manage, schedule, and orchestrate these Docker containers. Kubernetes is responsible for monitoring and managing the life cycle of containers, and automatically expanding, reducing, restarting, or reallocating container resources when necessary to ensure high availability and load balancing of applications. In this system, the Docker containerization unit is responsible for encapsulating Flink tasks into independent Docker containers, while the K8S container orchestration management unit is responsible for managing the life cycle, scheduling, and expansion of these containers. This combination ensures that Flink tasks can run efficiently and reliably in a distributed cloud environment, and has good scalability and fault tolerance.

[0044] The task performance monitoring unit is connected to the cloud monitoring unit, and the automated deployment unit is connected to the Docker containerization unit.

[0045] After receiving the data processing logic, the task processing system automatically allocates resources through the data source configuration unit and the resource allocation unit based on the status monitoring data of the task status monitoring unit to execute the Flink task.

[0046] It should be noted that the system achieves efficient execution and resource optimization of Flink tasks through automated management and containerized deployment. It uses Kubernetes and Docker containerization to ensure that tasks have elastic expansion and fault tolerance in a distributed cloud environment, while supporting user-defined personalized data processing logic to adapt to complex and diverse data processing needs. This system structure significantly improves the flexibility of task management, resource utilization, and real-time processing, providing reliable support for large-scale data processing.

[0047] In a possible implementation, the data source configuration unit is specifically used to configure the parallelism and window size of the Flink task.

[0048] It should be noted that the parallelism and window size of Flink tasks can be flexibly adjusted through the data source configuration unit, so that the task can dynamically adapt the processing capacity according to the changes in data flow, optimize resource allocation, improve the overall efficiency of the system and the accuracy of data processing, thereby meeting the needs of different business scenarios.

[0049] In a possible implementation, the Flink task processing module further includes an automatic fault-tolerance unit, wherein the automatic fault-tolerance unit is used to automatically restart the Flink task in the event of a Flink task failure.

[0050] It should be noted that the automatic fault-tolerance unit can automatically detect and trigger a restart when a Flink task fails, ensuring that the task continues to run without interruption. In this way, even if a failure occurs, the system can recover quickly, improving the reliability of data processing tasks and the stability of the system, and avoiding the risk of delays and data loss caused by human intervention.

[0051] In a possible implementation, the Flink task processing module further includes a version control unit, wherein the version control unit is used to roll back the version of the Flink task.

[0052] It should be noted that the version control unit allows Flink tasks to roll back to the previous stable version when errors or instability occur, ensuring task continuity and data processing accuracy. This version control mechanism improves the flexibility of the system, allowing users to quickly respond to and recover from abnormal situations, thereby significantly reducing downtime and data loss risks caused by task errors.

[0053] In a possible implementation, the cloud monitoring unit includes a prometheus tool and a pushgateway tool.

[0054] It should be noted that the cloud monitoring unit integrates Prometheus and Pushgateway tools to collect and store real-time performance indicators of Flink tasks, such as processing speed, latency, and resource usage. Prometheus provides powerful monitoring and alerting capabilities, while Pushgateway allows for relay storage of short-term data, ensuring that the system can continuously monitor and quickly respond to anomalies, helping users effectively manage and optimize task execution.

[0055] For example, suppose that a telecom operator needs to process a large amount of communication data generated in real time, including user call records, SMS records, data traffic, etc. The operator hopes to use the task processing system to clean, integrate and analyze these data in real time to improve operational efficiency and data quality. This step specifically includes: First, in the task management module, the operator uses the engine's user interface to create a real-time communication data processing task. In the task configuration, the task type is defined as real-time data processing, the input data source is the communication data stream, and the output target is the analysis result storage database. In addition, since the communication data needs to be processed in real time, the operator sets the task to run all the time to achieve continuous processing of the data stream. The operator can monitor the status of the task at any time, including the amount of data being processed, the progress of the task, etc. After the operator creates the task, the task will continue to run throughout the data processing life cycle to ensure the real-time nature of the data. Secondly, in the Flink task processing module, the Flink task is automated as a container and dynamically deployed through Kubernetes. The resource requirements of the task are automatically calculated by the engine and submitted to Kubernetes to ensure the scalability and reliability of the task. In actual operation, if operators find that the amount of data processing increases suddenly, they can dynamically adjust the parallelism of tasks through the engine interface to adapt to the change in data volume. Through user-defined Flink UDF, communication data is cleaned in real time, invalid records are filtered, and data is integrated into the analysis database. During the cleaning process, the operator wrote a custom Flink UDF to convert the communication data format into a standard format, remove abnormal communication records, etc. Flink UDF is applied to connect multiple data sources, including user call records, SMS records, and data traffic records, and integrate them into a data stream. Through Flink's real-time processing capabilities, operators have achieved real-time cleaning and integration of communication data to ensure the timeliness and accuracy of data. Throughout the data processing process, operators use the real-time performance monitoring interface provided by the system to keep abreast of key performance indicators such as task processing speed and latency. The logs during task execution are recorded in detail, including the execution status of each operator, data input and output, etc., which facilitates operators to gain in-depth understanding of the execution details of the task. Through machine learning and statistical analysis techniques, abnormal situations in task execution are automatically detected, and warnings are issued to operators in a timely manner, improving the reliability of the system. Furthermore, operators can retroactively review performance data and log records from past missions for long-term performance analysis, troubleshooting, and system optimization.

[0056] In the actual application process, the complexity and manual management cost of data processing tasks are effectively reduced through the Flink task processing module. Users do not need to manually create, start and stop tasks, which improves the automation of tasks and reduces the operational burden of users. The containerized deployment based on Kubernetes allows dynamic allocation of resources to meet the performance requirements of different tasks. This helps to optimize resource utilization, reduce the waste of cloud computing resources and save costs. It can be used for data cleaning, enabling users to improve data quality during data processing. Users can define complex cleaning and transformation logic to ensure the accuracy and credibility of data. It provides a scheduled task management function, allowing users to execute tasks according to a schedule. This is very important for periodic data processing tasks, allowing users to flexibly respond to different data processing needs. It allows users to dynamically adjust task parameters such as parallelism, window size, etc. when the task is running. This enables users to optimize and adjust in real time according to the execution of tasks, improving the flexibility of the system. Through the real-time performance monitoring interface, fine-grained task logging, intelligent anomaly detection and alarm system, the engine provides comprehensive and intuitive performance monitoring. Users can understand the execution status and performance indicators of tasks in a timely manner, strengthening the monitoring of system operation status. Users can customize Flink UDF according to specific needs and apply it to tasks. This makes the processing logic of tasks highly flexible and customizable to meet the needs of various data processing scenarios.

[0057] The embodiments of the present invention include the following advantages:

[0058] In the present invention, the task management module is used to obtain the necessary task data, and then the Flink task processing module is used to convert the task into the Flink task flow processing framework that is real-time, fault-tolerant and can meet the needs of large-scale data processing, and a custom UDF is set to provide users with fast and concise task logic settings, greatly improving the efficiency of task processing. In addition, the automatic fault-tolerant mechanism based on the Flink task flow processing framework improves the efficiency and reliability of task processing. After that, the cloud computing module is used to containerize the Flink task using the K8S container orchestration management unit and the Docker containerization unit, and the K8S container orchestration management unit is used to complete the automatic intelligent resource allocation and optimization of the entire Flink task process. Reduce the manual intervention of task processing, reduce the complexity of task data processing and the manual management cost, and improve the automation degree and efficiency of task processing.

[0059] Reference Figure 2 , shows a step flow chart of a task processing method provided in an embodiment of the present invention.

[0060] The specific steps may include:

[0061] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0062] The method applied to the task processing system of the system embodiment includes:

[0063] S1: Obtain task data of the target task through the task management module.

[0064] The task data includes task performance data obtained by a task performance monitoring unit.

[0065] In a possible implementation, the task performance data includes task processing speed, task processing delay, task processing throughput, and fine-grained task logs.

[0066] It should be noted that task performance data covers key indicators of task processing, including processing speed, latency, throughput, and fine-grained task logs. Processing speed and throughput help monitor the efficiency of tasks, latency reflects the real-time response capability of tasks, and fine-grained log records provide detailed information on task execution, making it easier to troubleshoot problems and optimize performance. These data enable the system to accurately monitor task performance, adjust resource allocation in a timely manner, and ensure the stability and efficiency of data processing.

[0067] In a possible implementation, the task data also includes the task type, task parameters, task input targets and task output targets created by the task creation unit, the task execution frequency and visual task execution schedule set by the task timing unit, the task execution status and task execution progress obtained by the task status monitoring unit, and the task creation instructions, task start instructions, task pause instructions, task stop instructions and task deletion instructions determined by the task lifecycle management unit.

[0068] It should be noted that the task management module obtains and manages detailed task data through various units, including performance data (such as processing speed, delay, throughput) and information such as task type, parameters, execution frequency and status. Through this meticulous task data collection, the system can accurately control the real-time status and execution progress of tasks, and support flexible task lifecycle management, so that operations such as task creation, execution, suspension, stop and deletion can be carried out efficiently and smoothly. This data structure provides the basis for the full monitoring and scheduling of tasks, ensuring that tasks are executed stably and efficiently in the system.

[0069] S2: Set the data processing logic of the task data to Flink task data through the Flink UDF management unit of the Flink task processing module.

[0070] It should be noted that the Flink UDF management unit applies user-defined data processing logic (i.e., UDF) to task data and converts it into the processing logic required by the Flink task. This process ensures that task data can be flexibly processed based on specific business requirements, enabling Flink tasks to perform personalized data operations such as filtering, conversion, and aggregation, thereby improving task customization and processing efficiency.

[0071] S3: In the cloud computing module, the Flink tasks are containerized using the K8S container orchestration management unit and the Docker containerization unit.

[0072] It should be noted that the cloud computing module packages Flink tasks into containers through the K8S container orchestration management unit and the Docker containerization unit for easy cross-platform deployment and management. Docker encapsulates Flink tasks and their dependent environments into standardized containers, while K8S is responsible for managing the scheduling, expansion, and fault tolerance of these containers to ensure that tasks can run efficiently and flexibly in a distributed environment. This containerization process improves the portability of tasks and the resource utilization of the system.

[0073] S4: Based on the task performance data, use the K8S container orchestration management unit to allocate resources for Flink tasks.

[0074] Specifically, the system dynamically adjusts the resource allocation of Flink tasks through the K8S container orchestration management unit based on the real-time performance data of the task (such as processing speed and latency). K8S will automatically allocate or recycle CPU, memory and other resources according to the current load situation to ensure that Flink tasks can obtain sufficient computing power during peak loads and save resources when the load decreases. This dynamic resource allocation mechanism improves the system's response speed and resource utilization efficiency, ensuring stable operation of tasks and optimizing costs.

[0075] S5: Return to step S4 until the target task is completed.

[0076] For example, in the process of data cleaning task processing, we can make full use of the powerful features of Flink and combine it with Flink's user-defined functions (UDF) to achieve efficient data processing.

[0077] Specifically, in the data cleaning and conversion of Flink UDF, first of all, the custom data format conversion, that is, users can write Flink UDF to convert data from one format to another format to adapt to the format requirements of different data sources. Secondly, in the complex data filtering logic setting, users can use Flink UDF to define complex filtering logic, or use the template logic pre-provided in the system to filter invalid or abnormal data according to business needs. In addition, Flink program application in data integration and inheritance, in the process of data integration and inheritance, users can flexibly apply Flink UDF. The following functions can be realized: Connecting multiple data sources: Users can write Flink UDF to handle the connection logic of multiple data sources and integrate data into one view. Complex logic in data inheritance: Users can define Flink UDF to apply to the data inheritance process and execute complex logic to improve data quality. Real-time data cleaning and integration: The engine combines Flink's real-time processing capabilities to support real-time data cleaning and integration: Real-time data cleaning: Using Flink's window operations, users can clean streaming data in real time to ensure the timeliness and accuracy of data. Real-time data integration: Users can define Flink UDF to integrate data from different sources into a data stream in real time, making data processing more real-time. Dynamically adjust the data processing flow: The engine provides the ability to dynamically adjust the data processing flow, combined with Flink's flexibility: Process monitoring and performance analysis: The engine monitors the data processing flow in real time and collects performance indicators, throughput and other information during task execution. Automated process adjustment suggestions: Based on the monitoring results, the engine can generate automated process adjustment suggestions to guide users on how to optimize the task flow, such as changing the order of operators, adjusting the window size, etc. Manual process adjustment: Users can manually adjust the process based on monitoring information, such as adding or deleting an operator, adjusting the parallelism of the task, etc., to optimize the performance and efficiency of the task.

[0078] In actual application, this task processing method obtains detailed task data through the task management module, and uses Flink's UDF to flexibly define data processing logic, thereby realizing personalized data cleaning, conversion, and integration. Through K8S and Docker containerization, this method enables Flink tasks to be deployed across platforms, and dynamically allocates resources through real-time performance data to ensure efficient and stable operation of tasks. In addition, the system automatically generates process optimization suggestions based on monitoring data, and supports users to manually adjust task processes, thereby improving the accuracy, real-time, and resource utilization efficiency of data processing, making task management more flexible and intelligent.

[0079] The embodiments of the present invention include the following advantages:

[0080] In the present invention, the task management module is used to obtain the necessary task data, and then the Flink task processing module is used to convert the task into the Flink task flow processing framework that is real-time, fault-tolerant and can meet the needs of large-scale data processing, and a custom UDF is set to provide users with fast and concise task logic settings, greatly improving the efficiency of task processing. In addition, the automatic fault-tolerant mechanism based on the Flink task flow processing framework improves the efficiency and reliability of task processing. After that, the cloud computing module is used to containerize the Flink task using the K8S container orchestration management unit and the Docker containerization unit, and the K8S container orchestration management unit is used to complete the automatic intelligent resource allocation and optimization of the entire Flink task process. Reduce the manual intervention of task processing, reduce the complexity of task data processing and the manual management cost, and improve the automation degree and efficiency of task processing.

[0081] As for the method embodiment, since it is basically similar to the system embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0082] In addition, an embodiment of the present invention further provides an electronic device, such as Figure 3 As shown, it includes a processor 1301, a communication interface 1302, a memory 1303 and a communication bus 1304, wherein the processor 1301, the communication interface 1302, and the memory 1303 communicate with each other through the communication bus 1304.

[0083] Memory 1303, used for storing computer programs;

[0084] The processor 1301 is used to implement the task processing method of the method embodiment when executing the program stored in the memory 1303.

[0085] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0086] The communication interface is used for communication between the above terminal and other devices.

[0087] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0088] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0089] like Figure 4 As shown, in another embodiment provided by the present invention, a computer-readable storage medium 1401 is also provided, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the task processing method described in the above embodiment.

[0090] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer executes the task processing method described in the above embodiment.

[0091] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0092] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0093] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0094] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A task processing system, characterized in that: The system includes a task management module, a Flink task processing module, and a cloud computing module; The task management module includes a task creation unit, a task timing unit, a task status monitoring unit, a task life cycle management unit and a task performance monitoring unit; The Flink task processing module includes a Flink UDF management unit, a data source configuration unit, a resource allocation unit, and an automatic deployment unit connected in sequence, wherein the Flink UDF management unit includes a custom UDF, wherein the custom UDF is used to receive the data processing logic of the Flink task; The task status monitoring unit is connected to the data source configuration unit; The cloud computing module includes a K8S container orchestration management unit, a Docker containerization unit and a cloud monitoring unit, wherein the K8S container orchestration management unit and the Docker containerization unit are used to containerize the Flink task; The task performance monitoring unit is connected to the cloud monitoring unit, and the automatic deployment unit is connected to the Docker containerization unit; After receiving the data processing logic, the task processing system automatically allocates resources through the data source configuration unit and the resource allocation unit based on the status monitoring data of the task status monitoring unit to execute the Flink task.

2. The task processing system according to claim 1, characterized in that: The data source configuration unit is specifically used to configure the parallelism and window size of the Flink task.

3. The task processing system according to claim 1, characterized in that: The Flink task processing module further includes an automatic fault-tolerant unit, wherein the automatic fault-tolerant unit is configured to automatically restart the Flink task in the event of a failure of the Flink task.

4. The task processing system according to claim 1, characterized in that: The Flink task processing module also includes a version control unit, wherein the version control unit is used to roll back the version of the Flink task.

5. The task processing system according to claim 1, characterized in that: The cloud monitoring unit includes a prometheus tool and a pushgateway tool.

6. A task processing method, characterized in that: Applied to the task processing system according to any one of claims 1 to 5, the method comprises: S1: Acquire task data of a target task through the task management module, wherein the task data includes task performance data acquired based on the task performance monitoring unit; S2: Setting the data processing logic of the task data as Flink task data through the Flink UDF management unit of the Flink task processing module; S3: In the cloud computing module, the Flink task is containerized by using the K8S container orchestration management unit and the Docker containerization unit; S4: Allocate resources for the Flink task using the K8S container orchestration management unit according to the task performance data; S5: Return to step S4 until the target task is completed.

7. The task processing method according to claim 6, characterized in that: The task performance data includes task processing speed, task processing delay, task processing throughput and fine-grained task logs.

8. The task processing method according to claim 6, characterized in that: The task data also includes the task type, task parameters, task input targets and task output targets created by the task creation unit, the task execution frequency and visual task execution schedule set by the task timing unit, the task execution status and task execution progress obtained by the task status monitoring unit, and the task creation instructions, task start instructions, task pause instructions, task stop instructions and task deletion instructions determined by the task lifecycle management unit.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the task processing method according to any one of claims 6 to 8 when executing the program stored in the memory.

10. A computer-readable medium having instructions stored thereon, which, when executed by one or more processors, causes the processors to execute the task processing method according to any one of claims 6 to 8.

Images