Big data real-time computing component deployment and operation method and system

Abstract

The invention mainly provides a big data real-time computing component deployment and operation method and system. The method comprises the steps that computing resource tenants are isolated, and a user configures Flink cluster parameters through a graphical configuration interface; after the platform receives the Flink cluster parameters, a resource list meeting the Kubernetes standard is dynamically generated in combination with a preset template library and a tenant strategy rule, and an interface packaged based on a Kubernetes API is called to complete deployment; after the cluster is started, Flink operation is continuously monitored and early warned, and the number of TaskManager instances is dynamically adjusted through an intelligent elastic telescopic controller according to the real-time load. According to the technical scheme, the deployment time can be shortened to the minute level, the universality and the transverse expansion capacity of the system are remarkably enhanced, an independent operation and maintenance system is prevented from being repeatedly constructed for each engine, complete monitoring during operation, abnormal early warning and an intelligent elastic expansion mechanism are integrated, and a closed-loop autonomous operation and maintenance system is constructed.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and specifically to a method and system for deploying and running a real-time big data computing component based on Kubernetes. Technical Background

[0002] In the current big data field, real-time computing has become a core driver of enterprise digital transformation. With the popularization of technologies such as 5G and the Internet of Things, the timeliness requirements for data processing in various industries have increased from hours to seconds or even milliseconds. Traditional batch processing models can no longer meet the needs of key scenarios such as real-time risk control and intelligent decision-making. This rigid demand for low latency and high throughput is driving real-time computing technology to evolve from an auxiliary system to a core infrastructure.

[0003] The deep integration of cloud-native technology stacks and stream processing frameworks has become a key industry requirement, while traditional technical solutions still have significant limitations. Traditional big data architectures deploy Flink directly on top of YARN or Mesos resource management systems, relying on HDFS to store checkpoint data. Although this approach achieves basic stream processing functionality, it exposes a series of structural defects: the weakness of resource isolation mechanisms leads to intense resource competition among tasks in multi-tenant environments, and a single abnormal task can cause cluster-level performance fluctuations; the lack of containerization elasticity means that the system can only cope with sudden traffic surges by over-provisioning resources, significantly increasing infrastructure costs. Even with basic cloud-native solutions using Kubernetes as infrastructure, the manual deployment mode still suffers from serious bottlenecks. Operations personnel need to spend hours writing complex YAML configuration files to define Session cluster parameters (including JobManager / TaskManager resource quotas, network policies, and persistent volume declarations). This not only results in low deployment efficiency and a high risk of configuration errors, but more importantly, the static configuration architecture is completely unable to respond to the dynamic characteristics of real-time computing loads: it lacks the ability to automatically scale the number of TaskManagers and cannot dynamically adjust container resources, ultimately leading to a surge in data processing latency and task failures. In addition, the integration with the Kubernetes scheduler is not deep enough, and Pod migrations triggered by node maintenance or resource rebalancing often cause task interruptions that cannot be self-healed. This semi-automated implementation mode has failed to truly unleash the core value of cloud-native technologies.

[0004] Technical content

[0005] The present invention aims to solve at least one of the above-mentioned technical problems.

[0006] To address the aforementioned problems, this invention provides a method and system for deploying and operating real-time big data computing components.

[0007] In a first aspect, the present invention provides a method for deploying and running a real-time big data computing component, comprising:

[0008] Computing resources are isolated by tenant, and users configure Flink cluster parameters through a graphical configuration interface;

[0009] After receiving the Flink cluster parameters, the platform dynamically generates a resource list that conforms to the Kubernetes standard by combining the preset template library and tenant policy rules, and calls the interface encapsulated based on the Kubernetes API to complete the deployment.

[0010] After the cluster starts, it continuously monitors and alerts Flink jobs, and dynamically adjusts the number of TaskManager instances based on real-time load using an intelligent elastic scaling controller.

[0011] Optionally, the scaling strategy of the intelligent elastic scaling controller includes:

[0012] Metric-driven mode: When throughput decreases or backpressure increases, the number of TaskManager replicas is automatically increased; when a performance bottleneck or persistently low resource utilization is detected, the number of TaskManager instances is automatically reduced; and the resource quota for individual Pods is adjusted simultaneously.

[0013] In addition, there is a time-driven mode: automatically scaling up the cluster to a predetermined size at a specified time and automatically reclaiming resources after the specified time has ended.

[0014] Optionally, the intelligent elastic scaling controller adjusts the quota by calling Kubernetes' Scale Subresource API to update the number of FlinkDeployment replicas or by modifying the Pod's Resource Template, and the Operator automatically completes the creation, termination, and resource configuration update of the Pod.

[0015] Optionally, after the cluster starts, it continuously monitors and provides early warnings for Flink jobs, including:

[0016] Collect multi-dimensional operational metrics data and identify abnormal behaviors based on preset thresholds and machine learning models; if an alarm is triggered, immediately execute a tiered response strategy. At the same time, the system supports integration with Kubernetes Pod Disruption Budget to protect critical Flink tasks from being mistakenly evicted during node maintenance or rolling upgrades.

[0017] Optionally, the warnings are categorized into Level 1, Level 2, and Level 3 warnings based on severity. When a Level 3 warning is triggered, a severe state is indicated, and the system automatically activates fault self-healing. Fault self-healing includes not only restarting but also:

[0018] For failures caused by insufficient resources, the intelligent elastic scaling controller will first attempt to perform vertical scaling, increasing the CPU / memory quota for the problematic Pod; if the problem persists, the intelligent elastic scaling controller will further trigger horizontal scaling, introducing new compute nodes; if an anomaly is detected in the entire Flink cluster, the current data is backed up and uploaded to the built-in Minio server, the Flink cluster is rebuilt, and the original cluster state is restored using the backup data.

[0019] Optionally, the collection of multi-dimensional operational metrics data includes: a job performance layer, including throughput, backpressure status, and watermark latency; a resource consumption layer, including CPU utilization of JobManager and TaskManager, heap memory / non-heap memory usage, and GC frequency; an infrastructure layer, including Pod status, node resource consumption, and network I / O; and a business customization layer, including business latency and error rate reported through Flink Metrics Reporter.

[0020] Optionally, the deployment is completed by calling the interface encapsulated by the Kubernetes API, including:

[0021] The generated resource configuration list is submitted to the target Kubernetes cluster through the encapsulated Kubernetes client interface; the platform uses the controller in the Operator pattern to listen to the status of custom resources and drive the actual creation process of the Flink cluster.

[0022] Optionally, the deployment is completed by calling the interface encapsulated by the Kubernetes API, including:

[0023] Start the JobManager Pod and expose the service;

[0024] Start a specified number of TaskManager Pods and add them to the cluster as needed;

[0025] Configure service discovery and load balancing;

[0026] Initialize the Checkpoint storage path and connect to the external system.

[0027] Optionally, the scaling strategy supports: custom scaling based on monitored data or manual scaling operations triggered by the user via API or management interface, or direct modification of resource quotas for specific nodes to cope with emergencies.

[0028] Secondly, this invention provides a system for deploying and operating real-time big data computing components, comprising:

[0029] The configuration module is used for tenant isolation of computing resources and provides a graphical configuration interface for users to configure Flink cluster parameters;

[0030] The resource generation module is used to receive the Flink cluster parameters and, in combination with the preset template library and tenant policy rules, dynamically generate a resource list that conforms to the Kubernetes standard.

[0031] The module is encapsulated and calls the interface encapsulated based on the Kubernetes API to complete the deployment;

[0032] The monitoring module is used to continuously monitor and issue alerts for Flink jobs after the cluster starts.

[0033] The intelligent elastic scaling controller is used to dynamically adjust the number of TaskManager instances based on real-time load.

[0034] The beneficial effects of this invention are: (1) Automated deployment: complex YAML configuration is abstracted into a declarative API, allowing Flink to run directly as an application on Kubernetes without relying on additional resource management frameworks like YARN or Mesos, reducing deployment time from hours to minutes; (2) Multi-tenant resource isolation: all resource allocation and isolation policies are uniformly managed by the self-developed platform, providing resource usage monitoring, quota warning and operation audit functions to meet security and compliance requirements; (3) Integration of complete real-time operation monitoring, anomaly warning and intelligent elastic scaling mechanism to build a closed-loop autonomous operation and maintenance system; and can automatically adjust the number of TaskManager instances to achieve precise resource scaling.

[0035] Instruction manual illustrations

[0036] Figure 1 This is a flowchart illustrating the deployment method of the real-time big data computing component according to an embodiment of the present invention;

[0037] Figure 2 This is a flowchart illustrating the operation method of the real-time big data computing component according to an embodiment of the present invention.

[0038] Figure 3 This is an architecture diagram of the deployment and operation system for the big data real-time computing component in an embodiment of the present invention. Detailed Implementation

[0039] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Although some embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the accompanying drawings and embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0040] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0041] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to"; the term "based on" means "at least partially based on"; the term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments"; and the term "optionally" means "optional embodiments". Definitions of other terms will be given in the following description. It should be noted that the concepts of "first," "second," etc., mentioned in this invention are used only to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0042] It should be noted that the terms "a" and "a plurality of" used in this invention are illustrative rather than restrictive. Those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0043] The names of the messages or information exchanged between the multiple devices in the embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0044] like Figure 1 and Figure 2 As shown in the figure, the big data real-time computing component deployment and method provided by the embodiments of the present invention mainly includes: abstracting complex Kubernetes YAML into business-level parameters through declarative cluster configuration to improve deployment efficiency; improving cluster resource utilization through intelligent elastic scaling, combined with time series prediction and rule engine; and establishing a multi-level recovery strategy from Pod restart to cluster reconstruction through hierarchical fault handling.

[0045] The entire process of the method of this invention is as follows:

[0046] S1. Tenant isolation of computing resources: Users configure Flink cluster parameters through a graphical configuration interface.

[0047] Specifically, based on Kubernetes' Namespace and RBAC (Role-Based Access Control) mechanisms, an independent logical isolation space is created for each tenant. The system predefines tenant-level resource quota policies at the platform level, limiting the total amount of resources such as CPU, memory, and the number of Pods through ResourceQuota, and setting the minimum / maximum resource requests and limits for individual Pods through LimitRange. This ensures that resources between tenants do not interfere with each other in a multi-tenant environment, preventing the "noisy neighbor" effect and meeting enterprise-level security and compliance requirements.

[0048] A graphical configuration interface is provided for users to configure core parameters of the Flink cluster, including but not limited to: (1) Computational resource configuration: specifying the number of nodes for JobManager and TaskManager, the number of CPU cores per node, and the memory size; (2) Version selection: supporting the selection of Flink version and the compatible Kubernetes API version; (3) Network policy configuration: setting the service exposure method, such as NodePort, LoadBalancer, or Ingress, and inbound / outbound rules, Egress or Ingress; (4) Persistent storage configuration: mounting persistent volumes to save Checkpoint and Savepoint data to ensure state consistency. In particular, the configuration provided in this embodiment is presented in the form of a structured form, which helps to reduce the user's learning curve and avoid the risk of errors caused by manually writing complex YAML files.

[0049] S2. After receiving the Flink cluster parameters, the platform dynamically generates a resource list that conforms to the Kubernetes standard by combining the preset template library and tenant policy rules, and calls the interface encapsulated based on the Kubernetes API to complete the deployment.

[0050] Specifically, the process begins by generating dynamic templates and a Kubernetes resource orchestration platform. This process is accomplished through the platform's internal template engine, which supports flexible extension of different Flink topology modes, such as deployment templates for Application or Session modes. After receiving user configuration, and combining it with a pre-defined deployment template library and tenant policy rules (such as naming conventions, tag policies, and security contexts), a resource manifest conforming to Kubernetes standards is dynamically generated. The resource manifest can be in YAML or JSON format and includes:

[0051] Custom resource (CR) objects: such as FlinkDeployment, based on Flink Operator CRD;

[0052] Native resource objects such as Deployment, Service, ConfigMap, Secret, and PersistentVolumeClaim;

[0053] Network policies and resource restriction policies.

[0054] Secondly, cluster deployment is completed by calling interfaces encapsulated based on the Kubernetes API. The entire deployment process can be completed within minutes, improving efficiency by over 90% compared to traditional manual deployment methods. The generated resource configuration manifest is submitted to the target Kubernetes cluster through an encapsulated Kubernetes client interface, such as client-go. The platform utilizes the Controller in the Operator pattern to monitor the status of custom resources and drive the actual creation process of the Flink cluster, including:

[0055] Start the JobManager Pod and expose the service;

[0056] Start a specified number of TaskManager Pods and add them to the cluster as needed;

[0057] Configure service discovery and load balancing;

[0058] Initialize the Checkpoint storage path and connect to external systems such as HDFS and S3.

[0059] Therefore, this invention abstracts the complex Flink cluster deployment process into a visual configuration interface and declarative API calls. Users only need to configure key parameters, such as the number of nodes, resource specifications, and Flink version, and the system can automatically complete resource configuration, template generation, and Kubernetes object deployment, reducing deployment time to minutes and improving operational efficiency and system availability.

[0060] S3 continuously monitors and alerts Flink jobs after the cluster starts, and dynamically adjusts the number of TaskManager instances based on real-time load using an intelligent elastic scaling controller.

[0061] Specifically, the platform continuously monitors Flink jobs during runtime and collects multi-dimensional runtime metrics, mainly including: (1) job performance layer, used to determine whether the computing capacity is saturated, including throughput (records / s), backpressure level, and watermark latency; (2) resource consumption layer, used to identify resource bottlenecks, including CPU utilization of JobManager and TaskManager, heap memory / non-heap memory usage, and GC frequency; (3) infrastructure layer, used to avoid scaling up when node resources are tight and to prioritize scheduling to low-load nodes, including Pod status (Ready / Pending), node resource consumption, and network I / O; (4) business customization layer, used to support business-level SLA-driven scaling, such as payment business requiring latency <500ms, including business latency, error rate, and other related metrics reported through Flink Metrics Reporter.

[0062] In this implementation, all metrics are uniformly collected and stored using Prometheus (a system monitoring and alerting toolkit), and visualized through the platform's monitoring page, supporting historical trend analysis and root cause analysis. Of course, other monitoring systems compatible with the OpenMetrics standard can also be used as alternatives, such as commercial monitoring platforms like Datadog and Alibaba Cloud ARMS. As long as they can obtain the metric interface data exposed by Flink and support alarm rule configuration, equivalent monitoring and alerting functions can be achieved.

[0063] The platform integrates Grafana's visual monitoring dashboard, combined with an alerting module engine to achieve multi-level anomaly detection and response. The visual monitoring dashboard displays key metrics such as TaskManager backpressure metrics, Checkpoint success rate, memory usage, and GC frequency. The dashboard features a customizable threshold panel, supporting dynamic adjustment of alert thresholds with immediate effect. It also displays historical data trend analysis, identifying periodic anomaly patterns through time-series charts.

[0064] The data acquired can be used to ensure that the system automatically identifies abnormal behaviors based on preset rules and machine learning models, such as: TaskManager continuously high backpressure (>75%) exceeding the threshold duration, Checkpoint failure rate exceeding the normal range for 3 consecutive times, and JVM memory usage exceeding the safety threshold accompanied by frequent GC.

[0065] For identified anomalies, this technical solution also has a graded response mechanism, with severity ranging from mild to severe: (1) Level 1 alert: Grafana panel alarm status changes, detailed logs are recorded and notifications are pushed (email / WeChat); (2) Level 2 alert: diagnostic scripts are automatically triggered, diagnostic reports are generated in Grafana, and stack analysis and log correlation are visualized; (3) Level 3 alert: severe status is marked, and the system will automatically enable fault self-healing. The platform is deeply integrated with Kubernetes PDB (PodDisruption Budget) to ensure that critical tasks run stably during node maintenance.

[0066] The intelligent elastic scaling controller in this technical solution can achieve three-dimensional elastic scaling from the macro-level of instance quantity to the micro-level of resource allocation, and is deeply integrated with a fault self-healing mechanism to build a closed-loop management system. The elastic scaling strategy of this intelligent elastic scaling controller includes two collaborative modes:

[0067] Metric-driven mode: The system continuously monitors key performance metrics, such as throughput, backpressure, and CPU / memory utilization. When a performance bottleneck or persistently low resource utilization is detected, it not only automatically increases or decreases the number of TaskManager instances, but also intelligently adjusts the resource quotas of individual Pods, such as the number of CPU cores and memory size, achieving dual optimization of both "horizontal scaling" and "vertical scaling".

[0068] Time-driven mode: To handle predictable business peaks, such as 9:00–11:00 daily, a timed strategy can be configured. The system can automatically scale up the cluster to the predetermined size at a specified time and automatically reclaim resources after the peak period ends, achieving the best balance between cost and performance.

[0069] Specifically, when the system triggers a Level 3 critical fault warning, the self-healing strategy is not limited to restarting. For faults caused by insufficient resources, the intelligent elastic scaling controller will first attempt to perform vertical scaling, increasing the CPU / memory quota for the affected Pods; if the problem persists, it will further trigger horizontal scaling, introducing new compute nodes to achieve a fundamental repair at the resource level; if an anomaly is detected in the entire Flink cluster, such as a JobManager cluster failure or an internal network partition anomaly, it will back up the current data and upload it to the built-in Minio (s3 server), then rebuild the Flink cluster and restore the original cluster state using the backup data.

[0070] In this embodiment, the intelligent elastic scaling controller updates the number of FlinkDeployment replicas by calling Kubernetes' Scale Subresource API or adjusts quotas by modifying the Pod's Resource Template. The Operator automatically completes the creation, termination, and resource configuration updates of Pods, all without manual intervention. Of course, the elastic scaling policy can also support uploading custom scaling rules based on Prometheus metrics, allowing the elastic policy to closely match diverse business scenarios. Simultaneously, users can also manually trigger scaling operations via API or management interface, or directly modify the resource quotas of specific nodes to handle emergency situations.

[0071] Therefore, the closed-loop management built by runtime monitoring, anomaly warning and intelligent elastic scaling mechanism can effectively improve system stability, resource utilization and the ability to cope with sudden traffic compared with the passive operation and maintenance mode that relies on manual intervention in the existing technology, and truly realize the "unattended" cloud-native real-time computing service.

[0072] Of course, the underlying principles of this technical solution can also be quickly adapted to other real-time big data computing components, such as Spark Streaming and Storm, requiring only the development of corresponding deployment templates and monitoring adapters for the target engine. This design significantly enhances the system's versatility and horizontal scalability, avoiding the need to repeatedly build independent operation and maintenance systems for each engine, and facilitating the construction of a unified enterprise-level big data computing platform.

[0073] This invention also provides a system 300 for deploying and running a real-time big data computing component, used to execute the method for deploying and running a real-time big data computing component as described above, including:

[0074] Configuration module 310 is used for tenant isolation of computing resources and provides a graphical configuration interface for users to configure Flink cluster parameters;

[0075] The resource generation module 320 is used to receive the Flink cluster parameters and, in combination with the preset template library and tenant policy rules, dynamically generate a resource list that conforms to the Kubernetes standard.

[0076] Encapsulation module 330 calls the interface encapsulated based on the Kubernetes API to complete the deployment;

[0077] The monitoring module 340 is used to continuously monitor and provide early warnings for Flink jobs after the cluster starts up;

[0078] The Smart Elastic Scaling Controller 350 is used to dynamically adjust the number of TaskManager instances based on real-time load.

[0079] While the present invention has been disclosed above, its scope of protection is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention, and all such changes and modifications will fall within the scope of protection of the present invention.

Claims

1. A method for deploying and running a real-time big data computing component, characterized in that... include: Computing resources are isolated by tenant, and users configure Flink cluster parameters through a graphical configuration interface; After receiving the Flink cluster parameters, the platform dynamically generates a resource list that conforms to the Kubernetes standard by combining the preset template library and tenant policy rules, and calls the interface encapsulated based on the Kubernetes API to complete the deployment. After the cluster starts, it continuously monitors and alerts Flink jobs, and dynamically adjusts the number of TaskManager instances based on real-time load using an intelligent elastic scaling controller.

2. The method for deploying and running a real-time big data computing component as described in claim 1, characterized in that... The scaling strategy of the intelligent elastic scaling controller includes: Metric-driven mode: When throughput decreases or backpressure increases, the number of TaskManager replicas is automatically increased; when a performance bottleneck or persistently low resource utilization is detected, the number of TaskManager instances is automatically reduced; and the resource quota for individual Pods is adjusted simultaneously. In addition, there is a time-driven mode: automatically scaling up the cluster to a predetermined size at a specified time and automatically reclaiming resources after the specified time has ended.

3. A method for deploying and running a real-time big data computing component as described in claim 1 or 2, characterized in that... The intelligent elastic scaling controller updates the number of FlinkDeployment replicas by calling Kubernetes' Scale Subresource API or modifies the Pod's Resource Template to adjust quotas, and the Operator automatically completes the creation, termination, and resource configuration updates of Pods.

4. The method for deploying and running a real-time big data computing component as described in claim 1, characterized in that... After the cluster starts, it continuously monitors and issues alerts for Flink jobs, including: Collect multi-dimensional operational metrics data and identify abnormal behaviors based on preset thresholds and machine learning models; if an alarm is triggered, immediately execute a tiered response strategy. At the same time, the system supports integration with Kubernetes Pod Disruption Budget to protect critical Flink tasks from being mistakenly evicted during node maintenance or rolling upgrades.

5. A method for deploying and running a real-time big data computing component as described in claim 1, 2, or 4, characterized in that... The warnings are categorized into Level 1, Level 2, and Level 3 warnings based on their severity. When a Level 3 warning is triggered, the system is marked as critical and automatically enables fault self-healing. The self-healing mechanism includes not only restarting, but also: In response to failures caused by insufficient resources, the intelligent elastic scaling controller will prioritize attempting vertical scaling to increase the CPU / memory quota for the problematic Pod; If the problem persists, the intelligent elastic scaling controller will further trigger horizontal scaling, introducing new computing nodes; if an anomaly is detected in the entire Flink cluster, the current data will be backed up and uploaded to the built-in Minio server, the Flink cluster will be rebuilt, and the original cluster state will be restored using the backup data.

6. The method for deploying and running a real-time big data computing component as described in claim 4, characterized in that... The collected multi-dimensional operational metrics data include: job performance layer, including throughput, backpressure status, and watermark latency; resource consumption layer, including CPU utilization of JobManager and TaskManager, heap memory / non-heap memory usage, and GC frequency; infrastructure layer, including Pod status, node resource consumption, and network I / O; and business customization layer, including business latency and error rate reported through Flink MetricsReporter.

7. The method for deploying and running a real-time big data computing component as described in claim 1, characterized in that... The deployment is completed by calling the interface encapsulated by the Kubernetes API, including: The generated resource configuration list is submitted to the target Kubernetes cluster through the encapsulated Kubernetes client interface; the platform uses the controller in the Operator pattern to listen to the status of custom resources and drive the actual creation process of the Flink cluster.

8. The system for deploying and operating a real-time big data computing component as described in claim 7, characterized in that... The deployment is completed by calling the interface encapsulated by the Kubernetes API, including: Start the JobManager Pod and expose the service; Start a specified number of TaskManager Pods and add them to the cluster as needed; Configure service discovery and load balancing; Initialize the Checkpoint storage path and connect to the external system.

9. The method for deploying and running a real-time big data computing component as described in claim 2, characterized in that... The scaling strategy supports: custom scaling based on monitored data, or manual scaling operations triggered by users via API or management interface, or direct modification of resource quotas for specific nodes to cope with emergencies.

10. The system for deploying and operating a real-time big data computing component as described in claim 1, characterized in that... include: The configuration module is used for tenant isolation of computing resources and provides a graphical configuration interface for users to configure Flink cluster parameters; The resource generation module is used to receive the Flink cluster parameters and, in combination with the preset template library and tenant policy rules, dynamically generate a resource list that conforms to the Kubernetes standard. The module is encapsulated and calls the interface encapsulated based on the Kubernetes API to complete the deployment; The monitoring module is used to continuously monitor and issue alerts for Flink jobs after the cluster starts. The intelligent elastic scaling controller is used to dynamically adjust the number of TaskManager instances based on real-time load.

Images