Containerization-fused data lake operation and maintenance management system and method

Abstract

The invention discloses a containerization-fused data lake operation and maintenance management system and method, and relates to the technical field of data lake management. The system comprises a container integration management module used for obtaining operation state information of a container cluster; the data life cycle management module is used for collecting a data processing process of each project group and a corresponding intermediate result, and storing the data processing process and the corresponding intermediate result in a graph database form; the container operation and maintenance sub-module is used for generating a management strategy instruction for the container cluster based on the running state information, and the container integrated management module is also used for executing corresponding management operation; and the data operation and maintenance sub-module is used for performing correlation analysis on the intermediate result and generating a construction strategy instruction, and the data life cycle management module is also used for generating a corresponding data spanning tree. The unified management of the container cluster and the data lake cluster is realized, the separation of the data lake and the containerized system is broken, the operation and maintenance complexity is reduced, and the operation and maintenance efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data lake management, in particular to a data lake operation and maintenance management system and method fusing containerization. BACKGROUND

[0002] With the rapid development of big data technology, enterprises have an increasing demand for data lakes. Data lakes are used to uniformly process and maintain multiple types and sources of data within enterprises. The core goal is to support efficient processing and analysis of structured, semi-structured, and unstructured data in the data life cycle through centralized data storage and intelligent management methods.

[0003] As an enterprise-level big data unified scheduling and management platform, data lakes can support data processing needs in multiple scenarios. However, traditional data lake systems have significant shortcomings in resource management, operation and maintenance efficiency, and data redundancy control. In order to achieve agile management of big data clusters, containerization technology is introduced based on cloud service management, so that cloud container engines can easily deploy, manage, and expand containerized applications on the cloud.

[0004] Although containerization technology optimizes the resource structure of data lakes, container clusters and data lake clusters rely on different protocol stacks, and lack unified monitoring and management, resulting in high operation and maintenance complexity and low operation and maintenance efficiency. SUMMARY

[0005] Based on the above problems, the present application provides a data lake operation and maintenance management system and method fusing containerization, which provides a unified management platform, realizes unified management of container clusters and data lake clusters, breaks the separation of data lakes and containerized systems, reduces operation and maintenance complexity, and improves operation and maintenance efficiency.

[0006] The present application embodiment discloses the following technical scheme:

[0007] In a first aspect, the present application embodiment provides a data lake operation and maintenance management system fusing containerization, comprising: a container integration management module, a data life cycle management module, and an intelligent operation and maintenance module; wherein the intelligent operation and maintenance module comprises a container operation and maintenance submodule and a data operation and maintenance submodule;

[0008] The container integration management module is configured to obtain running state information of a container cluster; wherein the running state information of the container cluster comprises real-time running state information and historical running state information of each container node;

[0009] The data life cycle management module is configured to collect the processing process of data by each project group and the corresponding intermediate results, and store them in the form of a graph database;

[0010] The container operation and maintenance submodule is configured to generate a management policy instruction for the container cluster based on the running state information of the container cluster; and the container integrated management module is further configured to perform a corresponding management operation in response to the management policy instruction.

[0011] The data operation and maintenance submodule is configured to perform correlation analysis on the intermediate results based on the processing process and the corresponding intermediate results stored in the form of a graph database, and generate a construction policy instruction; and the data lifecycle management module is further configured to generate a corresponding data generation tree in response to the construction policy instruction; wherein the data generation tree is used to represent a data processing path and a redundancy analysis result.

[0012] In a possible implementation, the container integrated management module is specifically configured to:

[0013] The running state information of each container node is collected in real time by a sidecar container of each container node; wherein the sidecar container and the container node are deployed in the same instance, and are used to monitor real-time running state information and historical running state information of the container node.

[0014] In a possible implementation, the container operation and maintenance submodule is specifically configured to:

[0015] The running state information of the container cluster is analyzed based on a preset machine learning module, and a management policy instruction for the container cluster is generated; wherein the management policy instruction includes an instance expansion instruction, an instance shrinkage instruction, an instance migration instruction, and a resource allocation adjustment instruction.

[0016] In a possible implementation, the data lifecycle management module is specifically configured to:

[0017] The processing process and the corresponding intermediate results of data by each project team are collected, and a data processing path is stored by a graph database; wherein the nodes in the graph database represent original data, intermediate results, or final output, the edges in the graph database represent data processing operations and metadata, and the attributes in the graph database represent operation parameters of the data processing operations.

[0018] In a possible implementation, the data operation and maintenance submodule is specifically configured to:

[0019] The intermediate results are analyzed based on the processing process and the corresponding intermediate results stored in the form of a graph database by a correlation rule mining algorithm, and a construction policy instruction is generated; wherein the construction policy instruction includes a data merging instruction, a storage format optimization instruction, and a data lifecycle marking instruction.

[0020] The data lifecycle management module is specifically configured to generate a corresponding data generation tree in response to the construction strategy instruction, and optimize the data generation tree based on a big data decision tree algorithm to obtain an optimized data generation tree.

[0021] In a possible implementation, the data lifecycle management module is further configured to:

[0022] The Hudi incremental management technology is used for data with less modification, and the incremental data is separated and stored in the containerized environment.

[0023] In a possible implementation, the intelligent operation and maintenance module further includes an exception early warning submodule; the exception early warning submodule is configured to:

[0024] When it is detected that the running state of the container cluster is abnormal based on the running state information of the container cluster, an alarm mechanism is triggered to notify a user that the container cluster is abnormal;

[0025] When it is detected that the data redundancy exceeds a preset threshold, an alarm mechanism is triggered to notify a user that the data lake cluster is abnormal.

[0026] In a possible implementation, the intelligent operation and maintenance module further includes a cross-cloud fusion submodule; the cross-cloud fusion submodule is configured to:

[0027] When it is detected that the container cluster resource is insufficient, an instance migration instruction is generated through a cross-cloud resource scheduling strategy; wherein the instance migration instruction includes an object storage (OBS) cross-cloud replication migration instruction or a containerized image migration instruction.

[0028] In a possible implementation, the system further includes a logical window unified view module;

[0029] The logical window unified view module is configured to integrate the monitoring data of the data lake cluster and the container cluster into the same view, and support monitoring units divided according to project groups;

[0030] The logical window unified view module is further configured to update configuration parameters according to component dependency relationships, and support batch operations.

[0031] In a second aspect, the embodiments of the present application provide a data lake operation and maintenance management method based on containerization, applied to the data lake operation and maintenance management system based on containerization as described in any of the possible implementation manners of the first aspect, and the method includes:

[0032] The running state information of the container cluster is obtained through the container integration management module; wherein the running state information of the container cluster includes real-time running state information and historical running state information of each container node;

[0033] The container operation and maintenance submodule generates management policy instructions for the container cluster based on the running state information of the container cluster;

[0034] The container integration management module performs corresponding management operations in response to the management policy instructions;

[0035] The data lifecycle management module collects the data processing process and corresponding intermediate results of each project group and stores them in the form of a graph database;

[0036] The data operation and maintenance submodule performs correlation analysis on the intermediate results based on the processing process and corresponding intermediate results stored in the form of a graph database, and generates construction policy instructions;

[0037] The data lifecycle management module generates corresponding data generation trees in response to the construction policy instructions; wherein the data generation trees are used to represent data processing paths and redundancy analysis results.

[0038] Compared with the prior art, the present application has the following beneficial effects: a unified management platform is provided, which realizes unified management of container clusters and data lake clusters, breaks the separation of data lakes and containerized systems, reduces the operation and maintenance complexity, and improves the operation and maintenance efficiency. The container operation and maintenance submodule and the container integration management module work together to realize automatic optimization of the container cluster, solve the problems of low resource utilization and slow fault handling in traditional containerized systems, and improve user experience and system stability; the data operation and maintenance submodule and the data lifecycle management module cooperate to generate and maintain data generation trees, support data reuse and improve storage efficiency, thereby avoiding repeated consumption of storage and computing resources. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0040] Figure 1 A structure diagram of a data lake operation and maintenance management system integrating containerization is provided for the embodiments of the present application;

[0041] Figure 2 Another structure diagram of a data lake operation and maintenance management system integrating containerization is provided for the embodiments of the present application;

[0042] Figure 3A flowchart of a containerized data lake operation and maintenance management method provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0043] As described above, although the containerization technology optimizes the resource structure of the database, the container cluster and the data lake cluster rely on different technology stacks (such as Kubernetes and Hadoop ecology), lack unified monitoring management, and the operation and maintenance personnel need to switch between multiple systems, resulting in high operation and maintenance complexity and low operation and maintenance efficiency.

[0044] Further, due to the reliance of the container cluster and the data lake cluster on different technology stacks, the lack of unified monitoring management, when cross-cloud data interaction, the data storage format and protocol of the container cluster and the data lake cluster are incompatible, and manual intervention is needed to complete data migration or sharing, increasing the operation cost and reducing the data availability.

[0045] For the process of containerization transformation, the current container cluster transformation mainly relies on the administrator's experience to manually judge whether the business scenario needs containerization, and lacks a unified identification method or rule. This leads to some business scenarios not being timely containerized, resulting in unreasonable resource allocation (such as non-containerized business occupying redundant resources) and low operation and maintenance efficiency. And the containerized system is easily affected by human experience bias in the resource evaluation stage, which may lead to resource evaluation errors. For example, during the business peak period, the resource shortage problem of the container cluster frequently occurs, which further causes business response delay, directly affecting user experience and system stability.

[0046] For data processing, the raw data stored in the data lake cannot be directly used and needs to be independently preprocessed by each project team. However, the preprocessing logic of different project teams for the same data source is highly repetitive (such as ETL rules, data cleaning steps), resulting in resource waste and low development efficiency. The preprocessed data files and raw data have high similarity (such as consistent structured fields, overlapping calculation logic), but each project team still stores the preprocessing results separately, forming a large amount of redundant data. This repeated storage not only occupies additional storage space, but also increases the complexity of data management and maintenance. And the current technology does not provide correlation analysis capabilities for cross-project data preprocessing results, which cannot identify the redundancy relationship between similar data. For example, the intermediate results generated by multiple project teams for the same data source may only have minor differences, but the system cannot automatically merge or share these results, resulting in repeated consumption of storage and computing resources.

[0047] The application provides a containerized data lake operation and maintenance management system, comprising a container integration management module, a data life cycle management module and an intelligent operation and maintenance module, wherein the intelligent operation and maintenance module comprises a container operation and maintenance submodule and a data operation and maintenance submodule. The container integration management module is used to acquire running state information of a container cluster. The data life cycle management module is used to collect processing processes and corresponding intermediate results of data of various project groups and store them in the form of a graph database. The container operation and maintenance submodule is used to generate management policy instructions for the container cluster based on the running state information of the container cluster. The container integration management module is also used to execute corresponding management operations in response to the management policy instructions. The data operation and maintenance submodule is used to perform correlation analysis on the intermediate results based on the processing processes and corresponding intermediate results stored in the form of the graph database, and generate construction policy instructions. The data life cycle management module is also used to generate corresponding data generation trees in response to the construction policy instructions. The application embodiment provides a unified management platform, realizes unified management of the container cluster and the data lake cluster, breaks the separation of the data lake and the containerized system, reduces the operation and maintenance complexity, and improves the operation and maintenance efficiency.

[0048] In order for those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of the present application.

[0049] Embodiment one:

[0050] The following will be combined Figure 1 and Figure 2 to introduce in detail the containerized data lake operation and maintenance management system provided by the embodiments of the present application.

[0051] As shown in Figure 1 , the containerized data lake operation and maintenance management system provided by the embodiments of the present application comprises the following modules: a container integration management module 110, a data life cycle management module 120 and an intelligent operation and maintenance module 130, wherein the intelligent operation and maintenance module 130 comprises a container operation and maintenance submodule 131 and a data operation and maintenance submodule 132.

[0052] The container integration management module 110 is used to acquire running state information of a container cluster.

[0053] A container cluster is a distributed system composed of multiple container nodes, providing a running environment for containerized applications, that is, the container cluster can be understood as a set of cloud resources required for container running.

[0054] A container node refers to a single computing unit in a container cluster, used to run a container instance (Pod). For example, a container node can be a cloud server (physical machine or virtual machine, providing computing and storage capabilities), a load balancer (used to distribute traffic, ensuring high availability and performance of the application), network resources, storage resources, etc.

[0055] As described above, a container cluster is composed of multiple container nodes, and a container node is the basic unit of a container cluster, responsible for running container instances and providing computing resources.

[0056] Specifically, the running state information of the container cluster includes real-time running state information and historical running state information of each container node.

[0057] Real-time running state information refers to the resource usage (such as CPU, memory, network traffic, disk I / O, etc.) and running state (such as whether it is running normally, whether there is an exception or failure) of each container node (such as Pod, instance) at a certain time. Historical running state information refers to the resource usage trend, peak and trough record, failure event record, etc. of each container node within a period of time (such as the past 24 hours, a week or longer).

[0058] In one possible implementation, the container integration management module 110 is specifically configured to: collect the running state information of each container node in real time through the sidecar container of each container node.

[0059] Among them, the sidecar container and its corresponding container node are deployed in the same instance, that is, the sidecar container is a kind of auxiliary container deployed in the same instance with the main container (container node), used to enhance the function of the main container, but the sidecar container itself does not directly participate in the business logic of the main container. For example, the sidecar container provides additional services such as monitoring, log collection, configuration management, security protection, etc. through shared network, storage and life cycle for the main container.

[0060] Specifically, the sidecar container and its corresponding container node share the network namespace (such as IP address and port) to facilitate direct communication; contribute storage volumes (such as log files, configuration files) to achieve data transfer; and the sidecar container runs an independent process to avoid conflicts with the business logic of its corresponding container node.

[0061] In the embodiments of the present application, the sidecar container and its corresponding container node are deployed in the same instance, which is used to monitor the real-time running state information and the historical running state information of the container node, thereby providing a dynamic resource mobilization basis for the intelligent operation and maintenance module.

[0062] The data lifecycle management module 120 is configured to collect the processing process and the corresponding intermediate result of the data of each project group, and store the processing process and the corresponding intermediate result in the form of a graph database.

[0063] The project group refers to a data processing team or department around a specific business requirement component. Each project group can independently develop data processing logic (such as ETL rules, data cleaning scripts).

[0064] The processing process refers to the complete operation chain of processing, converting, analyzing, etc. of data by the project group, for example: the processing process is "raw data -> cleaning -> field standardization -> aggregation calculation -> output".

[0065] The intermediate result refers to the temporary data set generated in the processing process, for example: cleaned transaction data, customer portrait label divided by area, statistical report in a certain time period, etc.

[0066] The graph database is a database system designed based on a graph structure (nodes and edges), which is used to efficiently store, query and analyze highly correlated data. Its core idea is to take entities (nodes) and their relationships (edges) in data as the core storage unit, and express complex relationship networks through a graph model.

[0067] In the embodiments of the present application, due to the similarity of business requirements, the processing steps of different project groups for the same data source may be highly repetitive. For example, multiple project groups may need to perform operations such as de-duplication and field standardization on raw data. Such repetition not only increases development costs, but also leads to waste of storage resources. In the embodiments of the present application, the data lifecycle management module 120 collects the processing process and the intermediate result of each project group, breaks the "data silos" between project groups, and provides a data basis for subsequent redundancy analysis and optimization. And storing the processing process and the corresponding intermediate result of the data in the form of a graph database can intuitively display the life cycle of the data, which is convenient for auditing, optimization and decision-making.

[0068] In a possible implementation, the data lifecycle management module 120 is specifically configured to: collect the processing process of data by each project group and the corresponding intermediate results, and store the data processing path through a graph database. A node of the graph database represents a data entity, including: raw data, intermediate results, and final output. An edge of the graph database represents a data processing operation (such as cleaning, standardization, calculation logic, etc.) and metadata (such as operation timestamp, executor, data redundancy evaluation result). A property of the graph database represents the operation parameters of the data processing operation. For example, the data processing path stored in the form of the graph database is: raw data A1->(cleaning operation)->intermediate result A2->(aggregation operation)->final output A3.

[0069] In a possible implementation, the data lifecycle management module 120 is further configured to: use a Hudi incremental management technology for data that is less frequently modified, separate the incremental data, and store the incremental data in a containerized environment. Thus, the storage space is reduced.

[0070] Hudi (Hadoop Upserts and Incremental Data Processing) is an open-source big data storage management framework designed for efficient processing of incremental updates of large-scale data sets. The Hudi incremental management technology can separate incremental data and divide the data into base files and delta log files, and only record the changed part (such as addition, update, and deletion operation) of the data, rather than rewriting the data in full.

[0071] The data that is less frequently modified refers to a data set that has a low update frequency and a small change range. For example, a static dimension table (such as a customer information table or a product classification table) is not changed for a long time after the data is generated, and only some fields are updated occasionally. Historical archive data is not frequently modified after business processing is completed, and only needs to be backed up or audited regularly. Low-frequency update intermediate results are temporary data generated in the data processing process, and only need to be modified a small amount later.

[0072] Further, the Hudi incremental management technology can also realize version management, that is, the data change history is marked by a timestamp, and quick rollback to any historical version is supported.

[0073] In the embodiments of the present application, by using the Hudi incremental management technology, only incremental data is stored instead of full value, which significantly reduces the storage space occupation; for example, if a 10TB static dimension table only modifies 1% of the fields, only about 100GB of incremental data needs to be stored by using the Hudi incremental management technology, instead of storing 10TB of full data. Further, it is suitable for scenarios that need to be frequently updated (i.e. real-time data lake), avoiding the limitation of immutability of traditional HDFS.

[0074] The container operation and maintenance submodule 131 is configured to generate a management policy instruction for the container cluster based on the running state information of the container cluster; and the container integrated management module 110 is further configured to execute a corresponding management operation in response to the management policy instruction.

[0075] In a possible implementation, the container operation and maintenance submodule 131 is specifically configured to analyze the running state information of the container cluster based on a preset machine learning model, and generate a management policy instruction for the container cluster. That is, the container operation and maintenance submodule 131 analyzes the running data of the container cluster by using the preset machine learning model, dynamically generates a corresponding management policy instruction, and realizes automatic operation and maintenance of the container cluster.

[0076] The management policy instruction includes an instance expansion instruction, an instance shrinkage instruction, an instance migration instruction, and a resource allocation adjustment instruction.

[0077] The container expansion instruction refers to triggering container instance expansion and increasing the number of Pods to share the load when resource shortage (such as CPU / memory continuously exceeding the limit) is detected.

[0078] The instance shrinkage instruction refers to automatically reducing redundant Pods and reducing resource waste during a resource idle period (such as a night business low peak period).

[0079] The instance migration instruction refers to migrating the affected Pods to other available nodes when a node fails or is resource deficient, to ensure business continuity.

[0080] The resource allocation adjustment instruction refers to dynamically adjusting the resource quota according to the business priority (such as allocating more CPU / memory for high-priority business).

[0081] Specifically, the container operations and maintenance submodule 131 is used to analyze real-time and historical runtime status information using preset machine learning models (such as open-source neural network models LSTM and Transformer) to obtain resource fluctuation trends and predict future resource demands. Based on the predicted future resource demands, it generates corresponding management policy instructions. The container integration management module 110 responds to the management policy instructions generated by the container operations and maintenance submodule 131 and executes specific management operations, thereby achieving unified management and dynamic resource adjustment of the container cluster. For example, in response to instance scaling up or scaling down instructions, the container integration management module 110 calls the Kubernetes API to dynamically adjust the number of Pods, thereby achieving instance scaling up or scaling down.

[0082] In this embodiment, the container operation and maintenance submodule 131 and the container integration management module 110 jointly construct an intelligent operation and maintenance system for the container cluster. The container operation and maintenance submodule 131 generates corresponding management policy instructions by analyzing the running status information, and the container integration management module 110 is responsible for efficiently executing these instructions and ensuring business stability. The collaborative work of the two solves the problems of low resource utilization and slow fault handling in traditional containerized systems.

[0083] Among them, the data operation and maintenance submodule 132 is used to perform correlation analysis on the intermediate results stored in the form of graph database and generate construction strategy instructions; the data lifecycle management module 120 is also used to generate the corresponding data generation tree in response to the construction strategy instructions.

[0084] Among them, the data generation tree is used to represent the data processing path and the results of redundancy analysis. That is, the data generation tree is a tree structure that represents the data processing path and the results of redundancy analysis.

[0085] Specifically, the root node of the data spanning tree represents the original data (such as transaction logs or customer information tables), the intermediate nodes represent intermediate results (such as cleaned data or aggregated statistical reports), the leaf nodes represent the final output (such as business reports or model training data), and the edges between nodes represent data processing operations and their corresponding metadata. In this embodiment, the data spanning tree, as the core carrier, not only records the complete data processing path but also significantly reduces storage costs and computational resource waste in the data lake through redundancy analysis and storage optimization.

[0086] In a possible implementation, the data operation and maintenance submodule 132 is specifically configured to: based on the processing process and the corresponding intermediate result stored in the form of the graph database, perform correlation analysis on the intermediate result by using an association rule mining algorithm, and generate a construction strategy instruction. Meanwhile, the data life cycle management module 120 is specifically configured to: in response to the construction strategy instruction, generate a corresponding data generation tree, and perform optimization on the data generation tree based on a big data decision tree algorithm, to obtain an optimized data generation tree.

[0087] The construction strategy instruction includes a data merging instruction, a storage format optimization instruction, and a data life cycle marking instruction.

[0088] The data merging instruction refers to generating a merging suggestion (such as a shared storage path or unified computing logic) for intermediate results with high similarity.

[0089] The storage format optimization instruction refers to selecting a suitable storage format (such as the MOR or COW mode of Hudi) according to the business scenario requirement. For example, the MOR (Merge-on-Read) mode is used for data with high update frequency, and the COW (Copy-on-Write) mode is used for data with low update frequency or static data.

[0090] The data life cycle marking instruction adds a life cycle label (such as “high-frequency access” or “low-frequency modification”) to the intermediate result, to facilitate subsequent storage strategy adjustment.

[0091] Specifically, the data operation and maintenance submodule 132 reads the processing process and the corresponding intermediate result of each project team from the graph database; an association rule mining algorithm (such as the Apriori algorithm or the FP-Growth algorithm) is used to analyze the similarity of the intermediate results of different project teams, for example, to identify the case that multiple project teams highly coincide in the preprocessing step of the same data source; based on the field distribution, the computing logic, and the storage format of the intermediate result, the correlation coefficient between the data is calculated, to generate the construction strategy instruction.

[0092] Specifically, the data life cycle management module 120 generates a corresponding data generation tree in response to the construction strategy instruction, based on the processing process and the corresponding intermediate result stored in the graph database; based on a big data decision tree algorithm (such as ID3 or C4.5), the data generation tree is pruned to remove redundant paths, for example, to merge intermediate result nodes with high similarity, to reduce storage overhead, to obtain an optimized data generation tree.

[0093] In the embodiments of the present application, the data operation and maintenance sub-module 132 identifies redundant paths and generates optimization schemes by association rule mining and construction of strategy instructions. The data lifecycle management module 120 generates and maintains a data generation tree by dynamic construction of a graph database and optimization of a decision tree algorithm, supporting data reuse and storage efficiency improvement.

[0094] In a possible implementation, the intelligent operation and maintenance module 130 further includes an abnormality early warning sub-module.

[0095] The abnormality early warning sub-module is configured to: when detecting, based on the running state information of the container cluster, that the running state of the container cluster is abnormal, triggering an alarm mechanism to notify a user that the container cluster is abnormal; and when detecting that the data redundancy exceeds a preset threshold, triggering the alarm mechanism to notify the user that the data lake cluster is abnormal.

[0096] The abnormality early warning sub-module is configured to monitor the states of the container cluster and the data lake cluster in real time and trigger the alarm mechanism when detecting an abnormality.

[0097] For the container cluster, the abnormality early warning sub-module determines an abnormality based on the running state information of the container cluster by using a preset threshold (such as CPU usage > 90%, or the number of consecutive Pod failures ≥ 3 times), and analyzes historical data fluctuation trends by using a machine learning model to predict potential risks (such as business delay caused by insufficient resources). When detecting an abnormality (such as node downtime or service unavailability), the abnormality early warning sub-module triggers an alarm mechanism (such as a short message, an email, or a platform notification) to notify an administrator to handle the abnormality in a timely manner.

[0098] For example, a certain container node causes a Pod to crash due to memory overflow, and the abnormality early warning sub-module immediately sends an alarm and migrates the faulty Pod according to an emergency handling plan.

[0099] For the data lake cluster, the abnormality early warning sub-module analyzes the redundancy of intermediate results (such as the similarity of similar intermediate results stored by multiple project teams ≥ 85%) by using a graph database in the data lifecycle management module, and identifies high-risk areas of redundant storage in combination with the pruning optimization result of the data generation tree. When the redundancy exceeds a preset threshold (such as storage space occupancy ≥ 80%), the abnormality early warning sub-module triggers an alarm mechanism to notify a user that the data lake cluster is abnormal.

[0100] Further, when triggering the alarm mechanism of the data lake cluster, the abnormality early warning sub-module is further configured to recommend optimization strategies (such as merging storage or adjusting the Hudi storage format). For example, the preprocessing results of the same data source by project teams A and B are highly repetitive, with a redundancy of 90%, and the abnormality early warning sub-module automatically sends an alarm and recommends a shared storage path.

[0101] In a possible implementation, the intelligent operation and maintenance module 130 further includes a cross-cloud fusion sub-module.

[0102] The cross-cloud fusion sub-module is configured to generate an instance migration instruction through a cross-cloud resource scheduling strategy when it is detected that the container cluster resource is insufficient.

[0103] Specifically, the cross-cloud fusion sub-module generates an instance migration instruction through a cross-cloud resource scheduling strategy when it is detected that the container cluster resource is insufficient.

[0104] The instance migration instruction includes an object storage (OBS) cross-cloud replication migration instruction or a containerized image migration instruction.

[0105] The object storage (OBS) cross-cloud replication migration instruction is suitable for a scenario in which a large amount of static data needs to be migrated, and the corresponding operation process is to copy the data in the OBS from the current cloud environment to the target cloud environment.

[0106] The containerized image migration instruction is suitable for a scenario in which a dynamic service (such as a micro-service Pod) needs to be quickly deployed or migrated, and the corresponding operation process is to push the container image from the current cloud environment to the image warehouse of the target cloud environment and redeploy the instance.

[0107] In the embodiments of the present application, the OBS cross-cloud replication is suitable for static data migration and guarantees data consistency, and the containerized image migration is suitable for dynamic service deployment and improves resource utilization. The data lake operation and maintenance management system provided in the embodiments of the present application coordinates cross-cloud operations and solves the data island problem in a multi-cloud environment.

[0108] In a possible implementation, the container integration management module 110 is further configured to store the running state information of the container cluster in an object storage (OBS) so as to be called by the container operation and maintenance sub-module 131.

[0109] The object storage (OBS) is a unified data storage platform that supports the container operation and maintenance sub-module 131 to quickly call historical data for resource scheduling decision and exception analysis.

[0110] In a possible implementation, the data lake operation and maintenance management system provided in the embodiments of the present application further includes a logical window unified view module.

[0111] For the convenience of understanding, the following describes the data lake operation and maintenance management system provided in the embodiments of the present application with reference to the accompanying drawings. Figure 2 Another data lake operation and maintenance management system provided in the embodiments of the present application is described by way of example and includes a logical window unified view module, a container integration management module, a data life cycle management module, an intelligent operation and maintenance module, and a storage object OBS. The intelligent operation and maintenance module includes a container operation and maintenance sub-module, a data operation and maintenance sub-module, an exception early warning sub-module, and a cross-cloud fusion sub-module.

[0112] The logical window unified view module is configured to integrate monitoring data of the data lake cluster and the container cluster into the same view and support division of monitoring units according to project groups.

[0113] Specifically, the logical window unified view module integrates monitoring information of the data lake cluster and the container cluster into the logical window to form a full-amount monitoring view. The monitoring data includes resource usage, service health status, data processing path, etc. The monitoring units are divided according to project groups, and the monitoring data of each project group is independently displayed (such as resource usage trend and data redundancy), which facilitates administrators to focus on specific business scenarios. If multiple project groups share the same data source or service, the logical window unified view module automatically associates the monitoring data thereof to reveal potential resource conflicts or redundancy risks. The logical window unified view module further simplifies the operation and maintenance process, breaks the fragmentation of multiple systems, and provides a one-stop management interface.

[0114] Further, the logical window unified view module is further configured to synchronously update configuration parameters according to component dependency relationships and support batch operations.

[0115] Specifically, when there is a component dependency relationship (such as A service depends on the configuration parameters of B service), the logical window unified view module automatically synchronously updates the related parameters to ensure configuration consistency. For example, if a project group adjusts the Hudi storage format of the data lake (such as switching from COW to MOR), the logical window unified view module will synchronously update the containerized components (such as the reading logic of the Spark task) that depend on the configuration.

[0116] Further, the logical window unified view module supports batch operations (such as batch modification of configuration parameters and batch start and stop of services) in the same logical window, which reduces the workload of administrators manually operating one by one. For example, the logical window unified view module can uniformly adjust the Pod resource quota of multiple project groups to avoid configuring each node one by one. In the embodiments of the present application, the logical window unified view module can effectively avoid configuration conflicts through dependency relationship synchronization, and batch operations and automatic linkage significantly reduce manual intervention and reduce operation and maintenance costs.

[0117] This application provides a containerized data lake operation and maintenance management system, including: a container integration management module 110, a data lifecycle management module 120, and an intelligent operation and maintenance module 130. The intelligent operation and maintenance module 130 includes: a container operation and maintenance submodule 131 and a data operation and maintenance submodule 132. The container integration management module 110 is used to acquire the running status information of the container cluster; the data lifecycle management module 120 is used to collect the data processing procedures and corresponding intermediate results from various project teams and store them in the form of a graph database; the container operation and maintenance submodule 131 is used to generate management policy instructions for the container cluster based on the running status information of the container cluster; and the container integration management module 110 is also used to execute corresponding management operations in response to the management policy instructions; the data operation and maintenance submodule 132 is used to perform correlation analysis on the intermediate results stored in the graph database and generate build policy instructions; the data lifecycle management module 120 is also used to generate a corresponding data spanning tree in response to the build policy instructions. This application provides a unified management platform that enables unified management of container clusters and data lake clusters, breaking down the separation between data lakes and containerized systems, reducing operational complexity, and improving operational efficiency.

[0118] Furthermore, the container operations and maintenance submodule generates corresponding management policy instructions by analyzing runtime status information. The container integration management module is responsible for efficiently executing these instructions and ensuring business stability, thus achieving automatic optimization of the container cluster. The collaborative work of the two solves the problems of low resource utilization and slow fault handling in traditional containerized systems, improving user experience and system stability.

[0119] Furthermore, the data operation and maintenance submodule identifies redundant paths and generates optimization solutions by mining association rules and constructing strategy instructions; the data lifecycle management module generates and maintains data generation trees by dynamically constructing graph databases and optimizing decision tree algorithms, supporting data reuse and improving storage efficiency, thereby avoiding the repeated consumption of storage and computing resources.

[0120] Example 2:

[0121] The following is combined Figure 3 This application provides a detailed description of a containerized data lake operation and maintenance management method.

[0122] like Figure 3 As shown in the figure, the data lake operation and maintenance management method integrating containerization provided in this application includes the following steps:

[0123] S301. Obtain the running status information of the container cluster through the container integration management module.

[0124] The running state information of the container cluster includes real-time running state information and historical running state information of each container node.

[0125] In a possible implementation, based on the container integration management module, the running state information of each container node is collected in real time by a sidecar container of each container node; the sidecar container and the container node are deployed in the same instance, and are used to monitor the real-time running state information and the historical running state information of the container node.

[0126] S302, based on the running state information of the container cluster, a management strategy instruction for the container cluster is generated by the container operation and maintenance submodule.

[0127] In a possible implementation, based on a preset machine learning module, the running state information of the container cluster is analyzed by the container operation and maintenance submodule, and the management strategy instruction for the container cluster is generated; the management strategy instruction includes an instance expansion instruction, an instance shrinkage instruction, an instance migration instruction, and a resource allocation adjustment instruction.

[0128] S303, in response to the management strategy instruction, a corresponding management operation is performed by the container integration management module.

[0129] S304, the data processing process and the corresponding intermediate result of each project team are collected by the data life cycle management module, and are stored in the form of a graph database.

[0130] In a possible implementation, the data processing process and the corresponding intermediate result of each project team are collected by the data life cycle management module, and the data processing path is stored by the graph database; the nodes in the graph database represent original data, intermediate results or final output, the edges in the graph database represent data processing operations and metadata, and the attributes in the graph database represent operation parameters of the data processing operation.

[0131] S305, based on the processing process and the corresponding intermediate result stored in the form of the graph database, the intermediate result is analyzed for correlation by the data operation and maintenance submodule, and a construction strategy instruction is generated.

[0132] In a possible implementation, based on the processing process and the corresponding intermediate result stored in the form of the graph database, the intermediate result is analyzed for correlation by the data operation and maintenance submodule by using an association rule mining algorithm, and a construction strategy instruction is generated; the construction strategy instruction includes a data merging instruction, a storage format optimization instruction, and a data life cycle marking instruction.

[0133] S306, in response to the construction strategy instruction, a corresponding data generation tree is generated by the data life cycle management module.

[0134] The data generation tree is used to represent a data processing path and a redundancy analysis result.

[0135] In a possible implementation, the data generation tree is generated by the data life cycle management module in response to the construction strategy instruction, and is optimized based on a big data decision tree algorithm to obtain an optimized data generation tree.

[0136] In a possible implementation, the method further includes: using, by the data life cycle management module, a Hudi incremental management technology for data with less modification, separating incremental data, and storing the incremental data in a containerized environment.

[0137] In a possible implementation, the method further includes: triggering an alarm mechanism to notify a user that the container cluster is abnormal when it is detected that the running state of the container cluster is abnormal based on the running state information of the container cluster; and triggering the alarm mechanism to notify the user that the data lake cluster is abnormal when it is detected that the data redundancy exceeds a preset threshold.

[0138] In a possible implementation, the method further includes: generating an instance migration instruction by a cross-cloud resource scheduling strategy when it is detected that the container cluster is insufficient in resources; wherein the instance migration instruction includes an object storage (OBS) cross-cloud replication migration instruction or a containerized image migration instruction.

[0139] The embodiments of the present application provide a containerized data lake operation and maintenance management method, including: obtaining, by a container integration management module, running state information of a container cluster; generating, by a container operation and maintenance submodule, management strategy instructions for the container cluster based on the running state information of the container cluster; executing, by the container integration management module, corresponding management operations in response to the management strategy instructions; collecting, by a data life cycle management module, data processing processes and corresponding intermediate results of each project group, and storing the data processing processes and the corresponding intermediate results in a graph database form; performing, by a data operation and maintenance submodule, correlation analysis on the intermediate results based on the processing processes and the corresponding intermediate results stored in the graph database form, to generate construction strategy instructions; and generating, by the data life cycle management module, corresponding data generation trees in response to the construction strategy instructions. The embodiments of the present application provide a unified management platform, which realizes unified management of the container cluster and the data lake cluster, breaks the separation of the data lake and the containerized system, reduces the operation and maintenance complexity, and improves the operation and maintenance efficiency.

[0140] Further, the corresponding management strategy instructions are generated by analyzing the running state information, and these instructions are efficiently executed to guarantee business stability, which realizes automatic optimization of the container cluster. The cooperation of the two solves the problems of low resource utilization and slow fault processing in the traditional containerized system, and improves user experience and system stability.

[0141] Further, by association rule mining and constructing strategy instructions, redundant paths are identified and optimization schemes are generated, and by dynamic construction of a graph database and optimization of a decision tree algorithm, a data generation tree is generated and maintained, data reuse and storage efficiency are supported, thereby avoiding repeated consumption of storage and computing resources.

[0142] It should be noted that each of the embodiments in the specification adopts a progressive manner for description, and the same and similar parts between the embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. Especially, since the method embodiment is basically similar to the system embodiment, the description is relatively simple, and the relevant part can be referred to the part of the method embodiment. The system embodiment described above is only schematic, and the units described as separate units can or can not be physically separate, and the units prompted as units can or can not be physical units, that is, they can be located in one place, or also can be distributed to multiple network units. According to the actual need, part or all of the modules can be selected to achieve the purpose of the embodiment scheme. Those skilled in the art can understand and implement without creative labor.

[0143] The above describes only a specific embodiment of the present application, but the protection scope of the present application is not limited to this. Any skilled person in the art can easily think of changes or replacements within the technical range disclosed in the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A converged containerized data lake operations management system, comprising: The application relates to a container integrated management system, comprising: a container integrated management module, a data life cycle management module and an intelligent operation and maintenance module; wherein the intelligent operation and maintenance module comprises a container operation and maintenance submodule and a data operation and maintenance submodule; the container integrated management module is used for acquiring running state information of a container cluster; wherein the running state information of the container cluster comprises real-time running state information and historical running state information of each container node; the data life cycle management module is used for collecting a data processing process and corresponding intermediate results of each project group and storing the data processing process and the corresponding intermediate results in the form of a graph database; the container operation and maintenance submodule is used for generating a management strategy instruction for the container cluster based on the running state information of the container cluster; and the container integrated management module is further used for executing a corresponding management operation in response to the management strategy instruction; the data operation and maintenance submodule is used for performing correlation analysis on the intermediate results based on the processing process and the corresponding intermediate results stored in the form of the graph database, and generating a construction strategy instruction; and the data life cycle management module is further used for generating a corresponding data generation tree in response to the construction strategy instruction; wherein the data generation tree is used for representing a data processing path and a redundancy analysis result.

2. The system of claim 1, wherein, The container integrated management module is specifically used for: collecting the running state information of each container node in real time through a sidecar container of each container node; wherein the sidecar container and the container node are deployed in the same instance and are used for monitoring real-time running state information and historical running state information of the container node.

3. The system of claim 1, wherein, The container operation and maintenance submodule is specifically used for: analyzing the running state information of the container cluster based on a preset machine learning module, and generating a management strategy instruction for the container cluster; wherein the management strategy instruction comprises an instance expansion instruction, an instance contraction instruction, an instance migration instruction and a resource allocation adjustment instruction.

4. The system of claim 1, wherein, The data life cycle management module is specifically used for: collecting a data processing process and corresponding intermediate results of each project group and storing a data processing path through a graph database; wherein nodes in the graph database represent original data, intermediate results or final outputs, edges in the graph database represent data processing operations and metadata, and attributes in the graph database represent operation parameters of the data processing operations.

5. The system of claim 1, wherein, The data operation and maintenance submodule is specifically used for: performing correlation analysis on the intermediate results through an association rule mining algorithm based on the processing process and the corresponding intermediate results stored in the form of the graph database, and generating a construction strategy instruction; wherein the construction strategy instruction comprises a data merging instruction, a storage format optimization instruction and a data life cycle marking instruction. The data life cycle management module is specifically used for generating a corresponding data generation tree in response to the construction strategy instruction, and optimizing the data generation tree based on a big data decision tree algorithm to obtain an optimized data generation tree.

6. The system of claim 1, wherein, The data life cycle management module is further used for: adopting a Hudi incremental management technology for data with less modification, separating incremental data and storing the incremental data in a containerized environment.

7. The system of claim 1, wherein, The intelligent operation and maintenance module further comprises an abnormality early warning submodule; the abnormality early warning submodule is configured to: trigger an alarm mechanism to notify a user that the container cluster is abnormal when detecting that the running state of the container cluster is abnormal based on the running state information of the container cluster; trigger an alarm mechanism to notify a user that the data lake cluster is abnormal when detecting that the data redundancy exceeds a preset threshold.

8. The system of claim 1, wherein, The intelligent operation and maintenance module further comprises a cross-cloud fusion submodule; the cross-cloud fusion submodule is configured to: generate an instance migration instruction through a cross-cloud resource scheduling strategy when detecting that the container cluster is insufficient in resources; wherein the instance migration instruction comprises an object storage (OBS) cross-cloud replication migration instruction or a containerized image migration instruction.

9. The system of claim 1, wherein, The system further comprises a logical window unified view module; The logical window unified view module is configured to integrate the monitoring data of the data lake cluster and the container cluster into the same view and support division of monitoring units according to project groups; The logical window unified view module is further configured to update configuration parameters synchronously according to component dependency relationships and support batch operations.

10. A converged containerized data lake operation and management method, characterized in that, The method is applied to the fusion containerized data lake operation and maintenance management system as claimed in any one of claims 1-9, and the method comprises: acquiring running state information of a container cluster through a container integration management module; wherein the running state information of the container cluster comprises real-time running state information and historical running state information of each container node; generating management strategy instructions for the container cluster based on the running state information of the container cluster through a container operation and maintenance submodule; performing corresponding management operations in response to the management strategy instructions through the container integration management module; collecting the processing process and corresponding intermediate results of data of each project group through a data life cycle management module and storing the processing process and corresponding intermediate results in the form of a graph database; performing correlation analysis on the intermediate results based on the processing process and corresponding intermediate results stored in the form of a graph database through a data operation and maintenance submodule to generate construction strategy instructions; generating corresponding data generation trees in response to the construction strategy instructions through the data life cycle management module; wherein the data generation trees are used to represent data processing paths and redundancy analysis results.

Images