A data lake service platform construction method based on stream-batch integration

Abstract

The application provides a data lake service platform construction method based on stream batch integration, which comprises the following steps: uniformly accessing data sources to a data job bottom layer by using a Flink CDC mode; constructing a data lake service platform and a data bus access job task, and uniformly accessing the data sources to the data lake; uniformly storing distributed files and metadata of data by the data lake service platform, and adopting a unified data lake table format for the accessed data; adopting Flink as a stream batch integrated computing engine, constructing a gateway of the unified data lake service platform based on Flink SQL GateWay, and uniformly adopting Flink SQL as a development language for stream batch data processing, so that stream batch processing is realized on the accessed data according to business timeliness; and when the job monitoring center judges that the index of metrics data reaches a preset alarm threshold, an alarm notification is sent to a relevant administrator. The application improves the data calculation result quality, improves the enterprise research and development efficiency and resource utilization rate, and reduces the maintenance cost.

Description

Technical Field

[0001] This invention relates to the field of computer software technology, and in particular to a method for constructing a data lake service platform based on integrated batch processing. Background Technology

[0002] Big data architecture solutions involve streaming data, batch data, and lake data to meet the processing needs of different types of data. Specifically:

[0003] Streaming data generally refers to real-time data streams, such as sensor data and log data. To process streaming data, streaming data processing engines can be used, such as Apache Kafka or Apache Flink. These engines can process data streams in real time and transform them into formats suitable for storage.

[0004] Batch data generally refers to data processed offline, such as data exported from a database or historical data. To process batch data, batch processing engines such as Apache Hadoop or Apache Spark can be used. These engines can perform parallel processing in a cluster and store the results in a data lake or data warehouse.

[0005] A data lake generally refers to a storage architecture used to store various types of raw and processed data, such as batch and streaming data. Data lakes typically employ distributed storage systems, such as Hadoop HDFS or Amazon S3. To manage and query the data in a data lake, query engines such as Apache Hive or Amazon Athena can be used.

[0006] Traditional stream processing and batch processing each use two separate pipelines, which has the following problems:

[0007] (1) There is a problem of data replay when processing the same set of data. Data replay will lead to data inconsistency and data recalculation will cause inconsistency in data caliber. Business indicators are easy to be inconsistent in offline and real-time task output, resulting in low data quality.

[0008] (2) Processing stream processing and batch processing in two separate links will also cause redundancy in the data link, resulting in problems such as data processing delay, low resource utilization, and high development and maintenance costs.

[0009] (3) There are two sets of code for stream processing and batch processing, which cannot be reused. Using two sets of code for the same processing logic in both stream and batch processing leads to reduced data validity and accuracy, resulting in low data quality. Summary of the Invention

[0010] The purpose of this invention is to at least address one of the aforementioned technical deficiencies.

[0011] Therefore, the purpose of this invention is to propose a method for constructing a data lake service platform based on integrated batch processing.

[0012] To achieve the above objectives, embodiments of the present invention provide a method for constructing a data lake service platform based on integrated stream and batch processing, comprising the following steps:

[0013] Step S1: Abstract each data source and data access behavior. Based on the abstracted data source and data access behavior, use Flink CDC to uniformly connect the data source to the underlying data job. Build a data lake service platform and a data bus access job task. The data bus access job task realizes unified management of the development job of data access and uniformly connects the data source to the data lake.

[0014] Step S2: The data lake service platform performs distributed file and metadata unified storage on the data accessed to the data lake, and adopts a unified data lake table format for the accessed data to perform unified layering and unified storage of offline data and real-time data.

[0015] Step S3: The data lake service platform uses Flink as the streaming and batch processing engine, builds a unified data lake service platform gateway based on Flink SQLGateway, and uniformly uses Flink SQL as the development language for streaming and batch data processing. This enables streaming and batch processing of the data after access based on business timeliness, thereby building a data lake service platform based on streaming and batch processing. The Flink jobs executed by the data lake service platform based on streaming and batch processing can freely switch between streaming and batch execution modes and use the same set of code.

[0016] In step S4, the job monitoring center uses passive push to obtain metrics data for stream processing and batch processing, queries and analyzes the metrics data, and sends an alarm notification to the relevant administrator when the metrics data reaches the preset alarm threshold. At the same time, the job monitoring display tool performs measurement analysis and dashboard display.

[0017] Furthermore, in step S1,

[0018] The abstraction of each data source includes the following steps: configuring a lightweight development interface and a directory tree-style management interface for each data source, so as to develop and manage the data source through the management interface;

[0019] The abstraction of data access behavior includes the following steps: abstracting the data access behavior of task execution configuration, data writing strategy, dirty data filtering strategy and access task execution time strategy for each data source.

[0020] Furthermore, in step S2, the data lake service platform uses the HDFS distributed file system to perform distributed file storage for the data in the data lake.

[0021] Furthermore, in step S2, the data lake service platform uses a MySQL database as the underlying storage and utilizes Hive Metastore artifacts to manage metadata in the Flink On Iceberg table format and provide services.

[0022] The data accessed adopts the Iceberg table format to unify the data lake table format. Based on the underlying storage of the MySQL database, the organization format of data and metadata is defined in the Iceberg table format to achieve unified layering and unified storage of offline and real-time data.

[0023] Furthermore, in step S3, a unified batch and stream script development console is constructed, and the data lake service platform is used to configure the unified batch and stream development script, including job configuration, execution configuration, savepoint configuration, and version history configuration.

[0024] Furthermore, the integrated streaming and batch script development console configures the data lake service platform for tasks, including: visually configuring the execution mode, cluster selection, task parallelism, and alarm groups for FlinkSQL, and displaying the save point path and creation time to facilitate recovery after the streaming job is interrupted.

[0025] Furthermore, in step S3, the data lake service platform manages user-defined UDF functions that integrate stream and batch processing.

[0026] Furthermore, in step S3, the data lake service platform publishes integrated stream and batch processing code, supporting the publication and execution of batch processing and stream processing jobs, so as to perform stream and batch processing on the data after access according to the business timeliness, and build a data lake service platform based on integrated stream and batch processing.

[0027] Furthermore, in step S4, the operation monitoring center periodically collects metrics data of stream processing and batch processing, and configures corresponding preset alarm thresholds for each metric.

[0028] When the metrics data reaches the preset alarm threshold, an alarm notification is sent to the administrator.

[0029] Furthermore, in step S4, the operation monitoring center provides an interface for querying, measuring, and graphically visualizing the metrics data.

[0030] The data lake service platform construction method based on integrated stream and batch processing according to embodiments of the present invention has the following beneficial effects:

[0031] (1) The data lake service platform supports offline and real-time data acquisition using a unified data acquisition method. Batch processing and stream processing are uniformly handled through CDC. Data is captured in real time, pushed to Kafka, and then loaded into Iceberg, thereby achieving unified storage of the data lake and avoiding the generation of data silos.

[0032] (2) The data lake service platform enables streaming and batch processing to use a unified ETL component and SQL syntax. Through underlying parsing, it adapts to the streaming and batch computing engines respectively, allowing Flink jobs to switch freely between streaming and batch execution modes and only need to maintain one set of code, thereby enhancing the reusability and maintainability of the job.

[0033] (3) Batch processing and stream processing use the same computing engine, which fundamentally avoids the problem of two sets of code for the same processing logic, thereby improving the validity of data and the accuracy of business calculations, and improving data quality.

[0034] (4) Based on the unified data storage of the data lake, the computing engine uses Flink so that the streaming and batch processing of data can use the same engine, the same SQL syntax and the same ETL components. While ensuring high throughput and low latency, it reduces data link redundancy, reduces the risk of data inconsistency caused by data replay and data inconsistency caused by data recalculation, and saves development and operation and maintenance costs.

[0035] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0036] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0037] Figure 1 This is a flowchart illustrating a method for constructing a data lake service platform based on integrated batch and stream processing according to an embodiment of the present invention.

[0038] Figure 2 This is a diagram illustrating the overall architecture of a data lake service platform based on integrated stream and batch processing, according to an embodiment of the present invention.

[0039] Figure 3 This is a data access flowchart for a data lake service platform based on integrated stream and batch processing, according to an embodiment of the present invention.

[0040] Figure 4This is a flowchart illustrating the unified data storage process of a data lake service platform based on integrated stream and batch processing, according to an embodiment of the present invention.

[0041] Figure 5 This is a flowchart illustrating the development process of batch and stream operations based on a data lake service platform integrating batch and stream operations according to an embodiment of the present invention.

[0042] Figure 6 This is a flowchart of the job monitoring process of a data lake service platform based on integrated batch processing according to an embodiment of the present invention. Detailed Implementation

[0043] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0044] The tools involved in the data lake service platform construction method based on integrated batch and stream processing of this invention will be described below:

[0045] Apache Flink is an open-source stream processing framework developed by the Apache Software Foundation. Its core is a distributed streaming data stream engine written in Java and Scala. Flink executes arbitrary streaming data programs in a data-parallel and pipelined manner. Flink's pipeline runtime system can execute both batch and stream processing programs. Meanwhile, FlinkSQLGateway provides a simple way to submit Flink jobs, look up metadata, and analyze data online, enabling multiple remote clients to execute SQL concurrently.

[0046] Apache Iceberg is an open table format for massive analytical datasets, designed specifically for petabyte-scale tables. Apache Iceberg aims to solve the correctness problem in eventually consistent cloud object storage, enabling fast reading of petabyte-scale tables without the need for distributed SQL.

[0047] This invention provides a method for constructing a data lake service platform based on integrated stream and batch processing. This method integrates FlinkSQL Gateway on a lake warehouse architecture to build a data architecture service platform that integrates stream and batch processing. This invention uses Flink as the integrated stream and batch computing engine, constructs a unified data lake service platform gateway based on FlinkSQL Gateway, and uniformly adopts FlinkSQL as the development language for stream and batch data processing. This enables the platform to collect static and dynamic data synchronously, and to perform stream and batch processing on the collected data according to business timeliness, thereby constructing a data lake service platform based on integrated stream and batch processing.

[0048] like Figure 1 and Figure 2 As shown in the figure, the method for constructing a data lake service platform based on integrated stream and batch processing according to an embodiment of the present invention includes the following steps:

[0049] Step S1 involves abstracting each data source and data access behavior. Based on the abstracted data sources and data access behaviors, the Flink CDC approach is used to uniformly connect the data sources to the underlying data job layer. A data lake service platform and data bus access job tasks are then constructed. These data bus access job tasks provide unified management of data access development jobs, ensuring that data sources are uniformly connected to the data lake.

[0050] The following is for reference. Figure 3 Step S1 will be explained in detail below:

[0051] Step S11 involves abstracting each data source, including the following steps: configuring a lightweight development interface and a directory tree-style management interface for the data source, so as to develop and manage the data source through the management interface.

[0052] Specifically, it provides a lightweight development interface and a directory tree-style management interface for reading common data sources such as JDBC, file systems, and message queues, making it convenient for users to develop and manage data sources in a unified manner.

[0053] In one embodiment of the present invention, data source management and access are supported, including relational databases such as MySQL and Oreacle, non-relational databases such as MongoDB and Elastic, and Kafka message queues.

[0054] It should be noted that the types of data sources that can be accessed are not limited to the examples above, and other types can also be included. These can be set as needed, and will not be elaborated here.

[0055] Step S12, abstract the data access behavior, including the following steps: abstract custom behaviors such as task execution configuration, data writing strategy, dirty data filtering strategy and access task execution time strategy for each data source, and use FlinkCDC to access all data sources uniformly.

[0056] Step S13: Unified access to the data lake. A lightweight development interface and a directory tree-style management interface are provided for reading and writing data lake data, making it convenient for users to develop and manage data lake storage in a unified manner.

[0057] Step S14: Construct a data bus access operation platform and provide a web console suitable for human-computer interaction.

[0058] Business data can be quickly imported into big data systems via the data bus, shortening the data analysis cycle. The development work for data access is managed in a unified manner, including functions such as data access, data publishing, and data subscription. With simple configuration, accessed data can be quickly synchronized to downstream storage systems, greatly reducing the workload of the data link.

[0059] Step S15: Connect to the job management console. This job management console supports page-based data source management, job management, task monitoring and alerting, and other functions. It also supports extending the functionality of the job management console using a low-code development framework.

[0060] In summary, this step establishes a data access bus through a unified data access method, enabling unified management of all data sources and data access jobs. It supports the management and access of data sources including relational databases such as MySQL and Oracle, non-relational databases such as MongoDB and Elastic, and Kafka message queues. The underlying data access jobs are uniformly implemented through Flink CDC.

[0061] In step S2, the data lake service platform performs distributed file and metadata unified storage on the data accessed to the data lake, and adopts a unified data lake table format for the accessed data to perform unified layering and unified storage of offline data and real-time data.

[0062] The following is for reference. Figure 4 Step S2 will be explained in detail below:

[0063] Step S21: The data lake service platform uses the HDFS distributed file system as the storage platform for data storage, and performs distributed file storage on the data in the data lake.

[0064] In step S22, the data lake service platform uses a MySQL database as the underlying storage and leverages Hive Metastore artifacts to manage metadata in the Flink On Iceberg table format and provide services.

[0065] Step S23: The Iceberg unified data lake table format is adopted, which supports unified querying of offline / real-time data and supports the expansion of real-time machine learning functions.

[0066] Specifically, the data accessed adopts the Iceberg table format to unify the data lake table format. Based on the underlying storage of the MySQL database, the Iceberg table format is used to define the organization format of data and metadata, so as to achieve unified layering and unified storage of offline and real-time data.

[0067] In summary, this step achieves unified storage of data and metadata by uniformly connecting the collected data to the data lake. The data storage uses HDFS, and the metadata storage uses MySQL, thereby realizing unified layering and storage of real-time and offline data, while ensuring data real-time performance and consistency.

[0068] Step S3: The data lake service platform adopts Flink as the unified stream and batch computing engine, builds a unified data lake service platform gateway based on Flink SQLGateway, and uniformly adopts Flink SQL as the development language for stream and batch data processing. This enables the processing of incoming data in both stream and batch modes according to business timeliness, thereby building a data lake service platform based on the unified stream and batch processing. Flink jobs executed on the unified stream and batch data lake service platform can freely switch between stream and batch execution modes and use the same set of code.

[0069] The following is for reference. Figure 5 Step S3 will be explained in detail below:

[0070] Step S31: Build a batch processing script development console that supports interactive, page-based code development, code inspection, process management, execution history, and other functions.

[0071] It should be noted that the integrated batch script development console is one of the modules of the data lake platform. It is mainly used as an IDE for writing scripting languages ​​such as SQL and Python. The subsequent management of UDF functions and code deployment are executed by the data lake platform.

[0072] Step S32: Configure the integrated stream and batch script development console for the data lake service platform, including job configuration, execution configuration, savepoint configuration, and version history configuration.

[0073] Specifically, the integrated batch and stream script development console configures the integrated batch and stream development scripts, including the environment required for script runtime, such as job configuration, execution configuration, savepoint configuration, and version history. In other words, it provides a configuration process for directly publishing development scripts based on the IDE; job configuration, execution configuration, savepoint configuration, and version history configuration are performed during script publishing.

[0074] For example, the integrated streaming and batch script development console configures jobs on the data lake service platform, including: visually configuring the execution mode, cluster selection, task parallelism, and alarm groups for FlinkSQL, and displaying the save point path and creation time to facilitate recovery after the streaming job is interrupted.

[0075] Step S33: The data lake service platform manages user-defined UDF functions that integrate batch and stream processing.

[0076] Specifically, it supports the use of different types of UDF functions such as Java, Scala, and Python during the code development process, including UDF function version management, editing, and deletion functions.

[0077] Step S34: The data lake service platform releases integrated stream and batch processing code, supporting the release and execution of batch processing and stream processing jobs, so as to perform stream and batch processing on the data after access according to the business timeliness, and build a data lake service platform based on integrated stream and batch processing.

[0078] In embodiments of the present invention, it is supported to publish and run batch processing and stream processing jobs, including functions such as task pre-dependency, task retries, task online, and task offline.

[0079] In summary, this step unifies the development APIs and paradigms for stream computing and batch computing by using a unified data computing engine and development language. The underlying computing engine for data development jobs is uniformly Flink, and the development language is uniformly FlinkSQL. This ensures data consistency in the processing and results of batch computing and stream computing, fundamentally avoiding the problem of having two sets of code for the same processing logic in batch jobs and stream jobs.

[0080] In step S4, the job monitoring center uses passive push to obtain metrics data for stream processing and batch processing, queries and analyzes the metrics data, and sends an alarm notification to the relevant administrator when the metrics data reaches the preset alarm threshold. At the same time, the job monitoring display tool performs measurement analysis and dashboard display.

[0081] In an embodiment of the present invention, the job monitoring center includes: a job data acquisition component, a job monitoring alarm component, and a job monitoring data display component.

[0082] The following is for reference. Figure 6 Step S4 will be explained in detail below:

[0083] In step S41, the job data acquisition component uses passive push to obtain metrics data for stream processing and batch processing, and exposes the data to the job monitoring and alarm component.

[0084] Step S42: The job monitoring center periodically collects metrics data from stream processing and batch processing, and configures corresponding preset alarm thresholds for each metric. When a metric data indicator reaches a preset alarm threshold, an alarm notification is sent to the administrator.

[0085] Specifically, the job monitoring and alarm component periodically pulls the exposed metrics from the job data collection component, configures alarm rules, and sends alarms when alarm threshold conditions are triggered.

[0086] It should be noted that the preset alarm threshold can be set based on empirical values ​​of metrics data and the past states of events requiring alarms. The preset alarm threshold can be updated as real-time status and events change.

[0087] In this step, the metrics data includes multiple metrics such as JM memory, TM memory, transmission latency, number of Sink records, number of Source records, number and size of Checkpoints, and size of State data.

[0088] Each metric has a corresponding alarm threshold. When one of the metrics reaches its corresponding alarm threshold, an alarm is triggered and an alarm notification is sent through internal communication.

[0089] In embodiments of the present invention, alarm notifications include, but are not limited to, the following methods: SMS, internal email, internal communication facilities, etc.

[0090] Step S43: The operation monitoring center uses the metrics data's indicator update query, metric analysis, and graphical visualization analysis interface.

[0091] Specifically, the graphical visualization analysis interface offers a variety of visualization methods, including heatmaps, line charts, and graphs.

[0092] In summary, this step involves building a monitoring metrics center to analyze the operational status and alarm configuration of jobs. The job monitoring center connects the Flink job's runtime metrics data to the job data acquisition component. The job monitoring alarm component pulls the metrics data exposed by the job data acquisition component to configure alarms, so that when the job's runtime metrics reach the threshold, relevant alarm information will be sent to the relevant responsible persons. At the same time, the job monitoring display tool performs measurement analysis and dashboard display.

[0093] This invention integrates FlinkSQLGateway on a lake-warehouse integrated architecture to build a data service lake platform that integrates streaming and batch processing. Compared with the traditional two-link processing of streaming and batch processing, the data service platform built on a lake-warehouse integrated architecture and integrating FlinkSQLGateway not only improves the quality of data calculation results, but also improves enterprise R&D efficiency and resource utilization, and reduces maintenance costs.

[0094] The data lake service platform construction method based on integrated stream and batch processing according to embodiments of the present invention has the following beneficial effects:

[0095] (1) The data lake service platform supports offline and real-time data acquisition using a unified data acquisition method. Batch processing and stream processing are uniformly handled through CDC. Data is captured in real time, pushed to Kafka, and then loaded into Iceberg, thereby achieving unified storage of the data lake and avoiding the generation of data silos.

[0096] (2) The data lake service platform enables streaming and batch processing to use a unified ETL component and SQL syntax. Through underlying parsing, it adapts to the streaming and batch computing engines respectively, allowing Flink jobs to switch freely between streaming and batch execution modes and only need to maintain one set of code, thereby enhancing the reusability and maintainability of the job.

[0097] (3) Batch processing and stream processing use the same computing engine, which fundamentally avoids the problem of two sets of code for the same processing logic, thereby improving the validity of data and the accuracy of business calculations, and improving data quality.

[0098] (4) Based on the unified data storage of the data lake, the computing engine uses Flink so that the streaming and batch processing of data can use the same engine, the same SQL syntax and the same ETL components. While ensuring high throughput and low latency, it reduces data link redundancy, reduces the risk of data inconsistency caused by data replay and data inconsistency caused by data recalculation, and saves development and operation and maintenance costs.

[0099] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0100] Although embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention without departing from the principles and spirit of the invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for constructing a data lake service platform based on integrated stream and batch processing, characterized in that, Includes the following steps: Step S1: Abstract each data source and data access behavior. Based on the abstracted data source and data access behavior, use Flink CDC to uniformly connect the data source to the underlying data job. The data lake service platform and data bus access tasks are constructed. The data bus access tasks enable unified management of data access development tasks and unify the access of data sources to the data lake. The abstraction of each data source includes the following steps: configuring a lightweight development interface and a directory tree-style management interface for each data source, so as to develop and manage the data source through the management interface; The abstraction of data access behavior includes the following steps: abstracting data access behavior for each data source, including task execution configuration, data writing strategy, dirty data filtering strategy, and access task execution time strategy. Step S2: The data lake service platform performs distributed file and metadata unified storage on the data accessed to the data lake, and adopts a unified data lake table format for the accessed data to perform unified layering and unified storage of offline data and real-time data. The data lake service platform uses a MySQL database as the underlying storage and utilizes Hive Metastore artifacts to manage metadata in the Flink On Iceberg table format and provide services. The data accessed adopts the Iceberg table format to unify the data lake table format. Based on the underlying storage of the MySQL database, the organization format of data and metadata is defined in the Iceberg table format to achieve unified layering and unified storage of offline and real-time data. Step S3: The data lake service platform uses Flink as the streaming and batch processing engine, builds a unified data lake service platform gateway based on Flink SQLGateway, and uniformly uses Flink SQL as the development language for streaming and batch data processing. This enables streaming and batch processing of the data after access based on business timeliness, thereby building a data lake service platform based on streaming and batch processing. The Flink jobs executed by the data lake service platform based on streaming and batch processing can freely switch between streaming and batch execution modes and use the same set of code. In step S4, the job monitoring center uses passive push to obtain metrics data for stream processing and batch processing, queries and analyzes the metrics data, and sends an alarm notification to the relevant administrator when the metrics data reaches the preset alarm threshold. At the same time, the job monitoring display tool performs measurement analysis and dashboard display.

2. The method for constructing a data lake service platform based on integrated stream and batch processing as described in claim 1, characterized in that, In step S2, the data lake service platform uses the HDFS distributed file system to perform distributed file storage for the data in the data lake.

3. The method for constructing a data lake service platform based on integrated stream and batch processing as described in claim 1, characterized in that, In step S3, a unified batch and stream script development console is constructed, and the data lake service platform is used to configure the unified batch and stream script development, including job configuration, execution configuration, savepoint configuration, and version history configuration.

4. The method for constructing a data lake service platform based on integrated stream and batch processing as described in claim 3, characterized in that, The integrated streaming and batch script development console configures the data lake service platform for tasks, including: visually configuring the execution mode, cluster selection, task parallelism, and alarm groups for FlinkSQL, and displaying the save point path and creation time to facilitate recovery after the streaming job is interrupted.

5. The method for constructing a data lake service platform based on integrated stream and batch processing as described in claim 3, characterized in that, In step S3, the data lake service platform manages user-defined UDF functions that integrate stream and batch processing.

6. The method for constructing a data lake service platform based on integrated stream and batch processing as described in claim 1, characterized in that, In step S3, the data lake service platform publishes integrated stream and batch processing code, which supports the publication and execution of batch processing and stream processing jobs to perform stream and batch processing on the data after access based on the business timeliness, thereby building a data lake service platform based on integrated stream and batch processing.

7. The method for constructing a data lake service platform based on integrated stream and batch processing as described in claim 1, characterized in that, In step S4, the job monitoring center periodically collects metrics data of stream processing and batch processing, and configures corresponding preset alarm thresholds for each metric. When the metrics data reaches the preset alarm threshold, an alarm notification is sent to the administrator.

8. The method for constructing a data lake service platform based on integrated stream and batch processing as described in claim 1, characterized in that, In step S4, the operation monitoring center uses the metrics data to query metrics updates, perform metric analysis, and access graphical visualization analysis interfaces.

Images