Hadoop-free cloud data warehouse in cloud original environment

Abstract

The invention discloses a Hadoop-free cloud data warehouse in an original cloud environment. The Hadoop-free cloud data warehouse comprises a multi-data source layer, a data acquisition layer, a data transmission layer, a data calculation layer, a data storage layer, a service layer and an operation environment, the multi-data source layer comprises various original data sources; the data acquisition layer is used for collecting, processing and storing various data sources; the data transmission layer is used for carrying out parallel processing and transmission on various types of data; the data calculation layer is used for realizing real-time data processing and analysis; the data storage layer realizes data storage and analysis; the business layer supports real-time data analysis, business monitoring, report generation and prediction analysis; and the operation environment constructs a data warehouse operation environment. By means of the cloud native data warehouse system, an efficient, flexible and extensible data processing and storage solution without a Hadoop framework is achieved, all aspects of data collection, transmission, calculation, storage, service application and operation environments are covered, and the requirements of enterprises for real-time data analysis and service optimization are met.

Description

Technical Field

[0001] The present invention relates to the technical field of data warehouse and cloud computing, and in particular to a Hadoop-free cloud data warehouse in a cloud native environment. Background Art

[0002] A data warehouse is a system used to store and manage large amounts of historical data to support business intelligence activities such as reporting, analysis, and decision support. A data warehouse integrates data from multiple heterogeneous data sources to provide a unified data view for effective data analysis and mining.

[0003] Traditional data warehouses are usually based on large relational database systems, which require a lot of hardware resources and complex software architecture to support. In order to process large-scale data, enterprises often rely on distributed processing frameworks such as Hadoop. However, this architecture faces the following challenges:

[0004] High complexity: The Hadoop system requires the configuration and management of multiple components (such as HDFS, MapReduce, YARN, etc.), which increases the complexity of the system.

[0005] High maintenance cost: Maintaining a Hadoop cluster requires a dedicated technical team, which increases manpower and operating costs.

[0006] Scalability limitations: Although Hadoop has advantages in processing large-scale data, its scalability is still limited, especially when faced with sudden data growth. Summary of the invention

[0007] In order to solve the existing problems, the present invention provides a method for constructing a Hadoop cloud data warehouse in a cloud native environment. The specific scheme is as follows:

[0008] A cloud data warehouse without Hadoop in a cloud native environment, including a multi-data source layer, a data collection layer, a data transmission layer, a data calculation layer, a data storage layer, a business layer and an operating environment;

[0009] The multiple data source layer includes various original data sources;

[0010] The data collection layer is used to collect multiple data sources from the multiple data source layer, and process and store the collected data;

[0011] The data transmission layer is used to use Apache Kafka to perform parallel processing and high-throughput transmission of various types of data collected by the data collection layer;

[0012] The data computing layer is used to achieve efficient, low-latency real-time data processing and analysis through Apache Flink;

[0013] The data storage layer uses Apache Doris and a combination of Apache Iceberg and object storage service OSS to achieve efficient, flexible and scalable data storage and analysis;

[0014] The business layer provides solutions by integrating the data collection layer, data transmission layer, data calculation layer and data storage layer, supporting real-time data analysis, business monitoring, report generation and predictive analysis;

[0015] The operating environment uses Kubernetes and Docker to build an efficient, scalable, and easy-to-manage data warehouse operating environment.

[0016] Preferably, the data in the multiple data source layer is cleansed and converted to meet the needs of the data warehouse.

[0017] Preferably, the data collection layer realizes efficient data collection and processing of multiple data sources through Fluent Bit and Flink CDC; wherein, Fluent Bit is responsible for collecting and processing log data, Flink CDC captures and processes database change data, and transmits the data to Apache Kafka for unified management, and combines Flink's real-time stream processing capabilities to further process the data and store it in Iceberg and Doris.

[0018] Preferably, in the data transmission layer, Fluent Bit and Flink CDC act as producers to send different types of data to Apache Kafka, and Flink acts as a consumer to read and process data from Apache Kafka in real time; through the topic and partition mechanism of Apache Kafka, parallel processing and high-throughput transmission of data are achieved, ensuring the timeliness and reliability of real-time data processing.

[0019] The beneficial effects of the present invention are:

[0020] Through innovative data warehouse architecture and technology combination, the present invention realizes the simplification, efficiency, flexibility and security of the system, significantly improves the enterprise data processing and analysis capabilities, reduces operating costs, improves business response speed and system stability, and provides strong support for the enterprise's data-driven decision-making and business optimization.

[0021] 1. Simplify system architecture and operation and maintenance

[0022] By not relying on the traditional Hadoop architecture, the present invention significantly simplifies the system architecture and reduces the complexity and cost of operation and maintenance. The simplified architecture makes the system easier to deploy and maintain, while reducing the need for professional Hadoop operation and maintenance personnel.

[0023] 2. Improve real-time data processing and analysis capabilities

[0024] The present invention uses Apache Flink for real-time data processing, ensuring that data can flow from the data source to the analysis layer in the shortest possible time, meeting the enterprise's demand for low-latency data processing. This capability is critical for business scenarios that require real-time decision-making and rapid response.

[0025] 3. Enhance data transmission efficiency and reliability

[0026] Fluent Bit and Flink CDC are used for data collection, combined with Kafka for data transmission, to build an efficient and reliable data pipeline. Kafka's high throughput and distributed characteristics ensure the stability and efficiency of data transmission, and maintain good performance even in a high-concurrency environment.

[0027] 4. Optimize storage and access performance

[0028] By combining Apache Doris and Apache Iceberg+OSS, an optimized storage solution for different data types and access modes is provided. Doris supports high-concurrency real-time query and analysis, while Iceberg combined with OSS supports the storage and management of large-scale historical data, providing a flexible data storage solution.

[0029] 5. Improve the scalability and flexibility of the system

[0030] Using Kubernetes and Docker for containerization and orchestration management makes the system highly available, automated, and flexible in expansion. Kubernetes's automatic expansion function dynamically adjusts resources according to business needs to ensure that the system can run smoothly even during peak periods.

[0031] 6. Achieve data security and compliance

[0032] Through the identity authentication and authorization mechanism of Kubernetes, the present invention can achieve strict access control and data protection, ensuring data security and compliance, which is crucial for protecting sensitive enterprise data and meeting regulatory requirements.

[0033] 7. Provide comprehensive monitoring and management capabilities

[0034] Combining Prometheus and Grafana, a comprehensive monitoring and log management system is built. Real-time monitoring and alarm functions help to discover and solve system problems in a timely manner, and log management provides detailed operation records and audit capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 It is a system principle block diagram of the present invention;

[0037] Figure 2 It is the data flow graph of Flink of the present invention;

[0038] Figure 3 This is the grafana monitoring chart of the doris cluster and flink cluster of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] like Figure 1 , a Hadoop-free cloud data warehouse in a cloud native environment, including a multi-data source layer, a data collection layer, a data transmission layer, a data computing layer, a data storage layer, a business layer and an operating environment.

[0041] Among them, the multi-data source layer, the data source layer is the bottom layer of the data warehouse and the starting point of the data warehouse. This layer mainly includes the original data sources such as various business systems, databases, and file systems of the enterprise. The data in these data sources are characterized by large scale and complex structure, and usually need to be cleaned and converted to meet the needs of the data warehouse.

[0042] The data collection layer uses Fluent Bit and Flink CDC to achieve efficient data collection and processing of multiple data sources. Fluent Bit is responsible for collecting and processing log data, and Flink CDC captures and processes database change data and transmits the data to Kafka for unified management. Combined with Flink's real-time stream processing capabilities, the data is further processed and stored in Iceberg and Doris, realizing an efficient, flexible, and scalable data collection and processing system.

[0043] At the data transmission layer, Apache Kafka is used to implement an efficient, low-latency, reliable, and scalable data transmission solution. Fluent Bit and Flink CDC act as producers to send different types of data to Kafka, and Flink acts as a consumer to read and process data from Kafka in real time. Through Kafka's topic and partition mechanism, parallel processing and high-throughput transmission of data are achieved, ensuring the timeliness and reliability of real-time data processing.

[0044] The data computing layer implements efficient, low-latency real-time data processing and analysis through Apache Flink. Flink reads real-time data streams from Kafka, performs complex data processing operations, and outputs the processed data to Kafka, Iceberg, or Doris. Flink's high throughput, low latency, rich operations, and fault tolerance ensure the accuracy and reliability of data processing. Through integration with Kafka, Iceberg, Doris, and Kubernetes, Flink builds a flexible, scalable, and efficient data computing solution.

[0045] In the data storage layer, Apache Doris and Apache Iceberg are combined with the object storage service (OSS) to achieve efficient, flexible, and scalable data storage and analysis. Doris provides high-performance real-time data storage and query capabilities, suitable for real-time analysis scenarios. The combination of Iceberg and OSS provides storage and management of large-scale data, supporting complex batch analysis and query. Through Flink's data processing and writing, efficient flow and storage of data between Doris and Iceberg are achieved, meeting various data analysis needs.

[0046] The business layer provides a complete solution by integrating data collection, transmission, computing and storage layers, supporting real-time data analysis, business monitoring, report generation and predictive analysis of enterprises. By using Doris and Iceberg as data storage engines and combining Apache Flink for data processing and analysis, an efficient, flexible and scalable data warehouse system is built. Through the implementation of the business layer, enterprises can better understand and use data, support business decision-making and operation optimization. Figure 2 It is the data flow diagram of Flink. The source end is Kafka and the sink end is Doris. This process is to read the data from Kafka, perform conversion calculations and other operations to form the target data and store it in the Doris database cluster.

[0047] By using Kubernetes and Docker, we can build an efficient, scalable, and easy-to-manage data warehouse operating environment. Docker containerization technology ensures the portability and consistency of applications. Kubernetes provides powerful container orchestration capabilities to achieve automated deployment, expansion, and management. Combined with Prometheus and Grafana, we have built a complete monitoring and log management system to ensure the high availability and stability of the system. Figure 3 They are grafana monitoring of doris cluster and flink cluster respectively.

[0048] The present invention realizes the simplification, efficiency, flexibility and security of the system through innovative data warehouse architecture and technology combination, significantly improves the enterprise data processing and analysis capabilities, reduces operating costs, improves business response speed and system stability, and provides strong support for enterprise data-driven decision-making and business optimization. The present invention has the following beneficial effects:

[0049] 1. Simplify system architecture and operation and maintenance

[0050] By not relying on the traditional Hadoop architecture, the present invention significantly simplifies the system architecture and reduces the complexity and cost of operation and maintenance. The simplified architecture makes the system easier to deploy and maintain, while reducing the need for professional Hadoop operation and maintenance personnel.

[0051] 2. Improve real-time data processing and analysis capabilities

[0052] The present invention uses Apache Flink for real-time data processing, ensuring that data can flow from the data source to the analysis layer in the shortest possible time, meeting the enterprise's demand for low-latency data processing. This capability is critical for business scenarios that require real-time decision-making and rapid response.

[0053] 3. Enhance data transmission efficiency and reliability

[0054] Fluent Bit and Flink CDC are used for data collection, combined with Kafka for data transmission, to build an efficient and reliable data pipeline. Kafka's high throughput and distributed characteristics ensure the stability and efficiency of data transmission, and maintain good performance even in a high-concurrency environment.

[0055] 4. Optimize storage and access performance

[0056] By combining Apache Doris and Apache Iceberg+OSS, an optimized storage solution for different data types and access modes is provided. Doris supports high-concurrency real-time query and analysis, while Iceberg combined with OSS supports the storage and management of large-scale historical data, providing a flexible data storage solution.

[0057] 5. Improve the scalability and flexibility of the system

[0058] Using Kubernetes and Docker for containerization and orchestration management makes the system highly available, automated, and flexible in expansion. Kubernetes's automatic expansion function dynamically adjusts resources according to business needs to ensure that the system can run smoothly even during peak periods.

[0059] 6. Achieve data security and compliance

[0060] Through the identity authentication and authorization mechanism of Kubernetes, the present invention can achieve strict access control and data protection, ensuring data security and compliance, which is crucial for protecting sensitive enterprise data and meeting regulatory requirements.

[0061] 7. Provide comprehensive monitoring and management capabilities

[0062] Combining Prometheus and Grafana, a comprehensive monitoring and log management system is built. Real-time monitoring and alarm functions help to discover and solve system problems in a timely manner, and log management provides detailed operation records and audit capabilities.

[0063] The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples and designs described herein, but should be granted the widest scope consistent with the principles and novel features disclosed herein.

[0064] Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent substitutions for some of the technical features therein; and these modifications or substitutions do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cloud data warehouse without Hadoop in a cloud native environment, characterized by: It includes multiple data source layer, data collection layer, data transmission layer, data calculation layer, data storage layer, business layer and operating environment; The multiple data source layer includes various original data sources; The data collection layer is used to collect multiple data sources from the multiple data source layer, and process and store the collected data; The data transmission layer is used to use Apache Kafka to perform parallel processing and high-throughput transmission of various types of data collected by the data collection layer; The data computing layer is used to achieve efficient, low-latency real-time data processing and analysis through Apache Flink; The data storage layer uses Apache Doris and a combination of Apache Iceberg and object storage service OSS to achieve efficient, flexible and scalable data storage and analysis; The business layer provides solutions by integrating the data collection layer, data transmission layer, data calculation layer and data storage layer, supporting real-time data analysis, business monitoring, report generation and predictive analysis; The operating environment uses Kubernetes and Docker to build an efficient, scalable, and easy-to-manage data warehouse operating environment.

2. The data warehouse according to claim 1, characterized in that: The data in the multiple data source layer is cleaned and converted to meet the needs of the data warehouse.

3. The data warehouse according to claim 1, characterized in that: The data collection layer uses Fluent Bit and Flink CDC to achieve efficient data collection and processing of multiple data sources; Fluent Bit is responsible for collecting and processing log data, Flink CDC captures and processes database change data, and transmits the data to Apache Kafka for unified management. Combined with Flink's real-time stream processing capabilities, the data is further processed and stored in Iceberg and Doris.

4. The data warehouse according to claim 3, characterized in that: In the data transmission layer, Fluent Bit and Flink CDC act as producers to send different types of data to Apache Kafka. Flink acts as a consumer to read and process data from Apache Kafka in real time. Through the topic and partition mechanism of Apache Kafka, parallel processing and high-throughput transmission of data are achieved, ensuring the timeliness and reliability of real-time data processing.

Images