Paimon real-time data-based data warehouse solving system and method

Abstract

The invention belongs to the technical field of data processing, and discloses a data warehouse solving system based on Paimon real-time data, and the data warehouse solving system comprises a data access module, a data processing module, a data verification module, and a data storage module. Wherein the data processing module comprises a real-time data processing unit and a data merging unit and is used for processing, analyzing and merging real-time data and synchronizing the merged data to the Hi ve table. Through support of Paimon, real-time processing and analysis of real-time data are realized, business requirements of rapid decision making and real-time feedback are met, processing is performed aiming at different data sources, unification of data models is realized, complexity of data processing is simplified, and through an off-line data verification mechanism in the early morning every day, data processing efficiency is improved. According to the system and the method, the consistency and the availability of real-time data are ensured, the accuracy of the data is improved, the system and the method support real-time data processing and are compatible with batch processing data, and the requirements of different service scenes are met.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing, and specifically relates to a data warehouse solution system and method based on paimon real-time data. Background Art

[0002] With the continuous development of the Internet industry, the amount of data has increased rapidly, which is called the era of big data. This means that companies need to better manage, store and analyze large-scale data to support business decisions and innovation. Traditional offline data processing is no longer sufficient to meet the real-time needs of the business. Many business scenarios require timely data to support real-time decision-making, such as advertising, personalized recommendations, and risk management. At present, common systems on the market have data redundancy and complex processes. The development cost of flink sql is high and there is no universality. The real-time data is separated from the offline data, and the data uniformity is poor. Therefore, this paper optimizes and improves this problem. Summary of the invention

[0003] The purpose of the present invention is to provide a data warehouse solution system and method based on paimon real-time data to solve the problems raised in the above background technology.

[0004] In order to achieve the above object, the present invention provides the following technical solutions: a data warehouse solution system based on paimon real-time data, the data warehouse solution system includes: a data access module, a data processing module, a data verification module, and a data storage module;

[0005] The data processing module includes: a real-time data processing unit and a data merging unit, which are used to process, analyze and merge real-time data, and synchronize the merged data to the Hive table;

[0006] The data access module is responsible for receiving data from different data sources, including real-time process and offline data; real-time process data is accessed to the system in real time through a specific interface for subsequent real-time model table development;

[0007] The real-time data processing unit: utilizes the feature of Paimon supporting real-time update to process and analyze the real-time access data immediately and build a real-time model table;

[0008] The data merging unit: merges the real-time model table data with the offline data to ensure the integrity and consistency of the data; at the same time, using the synchronization capability of Paimon and Hive, synchronizes the merged data to the Hive table to achieve the non-perceived use of real-time data by downstream businesses;

[0009] The data verification module is started at regular intervals every morning to perform consistency verification on real-time data using offline data to ensure the accuracy and reliability of data during real-time processing;

[0010] The data storage module stores the Meta and data of the day based on Paimon's data lake format, and supports synchronizing Meta to Hive, providing a unified data access interface for downstream businesses.

[0011] Preferably, the data warehouse solution includes the following steps:

[0012] Step 1: Real-time process data access: Receive real-time process data through the data access module and perform preliminary preprocessing;

[0013] Step 2: Real-time model table development: In the real-time data processing unit of the data processing module, develop a real-time model table according to business needs to process and analyze real-time data;

[0014] Step 3: Data synchronization and merging: synchronize the real-time model table data to Paimon and merge it with the offline data; synchronize the merged data to the Hive table through the data merging unit to ensure that downstream businesses can use the real-time updated data without being aware of it;

[0015] Step 4: Data consistency and availability verification: The data verification module uses offline data to perform consistency verification on real-time data every morning to ensure data consistency and availability. Once data anomalies are found, the data correction mechanism is triggered in time to ensure data accuracy.

[0016] Step 5: Data storage and access: Store the processed data in the Paimon data lake, while keeping Meta and Hive synchronized to provide efficient and unified data access services for downstream businesses.

[0017] Preferably, the data access module is used to receive data from different data sources, including real-time processes and offline data;

[0018] The data verification module uses offline data to perform consistency verification on real-time data every morning;

[0019] The data storage module stores the Meta and data of the day based on Paimon's data lake format, and synchronizes the Meta to Hive.

[0020] Preferably, the step 3 of data synchronization and merging includes synchronizing the real-time model table data to Paimon, merging it with the offline data, and then synchronizing the merged data to the Hive table;

[0021] The step 4 of data consistency and availability verification includes using offline data to perform consistency verification on real-time data in the early morning every day.

[0022] Preferably, the specific data flow steps of the data warehouse solution are:

[0023] S1, obtain click stream data: the system obtains user click stream data in real time through specific data collection tools or interfaces as the basis for subsequent processing;

[0024] S2, use Flume to clean the data in real time and enter Kafka: The collected raw click stream data is first cleaned in real time by Flume to remove invalid, erroneous or redundant data to ensure data quality; the cleaned data is pushed to the Kafka message queue for subsequent real-time processing;

[0025] S3, using Kafka SQL for real-time model processing: In Kafka, use Kafka SQL to process the real-time data into models; the processed data forms a real-time model table, which provides a basis for subsequent data storage and query;

[0026] S4, data is stored in the corresponding data model: the real-time model table data is stored in the Paimon data lake to form a unified data model; Paimon, as a high-performance lake storage, supports real-time updates and efficient queries;

[0027] S5, supports real-time streaming mode to read data: The system supports reading data from Paimon in real-time streaming mode to meet the needs of downstream businesses for real-time data;

[0028] S6, data consistency check and overwriting: Every morning, the system uses offline data to check the consistency of real-time data; if the check result meets the standard, the real-time data is overwritten or updated with offline data to ensure data consistency and accuracy.

[0029] Preferably, the specific business flow steps of the data warehouse solution are:

[0030] A1. Obtain data table information and write data logic according to requirements: Business personnel first obtain data table information and write data processing logic according to business requirements;

[0031] A2, verify whether the data logic is correct: verify the written data logic to ensure its correctness and feasibility; if the verification fails, return to rewrite the data logic until the verification passes;

[0032] A3, provide real-time and historical data: according to business needs, the system can provide real-time data or historical data of the day; real-time data is read from Paimon through real-time streaming mode, and historical data is obtained by querying Paimon or Hive tables.

[0033] Preferably, the data format conversion unit supports conversion of multiple data formats, including but not limited to CSV, JSON, XML, Parquet and Avro formats, to ensure that the received data can be seamlessly integrated into the system.

[0034] Preferably, the data processing module also includes a data cleaning unit for cleaning the data before real-time data processing and merging, removing duplicate data, correcting erroneous data, and filling missing data to improve data quality and the accuracy of subsequent analysis.

[0035] Preferably, the data cleaning unit adopts a rule-based method to automatically clean the received data according to preset data cleaning rules, and supports user-defined cleaning rules to meet the needs of different business scenarios.

[0036] Preferably, the system also includes a data monitoring module for real-time monitoring of data access, processing, storage and access processes, as well as data quality and performance. Once a problem is found, an alarm is issued in a timely manner and a corresponding processing mechanism is triggered to ensure stable operation of the system and high availability of data.

[0037] The beneficial effects of the present invention are as follows:

[0038] 1. With the support of Paimon, the present invention realizes the instant processing and analysis of real-time data, meeting the business needs of rapid decision-making and real-time feedback. Processing is performed on different data sources, achieving the unification of data models and simplifying the complexity of data processing. Through the offline data verification mechanism at dawn every day, the consistency and availability of real-time data are ensured, and the accuracy of data is improved. The system and method support both real-time data processing and batch data processing, meeting the needs of different business scenarios.

[0039] 2. The present invention ensures the timeliness and validity of data through real-time data cleaning and model processing. Data storage and real-time stream mode reading meet the needs of downstream businesses for real-time data. Data consistency verification and coverage mechanisms ensure data consistency and accuracy. The refinement of business flows makes data processing logic clearer and more controllable, improving business flexibility and response speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 The data flow diagram of the present invention is:

[0041] Figure 2 This is the business flow diagram of the present invention. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] like Figure 1 to Figure 2 As shown, the embodiment of the present invention provides a data warehouse solution system based on paimon real-time data, and the data warehouse solution system includes: a data access module, a data processing module, a data verification module, and a data storage module;

[0044] The data processing module includes: a real-time data processing unit and a data merging unit, which are used to process, analyze and merge real-time data, and synchronize the merged data to the Hive table;

[0045] Data access module: responsible for receiving data from different data sources, including real-time process and offline data; real-time process data is connected to the system in real time through a specific interface for subsequent real-time model table development;

[0046] Real-time data processing unit: using the real-time update feature of Paimon, it processes and analyzes the real-time access data immediately and builds a real-time model table;

[0047] Data merging unit: merges the real-time model table data with the offline data to ensure data integrity and consistency; at the same time, using the synchronization capability of Paimon and Hive, synchronizes the merged data to the Hive table, enabling downstream businesses to use real-time data without being aware of it;

[0048] Data verification module: It starts at a fixed time every morning and uses offline data to perform consistency verification on real-time data to ensure the accuracy and reliability of data during real-time processing;

[0049] Data storage module: Stores the current day's Meta and data in Paimon's data lake format, and supports synchronizing Meta to Hive, providing a unified data access interface for downstream businesses.

[0050] The data warehouse solution includes the following steps:

[0051] Step 1: Real-time process data access: Receive real-time process data through the data access module and perform preliminary preprocessing;

[0052] Step 2: Real-time model table development: In the real-time data processing unit of the data processing module, develop a real-time model table according to business needs to process and analyze real-time data;

[0053] Step 3: Data synchronization and merging: synchronize the real-time model table data to Paimon and merge it with the offline data; synchronize the merged data to the Hive table through the data merging unit to ensure that downstream businesses can use the real-time updated data without being aware of it;

[0054] Step 4: Data consistency and availability verification: The data verification module uses offline data to perform consistency verification on real-time data every morning to ensure data consistency and availability. Once data anomalies are found, the data correction mechanism is triggered in time to ensure data accuracy.

[0055] Step 5: Data storage and access: Store the processed data in the Paimon data lake, while keeping Meta and Hive synchronized to provide efficient and unified data access services for downstream businesses.

[0056] Among them, the data access module is used to receive data from different data sources, including real-time processes and offline data;

[0057] The data verification module uses offline data to perform consistency verification on real-time data every morning;

[0058] The data storage module stores the current day's Meta and data based on Paimon's data lake format and synchronizes the Meta to Hive.

[0059] Among them, step 3 data synchronization and merging includes synchronizing the real-time model table data to Paimon, merging it with the offline data, and then synchronizing the merged data to the Hive table;

[0060] Step 4: Data consistency and availability verification includes using offline data to perform consistency verification on real-time data every morning.

[0061] Among them, the specific data flow steps of the data warehouse solution are:

[0062] S1, obtain click stream data: the system obtains user click stream data in real time through specific data collection tools or interfaces as the basis for subsequent processing;

[0063] S2, use Flume to clean the data in real time and enter Kafka: The collected raw click stream data is first cleaned in real time by Flume to remove invalid, erroneous or redundant data to ensure data quality; the cleaned data is pushed to the Kafka message queue for subsequent real-time processing;

[0064] S3, use Kafka SQL for real-time model processing: In Kafka, use Kafka SQL to process the real-time data into models, such as aggregation, filtering, and conversion, to meet business needs. The processed data forms a real-time model table, which provides a basis for subsequent data storage and query;

[0065] S4, data is stored in the corresponding data model: the real-time model table data is stored in the Paimon data lake to form a unified data model; Paimon, as a high-performance lake storage, supports real-time updates and efficient queries;

[0066] S5, supports real-time streaming mode to read data: The system supports reading data from Paimon in real-time streaming mode to meet the needs of downstream businesses for real-time data;

[0067] S6, data consistency check and overwriting: Every morning, the system uses offline data to check the consistency of real-time data; if the check result meets the standard, the real-time data is overwritten or updated with offline data to ensure data consistency and accuracy.

[0068] The specific business flow steps of the data warehouse solution are as follows:

[0069] A1, obtain data table information and write data logic according to requirements: Business personnel first obtain data table information, including table structure, field meaning, etc. According to business requirements, write data processing logic, such as calculation indicators, filtering conditions, etc.

[0070] A2, verify whether the data logic is correct: verify the written data logic to ensure its correctness and feasibility; if the verification fails, return to rewrite the data logic until the verification passes;

[0071] A3, provide real-time and historical data: according to business needs, the system can provide real-time data or historical data of the day; real-time data is read from Paimon through real-time streaming mode, and historical data is obtained by querying Paimon or Hive tables.

[0072] Among them, the data format conversion unit supports the conversion of multiple data formats, including but not limited to CSV, JSON, XML, Parquet and Avro formats, to ensure that the received data can be seamlessly integrated into the system.

[0073] The data processing module also includes a data cleaning unit, which is used to clean the data before real-time data processing and merging, remove duplicate data, correct erroneous data, and fill in missing data to improve data quality and the accuracy of subsequent analysis.

[0074] Among them, the data cleaning unit adopts a rule-based approach to automatically clean the received data according to preset data cleaning rules, and also supports users to customize cleaning rules to meet the needs of different business scenarios.

[0075] The system also includes a data monitoring module, which is used to monitor the data access, processing, storage and access process, as well as the data quality and performance in real time. Once a problem is found, it will promptly alarm and trigger the corresponding processing mechanism to ensure the stable operation of the system and high availability of data.

[0076] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0077] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. Data warehouse solution system based on paimon real-time data, characterized by: The data warehouse solution system includes: data access module, data processing module, data verification module, and data storage module; The data processing module includes: a real-time data processing unit and a data merging unit, which are used to process, analyze and merge real-time data, and synchronize the merged data to the Hive table; The data access module is responsible for receiving data from different data sources, including real-time process and offline data; real-time process data is accessed to the system in real time through a specific interface for subsequent real-time model table development; The real-time data processing unit: utilizes the feature of Paimon supporting real-time update to process and analyze the real-time access data immediately and build a real-time model table; The data merging unit: merges the real-time model table data with the offline data to ensure the integrity and consistency of the data; at the same time, using the synchronization capability of Paimon and Hive, synchronizes the merged data to the Hive table to achieve the non-perceived use of real-time data by downstream businesses; The data verification module is started at regular intervals every morning to perform consistency verification on real-time data using offline data to ensure the accuracy and reliability of data during real-time processing; The data storage module stores the Meta and data of the day based on Paimon's data lake format, and supports synchronizing Meta to Hive, providing a unified data access interface for downstream businesses.

2. The data warehouse solution system based on paimon real-time data according to claim 1 is characterized by: The data access module is used to receive data from different data sources, including real-time processes and offline data; The data verification module uses offline data to perform consistency verification on real-time data every morning; The data storage module stores the Meta and data of the day based on Paimon's data lake format, and synchronizes the Meta to Hive.

3. The data warehouse solution system based on paimon real-time data according to claim 1 is characterized by: The data format conversion unit supports conversion of multiple data formats, including but not limited to CSV, JSON, XML, Parquet, and Avro formats, to ensure that the received data can be seamlessly integrated into the system.

4. The data warehouse solution system based on paimon real-time data according to claim 1 is characterized by: The data processing module also includes a data cleaning unit, which is used to clean the data before real-time data processing and merging, remove duplicate data, correct erroneous data, and fill in missing data to improve data quality and the accuracy of subsequent analysis.

5. The data warehouse solution system based on paimon real-time data according to claim 1 is characterized by: The data cleaning unit adopts a rule-based method to automatically clean the received data according to preset data cleaning rules, and also supports user-defined cleaning rules to meet the needs of different business scenarios.

6. The data warehouse solution system based on paimon real-time data according to claim 1 is characterized by: The system also includes a data monitoring module for real-time monitoring of data access, processing, storage and access processes, as well as data quality and performance. Once a problem is found, an alarm is issued in a timely manner and a corresponding processing mechanism is triggered to ensure stable operation of the system and high availability of data.

7. The data warehouse solution based on paimon real-time data is characterized by: The data warehouse solution includes the following steps: Step 1: Real-time process data access: Receive real-time process data through the data access module and perform preliminary preprocessing; Step 2: Real-time model table development: In the real-time data processing unit of the data processing module, develop a real-time model table according to business needs to process and analyze real-time data; Step 3: Data synchronization and merging: synchronize the real-time model table data to Paimon and merge it with the offline data; synchronize the merged data to the Hive table through the data merging unit to ensure that downstream businesses can use the real-time updated data without being aware of it; Step 4: Data consistency and availability verification: The data verification module uses offline data to perform consistency verification on real-time data every morning to ensure data consistency and availability; Once data anomalies are discovered, the data correction mechanism is triggered in a timely manner to ensure data accuracy; Step 5: Data storage and access: Store the processed data in the Paimon data lake, while keeping Meta and Hive synchronized to provide efficient and unified data access services for downstream businesses.

8. The data warehouse solution based on paimon real-time data according to claim 7 is characterized by: The step 3 data synchronization and merging includes synchronizing the real-time model table data to Paimon, merging it with the offline data, and then synchronizing the merged data to the Hive table; The step 4 of data consistency and availability verification includes using offline data to perform consistency verification on real-time data in the early morning every day.

9. The data warehouse solution based on paimon real-time data according to claim 7 is characterized by: The specific data flow steps of the data warehouse solution are: S1, obtain click stream data: the system obtains user click stream data in real time through specific data collection tools or interfaces as the basis for subsequent processing; S2, use Flume to clean the data in real time and enter Kafka: The collected raw click stream data is first cleaned in real time by Flume to remove invalid, erroneous or redundant data to ensure data quality; the cleaned data is pushed to the Kafka message queue for subsequent real-time processing; S3, using Kafka SQL for real-time model processing: In Kafka, use Kafka SQL to process the real-time data into models; the processed data forms a real-time model table, which provides a basis for subsequent data storage and query; S4, data is stored in the corresponding data model: the real-time model table data is stored in the Paimon data lake to form a unified data model; Paimon, as a high-performance lake storage, supports real-time updates and efficient queries; S5, supports real-time streaming mode to read data: The system supports reading data from Paimon in real-time streaming mode to meet the needs of downstream businesses for real-time data; S6, data consistency check and overwriting: Every morning, the system uses offline data to check the consistency of real-time data; if the check result meets the standard, the real-time data is overwritten or updated with offline data to ensure data consistency and accuracy.

10. The data warehouse solution based on paimon real-time data according to claim 7 is characterized by: The specific business flow steps of the data warehouse solution are as follows: A1. Obtain data table information and write data logic according to requirements: Business personnel first obtain data table information and write data processing logic according to business requirements; A2, verify whether the data logic is correct: verify the written data logic to ensure its correctness and feasibility; if the verification fails, return to rewrite the data logic until the verification passes; A3, provide real-time and historical data: according to business needs, the system can provide real-time data or historical data of the day; real-time data is read from Paimon through real-time streaming mode, and historical data is obtained by querying Paimon or Hive tables.

Images