Data processing method and device for data warehouse, equipment and medium

Abstract

The invention relates to the technical field of data processing and the field of financial science and technology, and discloses a data processing method for a data warehouse, which comprises the following steps of: acquiring business data from a business database, and synchronizing the business data to an operation data storage layer of a data lake processing framework; a distributed real-time data processing module is used for processing the business data in the operation data storage layer, the processed business data are incrementally written into the operation data storage layer in a transaction control mode, incremental data are read from the operation data storage layer and written into a data warehouse layer of the data lake processing framework, and the data warehouse layer is used for storing the incremental data. The incremental data is further transmitted to the data mart layer. And performing aggregation processing on the incremental data based on business requirements, storing the aggregated data in an aggregation index output library, and finally transmitting the aggregated data to an application platform database for business query and report generation. By adopting incremental data processing and transaction control, the problems of data processing delay and data consistency are solved, and efficient real-time data processing is realized.

Description

Technical Field

[0001] The present invention relates to the fields of data processing technology and financial technology, and in particular to a data processing method, device, equipment and storage medium for a data warehouse. Background Art

[0002] In the financial sector, with the continuous growth of data scale and the rapid changes in business needs, financial institutions have put forward higher requirements for the real-time, accuracy and processing efficiency of data. As a solution for large-scale data storage and processing, data lake technology has been widely used in the financial industry. However, the current batch-based data lake warehouse building solution still has some technical deficiencies that cannot meet actual needs in key business scenarios, especially in terms of incremental data processing, data consistency maintenance and data timeliness.

[0003] Existing data lake warehouse building solutions mainly rely on batch processing mode, which loads and updates data by periodically executing batch tasks, resulting in significant delays in data processing. In financial business, real-time performance is particularly important. For example, scenarios such as financial transaction analysis and risk assessment require rapid processing and response to the latest data. Since the batch processing mode cannot meet these high requirements for data timeliness, financial institutions face decision lags caused by data processing delays.

[0004] Current batch data lake solutions usually require full reading and updating of data, which has low efficiency in processing incremental data. In the financial industry, transaction data, customer behavior data and other data change frequently. Using full replacement to process these incremental data not only increases computing and storage costs, but also leads to reduced processing efficiency and inability to respond to business needs in a timely manner.

[0005] In the existing technology, data consistency is usually achieved by full replacement. Although this method is simple, it may make it difficult to ensure data consistency in the financial industry, especially when dealing with frequently updated financial transaction data. The full replacement method has a greater risk of data loss and inconsistency. Financial business has strict requirements on data accuracy and consistency, and the existing solutions cannot efficiently maintain data consistency, increasing potential risks. Summary of the invention

[0006] The main purpose of the present invention is to provide a data processing method, device, equipment and storage medium for a data warehouse, aiming to solve the technical problem that the existing technology is insufficient in real-time data processing and incremental data processing efficiency, and cannot meet the requirements for timely data processing and consistency.

[0007] To achieve the above object, the present invention provides a data processing method for a data warehouse, comprising:

[0008] Collect business data from the business database and synchronize the business data to the operational data storage layer of the data lake processing framework;

[0009] Processing the business data in the operation data storage layer using a distributed real-time data processing module, and incrementally writing the processed business data into the operation data storage layer in a transaction control manner;

[0010] Read incremental data from the operational data storage layer using a distributed real-time data processing module, and write the incremental data into a data warehouse layer of a data lake processing framework;

[0011] Transmitting the incremental data from the data warehouse layer to the data mart layer within the data lake processing framework;

[0012] Based on business needs, the incremental data is aggregated using a distributed real-time data processing module, and the aggregated data is stored in an aggregate indicator output library of the data lake processing framework;

[0013] The aggregated data is transmitted from the aggregated indicator output library to the application platform database for business query and report generation.

[0014] Furthermore, to achieve the above-mentioned purpose, the present invention provides a data processing device for a data warehouse, comprising:

[0015] A data collection module, used to collect business data from a business database and synchronize the business data to an operational data storage layer of a data lake processing framework;

[0016] A data processing module, used to process the business data in the operation data storage layer using a distributed real-time data processing module, and incrementally write the processed business data into the operation data storage layer in a transaction control manner;

[0017] A data warehouse writing module, configured to read incremental data from the operational data storage layer using a distributed real-time data processing module, and write the incremental data into a data warehouse layer of a data lake processing framework;

[0018] A data transmission module, used to transmit the incremental data from the data warehouse layer to the data mart layer in the data lake processing framework;

[0019] A data aggregation module is used to aggregate the incremental data using a distributed real-time data processing module based on business needs, and store the aggregated data in an aggregate indicator output library of a data lake processing framework;

[0020] The application platform interface module is used to transmit the aggregated data from the aggregated indicator output library to the application platform database for business query and report generation.

[0021] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer device, which includes a memory, a processor, and a data processing program for a data warehouse stored in the memory and executable on the processor, and the data processing program for the data warehouse, when executed by the processor, implements the steps of the data processing method for the data warehouse as described above.

[0022] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium, on which a data processing program for a data warehouse is stored, and when the data processing program for the data warehouse is executed by a processor, the steps of the data processing method for the data warehouse as described above are implemented.

[0023] Beneficial effects: The present invention relates to the fields of data processing technology and financial technology, and discloses a data processing method for a data warehouse, which collects business data from a business database and synchronizes the business data to the operational data storage layer of a data lake processing framework, uses a distributed real-time data processing module to process the business data in the operational data storage layer, and incrementally writes the processed business data to the operational data storage layer in a transaction-controlled manner, and then reads the incremental data from the operational data storage layer and writes it to the data warehouse layer of the data lake processing framework, and the incremental data is further transmitted to the data mart layer. Aggregate the incremental data based on business needs, store the aggregated data in an aggregated indicator output library, and finally transmit it to the application platform database for business query and report generation. The present invention solves the problems of data processing delay and data consistency by adopting incremental data processing and transaction control, realizes efficient real-time data processing, and improves the overall performance of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0025] Figure 1 A schematic diagram of an application environment of a data processing method of a data warehouse in an embodiment of the present invention;

[0026] Figure 2 A schematic diagram of a flow chart of an embodiment of a data processing method for a data warehouse of the present invention;

[0027] Figure 3 A schematic diagram of functional modules of a preferred embodiment of a data processing device for a data warehouse of the present invention;

[0028] Figure 4 A schematic diagram of the structure of a computer device in one embodiment of the present invention;

[0029] Figure 5 FIG. 4 is another schematic diagram of the structure of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0030] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0031] The data processing method of the data warehouse provided by the embodiment of the present invention can be applied in Figure 1 In an application environment, the user end communicates with the server end through a network. The server end can collect business data from the business database through the user end, and synchronize the business data to the operation data storage layer of the data lake processing framework, use a distributed real-time data processing module to process the business data in the operation data storage layer, and the processed business data is incrementally written to the operation data storage layer in a transaction control manner, and then the incremental data is read from the operation data storage layer, written to the data warehouse layer of the data lake processing framework, and the incremental data is further transmitted to the data mart layer. Aggregate the incremental data based on business needs, store the aggregated data in the aggregated indicator output library, and finally transmit it to the application platform database for business query and report generation. The present invention solves the problems of data processing delay and data consistency by adopting incremental data processing and transaction control, realizes efficient real-time data processing, and improves the overall performance of the system. Among them, the user end can be but is not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server end can be implemented with an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments.

[0032] See also Figure 2 , Figure 2 This is a flow chart of an embodiment of a data processing method for a data warehouse provided by the present invention. It should be noted that although a logical sequence is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0033] like Figure 2 As shown, the data processing method of the data warehouse proposed by the present invention includes the following steps:

[0034] S10, collecting business data from the business database, and synchronizing the business data to the operational data storage layer of the data lake processing framework;

[0035] In this embodiment, collecting business data refers to extracting the required data from the core business database of the financial industry. These databases may contain a large amount of transaction data, user information, bills, credit data, etc. The collection process needs to ensure the accuracy and completeness of the data, and the data is usually obtained by configuring the collection module. The collection module can use a variety of technical means, such as data capture based on API interface, incremental data extraction of database triggers, or the use of specialized ETL tools (such as Talend or Flink) to perform batch or real-time data extraction. In the financial field, the process of data collection must ensure the security and compliance of the data, so SSL / TLS encryption protocols are often used to ensure the security of data transmission.

[0036] Data synchronization refers to the transfer and update of data collected from the business database to the operational data storage layer of the data lake to ensure the consistency between the data lake and the business database. This synchronization process needs to be efficient and timely to ensure that business data can enter the data lake in real time for subsequent processing. Data synchronization is usually implemented using distributed data processing engines such as Flink or Kafka. These engines can synchronize data in real time or near real time, and support multiple data synchronization strategies, including incremental synchronization (CDC-Change Data Capture) or full synchronization. The specific implementation method can be through timestamp-based synchronization or database change log-based synchronization to reduce data latency and ensure the real-time performance of the system.

[0037] The operational data storage layer is part of the data lake and is specifically used to receive and store raw data or preliminarily processed data from the business database. This storage layer is usually designed for real-time data writing, supports large-scale data storage, and can provide transaction consistency. In the data lake architecture, the operational data storage layer usually uses object storage (such as HDFS or S3) or a database that supports distributed storage (such as Hudi). This storage layer provides high-throughput writing capabilities and supports partitioned storage of data, making parallel processing of large-scale data more efficient. In the financial field, this layer is mainly used to store data collected from sources such as transactions and user behaviors, and provides a basis for subsequent data analysis and processing.

[0038] Example description: In the financial industry, banks or financial institutions usually need to analyze and monitor massive amounts of transaction data in real time. For example, a credit card transaction system generates a large number of transaction records every day. If these data are not synchronized to the data lake in a timely manner, it may affect the accuracy and timeliness of subsequent fraud detection or risk analysis systems. Through this embodiment, it is possible to collect transaction data in the core system of the bank in real time and synchronize it to the operational data storage layer of the data lake. For example, the real-time stream processing mechanism of Flink can ensure that each transaction is synchronized to the data lake immediately after it occurs, so that transaction data can be analyzed in real time, potential abnormal transactions or risks can be quickly identified, and financial institutions can make decisions in the shortest time and reduce losses.

[0039] By adopting data collection modules and synchronization mechanisms, efficient collection and synchronization of business database data can be achieved. Incremental synchronization solves the data transmission delay and bandwidth occupation problems caused by full synchronization, while ensuring data consistency during the synchronization process. In addition, the synchronization method based on real-time stream processing technology can greatly improve the real-time performance of data, meet the needs of the financial field for transactions, risk monitoring and other scenarios, and realize real-time updates of the operational data storage layer in the data lake.

[0040] S20, using a distributed real-time data processing module to process the business data in the operation data storage layer, and incrementally writing the processed business data into the operation data storage layer in a transaction control manner;

[0041] In this embodiment, the distributed real-time data processing module is used to process the business data in the operational data storage layer, which may include financial transaction data, customer information, etc. During the processing, the distributed module can perform operations such as data cleaning, formatting, filtering, and merging to ensure that the data meets the quality standards required by the business. Distributed data processing frameworks such as Apache Flink and Apache Kafka Streams can be used, which can process large-scale data in real time. Specific processing operations may include:

[0042] Data cleaning: remove invalid data, erroneous data, process null values, etc.

[0043] Formatting: Convert data in different formats into a standard format for subsequent processing and storage.

[0044] Data filtering: Filter out qualified data according to business rules.

[0045] Data merging: combining or correlating data from different sources.

[0046] In the financial field, when processing financial transaction data, data cleaning can remove abnormal data points and ensure the accuracy of the data; the formatting step converts transaction data from different sources into a unified format for subsequent analysis.

[0047] Transaction control is a method used to ensure the consistency, integrity, and reliability of database operations. It ensures that when a set of operations is performed, these operations are treated as an indivisible whole (called a "transaction"). The main purpose of transaction control is to ensure that the system can maintain data consistency and integrity in the face of failures, errors, or concurrent operations. Transactions have the following ACID properties, each of which is used to ensure that transactions are handled correctly during execution:

[0048] Atomicity: Atomicity ensures that all operations in a transaction are either executed successfully or not executed at all. If any error occurs during the execution of a transaction, the system must roll back (undo) all operations that have been executed and restore to the state before the transaction started. This ensures the integrity of the transaction. Example: In a bank transfer operation, the two operations of reducing the amount of the outgoing account and increasing the amount of the incoming account must be treated as a transaction. If one of the operations fails, the entire transfer must be rolled back, and only one of the steps cannot be completed.

[0049] Consistency: Consistency ensures that the database is in a consistent state before and after the transaction is executed. That is, after the transaction is executed, the data should meet all database constraints (such as foreign keys, unique constraints, etc.), and there will be no violation of data rules. Example: In a financial system, the change in the total amount of funds should comply with business logic. If an error occurs during the transfer process (such as a negative balance), the transaction should be rolled back to ensure consistency.

[0050] Isolation: Isolation ensures that when multiple transactions are executed simultaneously, they do not interfere with each other. The intermediate state of each transaction is not visible to other transactions until the transaction is committed. This prevents concurrent operations between transactions from causing data inconsistency. Example: In a stock trading system, two users buy the same stock at the same time. The system needs to ensure that the two transactions are carried out independently, and that no user obtains more stocks than he can buy due to the simultaneous execution.

[0051] Durability: Durability ensures that once a transaction is committed, the changes made will be permanently saved, and the data will not be lost even if the system crashes or fails. Example: If a bank system fails after successfully completing a transfer operation, the submitted transaction will still be persistently stored and will not be lost.

[0052] Transaction control ensures the consistency and integrity of data during processing and writing. Through transaction control, it can ensure that data processing operations can be rolled back in the event of a failure, avoiding data inconsistency caused by partial writing. The implementation of transaction control depends on the support of distributed transaction managers or frameworks. Apache Flink and Apache Kafka support transactional data processing.

[0053] The implementation of transaction control usually relies on the database system or distributed processing framework. The specific implementation steps include:

[0054] Start a transaction: Before executing a set of related operations, the system will start a transaction to ensure that subsequent operations are included in the same transaction.

[0055] Execute operations: Each operation in a transaction is performed in a set order, such as reading, writing, and modifying data.

[0056] Commit the transaction: If all operations are executed successfully, the system commits the transaction and saves all changes permanently.

[0057] Rollback transaction: If an operation in a transaction fails, the system will roll back the executed operations and restore to the state before the transaction started.

[0058] Incremental writing means writing only new or changed data to the operational data storage layer of the data lake, without having to rewrite all the data every time. Incremental writing improves writing efficiency, especially when processing large-scale data, it can avoid repeatedly writing the same data and reduce the consumption of storage and computing resources. Incremental writing can be achieved by monitoring data changes (such as based on timestamps or version numbers). Common implementation methods include using Change Data Capture (CDC) technology to capture changes in the business database and write these incremental data to the data lake. Incremental writing not only increases the writing speed, but also ensures the real-time nature of the data. In financial data processing, incremental writing helps to update customer transaction data and credit scores and other information in real time to ensure the timeliness of the data.

[0059] For example, using a distributed real-time data processing module to process daily bank transaction data, the data cleaning and formatting steps can eliminate abnormal transactions and duplicate data. After processing, transaction control is used to ensure the consistency and integrity of each transaction data to prevent data loss or duplicate writing during write operations. Finally, the processed transaction data is incrementally written to the operational data storage layer of the data lake, and only the changed transaction records are updated, which reduces a large number of unnecessary data processing processes and provides efficient real-time data update support for subsequent analysis.

[0060] By using a distributed real-time data processing module to process business data in the operational data storage layer, combined with transaction control and incremental writing, the performance bottleneck problem caused by full data processing in the existing technology is solved. Transaction control ensures consistency and security during data processing, and incremental writing improves the processing efficiency of large-scale data and reduces storage costs.

[0061] S30, using a distributed real-time data processing module to read incremental data from the operational data storage layer, and write the incremental data into a data warehouse layer of a data lake processing framework;

[0062] In this embodiment, incremental data is extracted from the operational data storage layer through a distributed real-time data processing module. Incremental data refers to part of the data that has changed since the last synchronization or processing. The goal of reading incremental data is to reduce the occupation of storage and computing resources and ensure the timeliness of data processing. Incremental data reading is usually based on timestamps, version numbers or through Change Data Capture (CDC) technology. Distributed real-time data processing frameworks such as Apache Flink, Kafka Streams, etc. can continuously monitor changes in data sources and trigger processing flows when new data or updates are detected. By configuring a watermark mechanism or event time, it can be ensured that only new data since the last processing is read, thereby avoiding reading duplicate data. In financial scenarios, this process can be used to update transaction data, market data, etc. in real time to ensure the timeliness of analytical data.

[0063] The data warehouse layer is a core component of the data lake architecture, which is dedicated to storing processed data for subsequent analysis, query, and report generation. The process of writing incremental data to the data warehouse layer needs to ensure the integrity and consistency of the data, and transactional writes are usually used to ensure the reliability of the write operation. Incremental data writing can be done in batches or real-time streams. In a distributed environment, frameworks such as Flink or Kafka can process incremental data in parallel and write it to the data warehouse layer efficiently through partitioned storage technology. The data warehouse layer is usually based on columnar storage formats (such as Parquet, ORC) and supports efficient query performance. When writing, data compression and partitioning strategies can also be applied to improve storage efficiency and speed up subsequent queries.

[0064] By using a distributed real-time data processing module to read incremental data from the operational data storage layer and efficiently write incremental data to the data warehouse layer, the performance bottleneck caused by full data reading and writing in traditional data processing methods is solved. Incremental processing greatly reduces system resource usage, ensures the timeliness and real-time performance of data processing, and is particularly suitable for data processing scenarios that require rapid response. At the same time, based on the use of transactional writing methods, consistency and reliability of the data writing process are guaranteed.

[0065] S40, transmitting the incremental data from the data warehouse layer to the data mart layer in the data lake processing framework;

[0066] In this embodiment, the data warehouse layer stores processed incremental data, which needs to be further extracted and transmitted to the data mart layer for business use. Incremental data refers to data that has been added or updated since the last transmission, and its extraction process needs to ensure the consistency and integrity of the data. Data extraction is usually performed regularly or in real time through a scheduling mechanism. It can be determined which data is incremental data based on timestamps, markers, or Change Data Capture (CDC) technology. During the implementation process, the distributed data processing module scans the latest data in the data warehouse layer, avoids full data transmission by reading incremental data, and improves processing efficiency. In the financial field, this method can be used to extract the latest transaction data, market prices and other key information for subsequent analysis.

[0067] The DataMart Layer is a data storage layer for specific business needs. It provides customized data sets for specific business functions or departments. The DataMart Layer usually stores cleaned, aggregated, and formatted data for business systems to quickly query and analyze. Transferring incremental data to the DataMart Layer ensures that these business applications have the latest data and supports real-time analysis and decision-making. Incremental data transmission is usually completed in streaming or batch transmission. Streaming transfers incremental data from the data warehouse layer to the DataMart Layer through real-time data pipelines (such as Kafka or Flink). Batch transmission is performed under regular scheduling and is suitable for scenarios with low requirements for data timeliness. In the financial industry, the DataMart Layer may store daily transaction data, customer behavior data, or market analysis data for real-time or quasi-real-time queries by risk management or decision support systems.

[0068] For example, in a banking system, credit card transaction data is updated every day. The distributed real-time data processing module extracts the newly added credit card transaction data of the day from the data warehouse layer and transmits it to the data mart layer where the risk management system is located. The risk management department can quickly query the latest transaction status and track and handle suspicious transactions in a timely manner. This incremental data transmission method not only improves data transmission efficiency, but also supports real-time data analysis and improves the system's responsiveness.

[0069] By extracting incremental data from the data warehouse layer and efficiently transmitting it to the data mart layer, it ensures that the business system and data analysis platform can obtain the latest business data in a timely manner. Compared with full data transmission, incremental transmission greatly reduces the consumption of network and computing resources, improves data transmission efficiency, and meets the needs of real-time or quasi-real-time data analysis. The design of the data mart layer ensures that different business departments can quickly access customized data sets, supporting more accurate decision-making and data-driven business operations.

[0070] S50, based on business needs, using a distributed real-time data processing module to aggregate the incremental data, and storing the aggregated data in an aggregate indicator output library of a data lake processing framework;

[0071] In this embodiment, data aggregation refers to integrating, summarizing or classifying incremental data according to business needs for subsequent analysis. Business needs can be a summary of specific indicators, such as aggregation by time period, region, product type and other dimensions. Business needs in the financial field usually involve statistics on transaction data, user behavior analysis, etc. The system must first identify business needs and clarify the dimensions and calculation methods of aggregation. For example, in a financial scenario, a bank may need to summarize and count transaction data by month or product category to calculate the total transaction amount, number of transactions, etc. The data processing module will configure the corresponding aggregation strategy based on these needs, usually through SQL queries or the aggregation function of the distributed computing framework (such as Flink's window aggregation).

[0072] The distributed real-time data processing module is responsible for efficient aggregation operations on incremental data and supports parallel processing of massive data. The module can group data by different dimensions based on business needs, and then perform aggregation calculations in each group, such as sum, average, maximum, minimum, etc. Distributed real-time data processing modules (such as Apache Flink or Kafka Streams) can process constantly changing real-time data streams and perform aggregation operations based on predefined windows (such as time windows, event windows). Aggregation processing usually defines aggregation strategies and specifies the required grouping methods and aggregation functions. In financial scenarios, banks can summarize daily transaction amounts and customer transaction times in real time for statistics and analysis. The processing module can calculate data in different groups in parallel, greatly improving data processing efficiency.

[0073] The aggregated data is stored in the aggregate indicator output library for subsequent query and analysis. The aggregate indicator output library is a storage system specifically used to store aggregated summary data, supporting efficient reading and further business processing. Aggregate data can be stored in the aggregate indicator output library in the form of columnar storage (such as Parquet, ORC) or relational database. Usually, aggregated data is stored by partition with predefined indexes for subsequent fast query. In the financial system, banks can store daily transaction summary data in the aggregate indicator output library for business systems to generate reports or conduct further data analysis.

[0074] For example, banks can aggregate statistics by product category based on daily transaction volume. Through the real-time processing module, the system can classify and aggregate all transaction data of the day according to business needs every night, and store these aggregated data in the aggregate indicator output library for generating daily reports the next day or for management to view overall business trends. This can ensure the efficiency of data processing and meet the business department's needs for data timeliness.

[0075] By using a distributed real-time data processing module to aggregate incremental data and storing the aggregated data in the aggregate indicator output library, the efficiency of large-scale data processing can be effectively improved, especially in real-time data processing and big data scenarios. The system can flexibly define aggregation dimensions and strategies based on business needs to ensure that aggregated data is updated in a timely manner and can be quickly queried and used.

[0076] S60, transmitting the aggregated data from the aggregated indicator output library to the application platform database for business query and report generation.

[0077] In this embodiment, the aggregated data is stored in the aggregated indicator output library, which is usually the summary data generated according to business needs after aggregation processing. These data need to be further extracted and transmitted to the application platform database for the business system to query and generate reports. The goal of data extraction is to deliver the pre-processed summary data to the business system quickly and accurately to ensure that the downstream system can make business decisions and operations based on the latest data. The system extracts aggregated data from the aggregated indicator output library through regular scheduling or real-time triggering. This process can be completed through SQL queries, batch reading or streaming processing. Aggregate data in the financial industry usually involves transaction summaries, customer behavior statistics, etc. The efficiency and accuracy of data extraction are very critical.

[0078] The application platform database refers to the data storage layer used to store and support the operation of business application systems. As a data source directly accessed by the business system, it usually stores processed or aggregated data for business departments to query, analyze, generate reports or make real-time decisions.

[0079] The data in the application platform database has been cleaned, converted, aggregated, and processed to meet specific business needs. It is usually provided to users, management systems, or business departments for data query and analysis. For example, an application platform database in the financial field may store daily transaction records, customer information, financial statements, etc. for use by business departments.

[0080] The data in the application platform database usually needs to have high real-time and timeliness to ensure that the business system can obtain the latest data in a timely manner. For example, in the financial industry, the system may need to generate reports or conduct risk assessments based on the latest transaction data.

[0081] The application platform database provides query functions for business users, supporting multi-dimensional data query and report generation tools. For example, users can query transaction data by time, region, product category and other dimensions, and generate reports for analysis.

[0082] Application platform databases usually use optimized storage structures to improve query performance. Common technologies include index optimization, column storage, partitioning, etc., to speed up query speed under large data volumes and improve system responsiveness.

[0083] The application platform database supports simultaneous access by multiple business systems or users, allowing business departments to perform various operations through the system interface, such as query, report generation, statistical analysis, etc.

[0084] The application platform database is a data storage layer directly accessed by the business system, usually used to support query, report generation, and the operation of business applications. Transferring aggregated data to the database enables downstream business systems to quickly access these aggregated data to meet the needs of real-time query and report generation. The data transmission process is usually completed through batch or real-time data pipelines, specifically using ETL tools (such as Apache NiFi, Airflow) or stream processing frameworks (such as Apache Flink, Kafka Streams). During transmission, it is necessary to ensure the consistency and integrity of the data, especially in the financial field, the transmission process must avoid data loss or duplication. After the transmission is completed, the data in the application platform database will be available for query by the business department, generating various business reports or providing data support for the decision-making system.

[0085] The aggregated data transferred to the application platform database will be used by the business system as the basic data source for querying and generating reports. Different business departments can query this data according to their specific needs, such as querying summary results by time period, region or product type for decision support or business analysis. Query and report generation are usually performed by BI tools at the application layer (such as Tableau, PowerBI) or internal customized reporting systems. System users can obtain the latest aggregated data from the application platform database through predefined query statements or interactive query interfaces, generate reports and perform analysis. In the financial field, these reports may involve key business data such as customer behavior analysis, transaction aggregation, and credit risk assessment.

[0086] For example, in a credit card transaction system, banks can regularly extract summary data of daily transactions from the aggregate indicator output library and transmit this data to the application platform database. The risk management department can query the customer's transaction history in real time based on this data, generate relevant reports, and assess the customer's credit risk. This data transmission method ensures the timeliness and accuracy of the data, enabling banks to make timely decisions and prevent potential risks.

[0087] By transferring aggregated data from the aggregated indicator output library to the application platform database, the business system can obtain the latest aggregated data support for business query and report generation. Data transmission adopts an efficient and reliable method to meet the requirements of different business scenarios for data timeliness and integrity. In addition, based on the design of the application platform database, it supports multiple users to query and generate reports at the same time, which improves the efficiency and responsiveness of data analysis.

[0088] The present invention relates to the fields of data processing technology and financial technology, and discloses a data processing method for a data warehouse, which collects business data from a business database and synchronizes the business data to the operational data storage layer of a data lake processing framework, uses a distributed real-time data processing module to process the business data in the operational data storage layer, and incrementally writes the processed business data to the operational data storage layer in a transaction control manner, and then reads the incremental data from the operational data storage layer and writes it to the data warehouse layer of the data lake processing framework, and the incremental data is further transmitted to the data mart layer. Aggregate the incremental data based on business needs, store the aggregated data in an aggregated indicator output library, and finally transmit it to the application platform database for business query and report generation. The present invention solves the problems of data processing delay and data consistency by adopting incremental data processing and transaction control, realizes efficient real-time data processing, and improves the overall performance of the system.

[0089] In one embodiment, after the above S40, the method further includes:

[0090] S701, obtaining a query request, where the query request includes a query condition, an analysis dimension and / or an aggregation method;

[0091] S702, based on the query condition, determining target incremental data matching the query condition from the data mart layer;

[0092] S703, grouping the target incremental data based on the analysis dimension, and performing aggregation processing on each group of data according to the aggregation method to generate an aggregated query result;

[0093] S704, transmitting the query result to the application platform database for business query, report generation or business analysis.

[0094] In this embodiment, the query request is issued by the user or the business system, and contains specific query conditions, analysis dimensions (such as time, region, product category) and aggregation methods (such as sum, average, maximum value, etc.). These requests determine which data to obtain from the data mart layer and how to process the data. The system accepts query requests through an API, SQL query interface, or front-end user interface. Business system users can filter specific incremental data by specifying conditions. For example, financial users can query transaction records within a certain time period, credit card consumption data classified by region, etc. The dimensions and aggregation methods of the query request will be used in subsequent data processing steps.

[0095] The system selects the target incremental data that meets the conditions from the data mart layer according to the conditions in the query request. The data mart layer stores the incremental data that has been preliminarily cleaned and processed, has good query performance, and supports efficient multi-dimensional data retrieval. The system executes the query statement to extract the data that meets the conditions from the data mart layer. The data mart layer often uses indexing, partitioning and other technologies to speed up the query. The query results may include transaction data filtered by time or customer behavior data filtered by region. In the financial system, users can query all transaction data in any time period or specific region on demand.

[0096] After data filtering, the system groups the target data according to the specified analysis dimensions, and uses aggregation to summarize, sum, average, and other operations on each group of data. Analysis dimensions are specific dimensions that the business is concerned about, such as time, region, product category, etc. Use distributed data processing frameworks (such as Apache Flink, Spark) to group and process the filtered incremental data by dimension. For example, a bank can group transaction data by month, region, or product type, and then summarize each group to calculate the total amount, average transaction amount, etc. Through aggregation operations, a large amount of scattered data can be summarized into analysis results with business value.

[0097] The aggregated query results are stored in the application platform database for subsequent business system queries or report generation. The application platform database provides fast access and multi-dimensional query capabilities, supports report generation and further business analysis. The system transfers the aggregated results to the application platform database, usually through a SQL interface or API interface for data transmission. The business system can generate reports and perform analysis through the front-end interface or customized reporting tools (such as Tableau, PowerBI). For example, in a financial system, aggregated transaction data can be used to generate customer behavior reports, regional consumption trend reports, etc. to support business decisions.

[0098] For example, banks can extract the consumption data of the past month from customers' transaction records every day, group them by region and time, calculate the total consumption amount in each region, and generate a consumption behavior report. In this way, banks can understand customers' consumption habits in real time and adjust product strategies or identify potential risks in a timely manner.

[0099] In this embodiment, through query and aggregation processing based on business needs, the system can efficiently extract and process incremental data from the data mart layer. Through flexible setting of dimension grouping and aggregation methods, users can quickly obtain multi-dimensional aggregation analysis results to support business query, report generation and decision analysis.

[0100] In one embodiment, in the above S50, after storing the aggregated data in the aggregated indicator output library of the data lake processing framework, the following step is further included:

[0101] S801, extracting historical data from the data warehouse layer by executing offline scheduling tasks through scheduled tasks;

[0102] S802, transmitting the historical data to the application platform database for business query or report generation;

[0103] S803, using a distributed real-time data processing module to process the real-time data from the operation data storage layer, and perform cleaning and conversion operations on the real-time data;

[0104] S804, updating the processed real-time data to the application platform database in an incremental manner, and storing the data according to the same table structure and field type as the historical data.

[0105] In this embodiment, the scheduled task refers to the execution of data extraction and processing operations according to a predetermined time interval. Offline scheduling tasks are used to periodically extract historical data from the data warehouse layer to ensure that the business system can access complete historical data records. The system regularly executes data extraction tasks through a scheduler (such as Apache Airflow or Crontab) to extract historical data from the data warehouse layer. Historical data includes stored data records, such as past transactions, customer behavior and other information. Offline scheduling tasks can be set to execute daily, weekly or other periodic time intervals according to business needs, and are suitable for report generation or historical data analysis scenarios. In the financial industry, historical data typically includes past transaction records, capital flows and other information, which are used to generate business reports.

[0106] The extracted historical data is transferred to the application platform database so that the business system can query the historical data through the application platform database, generate business reports or conduct business analysis. The data transmission process can be implemented through ETL tools (such as Apache NiFi or Talend). The extracted historical data is converted and stored in the application platform database for downstream business systems to query and analyze. In the financial system, historical data transmission can be used to generate reports such as monthly transaction summaries and customer behavior analysis to help business departments analyze and make decisions.

[0107] The distributed real-time data processing module is responsible for processing incremental data that arrives in real time. The processing steps usually include data cleaning (removing noise, erroneous data), conversion (format standardization, etc.) and other operations to ensure data quality. Distributed data processing frameworks (such as Apache Flink or Kafka Streams) can monitor changes in the operational data storage layer in real time and process newly arrived incremental data. The system automatically performs cleaning steps, such as removing data that does not comply with the rules, or converting data into a format that meets system requirements. In the financial industry, real-time data processing includes cleaning and standardization of real-time transaction data, such as removing duplicate transaction records or formatting customer information.

[0108] Incremental update means adding or updating only newly arrived or changed real-time data to the application platform database without overwriting existing data. Incremental update can improve data writing efficiency and keep the latest data status in the database. The processed incremental data is written to the application platform database through the streaming processing framework or batch update operation. The data update process should maintain data consistency and integrity to ensure that newly arrived real-time data can be seamlessly merged into existing historical data. In the financial field, incremental update is applicable to daily new transaction data or customer operation records to ensure that the business system can access the latest business data in real time.

[0109] The storage format and structure of real-time data must be consistent with historical data to ensure that data can be processed uniformly during query, avoiding query errors or data analysis obstacles caused by inconsistent formats or field types. During the incremental update process, the system ensures that the table structure and field types of the data are consistent with historical data. Through the standardization of the data table structure, all newly written incremental data can be stored according to the predefined field type and structure. This process ensures that the business system will not cause errors due to inconsistent data when querying data. In the financial system, ensuring that the storage structure of historical transaction records and real-time transaction data is consistent can ensure data consistency and facilitate subsequent unified analysis and report generation.

[0110] In this embodiment, the system can extract historical data from the data warehouse layer regularly through scheduled tasks to ensure that the business system can access complete historical data. At the same time, the incremental update mechanism of real-time data ensures that the application platform database always contains the latest business data.

[0111] In one embodiment, the above S10 includes:

[0112] S101, configuring a data acquisition module, and extracting business data from a business database through the data acquisition module;

[0113] S102, transmitting the business data to the operation data storage layer through a data synchronization mechanism as initially stored business data.

[0114] In this embodiment, the data acquisition module is a component responsible for extracting business data from the business database. The business database is a database that stores daily business operations and transaction data, and the data acquisition module needs to extract these business data according to the set rules and conditions to ensure that the latest business activity data is captured in a timely manner. The data acquisition module can extract data from the business database through regular scheduling, real-time monitoring, or API calls. The data extraction process can be carried out by querying database tables, calling API interfaces, or monitoring database change logs. The business database in the financial system may store customer transaction records, credit card usage, etc. By configuring the data acquisition module, daily transaction data can be automatically extracted to ensure that the information in the data lake is synchronized with the actual business data.

[0115] The data synchronization mechanism is used to ensure that the extracted business data can be quickly and accurately transferred to the operational data storage layer of the data lake as the basis for further processing and analysis. The data synchronization mechanism includes two modes: incremental synchronization and full synchronization, depending on the size of the data and business needs. The data synchronization mechanism usually relies on data stream processing technology or batch transmission tools. Common synchronization mechanisms include:

[0116] Incremental synchronization: Through change data capture (CDC) technology, only the data that has changed since the last synchronization is transferred. For example, only the newly added customer transaction data is extracted every day.

[0117] Full synchronization: extract all data from the business database regularly and completely overwrite the existing data storage. This method is suitable for scenarios with less data changes.

[0118] In the financial field, the data synchronization mechanism can use stream processing frameworks such as Kafka and Flink, or batch transmission tools such as Apache NiFi to transfer the latest transaction records, account changes and other data to the operational data storage layer of the data lake.

[0119] After the business data is transferred to the operational data storage layer of the data lake processing framework, it becomes the initial stored data, which will be processed, analyzed, and summarized in subsequent steps. The operational data storage layer is mainly used to receive the original data synchronized from the business database. The operational data storage layer is usually designed as a scalable distributed storage system that can receive and store large-scale data sets. When the data enters the operational data storage layer, it has not undergone any complex processing and is the basic data for subsequent cleaning, transformation, and aggregation. The initially stored business data may include unprocessed customer transactions, business operation records, etc., waiting for further operations in the subsequent data processing process.

[0120] This embodiment configures the data acquisition module and uses the data synchronization mechanism to transfer the data in the business database to the operational data storage layer of the data lake processing framework, which can ensure the real-time and integrity of the data and meet the needs of large-scale data processing. As a unified data storage platform, the data lake can store business data from multiple sources and provide a stable foundation for subsequent data processing.

[0121] In one embodiment, the above S20 includes:

[0122] S201, performing a preprocessing operation on the business data in the operation data storage layer, wherein the preprocessing operation includes data cleaning and formatting;

[0123] S202, using a distributed real-time data processing module to perform transaction control on the business data after the pre-processing operation;

[0124] S203, incrementally updating the business data after transaction control to the operation data storage layer, overwriting or updating the initially stored business data.

[0125] In this embodiment, data preprocessing refers to cleaning and formatting the data extracted from the business database. Data cleaning includes deleting erroneous data, duplicate data, noise data, etc. to improve data quality. Formatting refers to standardizing the data to ensure data structure and field consistency for subsequent processing and storage. Data cleaning and formatting operations are usually performed by distributed real-time data processing modules (such as Apache Flink or Kafka Streams). The cleaning step may include deleting blank fields, records that do not match the format, or correcting errors in the data. For example, transaction data in a financial system may need to remove duplicate transaction records or correct incorrect transaction amounts. The formatting step unifies the field format of the data according to the system standards, such as unifying the date format into the form of "YYYY-MM-DD". In this process, distributed processing can speed up the preprocessing efficiency of large-scale data.

[0126] Transaction control refers to ensuring consistency and atomicity during data processing and writing. Transaction control ensures that when processing and writing data, even if the system fails or is interrupted, the data will not be partially written or inconsistent. Transaction control can ensure that each write operation is either completely successful or completely failed to avoid incomplete data. When processing large amounts of data, the distributed real-time data processing module manages data consistency through a distributed transaction mechanism. After each batch of data is processed, data consistency is ensured through commit or rollback operations. Distributed transaction control is usually implemented in combination with locking mechanisms and logging (such as WAL, write-ahead log) to ensure that when business data is written to the data lake processing framework, data consistency is guaranteed. For example, when a banking system processes transaction data, transaction control can ensure that the same transaction data will not be partially written or lost, maintaining the accuracy of the transaction.

[0127] Incremental update means writing only newly arrived or changed data to the operational data storage layer without rewriting all the data. Incremental update can improve write efficiency while ensuring that the data in the operational data storage layer is always up to date. Overwriting or updating the initially stored data means that during the update process, the old data needs to be overwritten based on the timestamp or version number of the data. After completing transaction control, the distributed real-time data processing module updates the processed business data to the operational data storage layer in an incremental manner. The system determines whether the existing data needs to be overwritten based on the unique identifier of each piece of data (such as the primary key, timestamp or version number). For example, if a customer's transaction record already exists in the system, the incremental update will only update its newly added transaction data without affecting the stored records. This process accelerates write operations in a distributed system through data sharding and indexing mechanisms to ensure the efficiency and consistency of the data storage layer.

[0128] This embodiment uses a distributed real-time data processing module to process business data, control transactions, and perform incremental updates, which can ensure consistency and real-time performance during data processing and storage. The data preprocessing stage improves data quality and consistency, transaction control ensures data atomicity and integrity, and the incremental update mechanism greatly improves data writing efficiency and reduces system resource usage.

[0129] In one embodiment, the above S30 includes:

[0130] S301, setting and configuring a distributed real-time data processing module, including setting trigger conditions and frequency for data reading;

[0131] S302, setting a watermark mechanism or a timestamp-based filtering strategy to identify and read incremental data added since the last synchronization;

[0132] S303, reading incremental data from the operational data storage layer using a distributed real-time data processing module based on the filtering strategy;

[0133] S304, performing data consistency verification on the read incremental data, wherein the data consistency verification includes verification of integrity, correctness and time sequence;

[0134] S305, writing the verified incremental data into the data warehouse layer of the data lake processing framework according to a predefined mode;

[0135] S306, updating the metadata information of the data warehouse layer, wherein the metadata information includes the writing time of the incremental data, the data version number, the data source information and the data processing history record.

[0136] In this embodiment, the distributed real-time data processing module is the key component responsible for extracting incremental data from the operational data storage layer and writing it to the data warehouse layer. The setting of trigger conditions and frequency determines when and how to start the data reading process. Trigger conditions can be based on time (such as scheduled tasks), events (such as data changes), etc. Configure a distributed data processing module (such as Apache Flink or Kafka Streams) in the system, set the frequency of data reading (such as once a minute) and trigger conditions (such as triggering when the data storage layer changes). In this way, the system can automatically and continuously read the newly added business data from the operational data storage layer to ensure the real-time nature of the data flow. For example, in a banking system, the trigger condition can be the update of a customer's transaction record, and the system will automatically start data reading according to the frequency of the transaction.

[0137] The watermark mechanism or timestamp-based filtering strategy is used to identify which data is incremental data added since the last synchronization, ensuring that only new data is read each time to avoid repeated reading. The watermark mechanism can mark the time order of data to ensure the processing order of the data stream. The system can filter based on the timestamp or event identifier of the data to ensure accurate reading of incremental data. For example, use the watermark mechanism to mark the boundaries of the data stream to process delayed data; or use timestamp filtering to only read new data generated since the last synchronization. In financial systems, the watermark mechanism can be used to ensure that delayed transaction records are processed in the correct time order, while timestamp filtering can help the system read only the new transaction records of the day.

[0138] The system uses the distributed data processing module to read incremental data from the operational data storage layer according to the configured filtering strategy. This step ensures that the system can continuously obtain incremental data and keep the information in the data lake synchronized with the data in the business system. The data processing module performs data reading operations and filters out new data generated since the last synchronization according to the set filtering strategy. For example, the system extracts new data from the operational data storage layer every minute through an event trigger mechanism. Incremental data in financial systems usually includes newly occurring transaction records or updated customer account information.

[0139] Data consistency verification ensures that the incremental data read is not erroneous or lost during transmission and processing. Verification includes checking the integrity of the data (whether the data is missing), correctness (whether the data conforms to the expected format), and time sequence (whether the data is correctly sorted by time). After reading the data, the system verifies the consistency of the data through the built-in verification mechanism. Data consistency verification in financial systems usually includes ensuring that the transaction amount is correct and the timestamp of the record is accurate. For example, the system checks whether each transaction contains the necessary fields (such as transaction amount, customer ID), and ensures that the transaction is processed in the correct time sequence to avoid data confusion or loss.

[0140] The incremental data that has been verified for consistency will be written to the data warehouse layer according to the predefined data storage mode (such as partition table, index, etc.) to facilitate subsequent query and analysis operations. The system writes the incremental data to the appropriate partition table or index according to the predefined storage mode of the data warehouse layer. This mode can be configured according to business needs, such as partitioning and storing data by dimensions such as time and customer ID. In the financial system, the bank's transaction records may be partitioned and stored by date to facilitate quick query and analysis of the daily transaction total.

[0141] After each data is written, the system will update the metadata information of the data warehouse layer to record the write time, data version number, data source information and processing history of the incremental data. Metadata records the source and change history of incremental data, which is convenient for subsequent data management and tracking. After the data is written to the data warehouse layer, the system will update the metadata table to record the detailed information of the write operation, including the write time, version number and source of the data. This helps to track the changes and processing history of the data and ensure the consistency and traceability of the data in subsequent operations. For example, in the financial system, the write time, source system (such as POS system or ATM) and other information of each transaction record will be recorded in the metadata table to ensure that the system can track the complete processing path of the transaction.

[0142] By using a distributed real-time data processing module, this embodiment enables the system to efficiently read incremental data from the operational data storage layer and ensure that only new data since the last synchronization is processed according to the set filtering policy. Through data consistency verification, the system can ensure the accuracy and integrity of the data. Ultimately, the verified data is reliably written to the data warehouse layer of the data lake processing framework, and the processing history of the data is managed through metadata records to ensure the traceability and consistency of the data throughout the storage and processing process.

[0143] In one embodiment, the above S50 includes:

[0144] S501, based on business requirements, identifying key business indicators in the incremental data, and defining aggregation dimensions based on the key business indicators, where the aggregation dimensions include a time dimension, a region dimension, or a product category dimension;

[0145] S502, setting a data aggregation strategy based on the aggregation dimension, including setting a grouping method and an aggregation time window;

[0146] S503, performing aggregation processing on the incremental data according to the aggregation strategy through a distributed real-time data processing module, and generating an aggregated data result according to the grouping method;

[0147] S504: Store the aggregated data results in an aggregated indicator output library of a data lake processing framework according to a predefined storage mode and index.

[0148] In this embodiment, the system analyzes incremental data according to business needs and identifies key business indicators, such as sales, number of customers, transaction volume, etc. Aggregation dimensions are used to define how to classify and summarize data, such as different dimensions such as time, region, product category, etc., for detailed analysis and report generation. Configure the system according to the business scenario and identify key business indicators. For example, a financial institution may need to analyze transaction volume by region or customer growth by time. In this case, the aggregation dimensions may include the two dimensions of region and time. In the incremental data, the system will identify the data fields that need to be aggregated based on these key indicators.

[0149] The aggregation strategy determines how to group and process the data. The grouping method defines how to classify the data according to the aggregation dimension, and the aggregation time window determines the time range of data processing (such as daily, hourly, or minute-by-minute aggregation). The system sets the aggregation strategy based on business needs, such as grouping customer data by region, or grouping sales data by month. The time window can be set to accumulate data within a certain period of time. For real-time data processing, strategies such as sliding windows and session windows may be used to continuously roll and aggregate incremental data. For example, a financial system can aggregate transaction data every hour and group the data by region to ensure that banks can analyze transactions in different regions.

[0150] The distributed real-time data processing module aggregates and calculates the incremental data according to the set aggregation dimensions and aggregation strategies, and generates aggregated data results. Each group of data will be processed according to the set grouping method to generate aggregated business indicators. The system performs aggregation processing through distributed real-time data processing modules (such as Flink or Kafka Streams). The system classifies the data according to the defined grouping method and performs aggregation operations such as summation, average, and count within each group. For example, a banking system can group transaction data by region and time, and count information such as the total transaction amount and number of transactions in each region. The distributed processing architecture makes real-time aggregation of large-scale data possible.

[0151] After the aggregation is completed, the system stores the aggregation results in the aggregate indicator output library according to the predefined storage mode (such as partition storage, index storage, etc.). This repository is specifically used to store data after aggregation processing to facilitate subsequent business queries and analysis. After the aggregated data results are generated, they will be written to the aggregate indicator output library according to the predefined storage mode. The storage mode may be designed as partitioned storage or a storage structure with indexes based on business needs to improve query performance. The system may store data of different dimensions (such as by time, by region) in different partitions to facilitate fast retrieval during query. In the financial system, the aggregated data results may include transaction summary data for each region, which are stored in the aggregate indicator output library of the data lake for subsequent report generation and decision analysis.

[0152] This embodiment uses a distributed real-time data processing module to aggregate incremental data, so that the system can effectively group, summarize and analyze large-scale business data. The aggregation strategy can flexibly adapt to different business needs and support multi-dimensional data analysis and report generation. The aggregated data results are stored in the aggregated indicator output library, which is convenient for subsequent business systems to query and analyze, thereby greatly improving the efficiency of data processing and the accuracy of business decisions.

[0153] In one embodiment, a data processing device for a data warehouse is provided, and the data processing device for the data warehouse corresponds one-to-one to the data processing method for the data warehouse in the above embodiment. Figure 3 , Figure 3 This is a functional module diagram of a preferred embodiment of a data processing device for a data warehouse of the present invention. A data acquisition module 10, a data processing module 20, a data warehouse writing module 30, a data transmission module 40, a data aggregation module 50 and an application platform interface module 60. Each functional module is described in detail as follows:

[0154] A data collection module 10 is used to collect business data from a business database and synchronize the business data to an operational data storage layer of a data lake processing framework;

[0155] The data processing module 20 is used to process the business data in the operation data storage layer using a distributed real-time data processing module, and incrementally write the processed business data into the operation data storage layer in a transaction control manner;

[0156] A data warehouse writing module 30, configured to read incremental data from the operational data storage layer using a distributed real-time data processing module, and write the incremental data into a data warehouse layer of a data lake processing framework;

[0157] A data transmission module 40, configured to transmit the incremental data from the data warehouse layer to the data mart layer within the data lake processing framework;

[0158] A data aggregation module 50 is used to aggregate the incremental data using a distributed real-time data processing module based on business needs, and store the aggregated data in an aggregate indicator output library of a data lake processing framework;

[0159] The application platform interface module 60 is used to transmit the aggregated data from the aggregated indicator output library to the application platform database for business query and report generation.

[0160] In one embodiment, the data transmission module 40 is specifically configured to:

[0161] Obtaining a query request, wherein the query request includes a query condition, an analysis dimension and / or an aggregation method;

[0162] Based on the query condition, determining target incremental data matching the query condition from the data mart layer;

[0163] Based on the analysis dimension, the target incremental data is grouped, and each group of data is aggregated according to the aggregation method to generate an aggregated query result;

[0164] The query results are transmitted to the application platform database for business query, report generation or business analysis.

[0165] In one embodiment, the data aggregation module 50 is specifically configured to:

[0166] Execute offline scheduling tasks through scheduled tasks to extract historical data from the data warehouse layer;

[0167] Transmitting the historical data to the application platform database for business query or report generation;

[0168] Using a distributed real-time data processing module to process the real-time data from the operational data storage layer, and to perform cleaning and conversion operations on the real-time data;

[0169] The processed real-time data is updated incrementally to the application platform database and stored according to the same table structure and field type as the historical data.

[0170] In one embodiment, the data acquisition module 10 is specifically used for:

[0171] Configure a data acquisition module, and extract business data from a business database through the data acquisition module;

[0172] The business data is transmitted to the operation data storage layer through a data synchronization mechanism as the initially stored business data.

[0173] In one embodiment, the data processing module 20 is specifically configured to:

[0174] Performing a preprocessing operation on the business data in the operation data storage layer, wherein the preprocessing operation includes data cleaning and formatting;

[0175] Use distributed real-time data processing modules to perform transaction control on business data after pre-processing operations;

[0176] The business data after transaction control is incrementally updated to the operational data storage layer, overwriting or updating the initially stored business data.

[0177] In one embodiment, the data warehouse writing module 30 is specifically used to:

[0178] Set up and configure the distributed real-time data processing module, including setting the trigger conditions and frequency of data reading;

[0179] Set up a watermark mechanism or timestamp-based filtering strategy to identify and read incremental data added since the last synchronization;

[0180] Reading incremental data from the operational data storage layer using a distributed real-time data processing module based on the filtering strategy;

[0181] Performing data consistency verification on the read incremental data, wherein the data consistency verification includes verification of integrity, correctness and time sequence;

[0182] Write the verified incremental data into the data warehouse layer of the data lake processing framework according to the predefined schema;

[0183] Update the metadata information of the data warehouse layer, wherein the metadata information includes the writing time of the incremental data, the data version number, the data source information and the data processing history record.

[0184] In one embodiment, the data aggregation module 50 is specifically configured to:

[0185] Based on business needs, identify key business indicators in the incremental data, and define aggregation dimensions based on the key business indicators, where the aggregation dimensions include a time dimension, a region dimension, or a product category dimension;

[0186] Setting a data aggregation strategy based on the aggregation dimension, including setting a grouping method and an aggregation time window;

[0187] Aggregate the incremental data according to the aggregation strategy through a distributed real-time data processing module, and generate an aggregated data result according to the grouping method;

[0188] The aggregated data results are stored in the aggregated indicator output library of the data lake processing framework according to the predefined storage mode and index.

[0189] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external user terminal through a network connection. When the computer program is executed by the processor, it realizes the functions or steps of the service side of a data processing method for a data warehouse.

[0190] In one embodiment, a computer device is provided. The computer device may be a user terminal, and its internal structure diagram may be as follows: Figure 5As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server through a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the user side of a data processing method for a data warehouse

[0191] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0192] Collect business data from the business database and synchronize the business data to the operational data storage layer of the data lake processing framework;

[0193] Processing the business data in the operation data storage layer using a distributed real-time data processing module, and incrementally writing the processed business data into the operation data storage layer in a transaction control manner;

[0194] Read incremental data from the operational data storage layer using a distributed real-time data processing module, and write the incremental data into a data warehouse layer of a data lake processing framework;

[0195] Transmitting the incremental data from the data warehouse layer to the data mart layer within the data lake processing framework;

[0196] Based on business needs, the incremental data is aggregated using a distributed real-time data processing module, and the aggregated data is stored in an aggregate indicator output library of the data lake processing framework;

[0197] The aggregated data is transmitted from the aggregated indicator output library to the application platform database for business query and report generation.

[0198] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0199] Collect business data from the business database and synchronize the business data to the operational data storage layer of the data lake processing framework;

[0200] Processing the business data in the operation data storage layer using a distributed real-time data processing module, and incrementally writing the processed business data into the operation data storage layer in a transaction control manner;

[0201] Read incremental data from the operational data storage layer using a distributed real-time data processing module, and write the incremental data into a data warehouse layer of a data lake processing framework;

[0202] Transmitting the incremental data from the data warehouse layer to the data mart layer within the data lake processing framework;

[0203] Based on business needs, the incremental data is aggregated using a distributed real-time data processing module, and the aggregated data is stored in an aggregate indicator output library of the data lake processing framework;

[0204] The aggregated data is transmitted from the aggregated indicator output library to the application platform database for business query and report generation.

[0205] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the user side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0206] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0207] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0208] It should be noted that if software tools or components other than those of the Company appear in the embodiments of the present application, they are only used for illustration and do not represent actual use. The above-described embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the above-mentioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the above-mentioned embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents; and these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A data processing method for a data warehouse, characterized in that: The following steps are involved: Collect business data from the business database and synchronize the business data to the operational data storage layer of the data lake processing framework; Processing the business data in the operation data storage layer using a distributed real-time data processing module, and incrementally writing the processed business data into the operation data storage layer in a transaction control manner; Read incremental data from the operational data storage layer using a distributed real-time data processing module, and write the incremental data into a data warehouse layer of a data lake processing framework; Transmitting the incremental data from the data warehouse layer to the data mart layer within the data lake processing framework; Based on business needs, the incremental data is aggregated using a distributed real-time data processing module, and the aggregated data is stored in an aggregate indicator output library of the data lake processing framework; The aggregated data is transmitted from the aggregated indicator output library to the application platform database for business query and report generation.

2. The data processing method of the data warehouse according to claim 1, characterized in that: After transferring the incremental data from the data warehouse layer to the data mart layer in the data lake processing framework, the method further includes: Obtaining a query request, wherein the query request includes a query condition, an analysis dimension and / or an aggregation method; Based on the query condition, determining target incremental data matching the query condition from the data mart layer; Based on the analysis dimension, the target incremental data is grouped, and each group of data is aggregated according to the aggregation method to generate an aggregated query result; The query results are transmitted to the application platform database for business query, report generation or business analysis.

3. The data processing method of the data warehouse according to claim 1, characterized in that: After storing the aggregated data in the aggregate indicator output library of the data lake processing framework, it also includes: Execute offline scheduling tasks through scheduled tasks to extract historical data from the data warehouse layer; Transmitting the historical data to the application platform database for business query or report generation; Using a distributed real-time data processing module to process the real-time data from the operational data storage layer, and to perform cleaning and conversion operations on the real-time data; The processed real-time data is updated incrementally to the application platform database and stored according to the same table structure and field type as the historical data.

4. The data processing method of the data warehouse according to claim 1, characterized in that: Collecting business data from the business database and synchronizing the business data to the operational data storage layer of the data lake processing framework includes: Configure a data acquisition module, and extract business data from a business database through the data acquisition module; The business data is transmitted to the operation data storage layer through a data synchronization mechanism as the initially stored business data.

5. The data processing method of the data warehouse as claimed in claim 1, characterized in that: Using a distributed real-time data processing module to process the business data in the operational data storage layer, and incrementally writing the processed business data to the operational data storage layer in a transaction control manner, including: Performing a preprocessing operation on the business data in the operation data storage layer, wherein the preprocessing operation includes data cleaning and formatting; Use distributed real-time data processing modules to perform transaction control on business data after pre-processing operations; The business data after transaction control is incrementally updated to the operational data storage layer, overwriting or updating the initially stored business data.

6. The data processing method of the data warehouse according to claim 1, characterized in that: Using a distributed real-time data processing module to read incremental data from the operational data storage layer and writing the incremental data into a data warehouse layer of a data lake processing framework includes: Set up and configure the distributed real-time data processing module, including setting the trigger conditions and frequency of data reading; Set up a watermark mechanism or timestamp-based filtering strategy to identify and read incremental data added since the last synchronization; Reading incremental data from the operational data storage layer using a distributed real-time data processing module based on the filtering strategy; Performing data consistency verification on the read incremental data, wherein the data consistency verification includes verification of integrity, correctness and time sequence; Write the verified incremental data into the data warehouse layer of the data lake processing framework according to the predefined schema; Update the metadata information of the data warehouse layer, wherein the metadata information includes the writing time of the incremental data, the data version number, the data source information and the data processing history record.

7. The data processing method of the data warehouse according to claim 1, characterized in that: Based on business needs, the incremental data is aggregated using a distributed real-time data processing module, and the aggregated data is stored in the aggregate indicator output library of the data lake processing framework, including: Based on business needs, identify key business indicators in the incremental data, and define aggregation dimensions based on the key business indicators, where the aggregation dimensions include a time dimension, a region dimension, or a product category dimension; Setting a data aggregation strategy based on the aggregation dimension, including setting a grouping method and an aggregation time window; Aggregate the incremental data according to the aggregation strategy through a distributed real-time data processing module, and generate an aggregated data result according to the grouping method; The aggregated data results are stored in the aggregated indicator output library of the data lake processing framework according to the predefined storage mode and index.

8. A data processing device for a data warehouse, characterized in that: The data processing device of the data warehouse comprises: A data collection module, used to collect business data from a business database and synchronize the business data to an operational data storage layer of a data lake processing framework; A data processing module, used to process the business data in the operation data storage layer using a distributed real-time data processing module, and incrementally write the processed business data into the operation data storage layer in a transaction control manner; A data warehouse writing module, configured to read incremental data from the operational data storage layer using a distributed real-time data processing module, and write the incremental data into a data warehouse layer of a data lake processing framework; A data transmission module, used to transmit the incremental data from the data warehouse layer to the data mart layer in the data lake processing framework; A data aggregation module is used to aggregate the incremental data using a distributed real-time data processing module based on business needs, and store the aggregated data in an aggregate indicator output library of a data lake processing framework; The application platform interface module is used to transmit the aggregated data from the aggregated indicator output library to the application platform database for business query and report generation.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and a data processing program for the data warehouse stored in the memory and executable on the processor. When the data processing program for the data warehouse is executed by the processor, the steps of the data processing method for the data warehouse as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The storage medium stores a data processing program for the data warehouse, and when the data processing program for the data warehouse is executed by the processor, the steps of the data processing method for the data warehouse according to any one of claims 1 to 7 are implemented.

Images