Lake warehouse integrated data processing system and electronic equipment

Abstract

The embodiment of the present application provides a lake warehouse integrated data processing system, relates to the field of big data processing, and data is accessed from a plurality of heterogeneous data sources by a data access layer; an object storage unit persistently stores initial data accessed from the data access layer in a raw data format, and constitutes a raw data area; a core storage engine unit adopts a large-scale parallel processing architecture, and manages a refined data area through an incremental data snapshot mechanism; a unified metadata service uniformly manages metadata, and records a derivative relationship and a structure mapping; a data processing layer extracts data from the raw data area, carries out conversion and loading operation, and writes processed data into the refined data area, and is integrated with a data quality checking module; a data service layer provides a unified data access interface, and can query and analyze data in the raw data area and the refined data area. The present application effectively solves problems such as data islands, performance bottlenecks, and management difficulties in the prior art.

Description

Technical Field

[0001] This invention belongs to the field of big data processing technology, specifically relating to a lake-warehouse integrated data processing system and electronic equipment. Background Technology

[0002] With the explosive growth of enterprise data and the increasing complexity of data analysis needs, how to efficiently and reliably store and process massive amounts of multi-source data has become a key challenge in the field of information technology. Against this backdrop, two mainstream technical architectures have emerged: data lakes and data warehouses.

[0003] A data lake is a centralized repository that allows massive amounts of structured, semi-structured, and unstructured data to be stored in their raw format. Its advantages include rapid data ingestion, low storage costs, and high flexibility, making it suitable for data exploration and discovery analysis. However, data lake architectures have significant limitations: because they store raw data, they lack strict data schema constraints and quality control mechanisms, easily evolving into "data swamps," making it difficult to guarantee data accuracy, consistency, and reliability.

[0004] In contrast to traditional data lakes, data warehouses focus on providing high-performance query and analysis capabilities for decision support. They typically employ highly structured models (such as star schemas and snowflake schemas) to store cleaned and transformed data, thereby optimizing query performance. However, their fixed data schema makes it difficult to adapt to rapidly changing business needs and flexible unstructured data processing; furthermore, traditional data warehouse solutions based on high-end hardware are costly and have limited scalability.

[0005] To balance the flexibility of data lakes with the management and performance of data warehouses, the industry has proposed the "lake-warehouse integration" architecture. However, existing lake-warehouse integration solutions mostly achieve this by overlaying a table format on top of the data lake, which is essentially still a loosely coupled integration. This architecture faces challenges in practical applications: First, the flow of data between the lake and the warehouse may still carry latency and consistency risks; second, it is prone to performance bottlenecks when dealing with high-concurrency queries and frequent data updates of large-scale data. Summary of the Invention

[0006] The purpose of this invention is to provide a lake-warehouse integrated data processing system that can realize the flexibility and low cost of a data lake and the high performance and strong governance capabilities of a data warehouse in a unified system, effectively solving the problems of data silos, performance bottlenecks, and governance difficulties in the existing technology.

[0007] In a first aspect, embodiments of the present invention provide a lake warehouse integrated data processing system, including a data access layer, a unified storage engine layer, a data processing layer, and a data service layer; the data access layer, the data processing layer, and the data service layer are all connected to the unified storage engine layer; the unified storage engine layer includes an object storage unit, a core storage engine unit, and a unified metadata service; The data access layer is configured to access data from multiple heterogeneous data sources; The object storage unit is configured to persistently store the initial data accessed from the data access layer in the original data format, forming the original data area; The core storage engine unit adopts a massively parallel processing architecture and is configured to manage the refined data area through an incremental data snapshot mechanism. The incremental data snapshot mechanism creates snapshots containing timestamp identifiers and corresponding data file sets for data updates to support querying the historical state of data based on time points or version numbers. The refined data area is used to store data processed by the data processing layer. The processed data is organized in a columnar storage format, compressed, and partitioned by time or business key. The unified metadata service is configured to uniformly manage the first metadata in the original data area and the second metadata in the refined data area, and record the derivation relationship and structural mapping between the first metadata and the second metadata; The data processing layer is configured to extract data from the raw data area, perform transformation and loading operations, and write the processed data into the refined data area. It also integrates a data quality verification module, which is used to execute data verification, cleaning and quality monitoring rules during the data processing process. The data service layer is configured to provide standardized SQL query interfaces and application programming interfaces, providing a unified data access interface that can query and analyze data in the raw data area and the refined data area.

[0008] Optionally, the data access layer is specifically implemented using the Apache Flink framework; The Apache Flink framework is configured as follows: Data is extracted in parallel from the multiple heterogeneous data sources through a unified source interface; During the data access process, the consumption offset and state information of the data stream are persistently stored in the checkpoint mechanism to achieve the exact-once semantic guarantee. Streaming data and batch data are processed through the data stream API and dataset API respectively, and the processing results are output to the unified storage engine layer.

[0009] Optionally, the core storage engine unit also employs a hybrid concurrency control mechanism, which specifically includes: Assign a transaction timestamp to each data read operation and read based on the latest committed data snapshot version to implement optimistic locking based on version snapshot; For data write and metadata change operations, a pessimistic locking mechanism based on two-phase locking is adopted to lock the relevant data partitions or metadata entries during operation execution. It also provides a conflict detection mechanism that automatically aborts and retryes the relevant transaction when a read / write conflict is detected.

[0010] Optionally, the incremental data snapshot mechanism also supports an automatic snapshot cleanup strategy, which specifically includes: Maintain a snapshot lifecycle management strategy table to record the creation time, data file size, and retention priority of each snapshot; Based on the time retention policy, snapshots that have exceeded the preset retention period are automatically marked as cleanable. According to the storage space policy, when the used storage space exceeds the threshold, snapshots are automatically cleaned up in order of creation time from earliest to latest until the storage space utilization rate is lower than the preset threshold. During the cleanup process, snapshot dependencies are maintained to ensure that data files that other snapshots depend on are not cleaned up.

[0011] Optionally, the unified metadata service also maintains a global data lineage graph, which specifically includes: Data asset nodes are used to represent data tables or data files in the raw data area and refined data area; Process nodes represent transformation operations performed by the data processing layer; Bloodline edges are used to connect related data asset nodes and processing nodes to form a complete data flow path; Furthermore, the global data lineage map also records the conversion operation identifier, execution timestamp, and data quality verification result score for each processing node.

[0012] Optionally, the data quality verification module supports a configurable data quality rule base, including integrity rules, consistency rules, accuracy rules, and timeliness rules; The integrity rule is used to verify whether the non-empty rate of the required field reaches the threshold. The consistency rules are used to verify whether data values ​​conform to a predefined range of values ​​or format specifications. The accuracy rules are used to verify the accuracy of the data by comparing it with authoritative data sources; The timeliness rules are used to verify whether the delay from data generation to access to the system is within the allowable range; Furthermore, the data quality verification module is configured to automatically repair the data by issuing real-time alarms, interrupting tasks, or calling preset data repair scripts when a rule is triggered, based on the severity level of the rule.

[0013] Optionally, the data service layer also provides data virtualization capabilities, the specific implementation process of which is as follows: Provides a logical view definition interface, allowing users to define a join query view across the raw data area and the refined data area using SQL statements; During query execution, the query optimizer resolves the logical view into subquery plans for different underlying data areas; By executing subqueries in parallel through a data federated query engine and performing join and aggregation operations on intermediate results in memory, a unified result set is finally returned, enabling cross-region joint queries without the need for physical data movement.

[0014] Optionally, the core storage engine unit supports a data hot caching layer, which specifically includes: The cache management module identifies hot data partitions in the refined data area based on the LRU-K algorithm; The cache storage area on a high-speed storage device is a columnar storage block used to store hot data partitions; A cache consistency mechanism ensures that when the underlying data is updated, the corresponding cached data is automatically invalidated or updated. The query execution engine prioritizes reading data from the cache storage area, and only accesses the main storage area when the cache is not hit.

[0015] Optionally, the system also integrates a data security module, which provides column-level encryption, dynamic data desensitization, and role-based access control capabilities for data in the unified storage engine layer; The column-level encryption is used to encrypt and store sensitive data columns using the AES-256 encryption algorithm, and the key is managed by an independent key management service; The dynamic data anonymization is used to partially hide or replace sensitive data in real time before the query results are returned, based on the visitor's role. The role-based access control is used to maintain a role permission table, define the access permissions of different roles to data areas, data tables, and data columns, and perform permission verification before query execution.

[0016] Secondly, an electronic device provided by an embodiment of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the functions of the integrated lake and warehouse data processing system described in the first aspect.

[0017] This invention provides a lake-warehouse integrated data processing system. Data enters the system via a data access layer, and raw data is first stored in a raw data area composed of object storage units. The data processing layer extracts data from the raw data area as needed, performs extraction, transformation, and loading, and stores the resulting data in a columnar format in a refined data area managed by the core storage engine unit. This process is monitored by a data quality verification module. The metadata and lineage of all data are uniformly managed by a unified metadata service. Finally, users can transparently access and analyze all data through the data service layer interface and can use the snapshot mechanism provided by the core storage engine unit to query the historical state of the data. Through the above architecture, this invention achieves the flexibility and low cost of a data lake and the high performance and strong governance capabilities of a data warehouse in a unified system, effectively solving the problems of data silos, performance bottlenecks, and governance difficulties in existing technologies. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the structure of a data processing system provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the architecture design of the integrated lake warehouse data processing system provided in an embodiment of the present invention. Detailed Implementation

[0019] The present invention will be described in detail below through embodiments.

[0020] With the explosive growth of enterprise data and the increasing complexity of data analysis needs, how to efficiently and reliably store and process massive amounts of multi-source data has become a key challenge in the field of information technology. Against this backdrop, two mainstream technical architectures have emerged: data lakes and data warehouses.

[0021] A data lake is a centralized repository that allows massive amounts of structured, semi-structured, and unstructured data to be stored in their raw format. Its advantages include rapid data ingestion, low storage costs, and high flexibility, making it suitable for data exploration and discovery analysis. However, data lake architectures have significant limitations: because they store raw data, they lack strict data schema constraints and quality control mechanisms, easily evolving into "data swamps," making it difficult to guarantee data accuracy, consistency, and reliability.

[0022] In contrast to traditional data lakes, data warehouses focus on providing high-performance query and analysis capabilities for decision support. They typically employ highly structured models (such as star schemas and snowflake schemas) to store cleaned and transformed data, thereby optimizing query performance. However, their fixed data schema makes it difficult to adapt to rapidly changing business needs and flexible unstructured data processing; furthermore, traditional data warehouse solutions based on high-end hardware are costly and have limited scalability.

[0023] To balance the flexibility of data lakes with the management and performance of data warehouses, the industry has proposed the "lake-warehouse integration" architecture. However, existing lake-warehouse integration solutions mostly achieve this by overlaying a table format on top of the data lake, which is essentially still a loosely coupled integration. This architecture faces challenges in practical applications: First, the flow of data between the lake and the warehouse may still carry latency and consistency risks; second, it is prone to performance bottlenecks when dealing with high-concurrency queries and frequent data updates of large-scale data.

[0024] To address the aforementioned technical problems in the existing technology, embodiments of the present invention provide a novel integrated lake-warehouse data processing system. For example... Figure 1 As shown, the lake warehouse integrated data processing system may include: a data access layer 110, a unified storage engine layer 120, a data processing layer 130, and a data service layer 140. The data access layer 110, data processing layer 130, and data service layer 140 are all connected to the unified storage engine layer 120. The unified storage engine layer 120 includes an object storage unit 121, a core storage engine unit 122, and a unified metadata service 123.

[0025] Data access layer 110 is configured to access data from multiple heterogeneous data sources.

[0026] Among them, multiple heterogeneous data sources include, but are not limited to, at least two of the following types: (1) Structured data source. This refers to a data source with a predefined schema, such as relational database management systems, including MySQL, Oracle, and PostgreSQL.

[0027] (2) Semi-structured data sources. These are data sources that do not have a strict schema but have a self-describing structure, such as NoSQL databases (e.g., MongoDB, Cassandra), JSON / XML files, and message queues such as Apache Kafka and RabbitMQ; (3) Unstructured data sources. These are data sources with variable formats and flexible patterns, such as log files, images, audio and video files, as well as Hadoop distributed file systems and object storage services.

[0028] The object storage unit 121 is configured to persistently store the initial data accessed from the data access layer in the original data format, forming the original data area.

[0029] The unified storage engine layer 120 is the core of the entire system, enabling deep integration of the data lake and data warehouse at the storage layer. The object storage unit 121 is built on a low-cost, highly scalable object storage service. It is configured to persistently store the initial data received from the data access layer 110 in raw formats (such as JSON, CSV, Avro, and Binaries), which constitutes the raw data area. The raw data area preserves the original appearance of the data, supporting low-cost data accumulation and exploratory analysis.

[0030] The core storage engine unit 122, employing a massively parallel processing architecture, is configured to manage the refined data area through an incremental data snapshot mechanism. This mechanism creates snapshots containing timestamp identifiers and corresponding sets of data files for each data update, enabling queries on the historical state of data based on time points or version numbers. The refined data area stores data processed by the data processing layer. The processed data is organized in a columnar storage format, compressed, and partitioned by time or business key.

[0031] Specifically, the core storage engine unit 122 employs a massively parallel processing (MPP) architecture, which is key to providing high-performance analytical queries. This unit manages the refined data area and implements several innovative technologies: (1) Storage optimization. The data in the refined data area is organized in an optimized columnar storage format (such as Apache Parquet or Apache ORC). The system applies efficient compression algorithms (such as Snappy and Zstandard) to columnar data to save storage space and I / O bandwidth, and adopts a data partitioning strategy based on time (such as dt=20231001) and / or business key (such as region=north), which enables the query engine to significantly reduce the amount of data scanned through partition pruning.

[0032] (2) Incremental Data Snapshot Mechanism. This mechanism is the core of supporting data version management. When data is inserted, updated, or deleted, the system does not directly overwrite the original data file. Instead, it generates a new data file using "copy-on-write" or "append" methods and creates a new snapshot. Each snapshot is identified by a globally unique, monotonically increasing timestamp identifier (such as a transaction ID or commit timestamp) and a set of pointers to all valid data files at that point in time. This mechanism allows users to query the complete state of data at a given historical point in time or snapshot version number, thus enabling the ability to query the historical state of data.

[0033] The unified metadata service 123 is configured to uniformly manage the first metadata in the raw data area and the second metadata in the refined data area, and record the derivation relationship and structural mapping of the first metadata and the second metadata.

[0034] Specifically, the unified metadata service manages the metadata (such as table structure, partition information, data file location, statistics, etc.) of all data in both the raw and refined data areas. More importantly, it records the derivation relationships and structural mappings from the metadata in the raw data area (referred to as "first metadata" for clarity) to the corresponding metadata in the refined data area (referred to as "second metadata" for clarity). For example, it records that "the user summary table in the refined data area is derived from the user click log table and user information table in the raw data area through JOIN and AGGREGATE operations." This global data lineage map is crucial for data governance, impact analysis, and troubleshooting.

[0035] The data processing layer 130 is configured to extract data from the raw data area, perform transformation and loading operations, and write the processed data into the refined data area. It also integrates a data quality verification module, which is used to execute data verification, cleaning and quality monitoring rules during the data processing process.

[0036] Specifically, the data processing layer 130 is responsible for data processing and quality assurance. This layer is preferably implemented using the Apache Spark framework, whose powerful distributed computing capabilities are suitable for ETL (Extract-Transform-Load) processing of large-scale data. The data processing layer 130 is configured to extract data from the raw data area, perform cleaning, transformation, and aggregation operations, and write the processed high-quality data into the refined data area. During this process, the integrated data quality verification module executes configurable quality rules (such as integrity, consistency, accuracy, and timeliness rules). If data does not conform to the rules, the system will issue alarms or automatically repair according to preset strategies to ensure that the data flowing into the refined data area is reliable.

[0037] The data service layer 140 is configured to provide a standardized SQL query interface and application programming interface, providing a unified data access interface, and is capable of querying and analyzing data in the raw data area and the refined data area.

[0038] Specifically, the data service layer 140 provides a unified data access point for end users and applications. This layer exposes standard SQL query interfaces and application programming interfaces. Through the query optimizer and federated query engine, the data service layer 140 understands the data distribution within the unified storage engine layer 120, providing a unified data access interface. Users do not need to care whether the data is physically located in the raw data area or the refined data area; they can complete cross-area join queries and analysis with a single SQL statement, achieving a logically unified view of the data.

[0039] This invention provides a lake-warehouse integrated data processing system. Data enters the system via a data access layer, and raw data is first stored in a raw data area composed of object storage units. The data processing layer extracts data from the raw data area as needed, performs extraction, transformation, and loading, and stores the resulting data in a columnar format in a refined data area managed by the core storage engine unit. This process is monitored by a data quality verification module. The metadata and lineage of all data are uniformly managed by a unified metadata service. Finally, users can transparently access and analyze all data through the data service layer interface and can use the snapshot mechanism provided by the core storage engine unit to query the historical state of the data. Through the above architecture, this invention achieves the flexibility and low cost of a data lake and the high performance and strong governance capabilities of a data warehouse in a unified system, effectively solving the problems of data silos, performance bottlenecks, and governance difficulties in existing technologies.

[0040] Based on the above embodiments, as one implementation of the present invention, the data access layer is specifically implemented using the Apache Flink framework.

[0041] The Apache Flink framework is configured as follows: Data is extracted in parallel from multiple heterogeneous data sources through a unified source interface.

[0042] During the data access process, the consumption offset and state information of the data stream are persistently stored in the checkpoint mechanism to achieve exactly-once semantic guarantee.

[0043] Streaming data and batch data are processed through the data stream API and dataset API respectively, and the processing results are output to the unified storage engine layer.

[0044] Specifically, the data access layer 110 is implemented using the Apache Flink framework. Apache Flink is an open-source unbounded and bounded data stream processing framework. Its core advantages lie in its integrated stream and batch processing capabilities and exactly-once state consistency guarantee, making it very suitable as the data access engine for the lakehouse integrated architecture of this invention.

[0045] The Apache Flink framework is configured to perform the following operations: 1. Parallel data extraction from multiple heterogeneous data sources via a unified source interface. The Flink framework provides a rich and extensible set of source connector APIs, allowing developers to easily adapt to various heterogeneous data sources. In this implementation, multi-source parallel extraction is achieved by configuring different Flink source connectors. For relational databases (such as MySQL and Oracle), a JDBC source connector is used to extract data from database tables in parallel through a standard JDBC interface. For message queues, a Kafka source connector is used to subscribe to and pull message streams from partitions of a Kafka topic in parallel as a consumer group. For static files in the file system, a FileSystem source connector is used to read file contents in parallel by monitoring a specified directory path. These source connectors run in parallel within the Flink job, with each parallel instance responsible for a portion of the data source reading tasks, thus achieving high-throughput data access.

[0046] 2. During data access, the consumption offset and state information of the data stream are persistently stored in the checkpoint mechanism to achieve exactly-once semantic guarantees. This is a key mechanism to ensure data reliability. The Flink runtime engine automatically initiates globally consistent checkpoints periodically (triggered at user-configurable intervals). The creation process of each checkpoint includes the following steps: The first step is barrier injection and propagation. Flink injects a special barrier signal into each parallel data stream. This barrier is propagated downstream with the data stream, logically dividing the stream into data before and after the checkpoint.

[0047] The second step is asynchronous snapshotting. When all barriers in all input streams of an operator are reached, the operator asynchronously snapshots its current state (e.g., the aggregation results in the window, the consumption offset of the data source) to a persistent storage system.

[0048] The third step is the confirmation mechanism. A checkpoint is marked as complete only after all operators in the entire job graph have successfully completed their snapshots. This mechanism ensures that the data source's consumption offset and the operator's internal state information are consistently preserved. In the event of a failure requiring recovery, Flink resets the entire job to the state of the most recent successful checkpoint and allows the data source to resume consuming data from the corresponding offset. This ensures that each piece of data is processed exactly once, even if a failure occurs during processing, preventing data loss or duplication and providing a solid foundation for data consistency in subsequent processing.

[0049] 3. Streaming and batch data are processed separately using the DataStream API and DataSet API, with the processing results output to a unified storage engine layer. Flink provides two core APIs to handle data of different types. Specifically, unbounded, real-time streaming data is processed using the DataStream API. For example, real-time user click streams from Kafka can be cleaned, filtered, and aggregated in real-time using operators such as map, filter, keyBy, and window on DataStream. Bounded, historical backlog batch data is processed using the DataSet API (or processed uniformly under the Table API / SQL). For example, historical log files from HDFS (Hadoop Distributed File System) can be fully computed using relevant operators on DataSet.

[0050] Despite using different APIs, Flink's unified stream and batch runtime can schedule and execute both types of computational tasks in a unified manner. The processed results (whether real-time results from stream processing or offline results from batch processing) are output to the unified storage engine layer 120 described in this invention through a unified receiver interface. For example, a custom Flink Sink can be developed to write the processed data to the refined data area of ​​the core storage engine unit 122 according to a specified format, or the original data can be written to the original data area of ​​the object storage unit 121 first. This design unifies stream and batch processing at the access layer, simplifying the technical architecture.

[0051] With the above configuration, the data access layer 110 using the Apache Flink framework provides a high-performance, highly reliable, and unified stream and batch data entry point for the lake warehouse integrated system of this invention.

[0052] Based on the above embodiments, as one implementation of the present invention, the core storage engine unit also adopts a hybrid concurrency control mechanism, which specifically includes: Each data read operation is assigned a transaction timestamp, and reads are performed based on the latest committed data snapshot version, thus implementing optimistic locking based on version snapshots.

[0053] For data write and metadata change operations, a pessimistic locking mechanism based on two-phase locking is adopted to lock the relevant data partitions or metadata entries during operation execution.

[0054] It also provides a conflict detection mechanism that automatically aborts and retryes the relevant transaction when a read / write conflict is detected.

[0055] Specifically, the core storage engine unit 122 also employs a hybrid concurrency control mechanism. This mechanism aims to efficiently coordinate concurrent access to system resources by multiple users and tasks, ensuring strong data consistency while guaranteeing high throughput and low latency. It innovatively applies optimistic locking based on multi-version concurrency control to read operations and pessimistic locking based on two-phase locking to write operations, resolving read-write conflicts through conflict detection and automatic retry mechanisms.

[0056] Specifically, the implementation process of this hybrid concurrency control mechanism includes the following steps: First, optimistic locking based on version snapshots handles read operations. When the system receives a data read request, the concurrency controller immediately assigns a globally unique, monotonically increasing read timestamp to the read transaction, such as a unique identifier generated based on the physical clock and node ID. This timestamp marks the logical point in time when the read operation began. Furthermore, the system binds this read timestamp to the latest committed data snapshot version in the incremental data snapshot mechanism, whose timestamp is less than or equal to this read timestamp. This snapshot contains data changes from all committed transactions before that point in time, thus providing a consistent, static view of the data for read operations.

[0057] Read operations are based on this bound snapshot version. Since the snapshot is read-only and immutable, the read operation can traverse the data without acquiring any locks. This means that a large number of read-only queries can be executed concurrently without blocking each other or ongoing write operations, resulting in extremely high read concurrency performance. This approach is inherently optimistic, as it assumes that read operations are unlikely to conflict with write operations that require abort.

[0058] Secondly, pessimistic locking based on two-phase locking handles write operations. When the system receives a data write or metadata change request, the following process is executed: 1. Acquiring a pessimistic lock. Before modifying any data or metadata, a write transaction must acquire an exclusive lock of appropriate granularity according to the scope of the operation. For example, to modify a row in a table, the data partition containing that row may be locked first. To perform an operation affecting the entire table (such as adding a column), a table-level lock is required. For metadata changes, the relevant metadata entries (such as catalog entries) are locked. 2. Two-phase locking protocol. Transactions strictly follow the two-phase locking protocol. In the growth phase, the transaction gradually acquires all the necessary locks and does not release any locks before acquiring all locks. Only when the transaction has completed all data modifications and is ready to commit (the shrinking phase) will all held locks be released at once. This effectively prevents other transactions from interfering with the modification process of this transaction, ensuring the atomicity and isolation of writes.

[0059] Finally, a conflict detection and transaction abort / retry mechanism is implemented. This mechanism is crucial for coordinating optimistic reads and pessimistic writes, and it is primarily triggered when a write transaction commits. When a write transaction completes all operations and is ready to commit, it does not commit immediately but enters a commit verification phase. The system checks whether the data version read by the transaction during its execution (i.e., the data snapshot version at the start) is still up-to-date. Specifically, the system verifies whether any other committed write transactions have modified data that this transaction has read since the transaction began. If no version conflict is detected, it means that the data state on which the transaction is based is still valid, and the system allows it to commit. The commit process includes: persisting its changes, creating a new global data snapshot version, and updating relevant metadata. If a conflict is detected (i.e., it is found that the data read by this transaction has been modified by other concurrently committed transactions), the system determines that this write may be based on outdated data, leading to an inconsistent state. At this point, the system will automatically abort the transaction and immediately release all locks held by it.

[0060] For aborted transactions, the system can automatically add them to the retry queue. Then, the system will re-execute all the logic of the transaction based on the latest data state. This process is transparent to the user, thus ensuring eventual data consistency.

[0061] This implementation's hybrid concurrency control mechanism cleverly balances performance and consistency by differentiating the characteristics of read and write operations and applying optimal concurrency strategies. Read operations utilize multi-version snapshots to achieve lock-free concurrency, significantly improving the system's query throughput. Write operations ensure correct data changes through strict pessimistic locking. Furthermore, conflict detection and automatic retry mechanisms at commit time guarantee eventual strong consistency in high-concurrency write scenarios. This design enables the lakeware system of this invention to efficiently support mixed workloads of complex analytical queries and frequent data update concurrency.

[0062] exist Figure 1 Based on the illustrated embodiment, as one implementation of this invention, the incremental data snapshot mechanism also supports an automatic snapshot cleanup strategy. This strategy aims to address the problem of excessive storage space consumption caused by the continuous accumulation of historical snapshots and their associated data files over time, thereby achieving automated and intelligent management of the storage lifecycle while ensuring data traceability. The automatic snapshot cleanup strategy specifically includes: Maintain a snapshot lifecycle management strategy table to record the creation time, data file size, and retention priority of each snapshot.

[0063] Based on the time retention policy, snapshots that have exceeded the preset retention period are automatically marked as cleanable.

[0064] According to the storage space policy, when the used storage space exceeds the threshold, snapshots are automatically cleaned up in order of creation time from earliest to latest until the storage space utilization rate is lower than the preset threshold.

[0065] During the cleanup process, snapshot dependencies are maintained to ensure that data files that other snapshots depend on are not cleaned up.

[0066] Specifically, the automatic snapshot cleanup policy is implemented through the following steps: The first step is to maintain the snapshot lifecycle management strategy table. The system maintains a snapshot lifecycle management strategy table in the unified metadata service 123. This table records detailed metadata information for each data snapshot, serving as the basis for cleanup decisions. Each record contains at least the following fields: snapshot identifier, creation timestamp, ... Data file size, retention priority, and snapshot dependencies. Retention priority is a configurable weight value used to identify the importance of the snapshot. For example, snapshots generated by important business operations can be given high priority to avoid premature cleanup. Snapshot dependencies record the relationship between this snapshot and other snapshots; for example, they indicate whether this snapshot is depended upon by subsequent snapshots.

[0067] The second step involves triggering cleanup based on the time-based retention policy. Specifically, the system supports time-based retention policies. Administrators can configure the retention period for snapshots globally or by table (e.g., retaining snapshots from the most recent 30 days). A scheduled task runs in the background, continuously scanning the snapshot lifecycle management policy table. For any snapshot whose creation timestamp is earlier than the current time minus the retention period, the system automatically marks its status as cleanable. This marking operation itself is lazy; it does not immediately delete the data, leaving room for subsequent space-based policies or manual intervention.

[0068] The third step involves triggering and executing cleanup based on storage space policies. Specifically, this policy serves as an emergency guarantee mechanism for storage resources. The system monitors the total storage space utilization of the unified storage engine layer 120 in real time. When the utilization exceeds a preset threshold (e.g., 85%), this policy is automatically triggered. The cleanup process can be as follows: The system filters all snapshots in a cleanupable state from the snapshot lifecycle management policy table and sorts them in ascending order of creation time, forming a cleanup queue. The earliest created snapshot will be considered for cleanup first. Alternatively, the system can start from the head of the cleanup queue (the oldest snapshot) and perform cleanup operations sequentially. For each selected snapshot, the system deletes its unique metadata entries and asynchronously reclaims data files unique to that snapshot and not referenced by any other active snapshots. This process continues until the total storage space utilization drops below a safe threshold (e.g., 75%).

[0069] The fourth step is maintaining the dependencies between snapshots and data files. This is a crucial step in ensuring data integrity. Before and after cleaning up any snapshot, the system strictly maintains the reference relationships between snapshots and underlying data files: before deleting a data file corresponding to a snapshot, the system queries the snapshot dependency graph to confirm whether these data files are shared and depended upon by any subsequently created snapshots. A data file is only deleted if it is no longer referenced by any snapshot that has not been marked as cleanable. This means that even if a snapshot itself has been marked for cleanup, as long as the data files it contains are still referenced by a snapshot that needs to be retained, these files will be preserved. This reference-count-based garbage collection mechanism effectively prevents the loss of historical data due to accidental cleanup.

[0070] This implementation's automatic snapshot cleanup strategy achieves an intelligent balance between storage costs and data retention requirements through automated management of both time and space dimensions. It not only prevents unlimited growth of storage space and reduces operational costs, but also ensures the reliability of core data historical query functions through rigorous dependency maintenance, making long-term stable system operation possible.

[0071] Based on the above embodiments, as one implementation of the present invention, the unified metadata service also maintains a global data lineage graph. This graph, in the form of a graph structure, records the complete lifecycle and flow path of data within the system from intake to consumption. It is a core infrastructure for achieving advanced data governance, impact analysis, fault tracing, and compliance auditing. The global data lineage graph specifically includes: Data asset nodes are used to represent data tables or data files in the raw data area and refined data area.

[0072] Process nodes represent transformation operations performed by the data processing layer.

[0073] Bloodline edges are used to connect related data asset nodes and processing nodes to form a complete data flow path.

[0074] Furthermore, the global data lineage graph also records the transformation operation identifier, execution timestamp, and data quality verification result score for each processing node.

[0075] Specifically, the global data lineage graph is a directed graph consisting of two types of nodes and one type of edge. The two types of nodes are data asset nodes and processing nodes. Data asset nodes represent static data entities stored in the system. Each node uniquely identifies a specific data object in the original or refined data area, and its attributes include basic metadata such as the data asset's unique identifier, storage location, data format, and creation time. Processing nodes represent instances of dynamic transformation operations performed on data. Each node corresponds to a specific task or operation step executed by the data processing layer 130.

[0076] Lineage edges are used to connect nodes, representing the flow of data between different states. There are two main types: input edges and output edges. Input edges point from data asset nodes to processing nodes, indicating that the data asset is the input source for this processing operation. Output edges point from processing nodes to data asset nodes, indicating that the data asset is the output of this processing operation. Through these edges, scattered nodes are organized into clear, end-to-end data flow paths from the original data source to the final data product.

[0077] To enhance the analytical value of pedigree graphs, the unified metadata service 123 also records detailed contextual information for each processing node within the graph. The transformation operation identifier uniquely identifies the job ID or task name of this transformation operation, facilitating precise location. The execution timestamp records the start and end times of the processing, used to analyze the timeline and efficiency of data processing. The data quality verification result score records the results of the data quality verification module's assessment of data quality before and after the processing. For example, it can record scores for completeness and accuracy. This directly links data quality information to the data processing process, allowing users to quickly identify which processing step might lead to a decline in data quality.

[0078] The mechanism for constructing and maintaining the global data lineage graph can be as follows: When the data processing layer 130 is executed, its runtime framework interacts with the unified metadata service 123, automatically registering processing nodes at the start and end of the job and recording their input and output data assets, thereby dynamically generating or updating lineage edges. The global data lineage graph is updated in real time as the data pipeline runs, ensuring that it always reflects the latest state of the system.

[0079] This implementation approach endows the system with strong observability by constructing and maintaining a global data lineage graph containing rich context. It makes the source of data clearly traceable, the scope of change impact predictable, and the root cause of failures quickly located, greatly improving the maintainability, reliability, and compliance of the entire lakeware system and meeting the stringent requirements of enterprise-level data governance.

[0080] Based on the above embodiments, as one implementation of the present invention, the data quality verification module supports a configurable data quality rule base, including integrity rules, consistency rules, accuracy rules and timeliness rules.

[0081] Integrity rules are used to verify whether the non-empty rate of required fields has reached a threshold.

[0082] Consistency rules are used to verify whether data values ​​conform to a predefined range of values ​​or format specifications.

[0083] Accuracy rules are used to verify the accuracy of data by comparing it with authoritative data sources.

[0084] Timeliness rules are used to verify whether the delay from data generation to access to the system is within the allowable range.

[0085] Furthermore, the data quality verification module is configured to automatically perform real-time alarms, interrupt tasks, or call preset data repair scripts for automatic repair when a rule is triggered, based on the severity level of the rule.

[0086] The aforementioned data quality rule base allows administrators to flexibly define quality rules for different data tables and fields through a declarative configuration interface or API. The rule base primarily includes the following four core rule categories and can be expanded according to business needs.

[0087] The integrity rule is used to verify the completeness of data records. This rule validates required fields. The system scans the target field and calculates the percentage of records with non-null values ​​out of the total number of records (non-null rate). Once the scan is complete, if the calculated non-null rate is lower than a preset threshold, the integrity rule is triggered. This ensures that the loss of critical information is effectively monitored.

[0088] Consistency rules are used to verify whether data values ​​conform to predefined business logic or format specifications. These rules are implemented by configuring SQL expressions or regular expressions. For example, a rule can be defined to verify that the value of the "Age" field must be between 0 and 150; or to verify that the value of the "Email" field must conform to a format defined by a regular expression. During data processing, the system performs this expression validation on each record, counts the proportion of records violating the consistency rules, and triggers an alarm if the proportion exceeds a tolerance threshold.

[0089] Accuracy rules are used to verify the accuracy of data by comparing it with authoritative data sources. This rule is suitable for dimensional data requiring high accuracy. For example, when processing "administrative division codes," the system can be configured to perform a lookup query on the code values ​​in the incoming data against an internally maintained, authoritative dictionary of administrative division codes during data processing. If an unknown code that cannot be matched is found, it is considered inaccurate data. The system calculates the proportion of inaccurate records, which serves as the basis for triggering the rule.

[0090] Timeliness rules are used to verify whether the delay from data generation to system access is within acceptable limits, ensuring data freshness. This rule requires data records to include a timestamp field indicating the data generation time. When data is accessed, the system calculates the difference between the current system time and the time in this field (i.e., the delay). If the average or maximum delay of a batch of data exceeds a preset threshold, the rule is triggered. This is crucial for real-time decision-making scenarios.

[0091] Furthermore, an automated response mechanism can be implemented when a rule is triggered. Specifically, the data quality verification module is configured to execute tiered automated response actions based on the pre-defined severity level for each rule when a rule is triggered.

[0092] For example, real-time alerts can be issued. Specifically, for rule violations with a severity level of "warning" (such as exceeding data latency limits), the module will log errors and send real-time alert notifications to the data operations team via integrated messaging systems (such as email, DingTalk, and WeChat Work). Data processing tasks will continue to execute without interruption, but relevant personnel will be aware of the potential risks.

[0093] Task interruption is also possible. Specifically, for rule violations with a severity level of "error" (such as critical field incompleteness), the module will immediately send an interrupt signal to the data processing engine, stopping the currently executing data processing task to prevent low-quality data from contaminating the downstream refined data area. The task status will be marked as failed, and an alarm will be triggered.

[0094] Automatic repair is also possible. Specifically, for certain types of rule violations, the module can be configured to automatically repair them by calling preset data repair scripts. For example, for the issue of "inconsistent phone number format" in the consistency rule, a script can be configured to automatically remove spaces and hyphens from the numbers, standardizing them. The repaired data will then undergo quality verification again, and only after passing the verification can it continue to flow. This mechanism improves the automation level of data processing while ensuring quality.

[0095] This implementation deeply embeds data quality assurance into the data processing pipeline through a configurable, multi-dimensional data quality rule base and an automated response mechanism. It achieves a shift from passive discovery to proactive prevention, significantly improving the credibility and reliability of core data assets flowing into the integrated lake warehouse system. This lays a solid foundation for accurate data analysis and decision-making at the upper level, and greatly reduces business risks and operational costs caused by data quality issues.

[0096] Based on the above embodiments, as one implementation of the present invention, the data service layer also provides data virtualization capabilities, the specific implementation process of which is as follows: Provides a logical view definition interface, allowing users to define a join query view across the raw data area and the refined data area using SQL statements; During query execution, the query optimizer resolves the logical view into subquery plans for different underlying data areas; By executing subqueries in parallel through a data federated query engine and performing join and aggregation operations on intermediate results in memory, a unified result set is finally returned, enabling cross-region joint queries without the need for physical data movement.

[0097] Specifically, the data service layer 140 provides a logical view definition interface to users (typically data analysts or applications). This interface supports view creation using standard data definition languages. Users can define complex query logic across extents using SQLCREATE VIEW statements.

[0098] For example, a user can define a view that associates the raw user click logs in the raw data area with the refined user dimension table in the refined data area. During this process, the user does not need to worry about the actual physical storage location, file format, or partitioning method of the user profile table and the raw click data table. The system persistently stores this view definition in the unified metadata service 123.

[0099] When a user queries this logical view, the query optimizer in the data service layer 140 begins operation. The optimizer first parses the SQL definition of the unified view of user behavior, verifying the existence and accessibility of the referenced tables in the refined data area and the original data area. The optimizer transforms the view's SQL logic into an initial logical execution plan, which includes cross-data area join operations. Then, based on underlying data statistics (such as table size, partition information, file format, etc.) obtained from the unified metadata service 123, the optimizer transforms the logical plan into an efficient, concrete physical execution plan. The core of this plan is to parse and push down a complex cross-data area query into multiple subquery plans that can be executed in parallel for different data areas. For example, it generates subquery A and subquery B. Subquery A is responsible for submitting the query to the core storage engine unit 122 to read the required user data from the refined data area. Subquery B is responsible for reading the raw click event data from the original data area through the interface of the object storage unit 121.

[0100] The generated physical execution plan is submitted to the data federated query engine of the data service layer 140 for execution. Specifically, it can be divided into: 1. Parallel execution of subqueries. The federated query engine dispatches subquery A and subquery B to their respective underlying storage units for execution in parallel. Since the refined data area uses columnar storage and may reside in the MPP engine, subquery A will be very fast. Subquery B, which reads the original data, may be slightly slower, but parallel execution minimizes the overall waiting time.

[0101] 2. In-Memory Data Processing. The result sets of the two subqueries are streamed into the memory of the node where the federated query engine resides. The engine then executes subsequent operations specified in the plan in memory, such as joining, filtering, or aggregating the two result sets.

[0102] 3. Return a unified result: After all data processing is complete, the engine will return the final unified result set to the user.

[0103] The key to this implementation is that throughout the entire process, the data in the original data area and the refined data area remain in their respective storage locations, without needing to be copied or moved to the same physical database. Data integration and computation are performed in the memory of the federated query engine using efficient algorithms.

[0104] This implementation completely breaks down the isolation caused by the physical location of data through data virtualization capabilities. It greatly simplifies the complexity of data analysis and enhances the agility of data exploration. Users can directly build data models based on business logic without waiting for a lengthy and error-prone ETL process. At the same time, it ensures data real-time performance because queries always directly apply to the latest data, providing strong technical support for enterprises to conduct rapid, cross-domain collaborative analysis.

[0105] Based on the above embodiments, as one implementation of this invention, the core storage engine unit supports a data hot caching layer. This layer aims to significantly reduce the number of accesses to the main storage area (typically based on low-cost but high-latency object storage or distributed file systems) by caching frequently accessed hot data on high-speed storage devices, thereby greatly improving the response speed of high-frequency queries and addressing high-concurrency, low-latency online analysis scenarios. The data hot caching layer specifically includes: The cache management module identifies hot data partitions in the refined data area based on the LRU-K algorithm; The cache storage area on a high-speed storage device is a columnar storage block used to store hot data partitions; A cache consistency mechanism ensures that when the underlying data is updated, the corresponding cached data is automatically invalidated or updated. The query execution engine prioritizes reading data from the cache storage area, and only accesses the main storage area when the cache is not hit.

[0106] The core responsibility of the cache management module is to dynamically identify and manage hot data in the refined data area.

[0107] This module continuously monitors all data read requests initiated by the query execution engine and calculates the access frequency and most recent access time for each data partition or more granular data block within a certain time window. This module employs the LRU-K algorithm as its core hotspot identification strategy. Unlike the traditional Least Recently Used (LRU) algorithm, which only focuses on the most recent access time, the LRU-K algorithm records the access history of each data item for the most recent K times (e.g., K=2). By analyzing access frequency and access time intervals, this algorithm can more accurately identify truly hot data, avoiding cache contamination by non-hotspot data due to accidental full table scans or other operations. Data partitions or data blocks determined by the LRU-K algorithm to be frequently accessed will be marked as hotspot data.

[0108] For the cache storage area, the system allocates a cache storage area locally on the compute nodes or on a dedicated high-speed storage cluster. This storage medium typically uses low-latency solid-state drives or non-volatile memory. Hot data selected from the refined data area of ​​the main storage area is loaded into this cache storage area, in units of its original columnar storage blocks. Maintaining the columnar storage format ensures that the query engine can directly apply optimization techniques such as predicate pushdown and column pruning without format conversion.

[0109] To ensure that the data in the cache remains synchronized with the data in the main storage area and to prevent queries from reading outdated or dirty data, the system implements a strict cache consistency mechanism. When the core storage engine unit 122 commits new data updates through the incremental data snapshot mechanism, this mechanism detects which update operations have affected the data blocks currently in the cache. For the affected data blocks, the system immediately marks their cached copies as invalid.

[0110] Regarding cached data update modes, the system supports two modes. One is lazy update, where the latest version of the data block is reloaded from the main storage only when a query request accesses expired cached data again. The other is pre-update, where for critical, frequently accessed data, the system can asynchronously push the updated data block to the cache in the background to ensure optimal performance on the next access.

[0111] Regarding cache-aware access in the query execution engine: The query execution engine is designed to be cache-aware. When processing a query, the engine first calculates the list of required data blocks based on the query conditions and prioritizes reading these data blocks from the cache storage. If the required data block is not in the cache (i.e., a cache miss), the engine will then access the main storage (i.e., the standard storage where the refined data area resides) to retrieve the data. Simultaneously, it sends a signal to the cache management module, which, according to the LRU-K policy, determines whether to load the newly read data block into the cache for use in subsequent queries.

[0112] For example, the workflow of the data hot caching layer can be as follows: A frequently executed daily report query needs to repeatedly read the sales data partitions from the most recent 7 days. The cache management module identifies these partitions as hot data using the LRU-K algorithm and loads them into the SSD cache. Subsequent daily report queries will directly read data from the high-speed SSD, reducing response time from seconds to milliseconds. When new sales data is added at midnight and updates the partitions for the next day, the cache consistency mechanism automatically invalidates the old data for the corresponding partitions in the cache. When a query the following day accesses this new partition, it will trigger a cache miss, thereby loading the latest data from the main storage and updating the cache.

[0113] This implementation introduces an intelligent data hot caching layer, providing near-in-memory database access performance for hot queries without altering the underlying data architecture. It effectively reduces the I / O pressure on the main storage layer from high-concurrency analysis loads, achieving a better allocation of computing and storage resources. This allows the integrated lakeware system to simultaneously execute two different workloads: offline analysis of historical data and online interactive queries of the latest hot data.

[0114] Based on the above embodiments, as one implementation of this invention, the system also integrates a data security module. This data security module provides multi-layered security protection for the data stored in the unified storage engine layer 120 throughout its data lifecycle, ensuring the confidentiality, integrity, and availability of sensitive data during storage, processing, and access. It provides column-level encryption, dynamic data anonymization, and role-based access control capabilities for the data in the unified storage engine layer. Column-level encryption is used to encrypt and store sensitive data columns using the AES-256 encryption algorithm, with the key managed by an independent key management service. Dynamic data anonymization is used to partially hide or replace sensitive data in real time before the query results are returned, based on the visitor's role. Role-based access control is used to maintain a role permission table, define the access permissions of different roles to data areas, data tables, and data columns, and perform permission verification before query execution.

[0115] For column-level encryption, this feature aims to encrypt specific sensitive data columns written to storage, ensuring that even if the data file is accessed illegally, its contents cannot be directly deciphered. The system encrypts designated sensitive data columns (such as ID card numbers, mobile phone numbers, bank card numbers, etc.) using the industry-leading AES-256 algorithm. The keys required for encryption and decryption are not stored internally but are managed, rotated, and audited by a separate, professional key management service. When the core storage engine unit 122 needs to encrypt or decrypt data, it requests a temporary key from this key management service through a secure API interface. This design achieves separate management of keys and data, greatly improving security.

[0116] This column-level encryption process is largely transparent to upper-layer applications. When the data processing layer 130 writes data containing sensitive columns to the refined data area, the security module intercepts the data stream, encrypts the specified columns using a key obtained from KMS, and then writes the ciphertext to storage. Conversely, when the query results from the data service layer 140 contain encrypted columns and the user has decryption privileges, the security module automatically decrypts the data before returning it to the user. Data always exists in ciphertext form on disk.

[0117] Dynamic data masking is performed in real time during data access, ensuring that users with different permissions see data of different security levels, achieving data usability without visibility. Administrators can predefine masking strategies, such as: "When a user with the role of 'Analyst' queries the 'User Table,' only the first 3 and last 4 digits of the 'Phone Number' column should be displayed, with asterisks used to fill in the middle." When a user submits a query through data service layer 140, the query execution engine calls the data security module in the final stage of generating the result set. Furthermore, the security module matches the corresponding masking rules based on the currently logged-in user's role and performs real-time transformation on each row of data in the result set in memory. The entire masking process is completed instantaneously before the data leaves the database engine; the original data remains unmodified in the storage layer, ensuring data authenticity while strictly limiting the excessive exposure of sensitive information.

[0118] Role-based access control (RBAC) is the cornerstone of data access, strictly controlling the scope of data a user can access through the principle of least privilege. The system maintains a role-based access control table in the unified metadata service 123. This table precisely defines the access permissions of different roles (such as data administrators, business analysts, and developers) to different levels of data resources. Permissions can be refined to the following levels: (1) Data area level, controls whether users can access the raw data area or the refined data area.

[0119] (2) Data table level, controls which specific tables users can access.

[0120] (3) Data column level: controls which columns in the table users can access (i.e., column-level permissions).

[0121] Furthermore, when a user initiates any data access request through the data service layer 140, the system does not execute a query first, but instead initiates an authorization verification process. This process parses the query statement, extracts the data objects (such as databases, tables, and columns) intended to be accessed, and then matches and verifies them against the current user's role in the role and permission table. Only if the authorization check passes will the query continue to be executed; otherwise, the query will be immediately rejected, and an "insufficient permissions" error message will be returned. This measure eliminates the possibility of unauthorized access from the very beginning.

[0122] The three functions mentioned above—column-level encryption, dynamic data anonymization, and role-based access control—work together to form a defense-in-depth system. For example, consider data containing sensitive user information: the ID number column is stored in encryption; when a regular analyst queries this data, their access is restricted to querying the table but not viewing the plaintext ID numbers. Therefore, the dynamic anonymization rule is triggered, completely hiding or replacing the column with the "**encrypted data**" label in the results; however, when a specially authorized data administrator queries this data, they are allowed access, and the security module calls KMS to decrypt the data and return the complete information.

[0123] This implementation provides end-to-end security protection from the storage layer to the access layer by integrating a data security module. It not only meets compliance requirements for sensitive data protection but also maximizes the analytical value of data while ensuring data security through granular access control and real-time data masking. This lays a solid technical foundation for enterprises to securely use data in a strictly controlled environment.

[0124] Figure 2 The architecture design of an integrated lake-warehouse data processing system according to an embodiment of the present invention is shown. Figure 2 As shown, the system adopts a layered architecture design, which includes an analysis layer, an access layer, a processing layer and a storage layer from top to bottom. The layers communicate and transfer data through standard interfaces, forming a complete and efficient data processing and analysis platform.

[0125] 1. Analysis layer The analytics layer, as the top layer of the system, directly faces end users and applications, providing diverse data consumption and interaction capabilities. This layer specifically includes: The BI tool supports mainstream business intelligence tools through standard interfaces, providing users with the ability to perform visual data analysis and generate reports.

[0126] The JDBC interface provides a standard database connection interface, allowing users to directly write SQL query statements to analyze data in the system using SQL client tools or custom applications.

[0127] The SPARK / FLINK computing engine integrates the Apache Spark and Apache Flink computing engines, allowing users to submit complex distributed computing tasks for advanced data analysis such as machine learning and graph computing.

[0128] The API interface provides a RESTful API, which makes it easy for applications to programmatically obtain data query results or trigger data processing tasks.

[0129] 2. Access Layer The access layer is responsible for extracting data from various heterogeneous data sources in real time or in batches, serving as the entry point for data into the system. The core component of this layer is Flink CDC, which is configured as follows: Change Data Capture technology captures incremental data changes in the source database (such as MySQL) in real time.

[0130] It also supports accessing data from streaming data sources such as message queues (e.g., Kafka), log files, and batch data files exported from traditional databases.

[0131] After the access layer performs preliminary formatting on the extracted data, it transmits it downstream. The diagram also illustrates a common layered model for data warehouses (Application Data Layer (ADS), Data Service Layer (DWS), and Data Warehouse Detail Layer (DWD), indicating that the accessed data will serve the construction of these standard data warehouse layers.

[0132] 3. Processing layer The processing layer receives raw data from the access layer and is responsible for data cleaning, transformation, integration, and quality control. The core of this layer includes: Traffic Data Lake: This area corresponds to the operational data storage layer, used to store the most raw business data introduced from the access layer. This data maintains the original state of the business system and supports rapid data backtracking and exploratory analysis.

[0133] Data processing logic: Based on a powerful distributed computing framework (the Spark engine implied in the figure), ETL jobs are performed on the raw data in the ODS layer, including data cleaning, format conversion, business logic calculation, data association, etc., and finally high-quality data that conforms to the data model specifications of each layer such as DWD, DWS, and ADS are produced.

[0134] 4. Storage layer and large-scale high-performance big data components The storage layer is the cornerstone of the entire architecture, consisting of the core large-scale, high-performance big data component of this invention. This component is designed as a unified storage engine, simultaneously carrying data from various layers processed by the processing layer, thus achieving the integration of data lake and data warehouse storage capabilities.

[0135] Unified storage: This component is built on a distributed file system or object storage, efficiently storing massive amounts of data in columnar format, and using data compression, partitioning and other technologies to optimize storage efficiency and query performance.

[0136] High-performance query: A built-in massively parallel processing analysis engine provides high-speed SQL query capabilities for the analysis layer. It directly interfaces with the data produced by the processing layer, avoiding redundant data migration between different storage systems.

[0137] Lake warehouse integration features: This component not only has the flexibility of storing raw and diverse data in a data lake, but also provides high-performance querying and strong data governance capabilities (such as transaction support and data version management) like a data warehouse, truly realizing lake warehouse integration.

[0138] The system workflow is as follows: The data flow follows a bottom-up direction: data from various data sources is extracted in real time or in batches through tools such as Flink CDC in the access layer; after entering the system, it is first stored in the ODS area of ​​the processing layer; then, after a series of ETL processes, it forms layered model data such as DWD, DWS, and ADS, and is uniformly stored in the underlying large-scale high-performance big data components; finally, various tools and interfaces in the analysis layer directly access the storage layer through standard protocols to query, analyze, and visualize the processed data.

[0139] This embodiment is illustrated by... Figure 2 The layered architecture shown decouples and organically integrates data access, processing, storage, and analysis capabilities. Its core advantage lies in using large-scale, high-performance big data components as a unified storage and computing foundation, eliminating the technical barriers between data lakes and data warehouses, simplifying the data architecture, and ensuring the timeliness of data processing, the standardization of data models, and the efficiency and flexibility of upper-level analysis.

[0140] This invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the functions of the integrated lake and warehouse data processing system provided in this invention.

Claims

1. A lake-warehouse integrated data processing system, characterized in that, It includes a data access layer, a unified storage engine layer, a data processing layer, and a data service layer; the data access layer, data processing layer, and data service layer are all connected to the unified storage engine layer; the unified storage engine layer includes an object storage unit, a core storage engine unit, and a unified metadata service. The data access layer is configured to access data from multiple heterogeneous data sources; The object storage unit is configured to persistently store the initial data accessed from the data access layer in the original data format, forming the original data area; The core storage engine unit adopts a massively parallel processing architecture and is configured to manage the refined data area through an incremental data snapshot mechanism. The incremental data snapshot mechanism creates snapshots containing timestamp identifiers and corresponding data file sets for data updates to support querying the historical state of data based on time points or version numbers. The refined data area is used to store data processed by the data processing layer. The processed data is organized in a columnar storage format, compressed, and partitioned by time or business key. The unified metadata service is configured to uniformly manage the first metadata in the original data area and the second metadata in the refined data area, and record the derivation relationship and structural mapping between the first metadata and the second metadata; The data processing layer is configured to extract data from the raw data area, perform transformation and loading operations, and write the processed data into the refined data area. It also integrates a data quality verification module, which is used to execute data verification, cleaning and quality monitoring rules during the data processing process. The data service layer is configured to provide standardized SQL query interfaces and application programming interfaces, and to provide a unified data access interface, enabling querying and analysis of data in the raw data area and the refined data area; The unified metadata service also maintains a global data lineage graph, which specifically includes: Data asset nodes are used to represent data tables or data files in the raw data area and refined data area; Process nodes represent transformation operations performed by the data processing layer; Bloodline edges are used to connect related data asset nodes and processing nodes to form a complete data flow path; Furthermore, the global data lineage map also records the conversion operation identifier, execution timestamp, and data quality verification result score for each processing node; The data service layer also provides data virtualization capabilities, the specific implementation process of which is as follows: Provides a logical view definition interface, allowing users to define a join query view across the raw data area and the refined data area using SQL statements; During query execution, the query optimizer resolves the logical view into subquery plans for different underlying data areas; By executing subqueries in parallel through a data federated query engine and performing join and aggregation operations on intermediate results in memory, a unified result set is finally returned, enabling cross-region joint queries without the need for physical data movement.

2. The system according to claim 1, characterized in that, The data access layer is specifically implemented using the Apache Flink framework; The Apache Flink framework is configured as follows: Data is extracted in parallel from the multiple heterogeneous data sources through a unified source interface; During the data access process, the consumption offset and state information of the data stream are persistently stored in the checkpoint mechanism to achieve an exactly-once semantic guarantee. Streaming data and batch data are processed through the data stream API and dataset API respectively, and the processing results are output to the unified storage engine layer.

3. The system according to claim 1, characterized in that, The core storage engine unit also employs a hybrid concurrency control mechanism, which specifically includes: Assign a transaction timestamp to each data read operation and read based on the latest committed data snapshot version to implement optimistic locking based on version snapshot; For data write and metadata change operations, a pessimistic locking mechanism based on two-phase locking is adopted to lock the relevant data partitions or metadata entries during operation execution. It also provides a conflict detection mechanism that automatically aborts and retryes the relevant transaction when a read / write conflict is detected.

4. The system according to claim 1, characterized in that, The incremental data snapshot mechanism also supports an automatic snapshot cleanup strategy, which specifically includes: Maintain a snapshot lifecycle management strategy table to record the creation time, data file size, and retention priority of each snapshot; Based on the time retention policy, snapshots that have exceeded the preset retention period are automatically marked as cleanable. According to the storage space policy, when the used storage space exceeds the threshold, snapshots are automatically cleaned up in order of creation time from earliest to latest until the storage space utilization rate is lower than the preset threshold. During the cleanup process, snapshot dependencies are maintained to ensure that data files that other snapshots depend on are not cleaned up.

5. The system according to claim 1, characterized in that, The data quality verification module supports a configurable data quality rule base, including integrity rules, consistency rules, accuracy rules, and timeliness rules; The integrity rule is used to verify whether the non-empty rate of the required field reaches the threshold. The consistency rules are used to verify whether data values ​​conform to a predefined range of values ​​or format specifications. The accuracy rules are used to verify the accuracy of the data by comparing it with authoritative data sources; The timeliness rules are used to verify whether the delay from data generation to access to the system is within the allowable range; Furthermore, the data quality verification module is configured to automatically repair the data by issuing real-time alarms, interrupting tasks, or calling preset data repair scripts when a rule is triggered, based on the severity level of the rule.

6. The system according to claim 1, characterized in that, The core storage engine unit supports a data hot caching layer, which specifically includes: The cache management module identifies hot data partitions in the refined data area based on the LRU-K algorithm; The cache storage area on a high-speed storage device is a columnar storage block used to store hot data partitions; A cache consistency mechanism ensures that when the underlying data is updated, the corresponding cached data is automatically invalidated or updated. The query execution engine prioritizes reading data from the cache storage area, and only accesses the main storage area when the cache is not hit.

7. The system according to claim 1, characterized in that, The system also integrates a data security module, which provides column-level encryption, dynamic data desensitization, and role-based access control capabilities for data in the unified storage engine layer; The column-level encryption is used to encrypt and store sensitive data columns using the AES-256 encryption algorithm, and the key is managed by an independent key management service; The dynamic data anonymization is used to partially hide or replace sensitive data in real time before the query results are returned, based on the visitor's role. The role-based access control is used to maintain a role permission table, define the access permissions of different roles to data areas, data tables, and data columns, and perform permission verification before query execution.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the functions of the integrated lake and warehouse data processing system as described in any one of claims 1-7.

Images