A dynamic hierarchical storage method and system based on lake-warehouse integrated architecture

Abstract

The application discloses a kind of dynamic hierarchical storage method and system based on lake warehouse integrated architecture, it is related to dynamic hierarchical storage technical field, including: data management layer: based on Apache Doris's Catalog, Database and Table three-layer logical structure;Physical storage layer: including cache layer, intermediate buffer layer and persistent layer;The cache layer is made of NVRAM / NVDIMM, is carried on the BE / FE node of Doris;The intermediate buffer layer is made of the Burst Buffer system of NVMe SSD construction;The persistent layer includes SSD array and disk array, for low-cost large-capacity data retention;Computing engine layer: integration Seatunnel, Spark and Flink.The technical problem to be solved in the application is to provide a kind of dynamic hierarchical storage method and system based on lake warehouse integrated architecture, fusion big data commonly used computing engine such as seatunnel, Flink, Spark, through the definition of hot spot data, the label definition of structured data and unstructured data, and the hierarchical storage and automatic migration of lake warehouse data are realized by data attribute.

Description

Technical Field

[0001] This invention relates to the field of dynamic hierarchical storage technology, and more specifically, to a dynamic hierarchical storage method and system based on a lakeware architecture. Background Technology

[0002] With the rapid development of big data technology, enterprise data is experiencing explosive growth, and traditional single data storage solutions can no longer meet the demands. Data lakes and data warehouses, as two mainstream data storage and processing methods, each have their own advantages and limitations. Data lakes excel at storing massive amounts of unstructured and semi-structured data, but lack efficient transaction processing capabilities and data analysis performance; while data warehouses perform exceptionally well in structured data processing and high-performance queries, but are less scalable and flexible. Therefore, the lake-warehouse integrated architecture has emerged, aiming to combine the advantages of both to provide unified data management and analysis capabilities.

[0003] Lakewarehouse architecture integrates diverse big data sources to meet enterprises' needs for storing, analyzing, and managing multi-source heterogeneous data. In traditional architectures, data lakes primarily store large-scale unstructured and semi-structured data at low cost, suitable for exploratory analysis, but lack efficient transaction processing capabilities and data governance mechanisms. Data warehouses, on the other hand, excel at handling high-value structured data, supporting complex queries and high-performance analysis, but have limited scalability and flexibility. Lakewarehouse storage technology effectively solves these problems through unified metadata management, separation of computation and storage, ACID transaction support, and compatibility with multiple computing engines.

[0004] Its core features include:

[0005] 1. Unified storage format: Adopting open file formats (such as Parquet, ORC) and a unified metadata layer to ensure seamless data flow between different tools and platforms.

[0006] 2. Elastic scalability: Based on a distributed architecture, it enables independent expansion of storage and computing resources to meet the needs of data scales from GB to PB.

[0007] 3. Multi-workload support: Simultaneously supports diverse tasks such as batch processing, stream processing, interactive querying, and machine learning, improving data utilization.

[0008] 4. Strong consistency and data governance: Introduce transaction logs and version control mechanisms to ensure data consistency, and provide fine-grained permission management and auditing functions.

[0009] Because different types of data have varying access frequencies, importance, and lifecycle characteristics, storing all data uniformly on high-performance storage media is neither economical nor efficient. A common practice is to distribute hot data (frequently accessed), warm data (occasional access), and cold data (long-term archiving) across different types of storage media (such as memory, SSDs, HDDs, or object storage), achieving a balance between performance and cost. This mechanism not only improves the overall efficiency of the system but also significantly reduces storage costs for enterprises.

[0010] Meanwhile, the maturity of distributed file systems and object storage technologies provides reliable technical support for lakehouse storage, ensuring data storage across different media. Through tiered storage strategies, the system can automatically identify data access patterns and migrate them to the most suitable storage tier, thereby optimizing resource utilization. Furthermore, this intelligent management approach reduces the need for manual intervention, improves operational efficiency, and provides more flexible and reliable underlying support for large-scale data analysis scenarios. However, current distributed file systems and object storage suffer from significant compatibility issues when integrated with big data tools. Differences exist between them and the native requirements of big data processing frameworks (such as Hadoop and Spark) in core aspects such as data access interfaces, metadata management models, and storage architecture design. This leads to reduced data flow efficiency, increased resource scheduling complexity, and difficulty in fully leveraging the parallel computing capabilities and efficient analytical features of big data tools, thus hindering the overall effectiveness of data value mining.

[0011] Driven by emerging technologies such as cloud computing and edge computing, the application scenarios for lake-warehouse integrated storage architectures are further expanding. Cloud-native technologies enable storage resources to scale elastically according to demand, while edge computing reduces latency through localized storage and processing.

[0012] Lakeware warehousing fully leverages cloud-native features to enable elastic scaling of storage systems: dynamically adjusting storage capacity and computing resources to adapt to changes in business workloads; supporting multi-tenant isolation to meet the needs of enterprise users; and providing automated operation and maintenance tools to reduce manual intervention and improve system availability.

[0013] Lakeware storage still has the following drawbacks in application expansion: high dependence on cloud-native technology stack, resulting in high enterprise migration and maintenance costs; complex multi-tenant isolation mechanisms, which may lead to resource contention and data security risks; limited ability of automated operation and maintenance tools to handle abnormal scenarios, and manual intervention is still required for complex faults; consistency issues are prone to occur when synchronizing local storage and cloud data in edge computing scenarios, increasing the difficulty of data management; resource configuration fluctuations may occur during elastic scaling, affecting system stability and service continuity.

[0014] The main shortcomings of existing technologies are as follows:

[0015] 1) Traditional data lakes and data warehouses lack flexibility when processing multi-source heterogeneous data, making it difficult to meet enterprises' dual needs for high performance and low cost at the same time.

[0016] 2) Existing tiered storage mechanisms often rely on static rules and cannot be dynamically adjusted according to actual access patterns, which leads to unreasonable resource allocation and performance bottlenecks.

[0017] 3) Data cleaning and archiving processes usually require manual intervention, which is inefficient and prone to errors.

[0018] 4) Distributed file systems and object storage have significant compatibility issues when combined with big data tools.

[0019] 5) Current storage utilization of local high-speed hard drives is low;

[0020] 6) In the existing solutions, the task definition languages ​​and configuration formats of Seatunnel, Spark, and Flink are completely independent: Seatunnel requires writing config.conf configuration files (based on YAML); Spark requires writing Scala / Python code (such as SparkSQL scripts); Flink requires writing Flink SQL or Java code; users need to master multiple sets of syntax, and task configurations cannot be reused (such as the migration logic of "MySQL→Hive", which needs to be rewritten in Spark and Flink), resulting in low standardization and high maintenance costs.

[0021] To address these issues, a dynamic tiered storage method and system based on a lakeware architecture is currently lacking. The aim is to provide an intelligent and automated solution that optimizes resource utilization through dynamic tiered storage strategies while enhancing system performance by combining local caching and prefetching technologies. The ultimate goal is to build a highly efficient, stable, and economical lakeware architecture that provides more powerful data management capabilities. Summary of the Invention

[0022] The technical problem to be solved by this invention is to provide a dynamic hierarchical storage method and system based on a lake warehouse integrated architecture, which integrates commonly used big data computing engines such as SeaTunnel, Flink, and Spark, and realizes hierarchical storage and automatic migration of lake warehouse data through the definition of hot data, the label definition of structured and unstructured data, and data attributes.

[0023] The present invention achieves its objective by employing the following technical solution:

[0024] A dynamic hierarchical storage method and system based on a lakeware architecture, characterized by comprising:

[0025] Metadata management layer: Based on the three-layer logical structure of Catalog, Database and Table of Apache Doris;

[0026] Physical storage layer: includes cache layer, intermediate buffer layer and persistent layer;

[0027] The cache layer consists of NVRAM / NVDIMM and is mounted on the BE / FE nodes of Doris;

[0028] The intermediate buffer layer consists of a Burst Buffer system built from NVMe SSDs;

[0029] The persistence layer includes SSD arrays and disk arrays, used for low-cost, high-capacity data retention;

[0030] Computing engine layer: Integrating Seatunnel, Spark and Flink, it builds a data migration system compatible with multiple scenarios, and realizes multi-engine collaborative scheduling through a unified adaptation layer;

[0031] Hierarchical control module: Monitors the access frequency of tables through distributed Catalog service, and automatically migrates frequently accessed tables to the cache layer or intermediate buffer layer according to the preset hot data definition, while low-frequency accessed tables are moved down to the persistence layer.

[0032] As a further limitation of this technical solution, the real-time synchronization mechanism between Catalog metadata and physical storage includes:

[0033] Low-latency updates and high reliability of Catalog metadata are ensured through persistent memory using NVRAM / NVDIMM;

[0034] Migration log recording at the table level enables data consistency verification and rollback across storage layers.

[0035] As a further limitation of this technical solution, the unified adaptation layer includes:

[0036] Standardized interface design, supporting dynamic selection of Seatunnel, Spark, or Flink as the migration tool;

[0037] The resource adaptation module automatically allocates computing resources based on data type, scale, and migration timeliness requirements.

[0038] As a further limitation of this technical solution, the unified adaptation layer also includes:

[0039] A standardized configuration abstraction layer and a unified API interface are implemented. A configuration abstraction model independent of specific engines is designed, mapping the unique parameters of tools like Seatunnel, Spark, and Flink to a unified configuration template. This enables one-time configuration for multiple engines. It overcomes the limitation of existing technologies requiring independent configuration for different engines, significantly lowering the barrier to entry for users and allowing business personnel to focus on data value mining rather than technical detail adaptation.

[0040] As a further limitation of this technical solution, the reliability assurance mechanism during the data migration process includes:

[0041] Field-level comparison and integrity verification, leveraging the metadata management capabilities of each engine to achieve multi-dimensional data consistency verification;

[0042] An exception retry strategy automatically restarts tasks that fail to migrate and logs the errors.

[0043] As a further limitation of this technical solution, in the hierarchical control module:

[0044] Access frequency calculation formula:

[0045] (1);

[0046] in: The access frequency of the Table within the current time window is the table access frequency calculated within the t-th statistical time window.

[0047] The access frequency of the previous time window is the Table access frequency calculated within the (t-1)th statistical time window, which is used to combine the time decay factor to achieve dynamic popularity assessment.

[0048] This is a time decay factor used to dynamically adjust the impact of historical access frequency on current popularity (to solve the problem of misjudgment due to instantaneous high frequency). Its value range is [0,1]. The closer it is to 1, the greater the impact of historical access frequency.

[0049] This represents the number of visits within the t-th time window;

[0050] Let t be the duration of the t-th time window;

[0051] Storage hierarchy decision formula:

[0052] (2);

[0053] Where: Th1 is the cache layer threshold, a preset critical value for the access frequency of hot data. When the access frequency of the Table is greater than or equal to Th1, it is determined to be high-frequency hot data and needs to be migrated to the cache layer.

[0054] Th2 is the threshold of the intermediate buffer layer, a preset critical value for the access frequency of low-to-medium frequency data. Data that is determined to be "medium frequency data" needs to be migrated to the intermediate buffer layer.

[0055] Th3 is a preset threshold for low-frequency data access in the persistence layer. When the table access frequency is less than or equal to Th3, it is considered low-frequency cold data and needs to be moved down to the persistence layer (such as HDD arrays or object storage service S3).

[0056] This is a hierarchical correction coefficient, a threshold correction factor based on the amount of data in the table, used to prevent very large tables from consuming too much cache resources;

[0057] L represents the storage level to which the output data should belong, with values ​​of {1,2,3}.

[0058] A storage method for a dynamic hierarchical storage system based on a lake warehouse architecture includes the following steps:

[0059] S1: Build the foundation of the lake warehouse integrated computing engine, integrating the Doris engine as the core. The Doris engine includes frontend nodes and backend nodes;

[0060] S2: Define data hierarchical attributes, including:

[0061] Hotspot data: High-frequency access data determined based on access frequency and real-time requirements;

[0062] Labels for structured and unstructured data: Set classification labels based on data format characteristics;

[0063] S3: Build a multi-level physical storage layer;

[0064] S4: Establish a mapping relationship between logical structure and physical storage based on the Doris metadata management layer;

[0065] S5: Perform dynamic migration based on the access frequency of the table: Monitor the access frequency of the table through the distributed Catalog service, migrate the hot table data with high frequency access to the high-speed cache layer or intermediate buffer layer, and migrate the cold table data with low frequency access to the persistence layer. During the migration process, the Catalog metadata and physical storage are kept synchronized in real time through a unified namespace.

[0066] The steps for standardizing the configuration abstraction layer and unifying the API interface are as follows:

[0067] S6: Unified Migration Task DSL (Domain-Specific Language): Designed as an engine-independent declarative DSL, users only need to describe: source end - target end - migration rules, without needing to care about the underlying engine;

[0068] S7: DSL parsing and engine code generation module;

[0069] The unified DSL is parsed, and the corresponding executable code / configuration is automatically generated based on the scheduling decision results of the configuration abstract model that is independent of the specific engine.

[0070] If scheduling Spark: Generate SparkSQL scripts (including logic for reading from MySQL, filtering, and writing to Hive);

[0071] If scheduling with Flink: Generate Flink SQL (including CDC real-time reading and state management logic);

[0072] If scheduling Seatunnel: Generate a YAML configuration file (including source data collection and intermediate storage writing logic);

[0073] S8: Task Configuration Template Library: Common migration scenarios (such as MySQL→Hive, Kafka→ClickHouse) are encapsulated into standardized templates, which users can reuse by simply modifying the source / target address.

[0074] Compared with related technologies, the dynamic hierarchical storage method and system based on lake warehouse architecture provided by this invention has the following beneficial effects:

[0075] (1) Lakeware hierarchical storage architecture driven by metadata: The core driving force is the three-layer metadata structure of Doris Catalog, Database and Table, which realizes the dynamic binding between logical structure and physical storage. Catalog maintains the mapping relationship between metadata and cross-media storage in real time, Database ensures global consistency in multi-tenant scenarios through namespace isolation, and Table automatically migrates between storage layers according to access frequency;

[0076] (2) Construct a four-level heterogeneous storage system that balances performance and cost: Construct a three-dimensional architecture of "NVRAM / NVDIMM high-speed cache layer - NVMe SSDBurst Buffer intermediate buffer layer - SSD / disk array persistence layer". The three layers work together to balance performance, capacity and cost, and adapt to the storage needs of EB-level data scale;

[0077] (3) Multi-engine integration full-scenario migration capability: Deeply integrate mainstream big data engines, build a unified adaptation and intelligent scheduling migration framework, and realize the automatic selection and collaborative scheduling of tools through a unified adaptation layer, solving the pain point of insufficient adaptability of single tools in complex data environments;

[0078] (4) Deeply integrate tools such as Seatunnel, Spark, and Flink to build a standardized data migration system compatible with multiple scenarios and solve the problem of insufficient adaptability of a single tool. Implement a data migration system that supports the collaborative work of multiple computing engines (Seatunnel, Spark, Flink), including a unified adaptation layer to abstract the interfaces of each engine, and automatically select the execution engine and schedule migration tasks according to the data source type, data volume and migration SLA. Attached Figure Description

[0079] Figure 1 This invention relates to a lake-warehouse integrated multi-level hierarchical storage architecture.

[0080] Figure 2 This is the Doris hierarchical storage driver process of the present invention.

[0081] Figure 3 This invention provides a standardized configuration abstraction layer and a unified API interface process. Detailed Implementation

[0082] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0083] A dynamic tiered storage system based on a lakeware architecture achieves efficient tiered data management through multi-layered storage collaboration and multi-engine adaptation. The system first establishes a dynamic mapping relationship between the physical storage layer and the logical structure based on Doris's three-layer metadata structure: Catalog, Database, and Table. The Catalog maintains the association between user permissions, table structure, and physical data in real time; the Database provides namespace isolation to support multi-tenant management; and the Table serves as the basic unit of tiered storage. During the data migration phase, the system performs fine-grained monitoring of Tables through a distributed Catalog service based on preset hot data definitions and access frequency analysis strategies, automatically identifying hot data (high-frequency accessed Tables) and cold data (low-frequency accessed Tables). For hot data, the system quickly locates and caches it in a Burst Buffer layer composed of NVMeSSDs using NVRAM / NVDIMM persistent memory, improving OLAP query performance; for cold data, it automatically migrates to low-cost storage media such as disk arrays. The migration process deeply integrates computing engines such as Seatunnel, Spark, and Flink: Seatunnel handles efficient synchronization of structured data, Spark processes large-scale unstructured data transformation, and Flink supports incremental migration of real-time data. The system achieves multi-engine collaborative scheduling through a unified adaptation layer, automatically selecting the optimal tool combination based on data source type, data volume, and migration timeliness requirements, and ensuring data consistency through a metadata verification mechanism during the migration process. This includes:

[0084] Metadata Management Layer: Based on the three-layer logical structure of Apache Doris: Catalog, Database, and Table; Catalog serves as the core metadata hub, storing the dynamic mapping relationship between user permissions, table structure, and physical data; Database provides namespace isolation to support multi-tenant management; Table serves as the smallest data storage unit, supporting automatic migration based on access frequency.

[0085] Apache Doris is a high-performance, real-time analytical database based on an MPP architecture.

[0086] Catalog: The core component for unified management of internal table metadata and external data source metadata. It enables cross-source federated queries without data migration and supports automatic metadata synchronization and unified access control.

[0087] Database: A logical namespace and table container used to classify, organize, and isolate tables (including internal and external tables).

[0088] A table is the basic unit for storing and organizing data. It belongs to a database (logical container) and is associated with a corresponding catalog.

[0089] NVRAM: Non-volatile random access memory.

[0090] NVDIMM: Non-volatile dual in-line memory modules.

[0091] Seatunnel, Flink, and Spark are open-source tools focused on data integration and synchronization, supporting data migration and transformation between multiple data sources.

[0092] Doris provides users with an excellent lakeware solution through its scalable connector framework, in-memory / compute separation architecture, high-performance data processing engine, and open data ecosystem. Doris's scalable connector framework supports access to mainstream data systems and formats, such as Hive, Iceberg, Hudi, and Paimon, and provides unified data analysis capabilities based on SQL. Apache Doris's in-memory / compute architecture is streamlined and easy to maintain. It includes the following two types of processes:

[0093] Frontend (FE): Primarily responsible for receiving user requests, query parsing and planning, metadata management, and node management.

[0094] Backend (BE): Primarily responsible for data storage and query plan execution. Data is split into data shards and stored in the BE using multiple replicas.

[0095] Physical storage layer: includes cache layer, intermediate buffer layer and persistent layer;

[0096] The cache layer consists of NVRAM / NVDIMM and is mounted on the BE / FE nodes of Doris. It utilizes its low latency characteristics to quickly respond to front-end read and write requests.

[0097] The intermediate buffer layer consists of a Burst Buffer system built on NVMe SSDs. As an intermediate buffer layer, it undertakes data transfer, adapts to the high-concurrency data transfer requirements, and ensures data transmission efficiency.

[0098] The persistence layer includes an SSD array (for storing metadata) and a disk array (for storing cold data), used for low-cost, high-capacity data retention;

[0099] Metadata storage relies on SSD arrays to ensure efficient retrieval, while large-capacity data storage uses disk arrays to handle massive amounts of cold data, balancing storage costs and data retention requirements. Through layered collaboration, efficient storage and flexible scheduling of data are achieved under the lake warehouse integration.

[0100] The computing engine layer (BE / FE nodes) relies on the Doris engine to process real-time queries, the cache layer (NVMe SSD BurstBuffer) quickly locates hot table data through catalog mapping, and the persistence layer (SSD / disk array) stores cold table data. The three layers achieve real-time synchronization of catalog metadata and physical storage through a unified namespace. Combined with an intelligent hierarchical strategy based on table access frequency, the flow path of hot data (high-frequency tables) to the cache layer and cold data (low-frequency tables) to the persistence layer is dynamically optimized.

[0101] This system innovatively uses the Doris metadata hierarchy as the core driver of tiered storage. The metadata mapping relationships of the Catalog guide the cross-tier migration of table data in real time, while the database's namespace management ensures global consistency and conflict avoidance of the table structure across multiple tiers. The system employs a distributed Catalog service for fine-grained monitoring and tiered decision-making on tables. Hot tables are preferentially housed in the NVMe SSD cache layer to improve OLAP performance, while cold tables are automatically migrated to low-cost disk arrays. Simultaneously, persistent memory using NVRAM / NVDIMM ensures highly reliable and low-latency updates of Catalog metadata. This design not only solves the migration overhead problem caused by the separation of metadata and storage in traditional architectures but also enhances the elastic scalability of resources across multiple business scenarios through the database's isolation mechanism. At EB-level data scales, it significantly reduces storage costs by more than 30% and improves query efficiency.

[0102] The computing engine layer integrates Seatunnel, Spark, and Flink to build a data migration system compatible with multiple scenarios. A unified adaptation layer enables collaborative scheduling of multiple engines, including:

[0103] Seatunnel is used for efficient synchronization of structured data;

[0104] Spark is used for distributed transformation and migration of unstructured data;

[0105] Flink is used for incremental migration and dynamic synchronization of real-time data.

[0106] This solution deeply integrates commonly used big data platform tools such as Seatunnel, Spark, and Flink to build a data migration system compatible with multiple scenarios. Addressing the technical characteristics of different tools, the solution achieves standardized interface design through a unified adaptation layer: leveraging Seatunnel's lightweight data synchronization capabilities to meet the efficient transmission of structured data between multi-source heterogeneous storage; utilizing Spark's distributed computing framework to handle the transformation and migration of large-scale unstructured data, improving the parallel processing efficiency of massive amounts of data; and relying on Flink's stream processing advantages to support real-time incremental migration and dynamic synchronization of data, ensuring timeliness and consistency during the data migration process. Simultaneously, the solution utilizes a collaborative scheduling mechanism between tools to achieve automatic tool selection and resource adaptation in different migration scenarios, solving the problem of insufficient adaptability of a single tool in complex data environments.

[0107] The unified adaptation layer also includes a standardized configuration abstraction layer and a unified API interface. It designs a configuration abstraction model independent of specific engines, mapping the unique parameters of tools such as Seatunnel, Spark, and Flink to a unified configuration template, enabling one-time configuration for multi-engine operation. This overcomes the limitation of existing technologies requiring independent configuration for different engines, significantly lowering the barrier to entry for users and allowing business personnel to focus on data value mining rather than technical adaptation details.

[0108] This invention further optimizes the reliability and scalability of the entire data migration process. In the data verification stage, a multi-dimensional data consistency verification mechanism is established by combining the metadata management capabilities of various tools. Through field-level comparison, integrity verification, and anomaly retry strategies, the accuracy of migrated data is ensured. In terms of architecture design, a modular and component-based approach is adopted, supporting flexible expansion and version iteration of tool plugins. New data sources and target storage types can be quickly integrated according to business needs. Furthermore, the solution provides a visual task monitoring and management interface, displaying the real-time running status, migration progress, and resource usage of each tool. This facilitates end-to-end management by operations personnel, effectively reducing the technical barriers and operational costs of data migration in big data environments.

[0109] Hierarchical control module: Monitors the access frequency of tables through distributed Catalog service, and automatically migrates frequently accessed tables to the cache layer or intermediate buffer layer according to the preset hot data definition, while low-frequency accessed tables are moved down to the persistence layer.

[0110] The real-time synchronization mechanism between catalog metadata and physical storage includes:

[0111] Low-latency updates and high reliability of Catalog metadata are ensured through persistent memory using NVRAM / NVDIMM;

[0112] Migration log recording at the table level enables data consistency verification and rollback across storage layers.

[0113] The unified adaptation layer includes:

[0114] Standardized interface design, supporting dynamic selection of Seatunnel, Spark, or Flink as the migration tool;

[0115] The resource adaptation module automatically allocates computing resources based on data type, scale, and migration timeliness requirements.

[0116] Reliability assurance mechanisms during data migration include:

[0117] Field-level comparison and integrity verification, leveraging the metadata management capabilities of each engine to achieve multi-dimensional data consistency verification;

[0118] An exception retry strategy automatically restarts tasks that fail to migrate and logs the errors.

[0119] In the hierarchical control module:

[0120] Access frequency calculation formula:

[0121] (1);

[0122] in: The access frequency of the Table within the current time window is the table access frequency calculated within the t-th statistical time window.

[0123] The access frequency of the previous time window is the Table access frequency calculated within the (t-1)th statistical time window, which is used to combine the time decay factor to achieve dynamic popularity assessment.

[0124] This is a time decay factor used to dynamically adjust the impact of historical access frequency on current popularity (to solve the problem of misjudgment due to instantaneous high frequency). Its value range is [0,1], and the closer it is to 1, the greater the impact of historical access frequency.

[0125] This represents the number of visits within the t-th time window;

[0126] Let t be the duration of the t-th time window;

[0127] Storage hierarchy decision formula:

[0128] (2);

[0129] Where: Th1 is the cache layer threshold, a preset critical value for the access frequency of hot data. When the access frequency of the Table is greater than or equal to Th1, it is determined to be high-frequency hot data and needs to be migrated to the cache layer.

[0130] Th2 is the threshold of the intermediate buffer layer, a preset critical value for the access frequency of low-to-medium frequency data. Data that is determined to be "medium frequency data" needs to be migrated to the intermediate buffer layer.

[0131] Th3 is a preset threshold for low-frequency data access in the persistence layer. When the table access frequency is less than or equal to Th3, it is considered low-frequency cold data and needs to be moved down to the persistence layer (such as HDD arrays or object storage service S3).

[0132] This is a hierarchical correction coefficient, a threshold correction factor based on the amount of data in the table, used to avoid excessive cache resources being consumed by very large tables;

[0133] L represents the storage level to which the output data should belong, with values ​​of {1,2,3}.

[0134] A storage method for a dynamic hierarchical storage system based on a lake warehouse architecture includes the following steps:

[0135] S1: Build the foundation of the lake warehouse integrated computing engine, integrating the Doris engine as the core. The Doris engine includes frontend (FE) nodes and backend (BE) nodes. The FE nodes are responsible for receiving user requests, query parsing and planning, metadata management and node management, while the BE nodes are responsible for data storage and query plan execution.

[0136] S2: Define data hierarchical attributes, including:

[0137] Hotspot data: High-frequency access data determined based on access frequency and real-time requirements;

[0138] Labels for structured and unstructured data: Set classification labels based on data format characteristics;

[0139] S3: Build a multi-level physical storage layer;

[0140] S4: Establish a mapping relationship between logical structure and physical storage based on the Doris metadata management layer;

[0141] S5: Perform dynamic migration based on the access frequency of the table: Monitor the access frequency of the table through the distributed Catalog service, migrate the hot table data with high frequency access to the high-speed cache layer or intermediate buffer layer, and migrate the cold table data with low frequency access to the persistence layer. During the migration process, the Catalog metadata and physical storage are kept synchronized in real time through a unified namespace.

[0142] The steps for standardizing the configuration abstraction layer and unifying the API interface are as follows:

[0143] S6: Unified Migration Task DSL (Domain-Specific Language): Designs an engine-independent declarative DSL where users only need to describe: source end - target end - migration rules, without needing to concern themselves with the underlying engine. For example:

[0144] -- Unified DSL Example: Full Migration of MySQL to Hive + Real-time Incremental Synchronization

[0145] MIGRATION_TASK {

[0146] SOURCE: mysql: / / user:pass@host:3306 / order_db.table_a

[0147] SINK: hive: / / host:10000 / ods_db.table_a

[0148] TYPE: BATCH+STREAM -- Batch and stream mixed type

[0149] RULES: -- Cleaning rules (not related to the engine)

[0150] FILTER: status='success' -- Filter successful orders

[0151] MAPPING: order_id→id, create_time→dt -- Field mapping

[0152] }

[0153] S7: DSL parsing and engine code generation module;

[0154] The unified DSL is parsed, and the corresponding executable code / configuration is automatically generated based on the scheduling decision results of the configuration abstract model that is independent of the specific engine.

[0155] If scheduling Spark: Generate SparkSQL scripts (including logic for reading from MySQL, filtering, and writing to Hive);

[0156] If scheduling with Flink: Generate Flink SQL (including CDC real-time reading and state management logic);

[0157] If scheduling Seatunnel: Generate a YAML configuration file (including source data collection and intermediate storage writing logic);

[0158] S8: Task Configuration Template Library: Common migration scenarios (such as MySQL→Hive, Kafka→ClickHouse) are encapsulated into standardized templates, which users can reuse by simply modifying the source / target address.

[0159] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A dynamic hierarchical storage system based on lake-warehouse integrated architecture, characterized in that, The application relates to a data migration system based on a lake warehouse integration computing engine, which comprises the following steps. A metadata management layer: a three-layer logical structure of Catalog, Database and Table based on Apache Doris; A physical storage layer: comprising a cache layer, an intermediate buffer layer and a persistent layer; The cache layer is composed of NVRAM / NVDIMM and is mounted on the BE / FE node of Doris; The intermediate buffer layer is composed of a Burst Buffer system constructed by NVMe SSD; The persistent layer comprises an SSD array and a disk array and is used for low-cost large-capacity data storage; A computing engine layer: integrating Seatunnel, Spark and Flink, a multi-scene compatible data migration system is constructed, and multi-engine cooperative scheduling is realized through a unified adaptation layer; A hierarchical control module: the access frequency of a Table is monitored through a distributed Catalog service, and high-frequency accessed Tables are automatically migrated to the cache layer or the intermediate buffer layer according to preset hot data definitions, and low-frequency accessed Tables are sunk to the persistent layer; The unified adaptation layer comprises the following: A standardized interface design supports dynamically selecting Seatunnel, Spark or Flink as a migration tool; A resource adaptation module automatically allocates computing resources according to data types, scales and timeliness requirements of migration; The unified adaptation layer further comprises the following: A standardized configuration abstraction layer and a unified API interface, a configuration abstraction model independent of specific engines is designed, specific parameters of Seatunnel, Spark and Flink tools are mapped into a unified configuration template, and once configuration is realized, multi-engine operation is realized; A reliability guarantee mechanism in the data migration process comprises the following: Field-level comparison and integrity check: multi-dimensional data consistency verification is realized through the metadata management capability of each engine; An abnormal retry strategy automatically restarts the task of migration failure and records error logs.

2. The dynamic tiered storage system based on the lake-warehouse integrated architecture according to claim 1, characterized in that: A real-time synchronization mechanism of Catalog metadata and physical storage comprises the following: Through NVRAM / NVDIMM persistent memory, low-delay update and high reliability of Catalog metadata are ensured; Based on the Table granularity, migration log recording is realized, and data consistency verification and rollback across storage layers are realized.

3. The dynamic tiered storage system based on the lake-warehouse integrated architecture according to claim 1, characterized in that: In the hierarchical control module: An access frequency calculation formula: （1）； wherein: is the current time window access frequency; frequency of access for the previous time window; is a time decay factor; is the number of visits for the tth time window; is the duration of the t-th time window; A storage level decision formula: （2）； Wherein: Th1 is a cache layer threshold value; Th2 is an intermediate buffer layer threshold value; Th3 is a low-frequency data access frequency critical value preset for the persistent layer; Data quantity correction coefficient; L is a storage level to which output storage data should belong, and the value is {1, 2, 3}.

4. The storage method of the dynamic hierarchical storage system based on the lake-warehouse integrated architecture according to claim 1, characterized in that, The application further relates to a data migration system based on a lake warehouse integration computing engine. S1: constructing a lake warehouse integration computing engine foundation, integrating a Doris engine as a core, and the Doris engine comprising a Frontend node and a Backend node; S2: defining data hierarchical attributes, comprising the following: Hot data: high-frequency accessed data determined based on access frequency and real-time requirement; Labels of structured data and unstructured data: classification labels are set according to data format characteristics; S3: building a multi-level physical storage layer. S4: Establish the mapping relationship between logical structure and physical storage based on Doris metadata management layer; S5: Perform dynamic migration according to the access frequency of Table: Monitor the access frequency of Table through distributed Catalog service, migrate the hot Table data with high access frequency to cache layer or intermediate buffer layer, and migrate the cold Table data with low access frequency to persistent layer. During the migration process, keep the real-time synchronization of Catalog metadata and physical storage through unified namespace.

5. The storage method of claim 4, wherein: The steps of the standardized configuration abstraction layer and unified API interface are: S6: Unified migration task DSL: Design a declarative DSL independent of the engine. Users only need to describe: source - target - migration rules, without worrying about the underlying engine; S7: DSL parsing and engine code generation module; Parse the unified DSL, and automatically generate executable code / configuration for the corresponding engine according to the scheduling decision result of the configuration abstraction model independent of the specific engine; If scheduling Spark: Generate SparkSQL script; If scheduling Flink: Generate Flink SQL; If scheduling Seatunnel: Generate YAML configuration file; S8: Task configuration template library: Encapsulate common migration scenarios as standardized templates, and users only need to modify the source / target address to reuse.

Images