Data hierarchical storage method

Abstract

The invention relates to a data hierarchical storage method. The method comprises the following steps: layering a data storage architecture into a hot data layer, a warm data layer and a cold data layer; taking in data from the hot data layer at regular time, and loading the data to a DuckDB inner table of the temperature data layer; performing conversion processing on the data in the temperature data layer in the DuckDB to form cold standby target data; transferring the cold standby target data to the cold data layer; wherein the hot data acquisition and loading task, the temperature data conversion task and the data cold standby task are coded according to a Dagster task framework, codes are constructed and published as container mirror images, and task scheduling configuration, task triggering and task execution are performed. Performing data conversion and low-frequency query on the temperature data by using DuckDB; the cold data layer stores historical data in a Parquet format through object storage, and the requirements of low cost and efficient analysis are met. The hierarchical storage is combined with a cloud native task scheduling technology, so that the problems of complexity of traditional task scheduling and low efficiency of resource allocation are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a data hierarchical storage method. Background Art

[0002] As the business scale expands, the amount of data generated by enterprises grows exponentially. Most enterprise data storage starts with transactional business database systems such as MySQL and PostgreSQL. Such databases usually adopt a single data storage mode, that is, all data is stored in the same location. Although this approach is simple and easy, it has many problems as the business and data scale expand:

[0003] First, low resource utilization: a large amount of inactive or historical data occupies valuable computing and storage resources.

[0004] Second, query performance decreases: As the amount of data increases, the speed of real-time queries becomes slower, affecting user experience.

[0005] Third, maintenance operations are difficult: When a certain amount of data is accumulated, changes to the database table structure and addition and deletion of indexes will be extremely slow. In scenarios where business continuity is required, such maintenance operations are unacceptable.

[0006] Fourth, storage costs are high: To store large amounts of data, the database requires higher CPU and memory, and takes up huge disk space.

[0007] In order to solve the above problems, the industry has proposed a solution for tiered storage based on the hot and cold characteristics of data. That is, data is divided into two categories: hot data and cold data according to the access frequency, update frequency, business importance and usage scenarios of the data. Hot data is frequently accessed in the business system and requires real-time or fast response, so it is suitable for storage in high-performance business databases and caches. Cold data is less frequently accessed in the business system, but needs to be stored for a long time, so it is suitable for storage in the form of data files in a cold storage system. At the same time, the implementation of hot and cold tiered storage of data relies on the ELT technology component to ingest, load and transform the business source data, and finally form data that conforms to the target format for storage. However, existing technical solutions often use traditional ELT or adopt complex technology combinations, which have problems such as high complexity, low resource utilization, insufficient real-time query performance, and high cost. Summary of the invention

[0008] The purpose of the present invention is to provide a data hierarchical storage method, which aims to solve the technical problems of existing data storage solutions, such as high complexity, low resource utilization, insufficient real-time query performance and high cost.

[0009] In order to solve the above technical problems, a data hierarchical storage method is provided, comprising:

[0010] Layer the data storage architecture into hot data tiers, warm data tiers, and cold data tiers;

[0011] Hot data ingestion and loading tasks: regularly ingest data from the hot data layer and load it into the DuckDB internal table of the warm data layer;

[0012] Warm data conversion task: convert the data in the warm data layer in DuckDB to form cold standby target data;

[0013] Data cold backup task: transferring the cold backup target data to the cold data layer;

[0014] Among them, the hot data ingestion and loading tasks, the warm data conversion tasks and the data cold standby tasks are coded according to the Dagster task framework, the code is built and published as a container image, and task scheduling configuration, task triggering and task execution are performed.

[0015] Furthermore, the task scheduling configuration includes: configuring the triggering time conditions of the hot data ingestion and loading tasks, and the process of loading data into the warm data layer; configuring the triggering conditions of the warm data conversion task, converting the warm data to form Parquet data files; configuring the triggering conditions of the data cold standby task, transferring the Parquet data files from the warm data layer to the cold data layer.

[0016] Furthermore, a coordination node is set to uniformly manage the scheduling and distribution of data tasks. When a task is triggered, the coordination node selects the image corresponding to the task execution code as needed to start the Kubernetes Job.

[0017] Furthermore, the task triggering includes: a webserver request initiated through a UI.

[0018] Furthermore, the task triggering includes: initiating periodically according to a set timing scheduling rule.

[0019] Furthermore, the hot data ingestion and loading task triggering time condition is configured to start from the first time node, the data ingested within the first time period is loaded into the warm data layer, and the data of the warm data layer is stored in monthly partitions.

[0020] Furthermore, the warm data conversion task trigger condition is configured to complete the ingestion of each data partition, process and lightly summarize the original data ingested by the warm data layer to form cold standby target data, and store the cold standby target data by monthly partitions.

[0021] Furthermore, the data cold standby task triggering condition is configured to start from the second time node, and the cold standby target data of the warm data layer is copied to the object storage through DuckDB.

[0022] Furthermore, DuckDB is used as the storage engine of the warm data layer.

[0023] Furthermore, transferring the cold standby target data to the cold data layer includes: the data of the warm data layer is aggregated and summarized, and retained in the warm data layer for a preset time for processing and querying the warm data. After retaining the preset time, if the warm data meets the cold standby conditions, the warm data is transferred to the cold data layer.

[0024] Implementing the embodiments of the present invention will have the following beneficial effects:

[0025] The data tiered storage method in this embodiment tiers the data storage architecture into hot data layer, warm data layer and cold data layer. The hot data layer is mainly used for real-time access and transaction processing to ensure the efficient operation of the core business of the enterprise; the warm data layer is used as a buffer for hot data and cold data, using DuckDB for data conversion and low-frequency query; the cold data layer stores historical data in Parquet format through object storage to meet the needs of low-cost and efficient analysis. This data storage architecture can significantly optimize data access performance while reducing storage costs.

[0026] The data hierarchical storage method in this embodiment uses Dagster as the data task scheduling framework, and realizes elastic management and isolation of task resources by deploying a dynamic task allocation mechanism on Kubernetes. The specific method is: coordinate nodes to uniformly schedule tasks and start independent Kubernetes Job Pods to ensure efficient resource utilization and task status synchronization. This method combines flexibility and security, and realizes rapid iteration and automated operation and maintenance through containerized delivery.

[0027] By using columnar storage formats (such as Parquet) and lightweight query engines (such as DuckDB), cold data can be directly queried efficiently and complexly from object storage, significantly reducing analysis costs and response times. Compared with traditional storage solutions, the data tiered storage method in this embodiment breaks through the performance bottleneck of cold data queries, while improving the availability and management economy of historical data. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0029] Figure 1 A flowchart of a data hierarchical storage method according to an embodiment of the present invention;

[0030] Figure 2 The overall framework diagram of data hierarchical storage and data scheduling described in the embodiment of the present invention;

[0031] Figure 3 The data layering data flow diagram described in the embodiment of the present invention;

[0032] Figure 4 This is a data scheduling framework diagram according to an embodiment of the present invention;

[0033] Figure 5 This is a data scheduling flow chart described in an embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. The preferred embodiments of the present invention are given in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive.

[0035] It should be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be a central element. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be a central element at the same time. The terms "vertical", "horizontal", "left", "right" and similar expressions used herein are for illustrative purposes only.

[0036] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0037] like Figure 1-Figure 5 As shown, an embodiment of the present invention provides a data hierarchical storage method, including:

[0038] The data storage architecture is layered into a hot data layer, a warm data layer, and a cold data layer; illustratively, the hot data layer is used to store hot data, the warm data layer is used to store warm data, and the cold data layer is used to store cold data. As a transactional database for the company's existing business, the hot data layer directly provides reading, writing, and analysis of the company's hot data. The warm data layer is a buffer and transitional data layer between the company's hot data and cold data. Ingest data that meets the company's definition of cold data into the warm data layer. The warm data layer still supports the reading and writing of data. The data in the warm data layer is written and converted to the main data format, and supports relatively low-frequency real-time queries to the outside world. In the cold data layer, the data has become historical data, will not be changed, and is rarely read.

[0039] Specifically, in this embodiment, hot data is defined as data within the last two natural years. The access frequency in the business system is high, and real-time or fast response is required. It is suitable for storage in high-performance business databases and caches to support the efficient operation of core businesses. Warm data is defined as data before hot data is processed and transferred to cold data. The data access frequency is lower than that of hot data, but it still needs to support certain query and processing requirements. It is retained in the warm data layer for six months for data processing and query. After aggregation and aggregation, the warm data meets the cold standby conditions and is then transferred to the cold data layer. Cold data is defined as data before the last two natural years. The access frequency in the business system is low, but it needs to be preserved for a long time. It is suitable for storage in a low-cost cold storage system in the form of data files. In the present invention, object storage is selected as the storage backend, and warm data is converted into Parquet data file storage. Of course, in specific applications, the definitions of hot data, warm data, and cold data can be set according to actual needs and circumstances. For example, as an alternative, hot data is defined as data within the last natural year, and cold data is defined as data before the last natural year, and no excessive restrictions are made here.

[0040] Hot data ingestion and loading tasks: regularly ingest data from the hot data layer and load it into the DuckDB internal table of the warm data layer;

[0041] Warm data conversion task: convert the data in the warm data layer in DuckDB to form cold standby target data;

[0042] Data cold backup task: transfer the cold backup target data to the cold data layer;

[0043] Among them, the hot data ingestion and loading tasks, warm data conversion tasks, and data cold standby tasks are coded according to the Dagster task framework, and the code build is published as a container image, and task scheduling configuration, task triggering, and task execution are performed. For example, as a container orchestration platform, Kubernetes can dynamically allocate resources according to the needs of tasks. This means that when the task load increases, resources can be automatically expanded, and when the task load decreases, resources can be reduced, thereby improving resource utilization. Each Kubernetes Job runs in an independent container, which provides good isolation for the tasks. This isolation prevents resource conflicts between tasks and ensures that each task can be executed in a stable environment. The container image contains all the dependencies and environment configurations required for task execution, which enables tasks to run in any environment that supports Kubernetes, improving the portability of tasks.

[0044] The data tiered storage method in this embodiment tiers the data storage architecture into hot data layer, warm data layer and cold data layer. The hot data layer is mainly used for real-time access and transaction processing to ensure the efficient operation of the core business of the enterprise; the warm data layer is used as a buffer for hot data and cold data, using DuckDB for data conversion and low-frequency query; the cold data layer stores historical data in Parquet format through object storage to meet the needs of low-cost and efficient analysis. This data storage architecture can significantly optimize data access performance while reducing storage costs.

[0045] like Figure 2 As shown in the figure, the data sources of the hot data layer include transactional databases such as MySQL, TDSQL, and MongoDB. The data of the hot data layer is ingested from these databases through the ELT (Extract, Load, Transform) process. The cold data layer uses low-cost object storage, such as Tencent Cloud COS, Alibaba Cloud OSS, AWS S3, or MinIO. Figure 2 The "k8s" in is the abbreviation of Kubernetes, which is an open source container orchestration platform for automating the deployment, scaling, and management of applications. Kubernetes (k8s) serves as the underlying platform and supports the operation of Dagster, a data task scheduling framework. Dagster is a tool for building data pipelines that can run on Kubernetes and use the features of Kubernetes to manage and schedule data tasks. In this way, data tasks can dynamically allocate resources based on demand, improving efficiency and flexibility. "Ingest-Load-Transform", referred to as ELT, is a data processing process that is used to extract data from a source system, then load it into a target system, and perform necessary conversions during the loading process to meet the data format and structure requirements of the target system.

[0046] Cloud-native ELT has the following advantages in data pipeline development, data task scheduling, and execution:

[0047] First, tasks have good flexibility in the use of computing resources, and can dynamically increase or decrease computing resources according to the load of the task.

[0048] Second, data tasks run in independent Kubernetes Pods, ensuring resource isolation and security.

[0049] Third, data tasks are delivered as container images, integrated with CI / CD, and easy to update and release.

[0050] like Figure 3 As shown in the figure, business data ingestion is the starting point of the data tiered storage process, which involves obtaining data from the business database, that is, regularly ingesting data from the hot data layer. When loading data, the ingested data is loaded into the warm data layer, where the DuckDB internal table is used. Data conversion is in the warm data layer, which is the warm data conversion task mentioned above, and the original data is processed and lightly summarized to form the cold standby target data. Data movement refers to the movement of the converted data from the warm data layer to the cold data layer, which is the data cold standby task mentioned above. The data cold standby task is triggered at a specific time point (such as October 1) and involves using DuckDB's Copy method to transfer Parquet data files to object storage, such as Tencent Cloud COS. Finally, when it is necessary to query and analyze the data in the cold data layer, columnar storage formats (such as Parquet) and lightweight query engines (such as DuckDB) can be used to reduce query costs and improve response time. The entire process embodies a cloud-native ELT (Extract, Load, Transform) data pipeline development and data task scheduling mode, which aims to achieve efficient tiered storage of data while reducing costs and improving query performance. The arrows in the flowchart indicate the direction of data flow, starting from the business database, processed by the warm data layer, and finally reaching the object storage of the cold data layer.

[0051] like Figure 4 As shown, Figure 4A cloud-native ELT data task scheduling framework based on Kubernetes is presented, which includes key components such as web services, schedulers, task metadata services, execution jobs, and state databases, which work together to achieve hierarchical storage of data and efficient scheduling of tasks. Specifically, the web service is responsible for receiving webserver requests initiated by users through the UI. The web service is connected to the Scheduler, which is responsible for managing the scheduling and distribution of data tasks. There are three task metadata services in the figure, marked as A, B, and C, which communicate with the web service through gRPC. Representing different data tasks or jobs, they need to communicate with the web service to obtain task information or status updates. The scheduler is responsible for triggering the execution of these Job tasks. All components are connected to the state database to store structured execution event data.

[0052] like Figure 5 The flowchart shown describes an automated data task scheduling and execution process. Specifically, configuring trigger conditions is the starting step of data task scheduling, which involves setting the conditions for task triggering. When the trigger conditions are met, the task is activated. The coordination node is a component in the Dagster framework that is responsible for receiving triggered tasks. Once the task is received, the coordination node will be responsible for creating a pod in the Kubernetes cluster to execute this task. The task pod is a working unit in Kubernetes that will perform specific data tasks such as data ingestion, transformation, or cold standby. After the task is completed, its results will be written to the state database.

[0053] like Figure 2 and Figure 4 As shown, in one possible implementation, the task scheduling configuration includes: configuring the trigger time conditions for hot data ingestion and loading tasks, and the process of loading data into the warm data layer; configuring the trigger conditions for the warm data conversion task, converting the warm data to form a Parquet data file; configuring the trigger conditions for the data cold standby task, transferring the Parquet data file from the warm data layer to the cold data layer. Exemplarily, by using a columnar storage format (such as Parquet) and a lightweight query engine (such as DuckDB), cold data can perform efficient and complex queries directly from object storage, significantly reducing analysis costs and response times. Compared with traditional storage solutions, the data tiered storage method in this embodiment breaks through the performance bottleneck of cold data queries, while improving the availability and management economy of historical data.

[0054] like Figure 2 and Figure 4As shown, in one possible implementation, a coordination node is set to uniformly manage the scheduling and distribution of data tasks. When a task is triggered, the coordination node selects the image corresponding to the task execution code as needed to start the KubernetesJob. By way of example, the data hierarchical storage method in this embodiment uses Dagster as a data task scheduling framework, and implements elastic management and isolation of task resources by deploying a dynamic task allocation mechanism on Kubernetes. The specific method is: the coordination node uniformly schedules tasks and starts an independent Kubernetes Job Pod to ensure efficient resource utilization and task status synchronization. This method combines flexibility and security, and achieves rapid iteration and automated operation and maintenance through containerized delivery.

[0055] like Figure 2 and Figure 4 As shown, in a possible implementation, the task trigger includes: a webserver request initiated through a UI. Exemplarily, the specific process includes: a user interacts with the system through a graphical user interface (UI). The user performs specific operations in the UI, such as clicking a button or submitting a form. The front-end logic converts the user's interaction into an HTTP request. The constructed HTTP request is sent to the back-end webserver through the Internet or an internal network. After receiving the request, the webserver will process it according to the type and content of the request. Specifically, it involves triggering data tasks, such as data ingestion, conversion, or cold standby operations. The webserver forwards the request to the task scheduling system, and the coordination node selects the corresponding container image according to the specific requirements of the task, and starts a new Job or Pod on Kubernetes to perform the task. The containerized task begins to execute, including ingesting data from the hot data layer, processing data in DuckDB, or transferring data from the warm data layer to the cold data layer. After the task is executed, the result will be fed back to the webserver, and then the webserver returns the processing result to the UI.

[0056] In a possible implementation, task triggering includes: initiating a task periodically through a set timed scheduling rule. Exemplarily, the timed scheduling rule can be based on different time intervals, such as minutes, hours, days, or weeks, to accommodate different data processing requirements and business scenarios. In addition to time-driven scheduling, the system can also be configured to respond to specific events, for example, triggering a task when a specific amount of data is detected to accumulate or the database reaches a preset threshold.

[0057] In a possible implementation, the hot data ingestion and loading task trigger time condition is configured to start from the first time node, the data ingested in the first time period is loaded into the warm data layer, and the data in the warm data layer is stored in monthly partitions. Exemplarily, monthly partition storage can help manage warm data more effectively, making the storage, retrieval and maintenance of warm data more orderly and efficient. Specifically, the first time node is configured as April 1 of each natural year.

[0058] In one possible implementation, the warm data conversion task trigger condition is configured to complete the ingestion of each data partition, the raw data ingested by the warm data layer is processed and lightly summarized to form the cold standby target data, and the cold standby target data is stored in monthly partitions. Exemplarily, monthly partitioned storage can help manage cold standby target data more effectively, making data storage, retrieval, and maintenance more orderly and efficient. By storing in monthly partitions, data for a specific time period can be quickly located, thereby improving the performance of data query and access, especially for the analysis and audit of historical data. Partitioning can make the management of cold data in object storage more flexible, and help implement different storage strategies, such as moving old data to a lower-cost storage tier.

[0059] like Figure 2 and Figure 4 As shown, in a possible implementation, the trigger condition of the configuration data cold standby task is to start from the second time node, and the cold standby target data of the warm data layer is copied to the object storage through DuckDB. Exemplarily, the second time node is configured as October 1 of each natural year. The DuckDB Copy method supports data transmission of the standard S3 protocol. Any object storage product compatible with the S3 protocol can be used as a cold data storage backend, such as Tencent Cloud COS, Alibaba Cloud OSS, AWSS3 or MinIO.

[0060] like Figure 2 and Figure 3 As shown, in one possible implementation, DuckDB is used as the storage engine of the warm data layer. Exemplarily, DuckDB is a lightweight in-process analytical database with low storage requirements, which enables it to process large amounts of data with limited resources, and is suitable as a storage engine for the warm data layer. DuckDB can handle the processing and analysis of large amounts of data, which is conducive to the conversion and query operations of data in the warm data layer. In addition, DuckDB supports data conversion so that data can be stored in formats such as Parquet, in preparation for subsequent cold data storage. The warm data layer needs to support relatively low-frequency real-time queries to the outside world, and DuckDB can meet this requirement, so that the warm data layer can respond quickly to query requests. DuckDB supports data partitioning, which is conducive to organizing and managing warm data by month or other logic, and improving data retrieval efficiency.

[0061] In a possible implementation, transferring the cold standby target data to the cold data layer includes: aggregating and summarizing the data in the warm data layer, and retaining it in the warm data layer for a preset time for processing and querying the warm data. After retaining it for the preset time, if the warm data meets the cold standby condition, the warm data is transferred to the cold data layer. Exemplarily, in this embodiment, the preset time is six months. It should be noted that the preset time can be set according to actual needs. For example, as an alternative, the preset time can also be configured as one year, eight months, or three months. There is no excessive restriction here. In this embodiment, the cold standby condition can be set according to the timeliness of the data. For example, the data is retained in the warm data layer for a certain period of time (such as six months) to support business query and analysis. Beyond this time limit, the real-time demand for the data is reduced and it can be transferred to the cold data layer. The cold standby condition can be set according to the frequency of data access. During the period when the data is in the warm data layer, if the access frequency decreases significantly, it indicates that the heat of the data is reduced, and it can be prepared to be transferred to the cold data layer. Specifically, a threshold of the number of accesses within a certain period of time is set. When the number of accesses to the data within a certain period of time is lower than the threshold of the number of accesses, it is determined that the data meets the cold standby condition. Of course, in specific applications, the cold standby condition may also be to set both data timeliness and data access frequency.

[0062] The storage method of this application can solve the problems of performance degradation, resource waste and high operation and maintenance costs caused by the growth of data volume in business databases by stratifying data into three layers: hot data layer, warm data layer and cold data layer. Hot data focuses on real-time reading and writing to ensure the efficient operation of core business; warm data buffers the migration needs between hot and cold data and supports low-frequency queries; cold data is stored in low-cost media, realizing the economic management of historical data.

[0063] Tiered storage combined with cloud-native task scheduling technology solves the complexity of traditional task scheduling and the inefficiency of resource allocation. The dynamic scheduling mechanism can flexibly allocate computing resources according to task load, ensuring that migration tasks are completed efficiently without interrupting business. At the same time, it improves system stability and task isolation, avoiding resource conflicts and performance bottlenecks.

[0064] In addition, cold data uses columnar storage format and lightweight query engine to significantly reduce the cost and response time of query and analysis. Combined with containerized deployment and automated operation and maintenance, it simplifies the task development and management process, allowing enterprises to flexibly respond to changes in data storage needs while significantly reducing long-term storage and operation and maintenance expenses.

[0065] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A data hierarchical storage method, characterized in that: include: Layer the data storage architecture into hot data tiers, warm data tiers, and cold data tiers; Hot data ingestion and loading tasks: regularly ingest data from the hot data layer and load it into the DuckDB internal table of the warm data layer; Warm data conversion task: convert the data in the warm data layer in DuckDB to form cold standby target data; Data cold backup task: transferring the cold backup target data to the cold data layer; Among them, the hot data ingestion and loading tasks, the warm data conversion tasks and the data cold standby tasks are coded according to the Dagster task framework, the code is built and published as a container image, and task scheduling configuration, task triggering and task execution are performed.

2. The data hierarchical storage method according to claim 1, characterized in that: The task scheduling configuration includes: configuring the triggering time conditions of the hot data ingestion and loading tasks, and the process of loading data into the warm data layer; configuring the triggering conditions of the warm data conversion task, converting the warm data to form Parquet data files; configuring the triggering conditions of the data cold standby task, transferring the Parquet data files from the warm data layer to the cold data layer.

3. The data layered storage method according to claim 2, characterized in that: A coordination node is set up to uniformly manage the scheduling and distribution of data tasks. When a task is triggered, the coordination node selects the image corresponding to the task execution code as needed to start the Kubernetes Job.

4. The data layered storage method according to claim 3, characterized in that: The task triggering includes: a webserver request initiated through the UI.

5. The data layered storage method according to claim 3, characterized in that: The task triggering includes: initiating periodically according to a set timing scheduling rule.

6. The data hierarchical storage method according to claim 2, characterized in that: The hot data ingestion and loading task triggering time condition is configured to start from the first time node, the data ingested within the first time period is loaded into the warm data layer, and the data of the warm data layer is stored in monthly partitions.

7. The data hierarchical storage method according to claim 6, characterized in that: The warm data conversion task trigger condition is configured to complete the ingestion of each data partition, process and slightly summarize the original data ingested by the warm data layer to form cold standby target data, and store the cold standby target data by monthly partitions.

8. The data hierarchical storage method according to claim 7, characterized in that: The data cold standby task trigger condition is configured to start from the second time node, and the cold standby target data of the warm data layer is copied to the object storage through DuckDB.

9. The data layered storage method according to claim 1, characterized in that: DuckDB is used as the storage engine of the warm data layer.

10. The data hierarchical storage method according to claim 1, characterized in that: The transferring of the cold standby target data to the cold data layer includes: the data of the warm data layer is aggregated and summarized, and retained in the warm data layer for a preset time for processing and querying the warm data. After retaining the preset time, if the warm data meets the cold standby conditions, the warm data is transferred to the cold data layer.

Images