Heterogeneous storage-oriented hot and cold data management method and system

Abstract

A heterogeneous storage-oriented hot and cold data management method and system, wherein hot and cold data are managed by means of the following steps: at table granularity, storing data from each table into an independent data bucket in the form of object storage, each data bucket comprising a plurality of partitions, the data being randomly distributed in each partition, and the consistency of replicas of the partitioned data being ensured by means of a distributed consensus algorithm; collecting statistics about the access frequencies of the partitions in one data bucket; when the access frequency of one partition reaches a set threshold, adding a new replica to a target storage medium by means of a membership change in the distributed consensus algorithm; and when data synchronization between the new replica and an old replica is completed, detecting the old replica. In this way, the migration of the partitioned data on different storage media is realized, and data cooling and heating are implemented. The method and system can maximize the use efficiency of high-performance storage media, ensure the access efficiency of users, and can reduce the costs of data migration and maintenance.

Description

A hot and cold data management method and system for heterogeneous storage TECHNICAL FIELD

[0001] The present application relates to the field of data storage and governance, in particular to a hot and cold data management method and system for heterogeneous storage media. BACKGROUND

[0002] With the growth of data generated by enterprises, how to efficiently store massive data and reduce data storage costs as much as possible while ensuring business is not affected is a problem that every enterprise is focusing on. The current mainstream storage formats include block storage, file storage, object storage, etc. Object storage, with its excellent horizontal expansion capability and economical and efficient storage format, eliminates the complexity, capacity limitations and cost barriers that plague traditional storage systems, and has increasingly become a storage solution in scenarios such as artificial intelligence, data lake, cloud-native applications, etc.

[0003] Considering that the access frequency of data in real business scenarios often differs greatly, the access frequency of recently generated data or hot historical data is often high, and these hot data should be stored on high-speed storage media to provide high-speed access speed; while some expired historical data is basically not accessed, and these cold data should be stored on low-speed storage media to reduce storage costs. TECHNICAL PROBLEM

[0004] The current data management system has made great progress in data volume and data access performance, but most lack efficient management and utilization of different performance heterogeneous storage media at the bottom, and cannot well combine the data characteristics of actual business and heterogeneous storage media to reduce data management costs.

[0005] Chinese invention patent with publication number CN 109857737A discloses a hot and cold data storage method and device, and electronic equipment, the method comprising: backing up hot data in a hot database to a cold database; wherein the hot database is used to store hot data, and the cold database is used to store cold data; the hot data is data with a data access frequency greater than or equal to a preset access frequency within a target time length, and the cold data is data with a data access frequency less than the preset access frequency within the target time length; determining whether there is target hot data that changes to cold data in the hot database; if there is target hot data, deleting the target hot data from the hot database. Although this method can achieve the management of hot and cold data to some extent, it still has the following problems:

[0006] 1) The data storage is implemented based on two database systems, and the consistency of data between the two database systems needs to be maintained, which requires additional cost and is difficult to implement.

[0007] 2) The hot and cold migration of data is perceptible to the user (application), and the user needs to know whether to access the hot database or the cold database to obtain data when accessing the data, thereby reducing the access efficiency and affecting the user experience. Technical solutions

[0008] In view of the above background, the present application provides a cold and hot data management method and system for heterogeneous storage media, which can realize the migration of partitioned data on different storage media, cool down and warm up the data, thereby maximizing the use efficiency of high-performance storage media; while ensuring the system storage size and performance, the user access efficiency is guaranteed, and the data transfer and maintenance cost is effectively reduced.

[0009] To achieve the above object, the technical scheme adopted by the present application is as follows:

[0010] The first aspect of the present application provides a cold and hot data management method for heterogeneous storage, comprising:

[0011] A cluster node configured with a heterogeneous storage device is established, the heterogeneous storage device includes a high-speed storage device and a low-speed storage device, and a storage directory list is configured in the startup configuration file of each node and a device type tag of each directory is marked;

[0012] According to the application scenario, a data table is created in the computing layer, and a storage parameter is configured, the storage parameter at least includes data bucket copy number, partition number and access frequency threshold of cold and hot data, and a data bucket associated with the data table is created on the cluster node of the storage layer;

[0013] Data is written to the data table through the computing layer interface, and the data table file after writing is completed is sent to the storage layer and stored in the form of object storage in the heterogeneous storage device of the node where the corresponding data bucket is located;

[0014] The storage layer triggers and executes a partition migration task based on the storage parameter, adds a new copy on the target storage device in the form of member change in the distributed consensus algorithm, and deletes the old copy after the new and old copies complete data synchronization; the partition migration task includes migrating the data of the target partition from the high-speed storage device to the low-speed storage device, or migrating the data from the low-speed storage device to the high-speed storage device.

[0015] In some embodiments, for newly written data, the default is hot data, and the corresponding data bucket partition is stored in the mounting directory corresponding to the high-speed storage device.

[0016] In some embodiments, the storage layer triggers and executes a partition migration task based on the storage parameter, comprising:

[0017] counting the frequency of access of each data bucket partition within a preset time;

[0018] when the access frequency of a partition within a preset time reaches the cold data access frequency threshold, and the data of the partition is located in the high-speed storage device directory, triggering a partition migration task to migrate the partition from the high-speed storage device directory to the low-speed storage device directory;

[0019] when the access frequency of a partition within a preset time reaches the hot data access frequency threshold, and the data of the partition is located in the low-speed storage device directory, triggering a partition migration task to migrate the partition from the low-speed storage device directory to the high-speed storage device directory.

[0020] In some embodiments, during the execution of the partition migration task, the partition is set to read-only, other partitions of the data bucket to which the partition belongs are still readable and writable, and the data table based on the data bucket is still readable and writable.

[0021] In some embodiments, when performing the partition migration task, among the storage device directories that meet the storage device type and available capacity size, the target storage device directory of the same node as the storage device directory of the current partition is preferentially selected; when the source device directory and the destination device directory are on the same node, the migration and copying of data are completed through local disk IO within the process, avoiding the overhead and latency caused by network IO.

[0022] In some embodiments, the sending of the data table file with the write completion to the storage layer, and the storage of the data table file in the heterogeneous storage device of the node corresponding to the data bucket in the form of object storage include:

[0023] generating an ID according to the path of the object file, and then randomly selecting a partition from the list of readable and writable partitions of the corresponding data bucket to write the object file to the storage device directory of the partition;

[0024] after the writing is completed, the mapping of the object file path to the ID, the mapping of the ID to the data partition, and the byte offset of the object file within the partition are stored as metadata information;

[0025] when the object file is read, the metadata is first queried to obtain the ID according to the file path, and then the specific partition and the byte offset within the partition are obtained according to the ID to access the file.

[0026] In some embodiments, the data of each partition is stored in multiple copies based on the number of copies configured by the storage parameter, and the consistency between the copies is ensured through a distributed consensus algorithm.

[0027] In some embodiments, the adding of the new copy on the target storage device in the manner of member change in the distributed consensus algorithm, and the deleting of the old copy after the new and old copies complete data synchronization, comprises:

[0028] Based on the member change feature of the distributed consensus algorithm, the new copy located in the directory of the target storage device is added to the replication group of the partition;

[0029] Triggering a partition copy data repair task to repair the data state of the newly added copy to be consistent with the old copy;

[0030] Based on the member change feature of the distributed consensus algorithm, the old copy located in the directory of the source storage device is removed from the replication group of the partition, and the old copy data is deleted.

[0031] The second aspect of the present application provides a cold and hot data management system for heterogeneous storage media, which adopts a distributed storage-computing separation architecture, comprising a computing layer configured with a computing engine module and a storage layer configured with a storage engine module;

[0032] The computing engine module performs statistical analysis on the data of the storage layer in units of data tables, and each data table corresponds to a data bucket of the storage engine module; the computing engine module performs dynamic scaling operation according to data volume and computing load;

[0033] The storage engine module is configured with a monitoring component, which is used to monitor the access frequency of each partition within a preset time, and trigger a partition migration task based on the access frequency threshold of cold and hot data; the storage engine module is based on distributed object storage, and manages the data stored in the storage layer by using different types of storage devices on the cluster node according to the method described in the first aspect of the present application. Advantages

[0034] The beneficial technical effects of the present application are as follows:

[0035] 1) The cold and hot data management method and system for heterogeneous storage media proposed by the present application can fully utilize different performance storage devices in the cluster node to manage cold and hot data according to the application scenario and data access frequency of the user. Under the conditions of meeting the specific storage capacity, access speed, etc., the hardware cost is reduced as much as possible.

[0036] 2) The cold and hot data management method and system for heterogeneous storage media proposed by the present application are implemented in a database, which avoids the problem of maintaining data consistency between two database systems, and only migrates data in the storage layer. The application of the database is connected by the computing layer, and the process of data migration is not perceived, which has better user experience.

[0037] 3) The cold and hot data management method and system for heterogeneous storage media proposed in the application adopts a distributed architecture, including a computing layer configured with a computing engine module and a storage layer configured with a storage engine module, and can respectively perform expansion and contraction operations on computing resources and storage resources according to changes in data volume and computing task load, and dynamically adjust the system size.

[0038] 4) The cold and hot data management method and system for heterogeneous storage media proposed in the application can greatly reduce the operation and maintenance cost of the system by monitoring the access frequency of the statistical data and automatically completing the data migration of cold and hot data on different storage media.

[0039] 5) The cold and hot data management method and system for heterogeneous storage media proposed in the application supports cold and warm operations of data at the granularity of partitions in data buckets, and can solve the performance problem caused by the sudden frequent access of some historical data after cold operation. BRIEF DESCRIPTION OF DRAWINGS

[0040] Fig. 1 is a flowchart of an embodiment of the cold and hot data management method for heterogeneous storage media proposed in the application.

[0041] Fig. 2 is a module diagram of an embodiment of the cold and hot data management system for heterogeneous storage media proposed in the application.

[0042] Fig. 3 is a schematic diagram of the migration of data partitions between different storage media directories before migration in an embodiment of the cold and hot data management method for heterogeneous storage media proposed in the application.

[0043] Fig. 4 is a schematic diagram of the migration process of data partitions between different storage media directories in an embodiment of the cold and hot data management method for heterogeneous storage media proposed in the application.

[0044] Fig. 5 is a schematic diagram of the migration of data partitions between different storage media directories after migration in an embodiment of the cold and hot data management method for heterogeneous storage media proposed in the application. BEST MODE FOR CARRYING OUT THE INVENTION

[0045] Enter the best mode for carrying out the invention description paragraph here. EMBODIMENTS OF THE INVENTION

[0046] Embodiments of the application will be described in greater detail below with reference to the accompanying drawings. Although some embodiments of the application are shown in the drawings, it should be understood that the application can be implemented in various forms and should not be interpreted as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided so that the application can be more thoroughly and completely understood. It should be understood that the drawings and embodiments of the application are only for exemplary purposes and are not intended to limit the scope of protection of the application.

[0047] Embodiment 1

[0048] The embodiment discloses a hot and cold data management method for heterogeneous storage media, comprising:

[0049] A cluster node configured with heterogeneous storage devices is established, the heterogeneous storage devices comprising high-speed storage devices and low-speed storage devices, and a storage directory list is configured in a startup configuration file of each node and a device type tag of each directory is marked;

[0050] A data table is created in a computing layer according to an application scenario, and storage parameters are configured, the storage parameters at least comprising a data bucket copy number, a partition number and an access frequency threshold of hot and cold data, and a data bucket associated with the data table is created on a cluster node of a storage layer;

[0051] Data is written to the data table through a computing layer interface, and a data table file written is sent to the storage layer to be stored in the heterogeneous storage devices of a node where the data bucket is located in the form of object storage;

[0052] The storage layer triggers and executes a partition migration task based on the storage parameters, adds a new copy on a target storage device in a member change manner in a distributed consensus algorithm, and deletes an old copy; the partition migration task comprises migrating data of a target partition from a high-speed storage device to a low-speed storage device, or migrating data of the target partition from the low-speed storage device to the high-speed storage device.

[0053] The method of the embodiment will be described in detail below in combination with FIG. 1. As shown in FIG. 1, in a specific example, the flow of the method comprises:

[0054] Step 1. Initialize the data management system. A cluster node configured with heterogeneous storage devices is established in a storage layer, wherein the heterogeneous storage devices comprise but are not limited to high-speed storage devices (such as solid state disks) and low-speed storage devices (such as mechanical hard disks) with inconsistent read-write performance. The storage devices are formatted and mounted to corresponding operating system storage directories. A storage directory list is configured in a startup configuration file of each node of the data management system, and a device type tag of each directory is marked.

[0055] Step 2. Create a data table. After the data management system is started, a data table is created in a computing layer according to a specific application scenario, and storage parameters such as a data bucket copy number, a partition number, an access frequency threshold of hot and cold data are configured, and a data bucket associated therewith is created on a cluster node of a storage layer.

[0056] Preferably, each partition of a newly created data bucket is preferentially distributed to a high-speed storage medium directory of a node with lower storage resource usage.

[0057] Step 3. Write data into the table. The application writes data into the data table through the interface provided by the computing layer, first generates a data file in the computing layer, and sends the data file to the storage layer when the data file reaches a certain size or a write operation is completed, and writes the data file into the heterogeneous storage device of the node where the corresponding data bucket is located as a hot data object.

[0058] Preferably, when writing object data into a data bucket, an ID is first generated according to the path of the object file, and then a partition is randomly selected from the list of readable and writable partitions to be responsible for the storage of the object file. After the object file is successfully written into the directory of the high-speed storage device of the partition, the mapping of the object file path to the ID, the mapping of the ID to the data partition, and the byte offset of the object file in the partition are stored as metadata information. When reading the file later, the metadata is first queried, the ID is obtained according to the file path, and then the specific partition and the byte offset in the partition are obtained according to the ID, so as to access the file.

[0059] Step 4. The monitoring component configured on the storage layer counts the access frequency of each data bucket partition. When the upper-layer application reads and writes the table data, the monitoring component records the access operation of each data bucket partition in real time, and sets a timing task according to the partition migration detection interval set by the user, periodically counts the access frequency of each data bucket partition and tries to trigger the partition migration task.

[0060] Step 5. When the timing task times out, first determine whether the system is running normally. If the system has been closed, the monitoring component exits execution, otherwise count the access frequency of each partition within a certain time period.

[0061] For example, the user can configure 0 times of access to the partition data within 1 month as the cold data threshold, and more than 10 times of access to the partition data within 1 day or more than 100 times of access to the partition data within 10 minutes as the hot data threshold. At this time, the monitoring component counts the number of accesses of each partition within 1 month, 1 day and 10 minutes respectively. If the access frequency of each partition does not reach the threshold, the monitoring component enters the next cycle. Otherwise, an asynchronous partition migration task is tried to be triggered.

[0062] Step 6. Try to trigger the partition data cooling task. When the access frequency of a partition reaches the cold data threshold within the configured time, the monitoring component tries to trigger the partition data cooling task.

[0063] Firstly, it is checked whether the partition data is located in the high-speed storage device directory. If not, the migration of the partition data is stopped. Then, a low-speed storage device directory that meets the remaining space size is selected as the target directory. At this time, the low-speed storage medium directory with the same node as the current partition directory is preferentially selected. The migration and copying of data are completed by local disk IO in the process, avoiding the overhead and latency caused by network IO. Then, the current partition is set to read-only mode. At this time, other partitions of the data bucket to which the partition belongs are still readable and writable. Therefore, the migration of the partition between different storage medium directories will not block the normal read and write requests of the entire storage bucket.

[0064] Finally, an asynchronous partition migration task is triggered, and the next cycle is entered. When the asynchronous partition migration task ends, the partition migrated to the low-speed storage device is re-set to readable and writable.

[0065] Step 7. Try to trigger the partition data warming task. Similar to step 6, when the access frequency of a partition reaches the hot data threshold within the configuration time, the monitoring component tries to trigger the partition data warming task.

[0066] Firstly, it is checked whether the partition data is located in the low-speed storage device directory. If not, the migration of the partition data is stopped.

[0067] Then, a high-speed storage device directory that meets the remaining space size is selected as the target directory. At this time, the high-speed storage device directory with the same node as the current partition directory is preferentially selected. Then, the current partition is set to read-only mode.

[0068] Finally, an asynchronous partition migration task is triggered, and the next cycle is entered. When the asynchronous partition migration task ends, the partition migrated to the high-speed storage device is re-set to readable and writable.

[0069] Embodiment 2

[0070] The embodiment discloses a cold and hot data management system for heterogeneous storage media, which adopts a distributed storage-computing separation architecture and includes a computing layer configured with a computing engine module and a storage layer configured with a storage engine module;

[0071] The computing engine module performs statistical analysis on the data of the storage layer in units of data tables. Each data table corresponds to a data bucket of the storage engine module. The computing engine module performs dynamic scaling operations according to the data volume and the computing load;

[0072] The storage engine module is configured with a monitoring component. The monitoring component is used to monitor the access frequency of each partition within a preset time and trigger a partition migration task based on the access frequency threshold of cold and hot data. The storage engine module is based on distributed object storage and manages the data stored in the storage layer by using different types of storage devices on the cluster nodes according to the method described in Embodiment 1.

[0073] Referring to FIG. 2, in one specific example, the system is deployed on a cluster node composed of at least 6 nodes, wherein node 1, node 2 and node 3 are respectively configured with a computing engine module to form a computing layer, node 4, node 5 and node 6 are respectively configured with different types of storage devices (including high-speed storage device SSD and low-speed storage device HDD) to form a storage layer, forming a distributed computing and storage separation architecture. It should be noted that in the hot and cold data management system of the present application, the computing engine module and the storage engine module can be located in the same node or different nodes, and the present example does not constitute a limitation.

[0074] The computing engine module is based on large-scale parallel processing technology and fully utilizes the processors, memories and other resources of the distributed cluster nodes to perform statistical analysis on the data of the storage layer. The storage engine module is based on distributed object storage technology and reasonably utilizes the heterogeneous storage media of the cluster nodes to perform safe and efficient persistent storage of data.

[0075] In this embodiment, the computing engine module is stateless and does not need to persist the state of the data, and can perform expansion and contraction operations of memory, CPU and other resources according to the data volume and computing load of the application. The storage engine module stores data in the form of object storage, data is stored in partitions, and a placement strategy based on storage resource utilization is adopted, and data partitions are preferentially placed on nodes with idle storage resources. When the system storage capacity reaches the upper limit, expansion of the storage capacity can be realized without moving data.

[0076] The hot and cold data management system in this embodiment manages data in units of tables. Each data table of the computing engine module corresponds to a data bucket of the storage engine module. The system exposes read and write interfaces to user applications through the logical table structure of the computing layer, and the user applications cannot directly operate the data buckets of the storage layer. When writing data into a data table, a data file is first generated in the computing engine module, and when the data file reaches a certain size or a write operation is completed, the data file is sent to the storage engine module, and the data file is stored in multiple copies in a certain partition of the corresponding data bucket of the storage module as a hot data object.

[0077] The partitioning of the data bucket is based on a distributed consensus algorithm to ensure the consistency of each data copy, and each partition will have a primary copy and other copies as secondary copies. When data is written to a partition, the data is first persisted to the log of the primary copy, and then the primary copy sends the log to other secondary copies for persistent storage. The system only confirms that the data is written successfully when more than half of the copies in the partition persist the data log. This mechanism can ensure that as long as more than half of the copies are available, the partition can provide normal services.

[0078] When the partition data is migrated between different storage device directories, first, based on the member change characteristics of the distributed consensus algorithm, a new copy located in the target storage device directory is added to the replication group of the partition. Then the partition copy data repair task is triggered to repair the data state of the newly added copy to be consistent with the old copy. Then based on the member change characteristics of the distributed consensus algorithm, the old copy located in the source storage device directory is removed from the replication group of the partition, and the old copy data is deleted. In this way, the migration of the partition data between different storage device directories is realized.

[0079] Embodiment 3

[0080] This embodiment discloses a detailed process example of the migration of data partitions between different storage device directories in the heterogeneous storage medium-oriented hot and cold data management method and system of the application.

[0081] Referring to FIGS. 3 to 5, taking a data table as an example, the storage content of which is as follows:

[0082]

[0083] The data table contains three partitions in the data bucket corresponding to the storage layer. Among them, partition 1 (including p1-1, p1-2, p1-3 three copies) and partition 3 (including p3-1, p3-2, p3-3 three copies) are hot data (shown in red) and are stored in high-speed storage devices such as SSD; partition 2 (including p2-1, p2-2, p2-3 three copies) is cold data (shown in gray) and is stored in low-speed storage devices such as HDD. Now the monitoring component triggers the cold task of partition 1, and the following steps are executed:

[0084] Step 1. Select the low-speed storage device directory where the new copy is located. As shown in FIG. 3, the first copy (p1-1) of partition 1 is in the SSD directory of node 1, and the HDD directory of the same node is selected as the migration target directory. The second copy (p1-2) is in the SSD directory of node 2, and the HDD directory of node 1 is selected as the migration target directory due to insufficient space in the HDD directory of node 2. The third copy (p1-3) is in the SSD directory of node 2, and the HDD directory of the same node is selected as the migration target directory.

[0085] Step 2. Mark partition 1 as read-only.

[0086] Step 3. As shown in FIG. 4, based on the three low-speed storage device directories selected in step 1, three new copies (p1-1, p1-2, p1-3 in gray) are created, and based on the member change characteristics of the distributed consensus algorithm, they are added to the replication group of partition 1. At this time, partition 1 has six copies. The new copy addition will trigger a copy repair task.

[0087] Step 4. Wait for the 6 replicas in the partition 1 replication group to reach consistency. At this time, the three newly added replicas will respectively receive data from the other three old replicas and recover to the consistent state with one of them. During the repair process, the data of the p1-1 and p1-3 replicas does not need to be transmitted across the network, and the data of the p1-2 replica needs to be transmitted across the network.

[0088] Step 5. As shown in FIG. 5, based on the member change feature of the distributed consensus algorithm, the three old replicas located in the directory of the high-speed storage device are removed from the replication group of the partition 1, and the data in the three old replicas is deleted. Finally, the partition 1 is marked as a readable and writable state.

[0089] By using the above scheme, the transmission amount of cold and hot data between different nodes during migration can be effectively reduced, the migration time can be reduced, the data management efficiency can be improved, and the management cost can be reduced.

[0090] Embodiment 4

[0091] The embodiment discloses an electronic device, comprising a memory and a processor, the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the disclosed cold and hot data management method.

[0092] It should be noted that the method of the embodiment of the present application can be executed by a single device, such as a computer or a server. The method of the embodiment can also be applied to a distributed scenario, and completed by multiple devices cooperating with each other. In this distributed scenario, one of the multiple devices can only execute one or more steps in the method of the embodiment of the present application, and the multiple devices can interact with each other to complete the method.

[0093] Embodiment 5

[0094] The embodiment discloses a computer readable storage medium, which stores a computer program. When the computer program is executed by a computer, the disclosed cold and hot data management method is implemented.

[0095] It should be noted that some embodiments of the present application have been described above, and other undisclosed embodiments are also included within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be executed in an order different from that in the above-described embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multi-task processing and parallel processing are possible or can be advantageous.

[0096] Those skilled in the art should understand that the above discussion of any embodiment is only exemplary, and is not intended to mean that the scope of the present application (including the claims) is limited to these examples; the technical features among the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other changes of different aspects of the embodiments of the present application as described above, which are not provided in details for the sake of brevity.

[0097] The embodiments of the present application are intended to cover all such alternatives, modifications, and variations of the present application falling within the broad scope of the appended claims. Accordingly, any and all such modifications, variations, omissions, and equivalents that are within the spirit and scope of the present application as defined by the claims are to be considered within the scope of the present application. Industrial Applicability

[0098] Enter an industrial applicability description paragraph here. Table Free Content

[0099] Enter a table free content description paragraph here.

Claims

1. A heterogeneous storage oriented cold-hot data management method, characterized in that, The application relates to a method for managing a data table in a distributed storage system. The method comprises the following steps: establishing a cluster node configured with heterogeneous storage devices, the heterogeneous storage devices comprising high-speed storage devices and low-speed storage devices, configuring a storage directory list in a startup configuration file of each node and marking a device type tag of each directory; creating a data table in a computing layer according to an application scenario and configuring storage parameters, the storage parameters at least comprising a data bucket copy number, a partition number and a cold and hot data access frequency threshold value, and simultaneously creating a data bucket associated with the data table on a cluster node of a storage layer; writing data into the data table through a computing layer interface and sending a written data table file to the storage layer to be stored in the heterogeneous storage devices of a node where the data bucket is located in the form of object storage; the storage layer triggers and executes a partition migration task based on the storage parameters, adds a new copy on a target storage device in a member change mode in a distributed consensus algorithm, and deletes an old copy after the new and old copies complete data synchronization; 2.The heterogeneous storage oriented cold-hot data management method of claim 1, wherein, the partition migration task comprises migrating data of a target partition from a high-speed storage device to a low-speed storage device or from a low-speed storage device to a high-speed storage device. 3.The heterogeneous storage oriented cold-hot data management method of claim 1, wherein, For newly written data, the data is hot data by default, and corresponding data bucket partitions are stored in a mounting directory of a high-speed storage device. the storage layer triggers and executes a partition migration task based on the storage parameters, which comprises the following steps: real-time statistics of a frequency of access of each data bucket partition within a preset time; when a partition has a cold data access frequency threshold value within a preset time and the partition data is located in a high-speed storage device directory, a partition migration task is triggered to migrate the partition from the high-speed storage device directory to a low-speed storage device directory; 4.The heterogeneous storage oriented cold-hot data management method of claim 3, wherein, when a partition has a hot data access frequency threshold value within a preset time and the partition data is located in a low-speed storage device directory, a partition migration task is triggered to migrate the partition from the low-speed storage device directory to a high-speed storage device directory. 5.The heterogeneous storage oriented cold-hot data management method of claim 3, wherein, During the execution of the partition migration task, the partition is set as read-only, other partitions of a data bucket to which the partition belongs are still readable and writable, and a data table based on the data bucket is still readable and writable. 6.The heterogeneous storage oriented cold-hot data management method of claim 1, wherein, When the partition migration task is executed, a target storage device directory of a same node as a storage device directory of a current partition is preferentially selected in a storage device directory meeting a storage device type and an available capacity size; when a source device directory and a target device directory are on a same node, data migration and copying are completed through local disk IO within a process. the method for managing a data table in a distributed storage system comprises the following steps: generating an ID according to a path of an object file, then randomly selecting a partition from a readable and writable partition list of a corresponding data bucket to write the object file into a storage device directory of the partition; after writing is completed, mapping an object file path to the ID, mapping the ID to a data partition and object file byte offset in the partition as metadata information for persistent storage; When reading the object file, first query the metadata, get the ID according to the file path, then get the specific partition and byte offset inside the partition according to the ID, and access the file. 7.The heterogeneous storage oriented cold-hot data management method of claim 1, wherein, The data of each partition is stored in multiple copies based on the number of copies configured by the storage parameter, and consistency is ensured between each copy through a distributed consensus algorithm. 8.The heterogeneous storage oriented cold-hot data management method of claim 1, wherein, The method for adding a new copy on the target storage device in the manner of member change in the distributed consensus algorithm, and deleting the old copy after the new and old copies complete data synchronization comprises: Based on the member change feature of the distributed consensus algorithm, add a new copy located in the directory of the target storage device to the replication group of the partition; Trigger the partition copy data repair task to repair the data state of the newly added copy to be consistent with the old copy; Based on the member change feature of the distributed consensus algorithm, remove the old copy located in the directory of the source storage device from the replication group of the partition, and delete the old copy data.

9. A hot and cold data management system for heterogeneous storage media, characterized by, A distributed storage-computing separation architecture is adopted, including a computing layer configured with a computing engine module and a storage layer configured with a storage engine module; The computing engine module performs statistical analysis on the data of the storage layer in units of data tables, and the data in each data table is finally persisted in the data bucket corresponding to the storage engine module; The computing engine module performs dynamic scaling operation according to the data volume and computing load; The storage engine module is configured with a monitoring component for monitoring the access frequency of each partition within a preset time, and triggering a partition migration task based on the access frequency threshold of hot and cold data; The storage engine module manages the data stored in the storage layer by using different types of storage devices on the cluster nodes based on the distributed object storage and the method according to any one of claims 1-8.

Images