Data separation storage methods, devices, storage media and computer program products

Abstract

This application proposes a data separation storage method, apparatus, storage medium, and computer program product. The method includes: when the configuration information of the column families in the data table of a file system includes data hierarchical timestamp information, obtaining multiple first access counts and multiple first access times for multiple data files; the multiple data files are data files generated when data is written to the file system; the multiple data files are stored in column families; the multiple data files are divided into first hot data files and cold data files according to the data hierarchical timestamp information, the multiple first access counts, and the multiple first access times; the first hot data files are merged and stored, and the cold data files are merged and stored. This method can automatically identify the hot and cold characteristics of data files, achieving data hot and cold separation, thereby reducing storage costs.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data separation and storage method, apparatus, storage medium and computer program product. Background Technology

[0002] In massive data scenarios, some business data becomes archived or is accessed very infrequently over time, and this data can be very large. Reducing the storage cost of this data can significantly reduce the overall data storage cost. A common solution is to separate hot and cold data, but this cannot automatically identify the hot and cold characteristics of the data. Summary of the Invention

[0003] This application provides a data separation and storage method, apparatus, storage medium, and computer program product that can automatically identify the hot and cold characteristics of data files, realize data hot and cold separation, and thus reduce storage costs.

[0004] To achieve the above objectives, the technical solution of this application embodiment is implemented as follows:

[0005] Firstly, this application proposes a data separation and storage method, the method comprising:

[0006] When the configuration information of the column family in the data table of the file system includes data hierarchical timestamp information, multiple first access counts and multiple first access times of the most recent access are obtained for multiple data files respectively; the multiple data files are data files generated when writing data to the file system; the multiple data files are stored in the column family;

[0007] Based on the data layer timestamp information, the multiple first access counts, and the multiple first access times, the multiple data files are divided into first hot data files and cold data files;

[0008] The first hot data file is merged and stored, and the cold data file is merged and stored.

[0009] Secondly, this application proposes a data separation storage device, the device comprising:

[0010] The acquisition unit is used to acquire, when the configuration information of the column family in the data table of the file system includes data hierarchical timestamp information, multiple first access counts corresponding to multiple data files and multiple first access times at the most recent access; the multiple data files are data files generated when writing data to the file system; the multiple data files are stored in the column family;

[0011] A partitioning unit is used to divide the multiple data files into a first hot data file and a cold data file based on the data layer timestamp information, the multiple first access times, and the multiple first access times.

[0012] The merging and storage unit is used to merge and store the first hot data file and the cold data file.

[0013] Thirdly, this application proposes a storage medium on which a computer program is stored, which, when executed by a processor, implements the steps of any of the methods described above.

[0014] Fourthly, this application proposes a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the methods described above.

[0015] This application proposes a data separation storage method, apparatus, storage medium, and computer program product. The method includes: when the configuration information of the column clusters in the data table of a file system includes data hierarchical timestamp information, obtaining multiple first access counts and multiple first access times for multiple data files; the multiple data files are data files generated when data is written to the file system; the multiple data files are stored in column clusters; the multiple data files are divided into first hot data files and cold data files according to the data hierarchical timestamp information, the multiple first access counts, and the multiple first access times; the first hot data files are merged and stored, and the cold data files are merged and stored. Using the above implementation scheme, the access counts and access times of multiple data files can automatically identify the hot and cold characteristics of the data files through the data hierarchical timestamp information in the column cluster configuration information, achieving data hot and cold separation, thereby reducing storage costs. Attached Figure Description

[0016] Figure 1 This application provides a schematic diagram of an exemplary primary / standby cluster data separation scheme.

[0017] Figure 2 A schematic diagram illustrating an exemplary implementation of hot and cold data separation using HDFS tiered storage, provided as an embodiment of this application;

[0018] Figure 3 A flowchart illustrating a data separation and storage method provided in an embodiment of this application;

[0019] Figure 4 A flowchart illustrating an exemplary method for storing key-value data in hot and cold categories for big data, provided as an embodiment of this application;

[0020] Figure 5 This is an exemplary flowchart illustrating the automatic identification of storage characteristics, provided as an embodiment of this application.

[0021] Figure 6 A schematic diagram illustrating an exemplary data layering and merging strategy provided in an embodiment of this application;

[0022] Figure 7 This is a schematic diagram of the structure of a data separation storage device provided in an embodiment of this application;

[0023] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0024] In order to gain a more detailed understanding of the features and technical content of the embodiments of this application, the implementation of the embodiments of this application will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for reference and illustration only and are not intended to limit the embodiments of this application.

[0025] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0026] In the following description, references to "some embodiments" refer to a subset of all possible embodiments. It is understood that "some embodiments" may be the same or different subsets of all possible embodiments and may be combined with each other without conflict. It should also be noted that the terms "first," "second," etc., used in the embodiments of this application are merely for distinguishing similar objects and do not represent a specific ordering of objects. It is understood that "first," "second," etc., may be interchanged in a specific order or sequence where permissible, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0027] Hadoop Database (HBase) is a popular distributed NoSQL (Not Only SQL) database for big data. It uses the Hadoop Distributed File System (HDFS) as its underlying storage, making it suitable for storing and querying large-scale data. Massive data storage involves cost issues, and reducing data storage costs is a pressing problem for enterprises. In massive data scenarios, some business data may become archived or be accessed infrequently over time, yet this data can be very large. Reducing the storage cost of this data can significantly save on enterprise costs. A common solution is to manage data through cold / hot data separation. Currently, HBase's cold / hot data separation solutions mainly include the following:

[0028] 1. A primary / standby cluster implementation of a hot / cold data separation scheme. The primary (hot) cluster uses solid-state drives (SSDs) for online queries, while the standby (cold) cluster uses inexpensive hard disk drives (HDDs) for historical data queries and parallel computing. Users write data to the primary cluster in real time, and the data is synchronized to the standby cluster via replication. The primary cluster has a time-to-live (TTL) setting, so that when the data cools down, the cold data only exists in the cold cluster. Figure 1 This application provides a schematic diagram of an exemplary primary / standby cluster data separation scheme. Figure 1 As shown, cluster A is in the hot table and cluster B is in the cold table. Cluster A can be understood as the primary cluster or the hot cluster; cluster B can be understood as the backup cluster or the cold cluster. Cluster A uses solid-state drives (SSDs) for storage, while cluster B uses hard disk drives (HDDs). When a client writes data, it writes the data to cluster A in real time and then synchronizes the data to cluster B through replication.

[0029] 2. Implement a hot / cold data separation solution using HDFS tiered storage. HDFS tiered storage allows specifying the storage medium for a directory or file, including various types such as HDD and SSD. Since HBase tables are stored on HDFS as directories and files, HDFS tiered storage can be used to specify the storage medium of the directory containing the table, thereby achieving hot / cold storage for that table. This enables hot / cold data separation for different tables within the same cluster. Figure 2This application provides an exemplary schematic diagram illustrating the use of HDFS tiered storage to achieve hot and cold data separation; as shown in the embodiments. Figure 2 As shown, in cluster A, when clients write data to hot and cold data tables, the data is stored on HDFS as DataNodes. DataNodes can also be understood as targets and files, with HDDs and SSDs on different DataNodes.

[0030] The above solution has the following drawbacks:

[0031] 1. Increased operational complexity, difficulty in maintaining data consistency, and high user query modification costs. Implementing a primary / standby cluster-based data hot / cold separation solution requires users to maintain both the primary (hot) cluster and the standby (cold) cluster, increasing operational complexity. Regardless of the data synchronization solution used, data consistency between the cold and hot databases is difficult to guarantee. For businesses, after implementing hot / cold separation, the data is no longer a "single database," and users need to decide whether to query the hot or cold database, increasing query modification costs.

[0032] 2. The actual storage capacity has not changed. Whether it is a hot-cold cluster solution or a tiered storage solution, after the data changes from hot to cold, only the storage medium changes, from more expensive high-performance storage to cheaper low-performance storage. The total size of the data capacity has not changed.

[0033] 3. Inability to automatically identify the hot / cold characteristics of data. The cloud HBase hot / cold separation solution relies on the hot / cold boundaries set by the user during use to transfer data from hot to cold and vice versa. However, some services may frequently access cold data during certain periods, requiring automatic conversion of cold data to hot data. This solution cannot automatically identify the hot / cold characteristics of the data and convert the data storage characteristics accordingly.

[0034] 4. The differences in storage characteristics between table data and column families were not considered. When using HDFS as a hot / cold data separation solution, the distinction between hot and cold data could only be made conceptually based on files, meaning only file attributes such as the last modification time could be used as the basis for hot / cold data separation. The differences in storage characteristics between table data itself and different column families were not considered. In HBase, each table (HTable) can correspond to one or more column families, and each column family can also correspond to one or more data files (HFiles). The hot / cold storage requirements differ between different column families. Even within the same HFile, some data may be hot and some cold, requiring further partitioning. Therefore, when storing data, it is not possible to store all HFiles from all column families in hot storage or all in cold storage.

[0035] Based on this, embodiments of this application provide a data separation and storage method. Figure 3 This is a flowchart illustrating a data separation and storage method provided in an embodiment of this application; as shown below. Figure 3 As shown, the method includes:

[0036] S301. If the configuration information of the column family in the data table of the file system includes data hierarchical timestamp information, obtain the multiple first access counts corresponding to multiple data files and the multiple first access times corresponding to the most recent access; the multiple data files are data files generated when writing data to files; the multiple data files are stored in the column family.

[0037] It should be noted that the file system can be any file system, determined based on the actual situation, and is not limited here; as an example, the file system can be HDFS. A data table can be understood as an HTable on HBase. There can be one or more data tables, the specific number depending on the actual situation. Each data table can correspond to one or more column families (CFs), the number of CFs depending on the actual situation. Data tiering timestamp information can be understood as the boundary between hot and cold data. In practical applications, data tiering timestamp information can be denoted as DATA_TIERED_TIMESTAMP. The configuration information of a CF includes data tiering timestamp information; that is, each CF's configuration information includes data tiering timestamp information, and the data tiering timestamp information included in the configuration information of each CF can be the same or different, depending on the actual situation. When creating a data table, the user can specify the data tiering timestamp information of the corresponding CF to determine the hot and cold data boundary of that CF; or the user can create the data table first and then modify the data tiering timestamp information as needed when it is required.

[0038] It's important to clarify that the first access count can be understood as the number of times a data file is accessed (read), or as the access frequency of a data file. Multiple first access counts for multiple data files can be understood as the first access count for each data file within the multiple data files. In practical applications, the first access count for each data file can be understood as the access frequency of each HFile. The first access time can be understood as the time when a data file was last accessed (read). Multiple first access times for multiple data files at their most recent access can be understood as the first access time for each data file at its most recent access. In practical applications, the first access time for each data file at its most recent access can be understood as the access time for each HFile at its most recent access. Storing multiple data files in a CF (CF Service) can be understood as storing multiple data files in the same CF, or storing multiple data files in different CFs.

[0039] S302. Based on the data layer timestamp information, multiple first access counts, and multiple first access times, divide multiple data files into first hot data files and cold data files.

[0040] It should be noted that, when multiple data files are stored in the same CF (Data Provider), the data files are divided into hot data files and cold data files based on the data layer timestamp information corresponding to that CF, as well as multiple first access counts and multiple first access times. When multiple data files are stored in different CFs, the data files stored under each CF are divided into hot data files and cold data files based on the time layer timestamp information corresponding to each CF, as well as the first access counts and first access times for each data file stored under each CF.

[0041] It should be noted that the first hot data file can be understood as containing hot data, and the cold data file can be understood as containing cold data. The first hot data file may include one or more hot data files, which can be determined according to the actual situation and is not limited here; the information in the cold data file may also include one or more cold data files, which can be determined according to the actual situation and is not limited here. In the embodiments of this application, the hot data file can be a hot file, and the cold data file can be understood as a cold file.

[0042] S303. Merge and store the first hot data file, and merge and store the cold data file.

[0043] It should be noted that merging and storing the first hot data file can be understood as merging one or more hot data files within the first hot data file into a single hot data file. The storage method can be determined based on the actual situation and is not limited here. Merging and storing cold data files can be understood as merging one or more cold data files within the cold data file into a single cold data file. The storage method can also be determined based on the actual situation and is not limited here; as an example, the EC method can be used for storage.

[0044] It should be noted that the first hot data file after merging and the cold data file after merging can be stored on the same storage medium. The specific storage medium can be determined according to the actual situation and is not limited here.

[0045] The solution in this application embodiment uses the data layering timestamp information in the CF configuration information to automatically identify the hot and cold characteristics of multiple data files based on the number of accesses and access times, thereby achieving data hot and cold separation and reducing storage costs.

[0046] In this embodiment of the application, the process of dividing multiple data files into first hot data files and cold data files based on data layer timestamp information, multiple first access counts, and multiple first access times specifically includes: dividing multiple data files into second hot data files and pending data files based on multiple first access counts and multiple first access times; dividing pending data files into third hot data files and cold data files based on data layer timestamp information; and merging the second hot data files and the third hot data files into a first hot data file.

[0047] It should be noted that pending data files can be understood as data files that require further classification. The process of dividing multiple data files into second-hot data files and pending data files based on multiple first access counts and multiple first access times specifically includes: designating data files that meet a first condition as second-hot data files, and designating the other data files besides the second-hot data files as pending data files; the first condition is that the first access count corresponding to the second-hot data file is greater than the average access count, and the difference between the first access time corresponding to the second-hot data file and the current time is less than or equal to the average access time.

[0048] It should be noted that the average number of accesses can be calculated based on the total number of data files in the data table and the sum of the accesses to each data file; the average access time can be calculated based on the sum of the differences between the access time of each data file in the data table and the current time of the file system. The average access time can also be understood as the average most recent access time. In practical applications, the average number of accesses can be denoted as C1, and the average access time can be denoted as C2.

[0049] It should be noted that the first condition can be understood as the first access count of the second hot data file being greater than C1, and the first access time being less than or equal to C2. The pending data files can be understood as other data files besides the first hot data file among multiple data files.

[0050] In this embodiment of the application, the process of dividing a pending data file into a third hot data file and a cold data file based on the data layering timestamp information specifically includes: obtaining the first timestamp information corresponding to each data file in the pending data file; if the first timestamp information is greater than the data layering timestamp information, the data file corresponding to the first timestamp information is regarded as a cold data file; if the first timestamp information is less than or equal to the data layering timestamp information, the data file corresponding to the first timestamp information is regarded as a third hot data file.

[0051] It should be noted that the first timestamp information can be understood as the timestamp information corresponding to each data file in each pending data file, or as the time information of the last modification or generation of each data file. If the first timestamp information is greater than the data layer timestamp information, the data file corresponding to the first timestamp information is considered a cold data file; if the first timestamp information is less than or equal to the data layer timestamp information, the data file corresponding to the first timestamp information is considered a third-hot data file. This can be understood as follows: if a data file in the pending data files has a first timestamp information greater than the data layer timestamp information, that data file is considered a cold data file; if a data file in the pending data files has a first timestamp information less than or equal to the data layer timestamp information, that data file is considered a third-hot data file.

[0052] It should be noted that the first timestamp information corresponding to each data file in the pending data file needs to be compared with the data layer timestamp information to determine whether each data file is a cold data file or a third hot data file.

[0053] It should be noted that the first hot data file includes the second hot data file and the third hot data file.

[0054] The solution in this application embodiment automatically identifies whether a data file is a hot data file or a cold data file by using the access count, access time, and data layer timestamp information of the CF corresponding to the data file in the created data table, thereby achieving data hot and cold separation and reducing the cost of data storage.

[0055] In this embodiment of the application, the process of merging and storing cold data files specifically includes: merging the cold data files and storing the merged cold data files using the file system's error correction coding (EC).

[0056] It should be noted that merging cold data files and using the file system's EC to store the merged cold data file can be understood as merging one or more cold data files into one cold data file and using EC to store the merged cold data file.

[0057] The solution in this application uses EC (Elastic Compute Service) to store cold data, rather than differentiating by storage medium. This allows for tiered storage of hot and cold data even within a storage cluster using the same media, eliminating reliance on the hardware environment.

[0058] In this embodiment of the application, after dividing multiple data files into second hot data files and pending data files based on multiple first access times and multiple first access times, the method further includes: determining that the storage characteristics of the second hot data files are hot storage characteristics and the storage characteristics of the pending data files are cold storage characteristics; the hot storage characteristics are used to merge the second hot data files with other hot data files in the file system; the cold storage characteristics are used to merge the cold data files in the pending data files with other cold data files in the file system.

[0059] It should be noted that determining the storage characteristics of the second hot data file as hot storage characteristics and the storage characteristics of the pending data file as cold storage characteristics can be understood as setting the storage characteristics of the second hot data file to hot storage characteristics, and setting the storage characteristics of the pending data file to hot storage characteristics. It should also be noted that data files with the same storage characteristics are merged and stored in the corresponding hot or cold data file.

[0060] The solution in this application embodiment can automatically identify the storage characteristics of data files and realize the separation of hot and cold data storage.

[0061] In this embodiment of the application, the method further includes: merging all data files in the file system when the configuration information of the column family does not include data hierarchical timestamp information.

[0062] It should be noted that if the data hierarchical timestamp information is not included in the column family configuration information, all data files in the file system will be merged into one data file.

[0063] To facilitate understanding, the above method is illustrated with examples. In practical applications, this data separation and storage method can be exemplified as a method for implementing hot and cold data separation storage for large-scale key-value data. This application embodiment supports users in setting different storage characteristics for each Column Family of HBase HTable, achieving hot and cold data separation at the Column Family level. It also allows for fine-grained hot and cold data splitting at the row level, achieving row-level hot and cold data separation. Furthermore, it utilizes the HFile access frequency and access time statistics of the Region on the HBase RegionServer to automatically identify the hot and cold characteristics of data, thereby achieving automatic hot and cold data conversion. During conversion, in addition to changing the storage medium, it also reduces the data volume, ensuring lower data storage usage under the same storage medium, i.e., reducing data storage costs without reducing HBase read and write performance.

[0064] After users create data tables according to their actual needs, they begin writing data to the HBase data tables. HBase adopts a Log-Structured Merge-Tree (LSM) architecture. Data is first written to the memory buffer (Memstore). Once the Memstore threshold is reached, the data is flushed to HDFS to form data files (HFiles). As time goes on, the amount of data written gradually increases, and the number of HFiles flushed to HDFS also gradually increases. Too many HFiles will lead to a decrease in HBase query performance. Therefore, HBase has an internal compaction mechanism that merges multiple HFiles under certain triggering conditions. This compaction process forms a new HFile file and removes the merged HFiles, thereby reducing the number of files and achieving the goal of optimizing read performance.

[0065] This application proposes a scheme for separating hot and cold data on HBase. By introducing a novel Data Tiered Compaction (DTCC) mechanism, HDFS's EC (Extended EC) function is used when writing to a new HFile to automatically split hot and cold data based on user-defined hot and cold data split lines and some metadata statistics. This application not only provides finer-grained management of hot and cold data separation based on tables and rows, but also reduces the storage space required for cold data.

[0066] First, users create tables in HBase and specify the hot / cold data boundary using the newly added DATA_TIERED_TIMESTAMP value in the CF attribute. When users write or read data, the access frequency of the accessed HFile files is recorded. When a compaction operation is triggered, if a hot / cold data boundary is set, Data Tiered Compaction will be used. This compaction strategy categorizes data according to certain conditions, and cold data will be stored using HDFS's EC error correction code, significantly reducing storage space.

[0067] For ease of understanding, Figure 4 This application provides an exemplary flowchart of a method for storing key-value data in hot and cold formats for large datasets. Figure 4 As shown, the specific steps are as follows:

[0068] 1. Begin.

[0069] 2. Create an HTable and write data to it.

[0070] 3. Trigger the merge.

[0071] 4. Whether to set a hot and cold dividing line.

[0072] It should be noted that the hot / cold boundary can be understood as the data layering timestamp information mentioned above, or it can be denoted as DATA_TIERED_TIMESTAMP; if no hot / cold boundary is set, proceed to step 5; if a hot / cold boundary is set, proceed to step 6.

[0073] 5. Using the default merging strategy, data will not be separated into hot and cold data.

[0074] It should be noted that step 11 is executed after step 5.

[0075] 6. Use a data tiered merging strategy.

[0076] 7. Traverse the data by row dimension.

[0077] 8. Is it cold data?

[0078] It should be noted that if the data is cold, proceed to step 9; if the data is not cold, proceed to step 10.

[0079] 9. Use error correction codes for data storage.

[0080] It should be noted that step 11 is executed after step 9.

[0081] 10. Stored using the original storage method.

[0082] It should be noted that step 11 is executed after step 10.

[0083] 11. End.

[0084] To make it easier to understand, the above steps are explained in detail below:

[0085] 1. Manually set the hot and cold boundary of CF.

[0086] This application's embodiment adds a new DATA_TIERED_TIMESTAMP attribute to the HBase CF configuration properties, which is the timestamp for data tiering. This attribute is a CF-level attribute, and different CFs can have different values. When creating a table, users can specify the DATA_TIERED_TIMESTAMP value of the corresponding CF to determine the cold / hot boundary that CF should use. Alternatively, the table can be created first, and then the value can be modified as needed.

[0087] 2. Automatically record data access frequency.

[0088] The original HBase design lacked statistics on HFile access frequency, making it difficult to determine whether a particular HFile was hot data. This embodiment of the application additionally records the read frequency of the HFile being read when a user reads data. When determining whether data is hot or cold, it comprehensively considers both DATA_TIERED_TIMESTAMP and the HFile read frequency to avoid situations where data exceeds the hot / cold boundary but is actually frequently read hot data.

[0089] When users use HBase to process business, the access frequency of HFiles will change with the changes in business. HFiles with low access frequency in one period may have high access frequency in another period. Therefore, this application embodiment supports automatically identifying the storage characteristics of HFiles based on two factors: access frequency and access timeliness. This function is triggered when the user manually sets the cold and hot decomposition at the column family level of the data table, and automatically identifies the storage characteristics that HFiles should use.

[0090] For ease of understanding, Figure 5 This application provides an exemplary flowchart of automatically identifying storage characteristics; as shown in the embodiments. Figure 5 As shown, the specific steps are as follows:

[0091] 1. The user sends a query request to HBase.

[0092] 2. HBase forwards the query request to the region server where the region is located.

[0093] 3. The data file records the number of times it was accessed and the time of the most recent access.

[0094] 4. Calculate the average number of accesses to the data file.

[0095] 5. Calculate the average last access time of the data file.

[0096] 6. If a data file is accessed more times than the average number of times it is accessed and the most recent access time is less than the average most recent access time, then it is considered hot data.

[0097] It should be noted that, Figure 5 The data file mentioned is an HFile; the region server where the region is located can be referred to as the RegionServer where the Region is located.

[0098] For ease of understanding, the above steps can be described in detail as follows: The user sends a read data request to HBase. HBase sends the user's read data request to the target RegionServer for processing. The RegionServer will search for the corresponding Region, look for the ColumnFamily within the Region, first search the Memstore, then the BlockCache, and finally traverse the HFiles. Once the data is found, it is returned. When an HFile is accessed, the number of HFile accesses and the access time are recorded. In this embodiment, the average value C1 is calculated based on the total number of HFiles in the HTable and the sum of the accesses of each HFile. The average value C2 is calculated based on the sum of the differences between the access times of each HFile in the HTable and the current system time. C1 and C2 will be used as threshold values ​​for automatically identifying the storage characteristics of HFiles. HFiles whose access counts in the HTable are higher than C1 and whose most recent access time is less than C2 are set to Hot storage characteristics; others are set to Cold. Finally, when the system performs a minor compaction or a major compaction, it compacts HFiles with the same storage characteristics and stores them in the corresponding Hot Data File or Cold Data File.

[0099] 3. Selection of merger strategy.

[0100] In this embodiment of the application, when a user continuously writes data to HBase and triggers compaction, the merging strategy is determined based on whether the DATA_TIERED_TIMESTAMP attribute is set. If the DATA_TIERED_TIMESTAMP attribute is set, the data tiered merging strategy of this embodiment will be adopted.

[0101] 4. Data layering and merging strategy.

[0102] The Data Tiered Compaction strategy proposed in this application differs from the default Compaction strategy. The default Compaction strategy merges all HFiles into one, while the Data Tiered Compaction strategy categorizes HFiles into two HFile files, one for cold storage and the other for hot storage.

[0103] For ease of understanding, Figure 6 A schematic diagram illustrating an exemplary data layering and merging strategy provided in this application embodiment; as shown Figure 6As shown, HTable is classified into column cluster 1 and column cluster 2. Both column cluster 1 and column cluster 2 contain multiple data files. A data hierarchical merging strategy is applied to these multiple data files. The multiple data files are automatically identified and classified into hot data files and pending data files based on their access frequency and most recent access time. Each data file in the pending data files is judged based on whether its row timestamp exceeds the hot / cold boundary (specifically, the timestamp of each data file is compared with the data hierarchical timestamp attribute of the column cluster). If it does not exceed the hot / cold boundary, the data file is treated as a hot data file and merged with the previously classified hot data files into a single hot data file, which is stored using three data copies. If it exceeds the hot / cold boundary, the data file is treated as a cold data file and stored using a data unit plus error correction unit.

[0104] The above method can be illustrated as follows: When compaction is triggered, Data Tiered Compaction first obtains the number of accesses and access times of the HFile based on the statistical information of the previous metadata. This is then compared with the average value to divide the HFiles into two categories: hot files and files awaiting further classification. Next, Data Tiered Compaction iterates through the HFiles in the awaiting list, obtaining the timestamp information of each row and comparing it with the set DATA_TIERED_TIMESTAMP. If the timestamp exceeds this value, the data is considered cold data; otherwise, it is considered hot data. When writing cold data, HDFS's error correction coding feature will be used to create the file and write the data.

[0105] 5. Storage method for hot and cold data.

[0106] Existing solutions for cold and hot storage primarily differ in the storage medium, such as switching from a more expensive solid-state drive (SSD) to a slightly cheaper hard disk drive (HDD), without changing the total amount of data stored. This application's embodiment uses Hadoop error correction codes to store cold data, transforming the data originally stored as three replicas on HDFS into a format of data units plus error correction units. This significantly reduces the total amount of data stored.

[0107] The solution in this application embodiment allows users to create data tables at the Column Family or Row level for hot and cold data separation storage according to actual conditions. At the same time, the system automatically identifies the storage characteristics of HFile based on the number of HFile accesses and access time, thereby realizing hot and cold data separation and reducing the cost of storing massive amounts of data for enterprises.

[0108] The solution in this application uses HDFS error correction codes to store cold data, rather than differentiating between storage media. This allows for tiered storage of hot and cold data even within a storage cluster using the same media, eliminating reliance on hardware conditions.

[0109] The solution in this embodiment only requires users to make simple attribute settings for the Column Family of the HBase table to achieve the purpose of separating hot and cold storage. The impact on the user layer is only that when creating the data table, the storage boundary at the Column Family level is selected according to the actual scenario. The operation is simple and convenient for users; and there are no additional requirements for the underlying hardware.

[0110] This application also provides a data separation storage device. Figure 7 This is a schematic diagram of the structure of a data separation storage device provided in an embodiment of this application; as shown below. Figure 7 As shown, the data separation storage device 700 includes:

[0111] The acquisition unit 701 is used to acquire, when the configuration information of the column family in the data table of the file system includes data hierarchical timestamp information, multiple first access counts corresponding to multiple data files and multiple first access times at the most recent access time; the multiple data files are data files generated when writing data to the file system; the multiple data files are stored in the column family;

[0112] The partitioning unit 702 is used to partition the multiple data files into a first hot data file and a cold data file according to the data layer timestamp information, the multiple first access times and the multiple first access times;

[0113] The merging and storage unit 703 is used to merge and store the first hot data file and the cold data file.

[0114] Optionally, the partitioning unit 702 is further configured to partition the plurality of data files into a second hot data file and a pending data file according to the plurality of first access times and the plurality of first access times; partition the pending data file into a third hot data file and the cold data file according to the data layer timestamp information; and merge the second hot data file and the third hot data file into the first hot data file.

[0115] Optionally, the partitioning unit 702 is further configured to designate the data file that meets the first condition among the plurality of data files as the second hot data file, and designate the other data files among the plurality of data files besides the second hot data file as the pending data files; the first condition is that the first access count corresponding to the second hot data file is greater than the average access count, and the difference between the first access time corresponding to the second hot data file and the current time is less than or equal to the average access time.

[0116] Optionally, the partitioning unit 702 is further configured to obtain first timestamp information corresponding to each data file in the pending data file; if the first timestamp information is greater than the data layer timestamp information, the data file corresponding to the first timestamp information is designated as the cold data file; if the first timestamp information is less than or equal to the data layer timestamp information, the data file corresponding to the first timestamp information is designated as the third hot data file.

[0117] Optionally, the merging and storage unit 703 is further configured to merge the cold data files and store the merged cold data files using the EC of the file system.

[0118] Optionally, after dividing the plurality of data files into second hot data files and pending data files according to the plurality of first access counts and the plurality of first access times, the data separation storage device 700 further includes a determining unit, used to determine that the storage characteristics of the second hot data file are hot storage characteristics and the storage characteristics of the pending data file are cold storage characteristics; the hot storage characteristics are used to merge the second hot data file with other hot data files in the file system; the cold storage characteristics are used to merge the cold data files in the pending data file with other cold data files in the file system.

[0119] Optionally, the data separation storage device 700 further includes a merging unit, used to merge all data files in the file system when the data hierarchical timestamp information is not included in the configuration information of the column family.

[0120] This application also provides an electronic device. Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application; as shown below. Figure 8 As shown, the electronic device 800 includes a processor 801 and a memory 803. Optionally, the electronic device 800 may also include a communication bus 802.

[0121] In specific embodiments, the processor 801 described above can be at least one of the following: Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), CPU, controller, microcontroller, and microprocessor. It is understood that for different devices, the electronic device used to implement the above processor function can also be other types, and this embodiment does not specifically limit it.

[0122] In this embodiment, the communication bus 802 is used to establish communication between the processor 801 and the memory 803; when the processor 801 executes the running program stored in the memory 803, it implements the following data separation storage method:

[0123] When the configuration information of the column family in the data table of the file system includes data hierarchical timestamp information, multiple first access counts and multiple first access times of the most recent access are obtained for multiple data files respectively; the multiple data files are data files generated when writing data to the file system; the multiple data files are stored in the column family; the multiple data files are divided into first hot data files and cold data files according to the data hierarchical timestamp information, the multiple first access counts and the multiple first access times; the first hot data files are merged and stored, and the cold data files are merged and stored.

[0124] Furthermore, the processor 801 is also configured to divide the plurality of data files into a second hot data file and a pending data file according to the plurality of first access times and the plurality of first access times; divide the pending data file into a third hot data file and the cold data file according to the data layer timestamp information; and merge the second hot data file and the third hot data file into the first hot data file.

[0125] Furthermore, the processor 801 is also configured to use the data file that meets the first condition in the plurality of data files as the second hot data file, and to use the other data files in the plurality of data files other than the second hot data file as the pending data files; the first condition is that the first access number corresponding to the second hot data file is greater than the average access number, and the difference between the first access time corresponding to the second hot data file and the current time is less than or equal to the average access time.

[0126] Furthermore, the processor 801 is also configured to obtain first timestamp information corresponding to each data file in the pending data file; if the first timestamp information is greater than the data layer timestamp information, the data file corresponding to the first timestamp information is designated as the cold data file; if the first timestamp information is less than or equal to the data layer timestamp information, the data file corresponding to the first timestamp information is designated as the third hot data file.

[0127] Furthermore, the processor 801 is also used to merge the cold data files and store the merged cold data files using the EC of the file system.

[0128] Furthermore, after dividing the plurality of data files into second hot data files and pending data files according to the plurality of first access counts and the plurality of first access times, the processor 801 is further configured to determine that the storage characteristics of the second hot data files are hot storage characteristics and the storage characteristics of the pending data files are cold storage characteristics; the hot storage characteristics are used to merge the second hot data files with other hot data files in the file system; the cold storage characteristics are used to merge the cold data files in the pending data files with other cold data files in the file system.

[0129] Furthermore, the processor 801 is also used to merge all data files in the file system when the data layer timestamp information is not included in the configuration information of the column family.

[0130] This application provides a storage medium on which a computer program is stored. The computer-readable storage medium stores one or more programs, which can be executed by one or more processors. The computer program implements the data separation storage method described above.

[0131] Based on the above embodiments, this application provides a computer program product, including a computer program that can be executed by one or more processors, and the computer program implements the data separation and storage method described above.

[0132] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0133] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause an image display device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this disclosure.

[0134] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application.

Claims

1. A data separation and storage method, characterized in that, The method includes: When the configuration information of the column family in the data table of the file system includes data layer timestamp information, multiple first access counts and multiple first access times of the most recent access are obtained for multiple data files respectively; the multiple data files are data files generated when writing data to the file system; the multiple data files are stored in the column family; the data layer timestamp information is the boundary between hot and cold data; Based on the data layer timestamp information, the multiple first access counts, and the multiple first access times, the multiple data files are divided into first hot data files and cold data files; The first hot data file is merged and stored, and the cold data file is merged and stored; the merged first hot data file and the merged cold data file are stored using the same storage medium; The step of dividing the multiple data files into first hot data files and cold data files based on the data layer timestamp information, the multiple first access counts, and the multiple first access times includes: The multiple data files are divided into second hot data files and pending data files based on the multiple first access counts and the multiple first access times; Based on the data layer timestamp information, the pending data file is divided into a third hot data file and the cold data file; The second thermal data file and the third thermal data file are merged into the first thermal data file.

2. The method according to claim 1, characterized in that, The step of dividing the multiple data files into second hot data files and pending data files based on the multiple first access counts and the multiple first access times includes: The data file that meets the first condition among the multiple data files is designated as the second hot data file, and the other data files among the multiple data files, excluding the second hot data file, are designated as the pending data files; the first condition is that the first access count corresponding to the second hot data file is greater than the average access count, and the difference between the first access time corresponding to the second hot data file and the current time is less than or equal to the average access time.

3. The method according to claim 1, characterized in that, The step of dividing the pending data file into a third hot data file and the cold data file based on the data layering timestamp information includes: Obtain the first timestamp information corresponding to each data file in the pending data files; If the first timestamp information is greater than the data layer timestamp information, the data file corresponding to the first timestamp information is regarded as the cold data file; If the first timestamp information is less than or equal to the data layer timestamp information, the data file corresponding to the first timestamp information is used as the third hot data file.

4. The method according to claim 1, characterized in that, The process of merging and storing the cold data files includes: The cold data files are merged, and the merged cold data files are stored using the error correction code EC of the file system.

5. The method according to claim 1, characterized in that, After dividing the multiple data files into second hot data files and pending data files based on the multiple first access counts and the multiple first access times, the method further includes: The storage characteristics of the second hot data file are determined to be hot storage characteristics, and the storage characteristics of the pending data file are determined to be cold storage characteristics; the hot storage characteristics are used to merge the second hot data file with other hot data files in the file system; the cold storage characteristics are used to merge the cold data files in the pending data file with other cold data files in the file system.

6. The method according to claim 1, characterized in that, The method further includes: If the data hierarchical timestamp information is not included in the configuration information of the column family, all data files in the file system are merged.

7. A data separation storage device, characterized in that, The device includes: The acquisition unit is used to acquire, when the configuration information of the column family in the data table of the file system includes data layering timestamp information, multiple first access counts corresponding to multiple data files and multiple first access times at the most recent access; the multiple data files are data files generated when writing data to the file system; the multiple data files are stored in the column family; the data layering timestamp information is a hot and cold data boundary line; A partitioning unit is used to divide the multiple data files into a first hot data file and a cold data file based on the data layer timestamp information, the multiple first access times, and the multiple first access times. The merging and storage unit is used to merge and store the first hot data file and the cold data file; the merged first hot data file and the merged cold data file are stored using the same storage medium. The partitioning unit is further configured to partition the plurality of data files into a second hot data file and a pending data file based on the plurality of first access counts and the plurality of first access times; partition the pending data file into a third hot data file and the cold data file based on the data layer timestamp information; and merge the second hot data file and the third hot data file into the first hot data file.

8. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images