Method and equipment for storing data in Hadoop distributed file system

Abstract

The embodiment of the invention aims to provide a method and equipment for storing data in a Hadoop distributed file system, a computer program product and a computer program storage medium. Wherein the table partitions are divided into cold data, temperature data or hot data according to the data access frequencies of the table partitions, and the data access frequencies of the cold data, the temperature data and the hot data are sequentially increased; according to the classification of each table partition, corresponding storage strategies are executed, and the storage efficiency of the cold data, the storage efficiency of the temperature data and the storage efficiency of the hot data are sequentially reduced. According to the embodiments of the invention, the system can provide corresponding storage strategies according to data with different temperatures, which is accurate to a table partition level, so that smooth migration of the data to EC storage is realized, normal access of services is not affected in the process, and high efficiency and availability of the system are ensured.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of Hadoop distributed file systems, and more particularly to a technology for storing data in a Hadoop distributed file system. Background Art

[0002] In order to achieve high availability, enterprises using Hadoop Distributed File System (HDFS) usually adopt a three-copy storage strategy, which means that each copy of data needs to occupy three times the disk space. However, starting from Hadoop 3.0, the system introduced the Erasure Coding (EC) technology. By using EC technology properly, it is possible to achieve high availability equivalent to three-copy storage with less storage space, thereby significantly reducing storage costs.

[0003] Although the EC storage technology in HDFS provides an effective compression method, the use of EC storage will significantly affect the read, write, and update performance of data because data needs to be compressed and decompressed when stored. If data is blindly converted to EC storage, it may lead to a significant decline in write, update, and query performance. In particular, when using an EC strategy with a higher compression ratio, the degradation of read and write performance will be more obvious. Summary of the invention

[0004] The purpose of each embodiment of the present disclosure is to provide a method, device, computer program product and computer program storage medium for storing data in a Hadoop distributed file system.

[0005] According to one aspect of the present disclosure, a method for storing data in a Hadoop distributed file system is provided, wherein the method comprises the following steps:

[0006] According to the data access frequency of each table partition, the partition is divided into cold data, warm data or hot data, wherein the data access frequency of the cold data, the warm data and the hot data increases in sequence;

[0007] Execute corresponding storage strategies according to the classification of the partitions of each table, wherein the storage efficiency of the cold data, the warm data, and the hot data decreases in sequence;

[0008] in:

[0009] For the table partition belonging to the cold data, a first EC storage strategy is adopted, where the first EC storage strategy is an RS-10-4-1024k EC strategy;

[0010] For the table partition belonging to the warm data, a second EC storage strategy is adopted, where the second EC storage strategy is an RS-3-2-1024k EC strategy;

[0011] For the table partition belonging to the hot data, a third EC storage strategy or multi-copy storage is adopted, wherein the storage efficiency of the third EC strategy is lower than the storage efficiency of the second EC strategy;

[0012] The step of executing the corresponding storage strategy specifically includes:

[0013] For the current first table partition, if it belongs to the cold data or the warm data, create a second table partition and configure a corresponding storage policy for the second table partition;

[0014] Writing the data of the first table partition into the second table partition according to the configured storage policy;

[0015] After the data is written, the access to the first table partition is locked;

[0016] Check whether the first table partition is updated. If not, rename the second table partition to the first table partition and delete the original first table partition.

[0017] The second table partition becomes a new first table partition to provide services related to writing data.

[0018] According to one aspect of the present disclosure, a device for storing data in a Hadoop distributed file system is also provided, wherein the device includes a memory and a processor, the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the device is configured to perform the following operations:

[0019] According to the data access frequency of each table partition, the partition is divided into cold data, warm data or hot data, wherein the data access frequency of the cold data, the warm data and the hot data increases in sequence;

[0020] Execute corresponding storage strategies according to the classification of the partitions of each table, wherein the storage efficiency of the cold data, the warm data, and the hot data decreases in sequence;

[0021] in:

[0022] For the table partition belonging to the cold data, a first EC storage strategy is adopted, where the first EC storage strategy is an RS-10-4-1024k EC strategy;

[0023] For the table partition belonging to the warm data, a second EC storage strategy is adopted, where the second EC storage strategy is an RS-3-2-1024k EC strategy;

[0024] For the table partition belonging to the hot data, a third EC storage strategy or multi-copy storage is adopted, wherein the storage efficiency of the third EC strategy is lower than the storage efficiency of the second EC strategy;

[0025] The operations of executing the corresponding storage policy specifically include:

[0026] For the current first table partition, if it belongs to the cold data or the warm data, create a second table partition and configure a corresponding storage policy for the second table partition;

[0027] Writing the data of the first table partition into the second table partition according to the configured storage policy;

[0028] After the data is written, the access to the first table partition is locked;

[0029] Check whether the first table partition is updated. If not, rename the second table partition to the first table partition and delete the original first table partition.

[0030] The second table partition becomes a new first table partition to provide services related to writing data.

[0031] According to one aspect of the present disclosure, a computer program product is also provided, comprising computer program instructions, wherein when the computer program instructions are executed by a computer device, the computer device is configured to perform a method for storing data in a Hadoop distributed file system, wherein the method comprises the following steps:

[0032] According to the data access frequency of each table partition, the partition is divided into cold data, warm data or hot data, wherein the data access frequency of the cold data, the warm data and the hot data increases in sequence;

[0033] Execute corresponding storage strategies according to the classification of the partitions of each table, wherein the storage efficiency of the cold data, the warm data, and the hot data decreases in sequence;

[0034] in:

[0035] For the table partition belonging to the cold data, a first EC storage strategy is adopted, where the first EC storage strategy is an RS-10-4-1024k EC strategy;

[0036] For the table partition belonging to the warm data, a second EC storage strategy is adopted, where the second EC storage strategy is an RS-3-2-1024k EC strategy;

[0037] For the table partition belonging to the hot data, a third EC storage strategy or multi-copy storage is adopted, wherein the storage efficiency of the third EC strategy is lower than the storage efficiency of the second EC strategy;

[0038] The step of executing the corresponding storage strategy specifically includes:

[0039] For the current first table partition, if it belongs to the cold data or the warm data, create a second table partition and configure a corresponding storage policy for the second table partition;

[0040] Writing the data of the first table partition into the second table partition according to the configured storage policy;

[0041] After the data is written, the access to the first table partition is locked;

[0042] Check whether the first table partition is updated. If not, rename the second table partition to the first table partition and delete the original first table partition.

[0043] The second table partition becomes a new first table partition to provide services related to writing data.

[0044] According to one aspect of the present disclosure, there is also provided a computer program storage medium, in which computer executable instructions are stored. When the computer executable instructions are executed by a computer device, the computer device is configured to execute a method for storing data in a Hadoop distributed file system, wherein the method comprises the following steps:

[0045] According to the data access frequency of each table partition, the partition is divided into cold data, warm data or hot data, wherein the data access frequency of the cold data, the warm data and the hot data increases in sequence;

[0046] Execute corresponding storage strategies according to the classification of the partitions of each table, wherein the storage efficiency of the cold data, the warm data, and the hot data decreases in sequence;

[0047] in:

[0048] For the table partition belonging to the cold data, a first EC storage strategy is adopted, where the first EC storage strategy is an RS-10-4-1024k EC strategy;

[0049] For the table partition belonging to the warm data, a second EC storage strategy is adopted, where the second EC storage strategy is an RS-3-2-1024k EC strategy;

[0050] For the table partition belonging to the hot data, a third EC storage strategy or multi-copy storage is adopted, wherein the storage efficiency of the third EC strategy is lower than the storage efficiency of the second EC strategy;

[0051] The step of executing the corresponding storage strategy specifically includes:

[0052] For the current first table partition, if it belongs to the cold data or the warm data, create a second table partition and configure a corresponding storage policy for the second table partition;

[0053] Writing the data of the first table partition into the second table partition according to the configured storage policy;

[0054] After the data is written, the access to the first table partition is locked;

[0055] Check whether the first table partition is updated. If not, rename the second table partition to the first table partition and delete the original first table partition.

[0056] The second table partition becomes a new first table partition to provide services related to writing data.

[0057] The embodiments of the present disclosure classify the data in HDFS, for example, cold, warm, and hot data can be determined according to the access frequency of the data, and the granularity of the data temperature can be accurate to the table partition level. For data of different temperatures, the system can provide corresponding storage strategies to achieve smooth migration of data to EC storage, which will not affect the normal access of the business in the process, ensuring the efficiency and availability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Other features, objects and advantages of the present disclosure will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0059] Figure 1 A flow chart of a method for storing data in a Hadoop distributed file system according to an embodiment of the present disclosure is shown;

[0060] Figure 2 A flowchart of a process of storing data in a Hadoop distributed file system according to an example of the present disclosure is shown.

[0061] The same or similar reference numerals in the drawings represent the same or similar components. DETAILED DESCRIPTION

[0062] The specific embodiments of the present disclosure will be further described below in conjunction with the accompanying drawings.

[0063] Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments of the present disclosure are described as devices represented by block diagrams and processes or methods represented by flow charts. Although the flow charts describe the operating processes of the various embodiments of the present disclosure as sequential processing, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the operations can be rearranged. The processes of the various embodiments of the present disclosure can be terminated when their operations are completed, but can also include additional steps not shown in the flow charts. The processes of the various embodiments of the present disclosure can correspond to methods, functions, procedures, subroutines, subprograms, etc.

[0064] The methods illustrated by the flow charts and the devices illustrated by the block diagrams discussed below may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, program code or code segments that perform the necessary tasks may be stored in a machine or computer readable medium such as a storage medium. (One or more) processors may perform the necessary tasks.

[0065] Similarly, it will also be understood that any flow charts, flow diagrams, state transition diagrams, and the like represent various processes that can be fully described as program code stored in a computer readable medium and thereby executed by a computer device or processor, whether or not such computer device or processor is explicitly shown.

[0066] As used herein, the term "storage medium" may refer to one or more devices for storing data, including read-only memory (ROM), random access memory (RAM), magnetic RAM, kernel memory, magnetic disk storage media, optical storage media, flash memory devices and / or other machine-readable media for storing information. The term "computer-readable medium" may include, but is not limited to, portable or fixed storage devices, optical storage devices, and various other media capable of storing and / or containing instructions and / or data.

[0067] A code segment may represent a procedure, function, subprogram, program, routine, subroutine, module, software package, class, or any combination of instructions, data structures, or program descriptions. A code segment may be coupled to another code segment or hardware circuit by passing and / or receiving information, data, arguments, parameters, or storage contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including storage sharing, information passing, token passing, network transmission, etc.

[0068] The "computer device" referred to in this context refers to an electronic device that can perform predetermined processing procedures such as numerical calculations and / or logical calculations by running predetermined programs or instructions, and may include at least a processor and a memory, wherein the processor executes program instructions pre-stored in the memory to perform the predetermined processing process, or the predetermined processing process is performed by hardware such as ASIC, FPGA, DSP, or a combination of the above two.

[0069] The above-mentioned "computer device" is usually expressed in the form of a general-purpose computer device, and its components may include but are not limited to: one or more processors or processing units, system memory. The system memory may include a computer-readable medium in the form of a volatile memory, such as a random access memory (RAM) and / or a cache memory. The "computer device" may further include other removable / non-removable, volatile / non-volatile computer-readable storage media. The memory may include at least one computer program product having a set (e.g., at least one) of program modules that are configured to perform the functions and / or methods of the various embodiments of the present disclosure. The processor executes various functional applications and data processing by running the program stored in the memory.

[0070] For example, a computer program for executing various functions and processes of multiple embodiments of the present disclosure is stored in the memory, and when the processor executes the corresponding computer program, multiple embodiments of the present disclosure are implemented.

[0071] Typically, the computer device may be, for example, a user device or a network device, or even a combination of the two. The user device includes, but is not limited to, a personal computer (PC), a laptop, a mobile terminal, etc. The mobile terminal includes, but is not limited to, a smart phone, a tablet computer, etc. The network device includes, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud consisting of a large number of computers or network servers based on cloud computing (Cloud Computing), wherein cloud computing is a type of distributed computing, a super virtual computer consisting of a group of loosely coupled computer sets. The computer device may be operated alone to implement the various embodiments of the present disclosure, or may be connected to a network and implement the various embodiments of the present disclosure through interactive operations with other computer devices in the network. The network in which the computer device is located includes, but is not limited to, the Internet, a wide area network, a metropolitan area network, a local area network, a VPN network, etc.

[0072] It should be noted that the user equipment, network equipment and network are only examples. Other existing or future computing devices or networks that are applicable to the embodiments of the present disclosure should also be included in the scope of protection of the present disclosure and are included here by reference.

[0073] The specific structural and functional details disclosed herein are merely representative and are for the purpose of describing exemplary embodiments of the present disclosure. However, the various embodiments of the present disclosure may be implemented in many alternative forms and should not be interpreted as being limited to only the embodiments set forth herein.

[0074] It should be understood that, although the terms "first", "second", etc. may be used herein to describe various units, these units should not be limited by these terms. These terms are used only to distinguish one unit from another unit. For example, without departing from the scope of the exemplary embodiments, the first unit may be referred to as the second unit, and similarly the second unit may be referred to as the first unit. The term "and / or" used herein includes any and all combinations of one or more of the listed associated items.

[0075] The terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments. Unless the context clearly indicates otherwise, the singular forms "one", "one" and "item" used herein are also intended to include plural numbers. It should also be understood that the terms "include" and / or "comprise" used herein specify the existence of stated features, integers, steps, operations, units and / or components, without excluding the existence or addition of one or more other features, integers, steps, operations, units, components and / or combinations thereof.

[0076] It should also be noted that, in some alternative implementations, the functions / actions mentioned may occur in a different order than that indicated in the accompanying drawings. For example, depending on the functions / actions involved, two figures shown in succession may actually be performed substantially simultaneously or may sometimes be performed in the reverse order.

[0077] See also Figure 1 , which shows the process of data storage in the Hadoop distributed file system according to an embodiment of the present disclosure.

[0078] See also Figure 1 In step S1, the computer device classifies each table partition into cold data, warm data or hot data according to the data access frequency of each table partition, wherein the data access frequency of cold data, warm data and hot data increases in sequence; in step S2, the computer device executes a corresponding storage strategy according to the classification of each table partition, wherein the storage efficiency of cold data, warm data and hot data decreases in sequence.

[0079] Specifically, in step S1, the computer device divides each table partition into cold data, warm data or hot data according to the data access frequency of the table partition.

[0080] In HDFS, table partitioning is usually associated with distributed data processing frameworks (such as Hive or HBase), and is mainly used to improve query efficiency and data management flexibility for large-scale data sets. Table partitioning optimizes performance by physically dividing data by a field or dimension to reduce the amount of data scanned during query.

[0081] Table partitioning is to divide a large table into smaller, independently stored data sets based on certain specific column values ​​(such as date, region, or category). Each partition corresponds to a specific directory, and the data is stored in HDFS in the form of files. The use of partitions can speed up queries, especially queries on specific ranges of data.

[0082] Partition columns are columns in a table that are used for partitioning. For example, when partitioning by date, the date column will be used to divide the data. Each partition will correspond to a different date value, and each partition stores a related subset of data.

[0083] Therefore, table partitioning in HDFS is to divide data into independent directories according to certain dimensions to improve query efficiency and management flexibility. Combined with EC strategy, different storage optimization methods can be used for partitioned data of different temperatures to further improve storage efficiency and performance.

[0084] DFS audit logs can be used to count cold, warm, and hot data in the cluster at the granularity of table partitions.

[0085] HDFS audit logs record all access operations to the file system, including query, delete, update, etc. These logs can be used to analyze the access frequency of files and directories and help determine which data is hot, warm, or cold.

[0086] To facilitate analysis, the audit logs are exported to a log system, such as ElasticSearch, Splunk, or other log collection and analysis tools. Log files within a period of time can be regularly transferred to ensure that there is enough long-term data for statistical analysis.

[0087] The output audit log is processed and counted through the log system, the access frequency of the data is analyzed, and cold, warm, and hot data with table partitions as the granularity are generated. For example, according to the file path and operation type (such as query, delete, update) in the audit log, the relevant access records are filtered out. Usually, the data of the table partition is stored in a specific HDFS path, such as / user / hive / warehouse / table_name / part_col=value. In some embodiments, the number of accesses to each partition is counted according to the file path. Partitions with high access frequency are hot data, those with lower frequency are warm data, and partitions with almost no access are cold data. In some other embodiments, data access trends can be analyzed by time period to capture dynamic changes in hot and cold data.

[0088] In order to accurately classify data, you can define access frequency thresholds to distinguish between cold, warm, and hot data. Specific standards can be customized according to business needs, for example:

[0089] Hot data: partitions that are frequently accessed in the recent period.

[0090] Warm data: Partitions that are occasionally accessed, usually used in specific analysis or query tasks.

[0091] Cold data: partitions that have not been accessed for a long time.

[0092] By enabling HDFS audit logs, outputting logs to the log system, and analyzing the frequency of data access based on table partitions, you can identify cold, warm, and hot data in the HDFS cluster. Then, based on the different characteristics of data access, apply appropriate storage optimization strategies to improve storage efficiency and maintain good system performance.

[0093] According to one example, the statistical decision rule is as follows:

[0094] The value of X is configurable, the default value is X=1;

[0095] The value of Y can be configured, the default value is Y=1;

[0096]

[0097] In some embodiments, the cold, warm and hot data can be classified according to the above default rules. In some other embodiments, the classification rules of the cold, warm and hot data can also be configured by the user.

[0098] Next, in step S2, the computer device performs corresponding storage strategies according to the data temperature classification of each table partition, wherein the storage efficiency of cold, warm and hot data decreases in sequence.

[0099] In some embodiments:

[0100] For table partitions belonging to cold data, the first EC strategy is adopted;

[0101] For the table partition belonging to warm data, the second EC strategy is adopted, and the storage efficiency of the second EC strategy is lower than the storage efficiency of the first EC strategy;

[0102] For table partitions that belong to hot data, multiple copies of storage are used.

[0103] These storage strategies are described in detail below.

[0104] EC strategy is a technology used for data storage and data protection. It achieves data reliability, fault tolerance and efficient storage space utilization by dividing data into multiple data blocks and generating redundant check blocks. EC strategy is particularly suitable for distributed storage systems or cloud storage systems. It can restore data when a storage device fails, while reducing the waste of redundant storage space.

[0105] The basic idea of ​​the EC strategy is to divide the original data into several data blocks and generate corresponding check blocks. When some data blocks are lost, the system can recover the lost data through the remaining data blocks and check blocks. Different EC strategies can define different numbers of data blocks and check blocks to meet different storage efficiency and fault tolerance requirements.

[0106] EC strategy is usually expressed in k+m form:

[0107] k: The number of data blocks. The original data will be divided into k data blocks.

[0108] m: The number of check blocks (redundant blocks). Generate m check blocks to recover lost data. Usually, EC uses methods in linear algebra, such as Reed-Solomon coding, to generate redundant blocks.

[0109] For example, the 4+2EC strategy means dividing the data into 4 data blocks and generating 2 check blocks, so that even if 2 blocks (either data blocks or check blocks) are lost, the data can still be recovered through the remaining blocks.

[0110] EC strategy has the following key features:

[0111] 1. Fault Tolerance

[0112] Fault tolerance refers to how many blocks (data blocks or check blocks) the system can tolerate and still recover data. The larger the m value, the stronger the fault tolerance. For example, the 4+2EC strategy can tolerate the loss of up to 2 blocks, while the 6+3EC strategy can tolerate the loss of up to 3 blocks.

[0113] 2. Storage efficiency

[0114] The storage efficiency depends on the ratio of data block k to check block m. As the number of check blocks increases, the system can provide better fault tolerance, but the storage efficiency decreases.

[0115] Storage efficiency = k / (k+m)

[0116] For example, for a 3+2 strategy, the storage efficiency is 3 / (3+2)=60%. This requires 5 blocks to store data, 2 of which are redundant blocks. The loss of 2 blocks can be tolerated.

[0117] For a 10+4 strategy, the storage efficiency is 10 / (10+4)≈71.43%. This provides efficient storage space usage while tolerating the loss of up to 4 blocks.

[0118] Generally, higher storage efficiency means lower redundancy. When choosing an EC strategy, you also need to balance storage overhead and fault tolerance. For cold data, EC strategies with higher storage efficiency (such as 10+4) are applicable; for warm data, strategies with higher redundancy (such as 3+2) are more appropriate.

[0119] 3. Read and write performance

[0120] Write performance: When writing data, the system needs to generate and write the check blocks at the same time. The more data blocks and check blocks there are, the greater the amount of calculation and delay for writing.

[0121] Read performance: If all data blocks are available, the system can directly read the data blocks. If data blocks are lost, the system needs additional calculations to recover the lost data, which increases the reading time.

[0122] 4. Advantages of EC strategy

[0123] Efficient storage: Compared with simple data replication (multi-copy storage), EC uses redundant blocks for data protection, significantly reducing the waste of storage space. Generally, the EC strategy can reduce the storage cost to 1 / 3 of data replication or less.

[0124] Stronger fault tolerance: EC policies can tolerate failures of multiple storage devices or nodes, not just one. For example, 4+2 can tolerate failures of two devices, and 6+3 can tolerate failures of three.

[0125] Flexibility: You can choose different k+m combinations to balance storage efficiency and fault tolerance according to business needs. For example, using the 10+4 strategy can maximize storage efficiency while ensuring good fault tolerance.

[0126] Cold data is data with a very low access frequency, which is usually stored for a long time and rarely read or updated. Typical cold data includes archived data, historical logs, backup data, etc. Warm data is data that is not frequently accessed but still has a certain read demand. It is often used in data storage systems and needs to maintain a certain performance while also considering storage costs.

[0127] For warm data, a balance needs to be struck between fault tolerance, storage efficiency, and read performance, and an EC policy with a medium redundancy rate is usually selected. For cold data, storage efficiency is the primary consideration, and a higher data block ratio policy can be selected to maximize storage efficiency and reduce storage costs. Based on this, an EC policy with higher storage efficiency can be selected for cold data.

[0128] In some examples, for table partitions belonging to cold data, the EC policy of RS-10-4-1024k is adopted; for table partitions belonging to warm data, the EC policy of RS-3-2-1024k is adopted.

[0129] RS-10-4-1024k EC Strategy

[0130] The RS-10-4-1024k EC policy allows up to 4 data blocks (either data blocks or redundant blocks) to be damaged or lost, and still be able to recover the complete data through the remaining blocks. This can improve data reliability and fault tolerance while reducing the additional storage space required compared to the simple copy policy.

[0131] Here, the parameters in RS-10-4-1024k are as follows:

[0132] RS: Uses the Reed-Solomon erasure code algorithm, which is a very powerful algorithm that can provide data redundancy and fault tolerance for data loss recovery.

[0133] 10: represents the number of data blocks, that is, under this strategy, the original data is divided into 10 data blocks.

[0134] 4: represents the number of parity blocks. In this strategy, the system generates 4 parity blocks to provide fault tolerance for data recovery. If up to 4 data blocks are damaged or lost, the complete data can still be recovered.

[0135] 1024k: Each data block is 1024KB (1MB). Therefore, the original data size is 10MB (10 data blocks, each 1MB).

[0136] For cold data table partitions, the RS-10-4-1024k EC strategy is used to reduce storage space while ensuring high data availability and fault tolerance. Specifically, this strategy has the following benefits for cold data management:

[0137] 1. Storage space optimization

[0138] Using the RS-10-4-1024k erasure coding strategy, the data is divided into 10 data blocks and 4 redundant blocks are generated. Compared with the traditional replication strategy, such as backing up data three times, this erasure coding scheme can significantly reduce storage space overhead.

[0139] For example, for the normal copy strategy, assuming the original data is 10MB, storing three copies requires 30MB of storage space. When the RS-10-4-1024k erasure code strategy is used, the storage of 10MB of data plus 4 redundant blocks totals 14MB, saving about 53% of storage space.

[0140] 2. Fault Tolerance

[0141] After adopting the RS-10-4-1024k strategy, the system can tolerate the loss or damage of up to 4 blocks (including data blocks or redundant blocks). Even if 4 blocks are lost, the original data can still be restored through the remaining 10 blocks, which greatly improves the reliability of the data.

[0142] For cold data, although the access frequency is low, the security and recoverability of the data are still important. This erasure coding scheme provides sufficient fault tolerance while reducing storage.

[0143] 3. Application scenarios of cold data

[0144] Cold data refers to data that does not need to be accessed frequently for a long time, such as historical data, archived data, etc. Since cold data is not frequently read, storage optimization technology is usually used to save resources. Using the RS-10-4-1024k erasure code strategy can effectively reduce storage costs while ensuring that the integrity of the data is maintained when it needs to be restored.

[0145] 4. Impact on performance

[0146] For cold data, the performance requirements are relatively low, the access frequency is not high, and the time and cost of data recovery are often not the main considerations. Although the encoding and decoding process of Reed-Solomon erasure codes is relatively time-consuming and resource-intensive, this cost is acceptable for cold data because reducing storage costs is the main priority.

[0147] Through the RS-10-4-1024k erasure coding strategy, cold data can be stored efficiently while ensuring that they can still be restored and maintain integrity if they need to be retrieved. This can achieve a good balance in the storage of cold data table partitions, greatly reducing storage costs while providing sufficient fault tolerance and data recovery capabilities.

[0148] RS-3-2-1024k EC Strategy

[0149] The RS-3-2-1024k EC strategy can recover data when up to 2 data blocks or redundant blocks are lost. Compared with the traditional copy backup method, this erasure coding strategy greatly reduces the use of storage space while maintaining a certain fault tolerance, which is suitable for application scenarios with low access frequency, such as storage management of warm data.

[0150] Here, the parameters in RS-3-2-1024k represent:

[0151] RS: Indicates that Reed-Solomon erasure code is used.

[0152] 3: represents the number of data blocks. That is, under this strategy, the original data is divided into 3 data blocks.

[0153] 2: represents the number of redundant blocks. In this strategy, the system generates 2 redundant blocks to provide fault tolerance for data recovery. If at most 2 data blocks are damaged or lost, the complete data can still be recovered.

[0154] 1024k: Each data block is 1024KB (1MB). Therefore, the original data size is 3MB (3 data blocks, each 1MB).

[0155] For table partitions of warm data, using the RS-3-2-1024k erasure coding strategy is a solution that balances storage costs and fault tolerance. Warm data is data between hot data (frequently accessed) and cold data (rarely accessed), with a certain access frequency, but not as frequent as hot data. Therefore, using this strategy in warm data can effectively optimize storage resources while ensuring sufficient fault tolerance. Specifically, this strategy has the following benefits for warm data management:

[0156] 1. Storage efficiency:

[0157] The traditional 3-copy strategy requires 3 copies of data to be backed up, which takes up 3 times the storage space of the original data. However, the RS-3-2-1024k erasure code strategy only takes up 1.67 times the storage space. For example, if the original data is 3MB, after adding 2 redundant blocks, the total storage capacity is 5MB.

[0158] Compared to a simple replication strategy (for example, 3 replicas will occupy 9MB of space), the RS-3-2-1024k erasure coding strategy only occupies 5MB, saving about 44% of storage space.

[0159] Since the access frequency of warm data is between that of cold data and hot data, storage space optimization is still an important consideration. The RS-3-2 erasure coding strategy reduces the storage space overhead (compared to the copy strategy), provides redundancy for data, and reduces the overall storage cost.

[0160] 2. Fault Tolerance:

[0161] The configuration of 3 data blocks + 2 redundant blocks in RS-3-2-1024k means that even if 2 blocks (either data blocks or redundant blocks) are lost or damaged, data can still be recovered through the remaining blocks. This fault tolerance capability is very suitable for warm data application scenarios because it can provide sufficient reliability without wasting a lot of storage resources like 3 copies.

[0162] 3. Application scenarios of warm data

[0163] In storage systems, warm data usually includes the following types:

[0164] Business data from the last few months: This data still needs to be accessed more frequently, but does not require extremely low access latency like the data from the current day.

[0165] Log files: system operation logs and application logs. The most recent logs may be accessed more frequently, while older logs have been archived as cold data.

[0166] Back up files regularly: such as data backup from a week or a month ago. The access frequency is not high, but a certain degree of recoverability is required.

[0167] In these scenarios, using the RS-3-2-1024k erasure coding strategy can significantly reduce storage resource overhead and provide reasonable fault tolerance and recoverability.

[0168] 4. Impact on performance

[0169] Compared to a simple copy replication strategy, Reed-Solomon erasure codes require an additional decoding process when reading data, so the reading performance is slightly reduced. However, for warm data, this performance overhead is acceptable. Every time data is written or updated, redundant blocks need to be generated, which may slightly reduce write performance. However, warm data is usually not updated as frequently as hot data, so this type of overhead is relatively small.

[0170] In some embodiments, for table partitions belonging to hot data, a high redundancy rate EC strategy can be adopted, and storage efficiency will also be reduced, such as the RS-8-8-1024k EC strategy. The more check blocks there are, the better the fault tolerance.

[0171] In some other embodiments, for table partitions belonging to hot data, data replication storage (Replication Storage), that is, multi-copy storage, may be used.

[0172] Data replication storage is a simple and direct way to protect data. It provides redundancy and fault tolerance by completely copying data to multiple copies (usually called mirrors or replicas). Multi-copy storage has the following characteristics:

[0173] 1. Redundancy: Each data block has one or more complete copies. For example, if a 3-copy replication strategy is used, each piece of data will be completely stored on 3 different nodes or devices.

[0174] 2. Fault tolerance: Data can be recovered by reading any replica. Therefore, if a storage node or device fails, as long as at least one replica is available, the system can recover the data normally.

[0175] 3. Storage efficiency: Data replication has low storage efficiency because redundant copies significantly increase storage requirements. The storage efficiency of the 3-copy strategy is 33.33%, which means that for every 1TB of original data stored, 3TB of storage space is actually required.

[0176] 4. Read and write performance:

[0177] Read performance: Replicated storage has better read performance because data can be read from multiple replicas, and reads can even be accelerated by accessing multiple replicas in parallel.

[0178] Write performance: When writing data, replicated storage usually needs to synchronize the data to multiple replicas, so the write latency is higher, especially when the number of replicas is large.

[0179] Typically, in some examples, a table partition belonging to hot data can be stored in three copies.

[0180] 3 copies of storage

[0181] In 3-copy storage, each data block is replicated three times and stored on three different nodes or disks. This means that even if two nodes fail, the system can still recover the data from the remaining replicas. This strategy is often used in distributed file systems such as Hadoop HDFS.

[0182] For "hot data" (i.e. frequently accessed data), it is common to use multiple copies of the data. The main reasons for using 3 copies of hot data are:

[0183] 1. High availability: Triple copy storage ensures that even if one copy fails, other copies can still provide data services, thus ensuring high data availability.

[0184] 2. Read performance optimization: Multiple replicas can share the read load and improve query concurrency and response speed. In this way, when data is frequently accessed, read operations can be more efficient.

[0185] 3. Data security: Multi-copy storage can protect data from loss, even in the event of hardware failure or other disaster events, it can ensure that data will not be lost.

[0186] 4. Fault tolerance: By retaining multiple copies, the system can tolerate the failure of a certain number of copies without affecting data integrity and service availability.

[0187] According to some embodiments, for each table partition in the HDFS system, if the current first table partition A is cold data or warm data, a second table partition B is created and a corresponding storage policy is configured for the second table partition B. The data of the first table partition A is written to the second table partition B according to the configured storage policy. If the first table partition A is cold data, the data is written to the second table partition B according to the EC policy corresponding to the cold data. If the first table partition A is warm data, the data is written to the second table partition B according to the EC policy corresponding to the warm data. Finally, the new table partition into which the data is written provides services related to writing data.

[0188] For some examples, see Figure 2 For each table partition in the HDFS system, the corresponding EC policy is executed based on the following process:

[0189] Step 201. Get "original data table partition A".

[0190] Step 202: Determine whether the table partition is cold data or warm data:

[0191] If it is not cold data or warm data, the process in step 203 ends directly without performing EC storage;

[0192] If it is cold data or warm data, continue to execute the next step 204.

[0193] Step 204. Create a new table partition B:

[0194] Create a new table partition B, and configure the corresponding EC policy based on the data temperature classification of the original table partition A. For example, configure the RS-10-4-1024k EC policy for cold data and the RS-3-2-1024k EC policy for warm data.

[0195] Step 205. Record metadata:

[0196] Record the latest update time and other information about table partition A in the HDFS metadata (information X).

[0197] Step 206. Data replication (A->B):

[0198] The data in table partition A is copied to the newly created table partition B. During the data writing process, the corresponding EC policy is executed on the data.

[0199] Step 207. Permission lock:

[0200] After the copy is complete, restrict access to table partition A. This means that ordinary users will no longer be able to read or modify the data in table partition A. Specifically, lock the current access permissions of table partition A through file user access permissions, prohibiting non-super users from accessing it. Locking permissions can ensure that other users will not accidentally modify the data during the data copying and verification period, ensuring data consistency. At the same time, it will not affect the user's normal access to the file.

[0201] Step 208. Check data consistency:

[0202] After the copy is complete, check whether the latest update time and other information about table partition A in the HDFS metadata are the same as the previously recorded information X to ensure data consistency.

[0203] Here, step 209 determines whether the metadata are the same:

[0204] If the metadata are the same, it means the data are consistent, and the next step 211 is executed.

[0205] If the metadata is not the same, step 210 deletes the data of table partition B, unlocks the user rights of table partition A, and returns to step 205 to re-execute a new round of data replication.

[0206] Step 211. Rename table partition A to A_del.

[0207] Step 212: Rename EC table partition B to A.

[0208] Step 213. Data / Service Migration:

[0209] New EC table partition A starts providing services.

[0210] Step 214. Delete the renamed old table partition:

[0211] Delete the renamed old table partition A_del to free up space and complete the process.

[0212] The core of the entire process is to apply erasure coding technology on cold and warm data to reduce the use of storage space while ensuring data consistency and integrity.

[0213] It should be noted that the various embodiments of the present disclosure may be implemented in software and / or a combination of software and hardware, for example, may be implemented using an application specific integrated circuit (ASIC), a general purpose computer, or any other similar hardware device. In one embodiment, the software program of the various embodiments of the present disclosure may be executed by a processor to implement the steps or functions described above. Similarly, the software program of the various embodiments of the present disclosure (including related data structures) may be stored in a computer-readable recording medium, for example, a RAM memory, a magnetic or optical drive, or a floppy disk and the like. In addition, some steps or functions of the various embodiments of the present disclosure may be implemented using hardware, for example, as a circuit that cooperates with a processor to perform various steps or functions.

[0214] In addition, at least a portion of the embodiments of the present disclosure may be applied as a computer program product, such as a computer program instruction, which, when executed by a computing device, can call or provide the method and / or technical solution according to the embodiments of the present disclosure through the operation of the computing device. The program instruction for calling / providing the method of the embodiments of the present disclosure may be stored in a fixed or removable recording medium, and / or transmitted through a data stream in a broadcast or other signal-bearing medium, and / or stored in a working memory of a computing device that runs according to the program instruction.

[0215] For those skilled in the art, it is obvious that the embodiments of the present disclosure are not limited to the details of the above exemplary embodiments, and the embodiments of the present disclosure can be implemented in other specific forms without departing from the spirit or basic features of the embodiments of the present disclosure. Therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the embodiments of the present disclosure is limited by the attached claims rather than the above description, so it is intended to include all changes within the meaning and scope of the equivalent elements of the claims in the embodiments of the present disclosure. Any figure mark in the claims should not be regarded as limiting the claims involved. In addition, it is obvious that the word "including" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the system claim can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any specific order.

Claims

1. A method for storing data in a Hadoop distributed file system, wherein: The method comprises the following steps: According to the data access frequency of each table partition, the partition is divided into cold data, warm data or hot data, wherein the data access frequency of the cold data, the warm data and the hot data increases in sequence; Execute corresponding storage strategies according to the classification of the partitions of each table, wherein the storage efficiency of the cold data, the warm data, and the hot data decreases in sequence; in: For the table partition belonging to the cold data, a first EC storage strategy is adopted, where the first EC storage strategy is an RS-10-4-1024k EC strategy; For the table partition belonging to the warm data, a second EC storage strategy is adopted, where the second EC storage strategy is an RS-3-2-1024k EC strategy; For the table partition belonging to the hot data, a third EC storage strategy or multi-copy storage is adopted, wherein the storage efficiency of the third EC strategy is lower than the storage efficiency of the second EC strategy; The step of executing the corresponding storage strategy specifically includes: For the current first table partition, if it belongs to the cold data or the warm data, create a second table partition and configure a corresponding storage policy for the second table partition; Writing the data of the first table partition into the second table partition according to the configured storage policy; After the data is written, the access to the first table partition is locked; Check whether the first table partition is updated. If not, rename the second table partition to the first table partition and delete the original first table partition. The second table partition becomes a new first table partition to provide services related to writing data.

2. A device for storing data in a Hadoop distributed file system, wherein: The device comprises a memory and a processor, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the device is configured to perform the following operations: According to the data access frequency of each table partition, the partition is divided into cold data, warm data or hot data, wherein the data access frequency of the cold data, the warm data and the hot data increases in sequence; Execute corresponding storage strategies according to the classification of the partitions of each table, wherein the storage efficiency of the cold data, the warm data, and the hot data decreases in sequence; in: For the table partition belonging to the cold data, a first EC storage strategy is adopted, where the first EC storage strategy is an RS-10-4-1024k EC strategy; For the table partition belonging to the warm data, a second EC storage strategy is adopted, where the second EC storage strategy is an RS-3-2-1024k EC strategy; For the table partition belonging to the hot data, a third EC storage strategy or multi-copy storage is adopted, wherein the storage efficiency of the third EC strategy is lower than the storage efficiency of the second EC strategy; The operations of executing the corresponding storage policy specifically include: For the current first table partition, if it belongs to the cold data or the warm data, create a second table partition and configure a corresponding storage policy for the second table partition; Writing the data of the first table partition into the second table partition according to the configured storage policy; After the data is written, the access to the first table partition is locked; Check whether the first table partition is updated. If not, rename the second table partition to the first table partition and delete the original first table partition. The second table partition becomes a new first table partition to provide services related to writing data.

3. A computer program product comprising computer program instructions, wherein: When the computer program instructions are executed by a computer device, the computer device is configured to perform the method as claimed in claim 1.

4. A computer program storage medium having computer executable instructions stored therein, wherein when the computer executable instructions are executed by a computer device, the computer device is configured to perform the method according to claim 1.

Images