A data processing method, device, medium and program product

Abstract

The application discloses a data processing method and device, medium and program product, relates to the technical field of storage, and comprises the following steps: obtaining to-be-stored data; encoding the to-be-stored data according to the size of a data file; determining a target data level according to the data attribute of the to-be-stored data; determining a target data protection strategy corresponding to the target data level according to the data attribute of the target data level; determining the number of data blocks and check blocks required for writing the to-be-stored data according to the target data protection strategy; determining target storage nodes corresponding to the data blocks and the check blocks according to the number of the data blocks and the check blocks required for writing the to-be-stored data; and storing the encoded to-be-stored data into the target storage nodes according to the target data level and the target data protection strategy. The application can take into account the utilization rate and data read-write performance of a disk, reduce resource calculation overhead, and improve file processing efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of storage, and in particular to a data processing method, device, medium and program product. BACKGROUND

[0002] With the rapid development of emerging technologies such as cloud computing, big data, artificial intelligence, data is growing explosively, and the massive data puts forward very high requirements on the capacity, performance, reliability and other aspects of the storage system. Distributed storage systems have become the mainstream choice for mass data storage due to their strong scalability, high fault tolerance, relatively low cost and other advantages.

[0003] In a distributed storage system, data protection is one of the core links, which aims to prevent data loss caused by hardware failure (such as disk damage, node failure, etc.) and ensure data availability and integrity. At present, the mainstream data protection mechanisms mainly include two schemes: multi-copy and erasure code. The multi-copy scheme is derived from the early centralized storage system, and its core is to copy the same data multiple times and store it on different physical nodes or disks. When one copy is damaged, the system can recover the data through other copies. This scheme is simple to implement and has excellent read-write operation performance, because when reading data, you can choose a node with lighter load from multiple copies, and when writing data, although you need to write multiple copies, the operation logic is simple and direct. The erasure code scheme is derived from the coding theory in the communication field, and its basic principle is to divide the original data into several data blocks and calculate the corresponding check blocks through a specific encoding algorithm. When some data blocks or check blocks are lost, the remaining blocks can be used to recover the lost data blocks through decoding algorithm. Compared with the multi-copy scheme, the erasure code scheme requires less storage space and higher disk utilization under the same reliability requirement.

[0004] However, with the continuous expansion of the scale of distributed storage systems and the increasing complexity of application scenarios, both the multi-copy scheme and the erasure code scheme cannot balance disk utilization and data read-write performance on the basis of ensuring data reliability. SUMMARY

[0005] The application provides a data processing method, device, medium and program product. The method comprises the following steps: obtaining to-be-stored data; encoding the to-be-stored data according to the size of a data file; determining a target data level according to the data attribute of the to-be-stored data, wherein the data attribute comprises one or more of data access frequency, data file size and data read-write performance requirement; determining a target data protection strategy corresponding to the target data level according to the data attribute of the target data level; determining the number of data blocks and check blocks required for writing the to-be-stored data according to the target data protection strategy; determining the target storage node corresponding to the data blocks and check blocks according to the number of data blocks and check blocks required for writing the to-be-stored data; and storing the encoded to-be-stored data into the target storage node according to the target data level and the target data protection strategy. The application can balance the utilization rate of the disk and the data read-write performance, reduce resource calculation overhead and improve file processing efficiency.

[0006] The application provides a data processing method. The method is applied to a distributed storage system, the system comprises a data processing module and a plurality of storage nodes, and the method comprises the following steps:

[0007] The data processing module obtains to-be-stored data.

[0008] The to-be-stored data is encoded according to the size of a data file.

[0009] A target data level is determined according to the data attribute of the to-be-stored data, wherein the data attribute comprises one or more of data access frequency, data file size and data read-write performance requirement.

[0010] A target data protection strategy corresponding to the target data level is determined according to the data attribute of the target data level.

[0011] The number of data blocks and check blocks required for writing the to-be-stored data is determined according to the target data protection strategy.

[0012] The target storage node corresponding to the data blocks and check blocks is determined according to the number of data blocks and check blocks required for writing the to-be-stored data.

[0013] The encoded to-be-stored data is stored into the target storage node according to the target data level and the target data protection strategy.

[0014] The application further provides a distributed storage system, which comprises a data processing module and a plurality of storage nodes (disks). The data processing module is used for:

[0015] Obtaining to-be-stored data.

[0016] Encoding the to-be-stored data according to the size of a data file.

[0017] determine a target data level according to data attributes of the data to be stored, wherein the data attributes comprise one or more of data access frequency, data file size, and data read-write performance requirement;

[0018] determine a target data protection strategy corresponding to the target data level according to data attributes of the target data level;

[0019] determine a number of data blocks and check blocks to which the data to be stored needs to be written according to the target data protection strategy;

[0020] determine target storage nodes corresponding to the data blocks and the check blocks according to the number of the data blocks and the check blocks to which the data to be stored needs to be written;

[0021] store the encoded data to be stored to the target storage nodes according to the target data level and the target data protection strategy.

[0022] The application further provides an electronic device, comprising a memory for storing a computer program and a processor for executing the computer program to implement steps of a data processing method, the method comprising:

[0023] obtaining data to be stored;

[0024] encoding the data to be stored according to a size of a data file;

[0025] determining a target data level according to data attributes of the data to be stored, wherein the data attributes comprise one or more of data access frequency, data file size, and data read-write performance requirement;

[0026] determining a target data protection strategy corresponding to the target data level according to data attributes of the target data level;

[0027] determining a number of data blocks and check blocks to which the data to be stored needs to be written according to the target data protection strategy;

[0028] determining target storage nodes corresponding to the data blocks and the check blocks according to the number of the data blocks and the check blocks to which the data to be stored needs to be written;

[0029] storing the encoded data to be stored to the target storage nodes according to the target data level and the target data protection strategy.

[0030] The application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement steps of a data processing method, the method comprising:

[0031] obtaining data to be stored;

[0032] encoding the data to be stored according to a size of a data file;

[0033] determine a target data tier according to data attributes of the data to be stored, wherein the data attributes comprise one or more of data access frequency, data file size, data read-write performance requirement;

[0034] determine a target data protection policy corresponding to the target data tier according to data attributes of the target data tier;

[0035] determine a number of data blocks and check blocks required for the data to be stored to be written according to the target data protection policy;

[0036] determine target storage nodes corresponding to the data blocks and the check blocks according to the number of the data blocks and the check blocks required for the data to be stored to be written;

[0037] store the encoded data to be stored to the target storage nodes according to the target data tier and the target data protection policy.

[0038] The application further provides a computer program product comprising a computer program, which, when executed by a processor, implements the steps of the data processing method, the method comprising:

[0039] obtain data to be stored;

[0040] encode the data to be stored according to a size of the data file;

[0041] determine a target data tier according to data attributes of the data to be stored, wherein the data attributes comprise one or more of data access frequency, data file size, data read-write performance requirement;

[0042] determine a target data protection policy corresponding to the target data tier according to data attributes of the target data tier;

[0043] determine a number of data blocks and check blocks required for the data to be stored to be written according to the target data protection policy;

[0044] determine target storage nodes corresponding to the data blocks and the check blocks according to the number of the data blocks and the check blocks required for the data to be stored to be written;

[0045] store the encoded data to be stored to the target storage nodes according to the target data tier and the target data protection policy.

[0046] According to the application, the method comprises: obtaining to-be-stored data; encoding the to-be-stored data according to the size of the data file; determining a target data level according to the data attribute of the to-be-stored data, wherein the data attribute comprises one or more of data access frequency, data file size, and data read-write performance requirement; determining a target data protection strategy corresponding to the target data level according to the data attribute of the target data level; determining the number of data blocks and check blocks required for writing the to-be-stored data according to the target data protection strategy; determining target storage nodes corresponding to the data blocks and check blocks according to the number of data blocks and check blocks required for writing the to-be-stored data; and storing the encoded to-be-stored data to the target storage nodes according to the target data level and the target data protection strategy. The application can balance the utilization rate of the disk and the data read-write performance, reduce resource calculation overhead, and improve file processing efficiency.

[0047] The technical scheme of the application uses different data protection strategies for data with different attributes through a dynamic hierarchical storage architecture, greatly improves the disk utilization rate under the premise of ensuring data reliability: the hot data layer uses a 2-replica + check block mode, which improves the disk utilization rate by about 33% compared with the traditional 3-replica scheme; the warm data layer uses a hybrid erasure code strategy, which improves the disk utilization rate by 40%-50% compared with the multi-replica scheme; the cold data layer uses an efficient erasure code strategy, and the disk utilization rate can reach more than 80%, which is more than 60% higher than the multi-replica scheme; therefore, the overall effective utilization rate of the disk of the application is improved by 30%-50%, significantly reducing the storage cost.

[0048] The intelligent I / O scheduling strategy of the technical scheme of the application can select the optimal data source for read requests, and the parallel reading of multiple replicas of hot data further improves the reading speed. Experimental data shows that the read performance of hot data is improved by 15%-20% compared with the traditional multi-replica scheme, and the read performance of warm data and cold data is improved by 20%-30% compared with the traditional erasure code scheme; the adaptive encoding switching mechanism uses a lightweight encoding algorithm for small files, reducing the encoding overhead of small files; the precomputed check block pool reduces the real-time computation time of the check block; the asynchronous writing of hot data reduces the client waiting time from the replica mode; the batch writing strategy of warm data and cold data reduces the disk seek and network transmission overhead; the I / O write performance of the application is improved by 10%-20% compared with the traditional multi-replica scheme, and by 30%-40% compared with the traditional erasure code scheme.

[0049] The technical scheme of the application adopts a pre-computed check block pool mechanism, uses system idle resources to pre-compute check blocks, and reduces real-time computation overhead during data writing and updating; an adaptive encoding switching mechanism uses a simple lightweight encoding algorithm for small files, reducing the encoding and decoding computation amount of the small files; and an improved RS encoding algorithm reduces the number of multiplication operations in the encoding process, and reduces the computation overhead of medium-sized files, so that the overall computation overhead of the application is reduced by 25%-35% compared with a traditional erasure code scheme.

[0050] The technical scheme of the application is targeted at the characteristics of small files, and the application adopts an optimized multi-copy strategy for small files in the hot data layer, and selects a lightweight XOR check encoding in the encoding mode, thereby reducing the encoding overhead and storage overhead of the small files. Here, the read-write performance of the small files is improved by 40%-50% compared with a traditional erasure code scheme, and the problem of low processing efficiency of traditional erasure code technology for small files is solved. BRIEF DESCRIPTION OF DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the application, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0052] Figure 1 A multi-copy data protection strategy diagram provided for the embodiments of the application;

[0053] Figure 2 An erasure code data protection strategy diagram provided for the embodiments of the application;

[0054] Figure 3 A first flowchart of data processing provided for the embodiments of the application;

[0055] Figure 4 A second flowchart of data processing provided for the embodiments of the application;

[0056] Figure 5 A specific flowchart of data processing provided for the embodiments of the application;

[0057] Figure 6 An exemplary system that can be used to implement various embodiments of the application is provided for the embodiments of the application. DETAILED DESCRIPTION

[0058] With reference to the drawings and specific embodiments described below, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0059] It should be noted that, in the description of the present application, the terms "comprise", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. The terms "first", "second" and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0060] In order to enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0061] In combination with the specific application environment architecture or specific hardware architecture on which the data processing method is dependent, the specific application environment architecture or specific hardware architecture is described herein.

[0062] As shown in Figure 1 Currently, there are mainly two schemes for data protection mechanism in distributed storage: multi-copy and erasure code. Multi-copy technology is one of the earliest widely used data protection schemes in distributed storage systems. Its typical implementation is to copy data into two or three or even more copies, which are stored in different nodes or disks. A common example is the three-copy scheme. In this scheme, when the client writes data, the data is first written to the primary copy and then synchronized to other secondary copies. Only when all copies are successfully written, the write success response is returned to the client. When reading data, the client can read data from any available copy, and the system will select the optimal read node according to the load of the physical node where each copy is located.

[0063] As shown in Figure 2As shown, the typical application of erasure code technology in distributed storage is to divide the original data into k data blocks, and then generate m parity blocks through encoding calculation, and store the k+m blocks on different physical nodes or disks; when no more than m blocks (data or parity blocks) are lost, the original data can be recovered through the remaining parity blocks; common erasure code schemes include Reed-Solomon (RS) code, such as RS(4,2), which means dividing data into 4 data blocks and generating 2 parity blocks, allowing up to 2 blocks to be lost (whether data blocks or parity blocks).

[0064] However, as the scale of distributed storage systems continues to expand and application scenarios become increasingly complex, both multi-copy technology and erasure code technology gradually expose their respective limitations; how to balance disk utilization and data read-write performance while ensuring data reliability has become a key problem in the field of distributed storage that needs to be solved.

[0065] The current two data protection mechanisms, although their implementation schemes are different, both have certain defects:

[0066] Multi-copy technology, although simple to implement and good read-write performance, has obvious shortcomings: first, the disk utilization is extremely low, for example, with a three-copy scheme, storing one copy of data consumes three times the disk space, and a large amount of storage space is wasted, as the amount of data increases, this waste of storage space will bring huge cost pressure; second, write performance is limited in large-scale data scenarios, although the write logic of a single copy of data is simple, when the amount of data is huge, writing multiple copies will occupy a large amount of network bandwidth and disk I / O resources, resulting in a decline in overall write performance; in addition, data consistency maintenance is complex, during the replication process, network failures, node crashes, and other abnormal situations may occur, requiring complex consistency protocols to ensure data consistency between copies, increasing the difficulty of system design and implementation.

[0067] Although the erasure code technology improves the disk utilization, it has two major shortcomings: first, the calculation overhead is large, and a large number of mathematical operations are required in the encoding and decoding process, especially when data is written and recovered, a large amount of processor CPU resources are consumed, affecting the overall performance of the system; second, the small file write performance is poor, for small files, erasure code needs to split the file into small blocks for encoding, and the calculation overhead of the encoding process is relatively large compared to the size of the file itself, resulting in low efficiency of small file writing; third, the data update cost is high, when some data blocks in the original data are updated, the related check blocks need to be recalculated, and for an erasure code scheme containing multiple check blocks, this update operation will bring a large calculation and I / O overhead; in addition, the recovery speed is slow, when data loss needs to be recovered, multiple remaining blocks need to be read for decoding calculation, not only a large amount of CPU resources are consumed, but also a large amount of network bandwidth is occupied, resulting in a long recovery process, and if data block loss occurs again during the recovery process, permanent data loss may occur.

[0068] It can be understood that the existing multiple copy technology has low disk utilization, resulting in serious waste of storage space and increased storage cost, especially in the massive data storage scenario, the cost pressure is more prominent;

[0069] The multiple copy technology occupies a large amount of network bandwidth and disk I / O resources when writing large-scale data, resulting in a decline in write performance and failing to meet the demand for high-performance storage;

[0070] The erasure code technology has large calculation overhead, and a large amount of CPU resources are consumed in the encoding and decoding process, affecting the overall performance of the system, especially during data writing and recovery, the performance bottleneck is obvious;

[0071] The erasure code technology has low processing efficiency for small files, and the encoding overhead of small files is relatively large, resulting in poor I / O read and write performance of small files;

[0072] The cost of data update and recovery in the erasure code technology is high, multiple check blocks need to be recalculated when data blocks are updated, and multiple blocks need to be read for decoding when data is recovered, which is time-consuming and resource-consuming;

[0073] Therefore, the related technical solutions cannot dynamically adjust the data protection strategy according to the characteristics of data access frequency, importance, size, etc., resulting in difficulty in simultaneously considering disk utilization and data read / write I / O performance in different scenarios.

[0074] Embodiments of the present application provide a data processing method, as shown in Figure 3 The method comprises the following steps:

[0075] Obtaining to-be-stored data;

[0076] encoding the data to be stored according to the size of the data file;

[0077] determining a target data level according to the data attribute of the data to be stored, wherein the data attribute comprises one or more of data access frequency, data file size, and data read-write performance requirement;

[0078] determining a target data protection strategy corresponding to the target data level according to the data attribute of the target data level;

[0079] determining the number of data blocks and check blocks required for the data to be stored to be written according to the target data protection strategy;

[0080] determining the target storage node corresponding to the data blocks and check blocks according to the number of data blocks and check blocks required for the data to be stored to be written;

[0081] storing the encoded data to be stored to the target storage node according to the target data level and the target data protection strategy.

[0082] It can be understood that, based on the current problems, the present application proposes a data processing method that takes into account the disk utilization rate and I / O read-write performance, introduces a dynamic hierarchical storage architecture, an adaptive encoding switching mechanism, a pre-computed check block pool, an intelligent I / O read-write scheduling and a data recovery mechanism, and makes up for the defects of the multi-copy and erasure code schemes.

[0083] Storing data information: input the data information that needs to be saved into the system, the data information includes various types of data such as text, image, video, etc.;

[0084] Adaptive encoding switching: before the data is stored, the system will select the most suitable encoding method according to the characteristics of the data, this adaptive mechanism can improve the compression efficiency of the data, reduce the occupation of the storage space, and at the same time ensure the integrity and readability of the data;

[0085] Dynamic hierarchical storage: the encoded data will be allocated to different storage levels, this hierarchical storage strategy can store the data on the most suitable medium according to the access frequency, importance and other factors of the data, such as high-frequency access data may be stored on faster solid state drive SSD, while low-frequency access data may be stored on lower-cost mechanical hard drive HDD;

[0086] Pre-computed check block pool: in order to ensure the integrity of the data, the system will pre-compute the stored data, generate check blocks and store them in the pre-computed check block pool, these check blocks can be used to verify the correctness of the data during data recovery, and ensure that the data is not damaged during transmission or storage;

[0087] Intelligent I / O scheduling strategy: Through intelligent I / O scheduling algorithm, the system can effectively manage data reading and writing operations, avoid I / O bottlenecks, and improve system response speed and overall performance.

[0088] Data recovery mechanism: When data is lost or damaged, the system can recover data through check blocks and other copies to ensure that data can be recovered in time even in the event of failure, ensuring data security and availability.

[0089] The whole process is a closed loop system, starting from data storage, through encoding, hierarchical storage, checking, scheduling and other links, finally realizing safe storage and efficient management of data.

[0090] Embodiments of the present application provide a data processing method, as shown in Figure 4 The method comprises the following steps:

[0091] Step S01, obtaining data to be stored;

[0092] According to the size of the data file, the data to be stored is encoded.

[0093] Here, the file size has a very large impact on processing efficiency during data storage, especially for small file processing, which not only consumes time, but also consumes a large amount of computing resources. In order to further optimize the processing efficiency of files of different sizes, the present application introduces an adaptive encoding switching mechanism to automatically select the appropriate encoding method according to the size of the file.

[0094] Step S011, when the memory space size value of the data file is less than the first preset value (32MB), the data to be stored is encoded by using the XOR check encoding algorithm;

[0095] When the memory space size value of the data file is greater than the first preset value (32MB) and less than the second preset value (1GB), the data to be stored is encoded by using the second forward error correction code encoding algorithm;

[0096] When the memory space size value of the data file is greater than the second preset value (1GB), the data to be stored is encoded by using the first forward error correction code encoding algorithm.

[0097] Specifically, for files less than 32MB, a lightweight encoding algorithm such as XOR check encoding is used. XOR check encoding is simple to calculate and has low overhead, suitable for fast encoding and decoding of small files, and can effectively improve the I / O read / write performance of small files.

[0098] For files of 32MB-1GB, an improved RS encoding algorithm (second forward error correction code encoding algorithm) is used, which optimizes the generation method of the encoding matrix, reduces the number of multiplication operations in the encoding process, and reduces the calculation overhead on the basis of the traditional RS encoding.

[0099] For files greater than 1GB, a standard RS encoding algorithm (first forward error correction code encoding algorithm) is used to ensure the reliability and efficient disk utilization of the encoding.

[0100] Step S012, when reading or recovering the stored data, decoding processing is performed according to the encoding mode corresponding to the stored data;

[0101] Create a data encoding mode mapping table;

[0102] Map the data file and the corresponding file encoding mode according to the size of the data file;

[0103] In response to a change in the memory space size of the data file, the encoding mode corresponding to the data file and the data encoding mode mapping table are updated.

[0104] Specifically, when writing a file, first determine the size of the file, then select the corresponding encoding mode according to the above standard for encoding processing; when reading or recovering a file, the corresponding decoding processing is performed according to the encoding mode recorded in the file metadata. In order to realize smooth switching of the encoding mode, the system maintains an encoding mode mapping table to record the encoding mode used by each file. When the file size changes (e.g. becomes larger through an append operation), the system will automatically trigger the reselection and switching of the encoding mode, and update the encoding mode mapping table.

[0105] Step S02, determine the target data level according to the data attributes of the data to be stored, wherein the data attributes include one or more of data access frequency, data file size, and data read / write performance requirements.

[0106] Specifically, dynamic layering means that data is divided into different levels according to the attributes of the stored data (including access frequency, file size, importance, and other performance indicators), and different data protection strategies are formulated for each level of data.

[0107] Step S021, store the data to be stored in the hot data layer if the data access frequency is greater than a third preset value (10 times) within a first preset interval of time and / or the data file size is less than a fifth preset value (64MB) and / or the read / write performance requirement for the first level.

[0108] the data access frequency number within the first preset interval of time (24 hours) is between the fourth preset value (3 times) and the third preset value (10 times) and / or the data file size is between the fifth preset value (64MB) and the second preset value (1GB) and / or the to-be-stored data requiring the second level of read-write performance requirement is stored in the warm data layer;

[0109] the data access frequency number within the first preset interval of time is less than the fourth preset value (3 times) and / or the data file size is greater than the second preset value (1GB) and / or the to-be-stored data requiring the third level of read-write performance requirement is stored in the cold data layer;

[0110] The data read-write performance requirement levels include the first level of read-write performance requirement, the second level of read-write performance requirement, and the third level of read-write performance requirement.

[0111] The first level of read-write performance requirement is a high level requirement, the second level of read-write performance requirement is a medium level requirement, and the first level of read-write performance requirement is a low level requirement.

[0112] Step S03, determining a target data protection strategy corresponding to the target data level according to the data attributes of the target data level;

[0113] Step S031, in response to the to-be-stored data being stored in the warm data layer, performing data protection on the data of the warm data layer through a hybrid multi-copy strategy;

[0114] Performing data protection on the data of the warm data layer through the hybrid multi-copy strategy includes:

[0115] dividing the stored data of the warm data layer into a plurality of warm data blocks, setting a plurality of copies for any one warm data block, and calculating a plurality of corresponding check blocks according to the warm data block;

[0116] in response to the to-be-stored data being stored in the warm data layer, performing data protection on the data of the warm data layer through a hybrid erasure code strategy;

[0117] Performing data protection on the data of the warm data layer through the hybrid erasure code strategy includes:

[0118] dividing the stored data of the warm data layer into a plurality of warm data blocks, performing data protection on part of the warm data blocks through an erasure code strategy, and performing data protection on part of the warm data blocks through a multi-copy strategy;

[0119] in response to the to-be-stored data being stored in the cold data layer, performing data protection on the data of the cold data layer through an erasure code strategy; performing data protection on the data of the cold data layer through the erasure code strategy includes:

[0120] The storage data of the cold data layer is divided into a plurality of cold data blocks, and a plurality of corresponding check blocks are calculated according to the plurality of cold data blocks.

[0121] Specifically, the application defines three data levels according to the access frequency, file size, and importance of the data:

[0122] The first layer is the hot data layer: data with high access frequency (such as more than 10 times in the last 24 hours), file size less than 64MB, and high I / O performance requirement is stored in the hot data layer; the layer adopts an improved multi-copy strategy to realize data protection; that is, a 2-copy + check block mode, for each hot data block, 2 complete copies are stored, and a small check block is calculated; this mode not only guarantees the high read-write performance of the hot data (when reading, 2 copies can be selected, and when writing, only 2 copies and 1 small check block need to be written), but also improves the disk utilization to a certain extent, and saves about 33% of space compared with the 3-copy scheme.

[0123] The second layer is the warm data layer: data with medium access frequency (such as 3-10 times in the last 24 hours), moderate size (such as 64MB-1GB), and low I / O performance requirement is stored in the warm data layer; the layer adopts a hybrid erasure code strategy, that is, after the data is divided into blocks, part of the data blocks are protected by the erasure code strategy, and part of the data blocks are protected by the copy strategy; for example, the data is divided into 8 data blocks, 6 of which are protected by RS(6,2) erasure code (that is, 6 data blocks, and 2 corresponding check blocks are calculated), and the other 2 data blocks are protected by 2 copies; this way not only utilizes the high disk utilization of erasure code, but also improves the read-write performance of the data through part of the copy protection.

[0124] The third layer is the cold data layer: data with low access frequency (such as less than 3 times in the last 24 hours), high storage capacity requirement (such as greater than 1GB), and low I / O performance requirement is stored in the cold data layer; the layer adopts an efficient erasure code strategy, such as RS(16,4) erasure code, which divides the data into 16 data blocks and calculates 4 corresponding check blocks; this way can maximize the disk utilization, and is suitable for storing cold data with low access frequency.

[0125] Here, the three layers of data are not fixed and will change with the number of accesses; the data monitoring module in the application will periodically (such as every hour) statistically analyze the access of the data, and when the properties of the data change, it will automatically migrate them from the current level to the corresponding target level and update the corresponding data protection policy. For example, when the access frequency of a hot data decreases to the standard of warm data within 24 hours, the system will migrate it from the hot data layer to the warm data layer, and convert its data protection policy from 2 copies + check blocks to a hybrid erasure code strategy; the data migration process uses incremental migration, only migrating the changed data part, reducing network bandwidth and I / O resource consumption during migration; at the same time, during the migration process, the data availability and consistency are guaranteed, and a read-write locking mechanism is used to prevent data inconsistency caused by modifying data during migration.

[0126] Step S04, creating a pre-computed check block pool;

[0127] Pre-compute the data to be stored to generate several corresponding check blocks of the data to be stored, and store the several corresponding check blocks of the data to be stored in the pre-computed check block pool.

[0128] Step S041, pre-computing the data to be stored in each data layer, dividing the data to be stored in each data layer into several data blocks;

[0129] According to the historical data access frequency and update, determine the data blocks that need to be updated in each data layer, and calculate the data blocks that need to be updated in each data layer by incremental calculation method according to the second preset interval time (such as every 10 minutes), to obtain several corresponding check blocks of the data blocks that need to be updated, and store the several corresponding check blocks of the data blocks that need to be updated in the pre-computed check block pool.

[0130] Specifically, in order to reduce the check block calculation overhead when writing and updating data, the application designs a pre-computed check block pool mechanism. The system predicts the data blocks that may need to be updated according to the historical data access frequency and update, and calculates the check blocks corresponding to these data blocks in advance and stores them in the pre-computed check block pool. For example, for frequently updated hot data blocks, the system will periodically (such as every 10 minutes) recalculate their check blocks and update them to the pool; when pre-computing the check blocks, incremental calculation method is used to calculate only the check blocks corresponding to the changed data part compared with the last calculation, to reduce the calculation amount. At the same time, the idle CPU resources of the system are used for pre-computation to avoid affecting the normal read-write operation of the system.

[0131] Step S05, updating the pre-computed check block pool;

[0132] The update of the pre-computed check block pool includes:

[0133] In response to the capacity of the pre-computed check block pool reaching a first threshold (set according to specific application scenarios, system resources and performance requirements, for example, 512 MB), the check block with the least usage frequency is deleted;

[0134] The validity of the check blocks in the pre-computed check block pool is checked at a third preset interval (for example, 30 seconds to 5 minutes);

[0135] The validity of the check blocks in the pre-computed check block pool is checked, including:

[0136] In response to a change in the original data block corresponding to the check block and the check block not being updated, the check block is marked as invalid, and the corresponding check block of the changed data block is recalculated;

[0137] In response to writing to the data to be stored, the data to be stored is divided into data blocks, and it is determined whether there is a valid check block corresponding to the data block in the pre-computed check block pool;

[0138] When there is a valid check block corresponding to the data block in the pre-computed check block pool, the valid check block is obtained;

[0139] When there is no valid check block corresponding to the data block in the pre-computed check block pool, the corresponding valid check block is recalculated according to the data block, and the valid check block is stored in the pre-computed check block pool.

[0140] Specifically, the pre-computed check block pool uses an LRU (Least Recently Used) eviction policy. When the capacity of the pool reaches the upper limit, the least recently used check block is deleted to make room for newly calculated check blocks.

[0141] At the same time, the system regularly checks the validity of the check blocks in the pool. When the corresponding original data block is updated and the check block is not updated in time, the check block is marked as invalid and re-computation is triggered.

[0142] When data is written or updated, the system first checks whether there is a valid check block in the pre-computed check block pool. If there is, the check block is used directly without the need for re-computation. If there is not, the check block is calculated in real time and added to the pre-computed check block pool.

[0143] Step S06, dynamically scheduling read and write requests for stored data according to the data layer type of data storage, data protection policy and load information of data storage nodes.

[0144] Specifically, in order to improve the overall I / O read-write performance of the system, the application proposes an intelligent I / O scheduling strategy, which reasonably schedules I / O requests according to the storage hierarchy of data, data protection mode, data file node load, and the like.

[0145] Step S061, in response to a read request for stored data, the number of data sources of the stored data is determined according to the data layer type and the data protection strategy type of the stored data, wherein the data source is the node storing the data block and / or the check block and / or the copy of the stored data;

[0146] Obtain the current load information corresponding to each data source node, wherein the load information includes the processor utilization and the memory usage;

[0147] According to the current load information corresponding to each data source node, the response time prediction value of the corresponding data source is calculated;

[0148] The data source with the shortest response time prediction value is taken as the read target node of the read request data;

[0149] The read request for the stored data is scheduled to the read target node.

[0150] Specifically, for a read request, the system first determines the possible data sources (such as copy nodes, nodes storing data blocks and check blocks) according to the storage hierarchy of the data and the data protection strategy, then collects the current load information (such as CPU utilization, memory usage, disk I / O busy degree, etc.) of the nodes of each data source, calculates the response time prediction value of each data source, and finally selects the data source with the shortest response time as the read target, and schedules the read request to the node. For a read request of hot data, since there are multiple copies, the system can implement parallel reading, simultaneously reading different parts of the data from multiple copies, and then merging the complete data to return to the client, further improving the reading speed.

[0151] Step S062, in response to a write request for stored data, the number of data blocks and check blocks to which the to-be-stored data needs to be written is determined according to the target data protection strategy;

[0152] According to the number of data blocks and check blocks to which the to-be-stored data needs to be written, the target storage nodes corresponding to the data blocks and check blocks are determined;

[0153] According to the target data layer and the target data protection strategy, the encoded to-be-stored data is stored to the target storage node;

[0154] Obtain the current load information corresponding to each target storage node;

[0155] The write request of the to-be-stored data is dynamically scheduled according to the current load information of each target storage node;

[0156] The encoded to-be-stored data is stored to the target storage node according to the target data level and the target data protection strategy, including:

[0157] In response to storing the hot data to the target storage node, the hot data is written to the master copy, the hot data is asynchronously written to the slave copy, and the data block corresponding to the hot data is written to the check block, wherein the hot data includes a plurality of copies;

[0158] In response to storing the warm data and the cold data to the target storage node, the warm data and the cold data are written by a batch write strategy.

[0159] Specifically, for the write request, the system determines the number of data blocks and check blocks and target nodes to be written according to the storage level of the data and the data protection strategy, and then dynamically allocates the write task according to the current load of each target node, avoiding concentrating too many write requests to the node with high load. For the write request of the hot data, the asynchronous write mode of the slave copy is adopted, that is, the data is first written to the master copy, and then the write success response is returned to the client immediately, while the data is synchronized to the slave copy and written to the check block in the background, reducing the waiting time of the client; for the write request of the warm data and the cold data, a batch write strategy is adopted, a plurality of small write requests are combined into one large write request for processing, reducing the disk seek time and network transmission overhead.

[0160] Step S07, in response to the loss or damage of the stored data, the stored data is recovered according to the data layer type of the data storage and the data protection strategy corresponding to the data layer.

[0161] Step S071, the data layer type and the data protection strategy type corresponding to the data layer of the lost data or the damaged data are obtained;

[0162] The data block copy and / or the check block of the data recovery are determined according to the data protection strategy type corresponding to the data layer of the lost data or the damaged data;

[0163] The data is recovered through the data block copy and / or the check block of the data recovery.

[0164] Specifically, the data recovery mechanism of the present application combines the dynamic hierarchical storage architecture and the pre-computed check block pool to improve the speed and efficiency of data recovery.

[0165] When a data block is detected to be lost (e.g., due to disk damage, the stored data block cannot be accessed), the system first determines the data storage layer where the lost data block is located and the corresponding data protection strategy, and then recovers the lost data block from the remaining valid data blocks and the check blocks (including the check blocks in the pre-computed check block pool) according to the data protection strategy.

[0166] Step S072, in response to the lost or damaged data being the data of the hot data layer, the data of the hot data layer is preferentially recovered through the data block primary copy;

[0167] When the data block primary copy is also lost or damaged, the data of the hot data layer is recovered through the corresponding check block of the data block and the data block secondary copy;

[0168] In response to the lost or damaged data being the data of the warm data layer and the cold data layer, the data of the warm data layer and the cold data layer is recovered through the corresponding check block of the data of the warm data layer and the cold data layer in the pre-computed check block pool.

[0169] Specifically, for the lost data block of the hot data layer, since there are multiple copies and check blocks, the system can preferentially recover the data from the copies, and if the copies are also lost, the check blocks and other copies are used for recovery, which is fast; for the lost data block of the warm data layer and the cold data layer, the corresponding erasure code encoding mode is used, and the check blocks in the pre-computed check block pool are combined to reduce the calculation overhead in the recovery process and improve the recovery efficiency.

[0170] Step S073, obtaining the load information of all storage nodes;

[0171] Determining the storage node with the minimum load rate according to the load information of all storage nodes;

[0172] Recovering the lost or damaged data according to the incremental recovery mode;

[0173] Storing the recovered data in the storage node with the minimum load rate.

[0174] Specifically, during the data recovery process, the system will assign the recovery task to the node with the lightest load according to the load condition of the storage node to process, avoiding affecting the normal read and write operations of the system; at the same time, the incremental recovery mode is adopted to recover only the lost data block, reducing the data transmission amount.

[0175] As shown in FIG. Figure 5 The technical key point of the technical scheme of the present application is to realize the effective balance of the disk utilization rate and the I / O read-write performance in the distributed storage through the cooperative work of multiple mechanisms, which specifically includes the following aspects:

[0176] 1. Intelligent partitioning and migration of dynamic hierarchical storage architecture: Based on the access frequency, size and other attributes of data, accurate hierarchical partitioning is performed, and dynamic and efficient migration of data between layers is realized. This hierarchical partitioning is not fixed, but can be adjusted in real time according to the actual usage of data, ensuring that the data is always under the optimal protection strategy. The quantified indicators of hierarchical partitioning (such as specific numerical range of access frequency, file size partitioning boundary) and the incremental migration and consistency guarantee mechanism in the migration process are the core of the effective operation of this storage architecture.

[0177] 2. Flexible adaptation of adaptive encoding switching mechanism: Different encoding methods are automatically selected according to file size, realizing the best match between encoding algorithm and file characteristics. The reasonable application range of lightweight encoding algorithm, improved RS encoding algorithm and standard RS encoding algorithm, as well as the smooth transition and mapping table management in the encoding method switching process, ensure that different size files can obtain efficient encoding and decoding performance.

[0178] 3. Precise prediction and management of pre-computed check block pool: Through analysis of historical data, the data blocks that may need to be updated are predicted, and the check blocks are calculated in advance and stored in the pre-computed check block pool, greatly reducing the real-time calculation overhead. The judgment of update frequency, the application of incremental calculation method, and the LRU eviction policy and effectiveness checking mechanism of the check block pool in the pre-computation strategy ensure the efficient utilization and accuracy of the check blocks in the pool.

[0179] 4. Optimized allocation of intelligent I / O scheduling strategy: Considering factors such as data storage hierarchy, data protection strategy and node load, read requests and write requests are reasonably scheduled. In read requests, a response time prediction model is constructed and parallel reading is implemented, while in write requests, asynchronous writing, batch writing and other strategies are applied, effectively improving the overall I / O read / write performance of the system.

[0180] 5. Efficient coordination of data recovery mechanism: Combined with the dynamic hierarchical storage architecture and the pre-computed check block pool, the optimal recovery method is selected according to the data storage hierarchy and data protection strategy, and the task allocation is considered in the storage node load during the recovery process. Incremental recovery is used to reduce data transmission, improving the speed and efficiency of data recovery.

[0181] In addition, through the incremental calculation method, the data blocks that need to be updated are calculated to obtain the corresponding check blocks of the data blocks, including:

[0182] The corresponding check blocks of the data blocks that need to be updated are calculated by XOR check method;

[0183] Obtain multiple data blocks D1, D2,..., Dn that need to be updated;

[0184] The check block corresponding to the plurality of data blocks is calculated by a formula: P=D1⊕D2⊕⋯⊕Dn;

[0185] When any one data block Di is modified to Di', the check block is updated in an incremental manner;

[0186] The check block is updated in an incremental manner, comprising:

[0187] The updated check block P' is calculated by a formula: P'=P⊕(Di⊕Di');

[0188] The check block corresponding to the data block to be updated is calculated by a linear encoding check manner;

[0189] The data block D=[D1, D2,..., Dk] is obtained, and a check part Gp of a data block matrix is generated;

[0190] The check block of the data block is calculated by a formula: P=D⋅Gp;

[0191] When any one Di is changed to Di', the data block change amount ΔDi is calculated by a formula: ΔDi=Di'-Di;

[0192] The check coefficient vector gi corresponding to the i-th column in the generated data block matrix is obtained;

[0193] The check block change ΔP is calculated by a formula: ΔP=ΔDi⋅gi;

[0194] The new check block P' is calculated by a formula: P'=P⊕(ΔDi⋅gi).

[0195] Here, the check block corresponding to the plurality of data blocks is calculated by the incremental calculation manner, which can avoid full reading of all data blocks, greatly reduce I / O delay, only process the change part, save CPU and network resources, reduce the number of disk / SSD read / write times, prolong the service life of the device, quickly synchronize the check information, reduce the risk of silent errors, and guarantee data consistency.

[0196] The method for data processing provided in the embodiments of the application can be improved and optimized without departing from the technical scheme of the application, and these improvements and optimizations should also be considered as the protection scope of the application.

[0197] The technical scheme provided in the embodiments of the application has the following beneficial effects:

[0198] The application can balance the utilization rate of the disk and the data read / write performance, reduce resource calculation overhead, and improve file processing efficiency.

[0199] The technical scheme of the application uses different data protection strategies for data with different attributes through a dynamic hierarchical storage architecture, greatly improves the disk utilization rate under the premise of ensuring data reliability: the hot data layer uses a 2-copy + check block mode, which improves the disk utilization rate by about 33% compared with the traditional 3-copy scheme; the warm data layer uses a hybrid erasure code strategy, which improves the disk utilization rate by 40%-50% compared with the multi-copy scheme; the cold data layer uses an efficient erasure code strategy, and the disk utilization rate can reach more than 80%, which is more than 60% higher than the multi-copy scheme; therefore, the effective utilization rate of the disk of the application is improved by 30%-50%, which significantly reduces the storage cost.

[0200] The intelligent I / O scheduling strategy of the technical scheme of the application can select the optimal data source for read requests, and the parallel reading of multiple copies of hot data further improves the reading speed. Experimental data shows that the read performance of hot data is improved by 15%-20% compared with the traditional multi-copy scheme, and the read performance of warm data and cold data is improved by 20%-30% compared with the traditional erasure code scheme; the adaptive coding switching mechanism uses a lightweight coding algorithm for small files, reducing the coding overhead of small files; the precomputed check block pool reduces the real-time computation time of the check block; the asynchronous writing of hot data reduces the client waiting time from the copy mode; the batch writing strategy of warm data and cold data reduces the disk seek and network transmission overhead; the I / O write performance of the application is improved by 10%-20% compared with the traditional multi-copy scheme, and by 30%-40% compared with the traditional erasure code scheme.

[0201] Embodiments of the application also provide a distributed storage system, which comprises a data processing module and a plurality of physical nodes (disks), and the data processing module is configured to:

[0202] obtain data to be stored;

[0203] encode the data to be stored according to the size of the data file;

[0204] determine a target data level according to the data attribute of the data to be stored, wherein the data attribute comprises one or more of data access frequency, data file size, and data read-write performance requirement;

[0205] determine a target data protection strategy corresponding to the target data level according to the data attribute of the target data level;

[0206] determine the number of data blocks and check blocks required for writing the data to be stored according to the target data protection strategy;

[0207] determine the target storage node corresponding to the data blocks and check blocks according to the number of data blocks and check blocks required for writing the data to be stored;

[0208] The encoded data to be stored is stored to the target storage node according to a target data level and a target data protection policy.

[0209] In one of the embodiments, the data processing module is configured to encode the data to be stored by using an XOR check code algorithm when the memory space size value of the data file is less than a first preset value.

[0210] Encode the data to be stored by using a second forward error correction code algorithm when the memory space size value of the data file is greater than the first preset value and less than a second preset value.

[0211] Encode the data to be stored by using a first forward error correction code algorithm when the memory space size value of the data file is greater than the second preset value.

[0212] In one of the embodiments, the data processing module is configured to store the data to be stored in a hot data layer when the data access frequency within a first preset interval of time is greater than a third preset value and / or the data file size is less than a fifth preset value and / or the data to be stored has a first level of read-write performance requirement.

[0213] Store the data to be stored in a warm data layer when the data access frequency within the first preset interval of time is between the fourth preset value and the third preset value and / or the data file size is between the fifth preset value and the second preset value and / or the data to be stored has a second level of read-write performance requirement.

[0214] Store the data to be stored in a cold data layer when the data access frequency within the first preset interval of time is less than the fourth preset value and / or the data file size is greater than the second preset value and / or the data to be stored has a third level of read-write performance requirement.

[0215] The data read-write performance requirement levels include the first level of read-write performance requirement, the second level of read-write performance requirement, and the third level of read-write performance requirement.

[0216] In one of the embodiments, the data processing module is configured to perform data protection on the data in the hot data layer by using a hybrid multi-copy strategy in response to the data to be stored being stored in the hot data layer.

[0217] Performing data protection on the data in the hot data layer by using the hybrid multi-copy strategy includes:

[0218] Divide the stored data in the hot data layer into a plurality of hot data blocks, set a plurality of copies for any one of the hot data blocks, and calculate a plurality of corresponding check blocks according to the hot data block.

[0219] Perform data protection on the data in the warm data layer by using a hybrid erasure code strategy in response to the data to be stored being stored in the warm data layer.

[0220] Data protection is performed on the data of the warm data layer by using a mixed erasure code strategy, including:

[0221] The stored data of the warm data layer is divided into a plurality of warm data blocks, data protection is performed on part of the warm data blocks by using an erasure code strategy, and data protection is performed on part of the warm data blocks by using a multiple copy strategy.

[0222] In response to the to-be-stored data being stored in the cold data layer, data protection is performed on the data of the cold data layer by using an erasure code strategy.

[0223] Data protection is performed on the data of the cold data layer by using an erasure code strategy, including:

[0224] The stored data of the cold data layer is divided into a plurality of cold data blocks, and a plurality of corresponding check blocks are calculated according to the plurality of cold data blocks.

[0225] In one embodiment, the data processing module is configured to create a pre-computed check block pool.

[0226] The to-be-stored data is pre-computed to generate a plurality of corresponding check blocks of the to-be-stored data, and the plurality of corresponding check blocks of the to-be-stored data are stored in the pre-computed check block pool.

[0227] The to-be-stored data is pre-computed to generate a plurality of corresponding check blocks of the to-be-stored data, and the plurality of corresponding check blocks of the to-be-stored data are stored in the pre-computed check block pool, including:

[0228] The to-be-stored data of each data layer is pre-computed, and the to-be-stored data of each data layer is divided into a plurality of data blocks.

[0229] According to the historical data access frequency and update situation, the data blocks that need to be updated of each data layer are determined, and the data blocks that need to be updated of each data layer are calculated by using an incremental calculation method according to a second preset interval time to obtain a plurality of corresponding check blocks of the data blocks that need to be updated.

[0230] The plurality of corresponding check blocks of the data blocks that need to be updated are stored in the pre-computed check block pool.

[0231] In one embodiment, the data processing module is configured to update the pre-computed check block pool.

[0232] The pre-computed check block pool is updated, including:

[0233] In response to the capacity of the pre-computed check block pool reaching a first threshold, the check block with the least frequency of use is deleted.

[0234] The validity of the check blocks in the pre-computed check block pool is checked according to a third preset interval time.

[0235] checking the validity of the pre-computed check block pool, comprising:

[0236] in response to the original data block corresponding to the check block changing and the check block not being updated, marking the check block as invalid, and re-computing the corresponding check block of the changed data block;

[0237] in response to writing to the data to be stored, dividing the data to be stored into a plurality of data blocks, and determining whether there is a valid check block corresponding to the data block in the pre-computed check block pool;

[0238] when there is a valid check block corresponding to the data block in the pre-computed check block pool, obtaining the valid check block;

[0239] when there is no valid check block corresponding to the data block in the pre-computed check block pool, re-computing the corresponding valid check block according to the data block, and storing the valid check block in the pre-computed check block pool.

[0240] In one embodiment, the data processing module is configured to dynamically schedule read requests for stored data according to the data level type of the data storage, the data protection strategy corresponding to the data level, and the load information of the data storage node;

[0241] dynamically scheduling read requests for stored data according to the data level type of the data storage, the data protection strategy corresponding to the data level, and the load information of the data storage node, comprising:

[0242] in response to a read request for stored data, determining a plurality of data sources of the stored data according to the data level type of the data storage and the data protection strategy type, wherein the data source is a storage node storing the data block and / or the check block and / or the copy corresponding to the stored data;

[0243] obtaining the current load information of the storage node corresponding to each data source, wherein the load information includes processor utilization and memory usage;

[0244] calculating the response time prediction value of the corresponding data source according to the current load information of the storage node corresponding to each data source;

[0245] selecting the data source with the shortest response time prediction value as the read target storage node of the read request data;

[0246] scheduling the read request for the stored data to the read target storage node.

[0247] In one embodiment, the data processing module is configured to obtain the current load information of each target storage node;

[0248] The write request of the to-be-stored data is dynamically scheduled according to the current load information corresponding to each target storage node;

[0249] In response to storing the hot data to the target storage node, the hot data is written into the master copy, the hot data is asynchronously written into the slave copy, and the data block corresponding to the hot data is written into the check block, wherein the hot data comprises a plurality of copies;

[0250] In response to storing the warm data and the cold data to the target storage node, the warm data and the cold data are written by using the batch write strategy.

[0251] In one embodiment, the data processing module is configured to, in response to the lost or damaged data, recover the stored data according to the data level type of the stored data and the data protection strategy corresponding to the data level;

[0252] Recovering the stored data according to the data level type of the stored data and the data protection strategy corresponding to the data level comprises:

[0253] Obtaining the data level type of the stored data and the data protection strategy type corresponding to the data level of the lost or damaged data;

[0254] Determining the data block copy and / or the check block for data recovery according to the data protection strategy type corresponding to the data level of the lost or damaged data;

[0255] Recovering the data by using the data block copy and / or the check block for data recovery.

[0256] In one embodiment, the data processing module is configured to, in response to the lost or damaged data being the data of the hot data level, preferentially recover the data of the hot data level by using the master copy of the data block;

[0257] When the master copy of the data block is also lost or damaged, the data of the hot data level is recovered by using the check block corresponding to the hot data block and the slave copy of the hot data block;

[0258] In response to the lost or damaged data being the data of the warm data level and the cold data level, the data of the warm data level and the cold data level is recovered by calculating the corresponding check block in the pre-computed check block pool.

[0259] In one embodiment, the data processing module is configured to obtain the load information of all storage nodes;

[0260] Determining the storage node with the minimum load rate according to the load information of all storage nodes;

[0261] Recovering the lost or damaged data according to the incremental recovery mode;

[0262] The data after loss or loss recovery is stored in the storage node with the minimum load rate.

[0263] In one of the embodiments, the data processing module is configured to decode the stored data according to the corresponding encoding mode of the stored data when reading or recovering the stored data.

[0264] The data encoding mode mapping table is created.

[0265] The data file and the corresponding file encoding mode are mapped according to the size of the data file.

[0266] In response to a change in the memory space size of the data file, the encoding mode corresponding to the data file and the data encoding mode mapping table are updated.

[0267] Specifically, the dynamic hierarchical storage includes specific division criteria for dividing data into hot data layer, warm data layer, and cold data layer based on data access frequency, size, and other attributes; data protection strategies such as 2 replicas + check block, hybrid erasure code, and efficient erasure code used by each layer; and mechanisms for dynamic migration of data between layers, including migration trigger conditions, incremental migration methods, and data consistency guarantee measures.

[0268] Adaptive encoding switching: specific selection criteria for selecting lightweight encoding algorithms, improved RS encoding algorithms, or standard RS encoding algorithms according to file size; mechanisms for smooth switching of encoding modes, including maintenance and updating methods of the encoding mode mapping table; and reselection and switching process of encoding modes when file size changes.

[0269] Construction and management of precomputed check block pool: precompute strategy based on historical data access and update to predict data blocks that need to be updated and compute check blocks in advance, including precompute period, incremental computation method, and implementation method using system idle resources; LRU eviction policy for managing check block pool, including setting of pool capacity upper limit, invalid check block marking, and recompute trigger mechanism; and query and use process of precomputed check blocks when data is written or updated.

[0270] Intelligent I / O scheduling: in read request scheduling, determine data source, collect node load information, calculate response time prediction value, and select optimal data source, as well as parallel read implementation for hot data read requests; in write request scheduling, determine write target node and dynamically allocate tasks according to node load; asynchronous write from replica for hot data write requests, and batch write strategy for warm data and cold data write requests.

[0271] Data recovery: when a data block is lost, a method for recovering the lost data block from the remaining valid blocks and the pre-computed check block pool according to the data storage hierarchy and data protection strategy; a mechanism for allocating recovery tasks according to node load during the recovery process; and an implementation of an incremental recovery method to reduce data transmission.

[0272] The technical scheme provided by the embodiments of the present application has the following beneficial effects:

[0273] The present application can balance the utilization rate of the disk and the data read-write performance, reduce resource calculation overhead, and improve file processing efficiency.

[0274] The technical scheme of the present application uses different data protection strategies for data with different attributes through a dynamic hierarchical storage architecture, greatly improves the disk utilization rate under the premise of ensuring data reliability: the hot data layer uses a 2-copy + check block method, which improves the disk utilization rate by about 33% compared with the traditional 3-copy scheme; the warm data layer uses a hybrid erasure code strategy, which improves the disk utilization rate by 40%-50% compared with the multiple copy scheme; the cold data layer uses an efficient erasure code strategy, and the disk utilization rate can reach more than 80%, which is more than 60% higher than the multiple copy scheme; therefore, the overall effective utilization rate of the disk of the present application is improved by 30%-50%, significantly reducing the storage cost.

[0275] The technical scheme of the present application uses a pre-computed check block pool mechanism to calculate the check block in advance using idle system resources, reducing the real-time calculation overhead during data writing and updating; an adaptive coding switching mechanism uses a simple lightweight coding algorithm for small files, reducing the coding and decoding calculation amount of small files; an improved RS coding algorithm reduces the number of multiplication operations in the coding process, reducing the calculation overhead of medium-sized files, so that the overall calculation overhead of the present application is reduced by 25%-35% compared with the traditional erasure code scheme.

[0276] The technical scheme of the present application is aimed at the characteristics of small files, and the present application uses an optimized multiple copy strategy for small files in the hot data layer, and selects a lightweight XOR check coding in the coding method, reducing the coding overhead and storage overhead of small files. The read-write performance of small files is improved by 40%-50% compared with the traditional erasure code scheme, solving the problem of low processing efficiency of small files in the traditional erasure code technology.

[0277] The description of the features in the embodiment of the data processing module can be referred to the related description of the embodiment of the data processing method, which will not be repeated here.

[0278] The embodiments of the present application also provide an electronic device comprising a memory and a processor, the memory storing a computer program, and the processor being configured to run the computer program to perform the steps in the data processing method embodiment, the method comprising:

[0279] obtaining data to be stored;

[0280] encoding the data to be stored according to a size of a data file;

[0281] determining a target data tier according to a data attribute of the data to be stored, wherein the data attribute comprises one or more of a data access frequency, a data file size, and a data read-write performance requirement;

[0282] determining a target data protection strategy corresponding to the target data tier according to a data attribute of the target data tier;

[0283] determining a number of data blocks and check blocks required for writing the data to be stored according to the target data protection strategy;

[0284] determining target storage nodes corresponding to the data blocks and the check blocks according to the number of data blocks and check blocks required for writing the data to be stored;

[0285] storing the encoded data to be stored to the target storage nodes according to the target data tier and the target data protection strategy.

[0286] As shown in Figure 6 embodiments of the present application also provide a computer readable storage medium having a computer program stored therein, wherein the computer program is configured to execute the steps in the data processing method embodiments when running, and the method comprises:

[0287] obtaining data to be stored;

[0288] encoding the data to be stored according to a size of a data file;

[0289] determining a target data tier according to a data attribute of the data to be stored, wherein the data attribute comprises one or more of a data access frequency, a data file size, and a data read-write performance requirement;

[0290] determining a target data protection strategy corresponding to the target data tier according to a data attribute of the target data tier;

[0291] determining a number of data blocks and check blocks required for writing the data to be stored according to the target data protection strategy;

[0292] determining target storage nodes corresponding to the data blocks and the check blocks according to the number of data blocks and check blocks required for writing the data to be stored;

[0293] storing the encoded data to be stored to the target storage nodes according to the target data tier and the target data protection strategy.

[0294] In an example embodiment, the computer readable storage medium described above can include, but is not limited to, a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store computer programs.

[0295] Embodiments of the present application also provide a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps in the data processing method embodiment, and the method comprises:

[0296] Obtaining data to be stored;

[0297] Encoding the data to be stored according to the size of the data file;

[0298] Determining a target data level according to data attributes of the data to be stored, wherein the data attributes include one or more of data access frequency, data file size, and data read-write performance requirement;

[0299] Determining a target data protection strategy corresponding to the target data level according to the data attributes of the target data level;

[0300] Determining the number of data blocks and check blocks to which the data to be stored needs to be written according to the target data protection strategy;

[0301] Determining target storage nodes corresponding to the data blocks and check blocks according to the number of data blocks and check blocks to which the data to be stored needs to be written;

[0302] Storing the encoded data to be stored to the target storage nodes according to the target data level and the target data protection strategy.

[0303] Embodiments of the present application also provide another computer program product, which comprises a non-volatile computer readable storage medium, and the non-volatile computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps in the data processing method embodiment, and the method comprises:

[0304] Obtaining data to be stored;

[0305] Encoding the data to be stored according to the size of the data file;

[0306] Determining a target data level according to data attributes of the data to be stored, wherein the data attributes include one or more of data access frequency, data file size, and data read-write performance requirement;

[0307] Determining a target data protection strategy corresponding to the target data level according to the data attributes of the target data level;

[0308] determining the number of data blocks and check blocks to which the to-be-stored data needs to be written according to a target data protection policy;

[0309] determining target storage nodes corresponding to the data blocks and the check blocks according to the number of the data blocks and the check blocks to which the to-be-stored data needs to be written;

[0310] storing the encoded to-be-stored data to the target storage nodes according to the target data hierarchy and the target data protection policy.

[0311] The application can balance the utilization rate of the disk and the data read-write performance, reduce resource calculation overhead, and improve file processing efficiency.

[0312] The technical scheme of the application uses different data protection policies for data with different attributes through a dynamic hierarchical storage architecture, greatly improves the disk utilization rate under the premise of ensuring data reliability: the hot data layer uses a 2-replica + check block mode, which improves the disk utilization rate by about 33% compared with the traditional 3-replica scheme; the warm data layer uses a hybrid erasure code strategy, which improves the disk utilization rate by 40%-50% compared with the multi-replica scheme; the cold data layer uses an efficient erasure code strategy, and the disk utilization rate can reach more than 80%, which is more than 60% higher than the multi-replica scheme; therefore, the effective utilization rate of the disk of the application is improved by 30%-50% as a whole, significantly reducing the storage cost.

[0313] Those skilled in the art will further appreciate that the functions of the examples described herein-based units and algorithm steps can be implemented using electronic hardware, computer software, or any combination of the two. To clearly illustrate the interchangeability of hardware and software, the above description has generally been described in terms of examples of functionally-oriented components and steps. Whether such functions are implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Those skilled in the art can implement the described functions in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the application.

[0314] The above provides a detailed introduction to the data processing method, device, medium and program product provided by the application. The principles and implementation modes of the application are described herein by applying specific examples, and the above description of the examples is only applicable to help understand the method and core idea of the application. It should be noted that, for those skilled in the art, without departing from the principles of the application, some improvements and modifications can be made to the application, and these improvements and modifications also fall within the protection scope of the application.

Claims

1. A data processing method, characterized by, The method comprises: acquiring data to be stored; encoding the data to be stored according to the size of the data file; determining a target data level according to the data attributes of the data to be stored, wherein the data attributes comprise one or more of data access frequency, data file size, and data read-write performance requirement; determining a target data protection strategy corresponding to the target data level according to the data attributes of the target data level; determining the number of data blocks and check blocks to which the data to be stored needs to be written according to the target data protection strategy; determining the target storage nodes corresponding to the data blocks and check blocks according to the number of data blocks and check blocks to which the data to be stored needs to be written; storing the encoded data to be stored to the target storage nodes according to the target data level and the target data protection strategy; after determining the target data protection strategy corresponding to the target data level according to the data attributes of the target data level, the method further comprises: creating a pre-computed check block pool; pre-computing the data to be stored to generate a plurality of corresponding check blocks of the data to be stored, and storing the plurality of corresponding check blocks of the data to be stored in the pre-computed check block pool; the pre-computing the data to be stored to generate a plurality of corresponding check blocks of the data to be stored, and storing the plurality of corresponding check blocks of the data to be stored in the pre-computed check block pool, comprises: pre-computing the data to be stored of each data level to divide the data to be stored of each data level into a plurality of data blocks; determining the data blocks that need to be updated of each data level according to historical data access frequency and update situation, and computing the data blocks that need to be updated of each data level by incremental computing mode according to a second preset interval time to obtain a plurality of corresponding check blocks of the data blocks that need to be updated; storing the plurality of corresponding check blocks of the data blocks that need to be updated in the pre-computed check block pool.

2. The data processing method according to claim 1, characterized in that, The encoding the data to be stored according to the size of the data file comprises: when the memory space size value of the data file is less than a first preset value, encoding the data to be stored by an exclusive or check encoding algorithm; when the memory space size value of the data file is greater than the first preset value and less than a second preset value, encoding the data to be stored by a second forward error correction code encoding algorithm; when the memory space size value of the data file is greater than the second preset value, encoding the data to be stored by a first forward error correction code encoding algorithm.

3. The data processing method of claim 1, wherein, The determining a target data level according to the data attributes of the data to be stored comprises: storing the data to be stored of a first grade read-write performance requirement in a hot data level when the data access frequency is greater than a third preset value and / or the data file size is less than a fifth preset value within a first preset interval time; storing the data to be stored of a second grade read-write performance requirement in a warm data level when the data access frequency is between the fourth preset value and the third preset value and / or the data file size is between the fifth preset value and the second preset value within the first preset interval time; storing the to-be-stored data in the cold data layer when the data access frequency is less than a fourth preset value in a first preset interval and / or the data file size is greater than a second preset value and / or the to-be-stored data requires a third level of read-write performance; wherein the data read-write performance requirement level includes a first level of read-write performance requirement, a second level of read-write performance requirement, and a third level of read-write performance requirement.

4. The data processing method of claim 1, wherein, The method further includes: in response to the to-be-stored data being stored in the hot data layer, performing data protection on the data of the hot data layer by using a hybrid multi-copy strategy; The method further includes: dividing the stored data of the hot data layer into a plurality of hot data blocks, setting a plurality of copies for any one hot data block, and calculating a plurality of corresponding check blocks according to the hot data block; in response to the to-be-stored data being stored in the warm data layer, performing data protection on the data of the warm data layer by using a hybrid erasure code strategy; The method further includes: dividing the stored data of the warm data layer into a plurality of warm data blocks, performing data protection on part of the warm data blocks by using an erasure code strategy, and performing data protection on part of the warm data blocks by using a multi-copy strategy; in response to the to-be-stored data being stored in the cold data layer, performing data protection on the data of the cold data layer by using an erasure code strategy; The method further includes: dividing the stored data of the cold data layer into a plurality of cold data blocks, and calculating a plurality of corresponding check blocks according to the plurality of cold data blocks.

5. The data processing method of claim 1, wherein, The method further includes: updating the pre-computed check block pool; The method further includes: in response to the capacity of the pre-computed check block pool reaching a first threshold, deleting the check block with the least frequency of use; checking the validity of the check blocks in the pre-computed check block pool according to a third preset interval; The method further includes: in response to the corresponding original data block of the check block changing and the check block not being updated, marking the check block as invalid, and re-computing the corresponding check block of the changed data block; in response to writing the to-be-stored data, dividing the to-be-stored data into a plurality of data blocks, and determining whether there is a valid check block corresponding to the data blocks in the pre-computed check block pool; when there is a valid check block corresponding to the data blocks in the pre-computed check block pool, obtaining the valid check block; when there is no valid check block corresponding to the data blocks in the pre-computed check block pool, re-computing a valid check block corresponding to the data blocks according to the data blocks, and storing the valid check block in the pre-computed check block pool.

6. The data processing method of claim 1, wherein, The method further includes: The read request of the stored data is dynamically scheduled according to the data level type of the data storage, the data protection strategy corresponding to the data level, and the load information of the data storage node. The read request of the stored data is dynamically scheduled according to the data level type of the data storage, the data protection strategy corresponding to the data level, and the load information of the data storage node. In response to the read request of the stored data, the data source of the stored data is determined according to the data level type and the data protection strategy type of the data storage, wherein the data source is a data block and / or a check block and / or a storage node storing a copy of the stored data. The current load information corresponding to each data source is obtained, wherein the load information includes processor utilization and memory usage. The response time prediction value of each data source is calculated according to the current load information corresponding to each data source. The data source with the shortest response time prediction value is taken as the read target storage node of the read request data. The read request of the stored data is scheduled to the read target storage node.

7. The data processing method of claim 1, wherein, The current load information corresponding to each target storage node is obtained. The write request of the stored data is dynamically scheduled according to the current load information corresponding to each target storage node. In response to storing the hot data to the target storage node, the hot data is written into the master copy, asynchronously written into the slave copy, and written into the check block corresponding to the data block of the hot data, wherein the hot data includes several copies. In response to storing the warm data and the cold data to the target storage node, the warm data and the cold data are written by a batch writing strategy. The method further comprises:

8. The data processing method of claim 1, wherein, In response to the loss or damage of the stored data, the stored data is recovered according to the data level type of the data storage and the data protection strategy corresponding to the data level. The stored data is recovered according to the data level type of the data storage and the data protection strategy corresponding to the data level. The data level type and the data protection strategy type corresponding to the data level of the lost data or the damaged data are obtained. The data block copy and / or the check block of the data recovery are determined according to the data protection strategy type corresponding to the data level of the lost data or the damaged data. The data is recovered through the data block copy and / or the check block of the data recovery. In response to the lost or damaged data being the data of the hot data level, the data of the hot data level is preferentially recovered through the data block master copy.

9. The data processing method according to claim 8, characterized in that, When the data block master copy is also lost or damaged, the data of the hot data level is recovered through the check block corresponding to the hot data block and the slave copy of the hot data block. ​ ​ In response to missing or damaged data of the warm data layer and the cold data layer, the corresponding check blocks are calculated through the pre-computed check block pool of the warm data layer and the cold data layer data to recover the data of the warm data layer and the cold data layer.

10. The data processing method according to claim 8, characterized in that, The method further comprises: acquiring load information of all storage nodes; determining a storage node with the minimum load rate according to the load information of all storage nodes; recovering missing or lost data according to an incremental recovery mode; storing the recovered missing or lost data in the storage node with the minimum load rate.

11. The data processing method according to claim 2, characterized in that, The method further comprises: when reading or recovering the stored data, decoding the stored data according to the corresponding encoding mode of the stored data; creating a data encoding mode mapping table; mapping the data file and the corresponding file encoding mode according to the size of the data file; in response to a change in the memory space size of the data file, updating the encoding mode of the data file and the data encoding mode mapping table.

12. An electronic device, comprising: comprises: a memory for storing a computer program; a processor for executing the computer program to implement the steps of the data processing method according to any one of claims 1 to 11.

Images