Data processing method and device, medium and program product

Abstract

The invention discloses a data processing method and device, a medium and a program product, and relates to the technical field of storage. Encoding the data to be stored according to the size of the data file; determining a target data hierarchy according to the data attribute of the to-be-stored data; determining a target data protection strategy corresponding to the target data hierarchy according to the data attribute of the target data hierarchy; according to the target data protection strategy, determining the number of data blocks and verification blocks needing to be written into the to-be-stored data; determining target storage nodes corresponding to the data blocks and the verification blocks according to the number of the data blocks and the verification blocks needing to be written in the to-be-stored data; and storing the coded data to be stored to the target storage node according to the target data hierarchy and the target data protection strategy. The utilization rate of the disk and the data read-write performance can be considered, the resource calculation overhead is reduced, and the file processing efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of storage technology, and in particular to a data processing method, device, medium and program product. Background Technology

[0002] With the rapid development of emerging technologies such as cloud computing, big data, and artificial intelligence, data is experiencing explosive growth. This massive amount of data places extremely high demands on the capacity, performance, and reliability of storage systems. Distributed storage systems, with their advantages of strong scalability, high fault tolerance, and relatively low cost, have become the mainstream choice for storing massive amounts of data.

[0003] In distributed storage systems, data protection is a core component, aiming to prevent data loss due to hardware failures (such as disk damage or node failure) and ensure data availability and integrity. Currently, mainstream data protection mechanisms mainly include two schemes: multiple replication and erasure coding. The multiple replication scheme originates from early centralized storage systems. Its core principle is to copy the same data multiple times and store them on different physical nodes or disks. When one copy fails, the system can recover the data using the other copies. This scheme is simple to implement and offers excellent read / write performance because when reading data, it can select the node with the lightest load from multiple copies. Although writing data requires writing to multiple copies, the operation logic is simple and direct. The erasure coding scheme, on the other hand, originates from coding theory in the field of communications. Its basic principle is to divide the original data into several data blocks and calculate corresponding check blocks using a specific coding algorithm. When some data blocks or check blocks are lost, the lost data blocks can be recovered using the remaining blocks through a decoding algorithm. Compared to the multiple replication scheme, the erasure coding scheme requires less storage space and has higher disk utilization under the same reliability requirements.

[0004] However, as the scale of distributed storage systems continues to expand and application scenarios become increasingly complex, neither multi-replica schemes nor erasure coding schemes can simultaneously guarantee disk utilization and data read / write performance while ensuring data reliability. Summary of the Invention

[0005] This application provides a data processing method, apparatus, medium, and program product. The method includes acquiring data to be stored; encoding the data to be stored according to the size of the data file; determining a target data level based on the data attributes of the data to be stored, wherein the data attributes include one or more of data access frequency, data file size, and data read / write performance requirements; determining a target data protection policy corresponding to the target data level based on the data attributes of the target data level; determining the number of data blocks and check blocks to be written to the data to be stored based on the target data protection policy; determining the target storage node corresponding to the data blocks and check blocks based on the number of data blocks and check blocks to be written to the data to be stored; and storing the encoded data to be stored to the target storage node according to the target data level and the target data protection policy. This application can balance disk utilization and data read / write performance, reduce resource computation overhead, and improve file processing efficiency.

[0006] This application provides a data processing method applied to a distributed storage system. The system includes a data processing module and several storage nodes. The method includes: The data processing module acquires the data to be stored; The data to be stored is encoded according to the size of the data file; The target data level is determined based on the data attributes of the data to be stored. The data attributes include one or more of the following: data access frequency, data file size, and data read / write performance requirements. Determine the target data protection strategy corresponding to the target data level based on the data attributes of the target data level; Determine the number of data blocks and check blocks that need to be written to the data to be stored based on the target data protection strategy; The target storage node corresponding to the data block and the check block is determined based on the number of data blocks and check blocks that need to be written to the data to be stored. The encoded data to be stored is stored to the target storage node according to the target data hierarchy and the target data protection strategy.

[0007] This application also provides a distributed storage system, which includes a data processing module and several storage nodes (disks). The data processing module is used for: Retrieve the data to be stored; The data to be stored is encoded according to the size of the data file; The target data level is determined based on the data attributes of the data to be stored. The data attributes include one or more of the following: data access frequency, data file size, and data read / write performance requirements. Determine the target data protection strategy corresponding to the target data level based on the data attributes of the target data level; Determine the number of data blocks and check blocks that need to be written to the data to be stored based on the target data protection strategy; The target storage node corresponding to the data block and the check block is determined based on the number of data blocks and check blocks that need to be written to the data to be stored. The encoded data to be stored is stored to the target storage node according to the target data hierarchy and the target data protection strategy.

[0008] This application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement a data processing method, the method comprising: Retrieve the data to be stored; The data to be stored is encoded according to the size of the data file; The target data level is determined based on the data attributes of the data to be stored. The data attributes include one or more of the following: data access frequency, data file size, and data read / write performance requirements. Determine the target data protection strategy corresponding to the target data level based on the data attributes of the target data level; Determine the number of data blocks and check blocks that need to be written to the data to be stored based on the target data protection strategy; The target storage node corresponding to the data block and the check block is determined based on the number of data blocks and check blocks that need to be written to the data to be stored. The encoded data to be stored is stored to the target storage node according to the target data hierarchy and the target data protection strategy.

[0009] This application also provides a computer-readable storage medium storing a computer program, wherein when executed by a processor, the computer program implements the steps of a data processing method, the method comprising: Retrieve the data to be stored; The data to be stored is encoded according to the size of the data file; The target data level is determined based on the data attributes of the data to be stored. The data attributes include one or more of the following: data access frequency, data file size, and data read / write performance requirements. Determine the target data protection strategy corresponding to the target data level based on the data attributes of the target data level; Determine the number of data blocks and check blocks that need to be written to the data to be stored based on the target data protection strategy; The target storage node corresponding to the data block and the check block is determined based on the number of data blocks and check blocks that need to be written to the data to be stored. The encoded data to be stored is stored to the target storage node according to the target data hierarchy and the target data protection strategy.

[0010] This application also provides a computer program product, including a computer program, wherein when the computer program is executed by a processor, it implements the steps of a data processing method, the method comprising: Retrieve the data to be stored; The data to be stored is encoded according to the size of the data file; The target data level is determined based on the data attributes of the data to be stored. The data attributes include one or more of the following: data access frequency, data file size, and data read / write performance requirements. Determine the target data protection strategy corresponding to the target data level based on the data attributes of the target data level; Determine the number of data blocks and check blocks that need to be written to the data to be stored based on the target data protection strategy; The target storage node corresponding to the data block and the check block is determined based on the number of data blocks and check blocks that need to be written to the data to be stored. The encoded data to be stored is stored to the target storage node according to the target data hierarchy and the target data protection strategy.

[0011] This application provides a method that includes: acquiring data to be stored; encoding the data to be stored according to the size of the data file; determining a target data level based on the data attributes of the data to be stored, wherein the data attributes include one or more of data access frequency, data file size, and data read / write performance requirements; determining a target data protection strategy corresponding to the target data level based on the data attributes of the target data level; determining the number of data blocks and check blocks to be written to the data to be stored based on the target data protection strategy; determining the target storage node corresponding to the data blocks and check blocks based on the number of data blocks and check blocks to be written to the data to be stored; and storing the encoded data to be stored to the target storage node according to the target data level and the target data protection strategy. This application can balance disk utilization and data read / write performance, reduce resource computation overhead, and improve file processing efficiency.

[0012] This application's technical solution employs a dynamic hierarchical storage architecture, using different data protection strategies for data with different attributes. While ensuring data reliability, it significantly improves disk utilization: the hot data layer uses two replicas plus a checksum block, increasing disk utilization by approximately 33% compared to the traditional three-replica scheme; the warm data layer uses a hybrid erasure coding strategy, increasing disk utilization by 40%-50% compared to the multi-replica scheme; and the cold data layer uses an efficient erasure coding strategy, achieving disk utilization of over 80%, more than 60% higher than the multi-replica scheme. Therefore, this application improves overall disk utilization by 30%-50%, significantly reducing storage costs.

[0013] The intelligent I / O scheduling strategy of this application can select the optimal data source for read requests. The parallel reading of multiple replicas of hot data further improves the read speed. Experimental data shows that the read performance of hot data is 15%-20% higher than that of traditional multi-replica schemes, and the read performance of warm and cold data is 20%-30% higher than that of traditional erasure coding schemes. The adaptive encoding switching mechanism uses a lightweight encoding algorithm for small files, reducing the encoding overhead of small files. The pre-computation check block pool reduces the real-time calculation time of check blocks. The asynchronous writing of hot data from replicas reduces the client waiting time. The batch writing strategy of warm and cold data reduces disk seek and network transmission overhead. The I / O write performance of this application is 10%-20% higher than that of traditional multi-replica schemes and 30%-40% higher than that of traditional erasure coding schemes.

[0014] The technical solution of this application adopts a pre-computation check block pool mechanism, which utilizes idle system resources to pre-compute check blocks, reducing the real-time computation overhead during data writing and updating; the adaptive encoding switching mechanism uses a computationally simple lightweight encoding algorithm for small files, reducing the computational load of encoding and decoding for small files; the improved RS encoding algorithm reduces the number of multiplication operations in the encoding process, reducing the computational overhead of medium-sized files, making the overall computational overhead of this application 25%-35% lower than that of traditional erasure coding schemes.

[0015] This application's technical solution addresses the characteristics of small files by employing an optimized multi-copy strategy at the hot data layer and selecting lightweight XOR checksum encoding, thereby reducing the encoding and storage overhead of small files. The read / write performance of small files is improved by 40%-50% compared to traditional erasure coding schemes, solving the problem of low efficiency in small file processing by traditional erasure coding technology. Attached Figure Description

[0016] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram illustrating a multi-copy data protection strategy provided in an embodiment of this application. Figure 2 This is a schematic diagram of the erasure coding data protection strategy provided in the embodiments of this application; Figure 3 This is a first flowchart of data processing provided for an embodiment of this application; Figure 4 This is a second flowchart of data processing provided for an embodiment of this application; Figure 5 A detailed flowchart of data processing provided for embodiments of this application; Figure 6 Exemplary systems provided for embodiments of this application that can be used to implement the various embodiments of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0019] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0020] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0021] The specific application environment architecture or specific hardware architecture on which the execution of the data processing method depends is described here.

[0022] like Figure 1 As shown, currently, distributed storage mainly employs two data protection mechanisms: multi-replica and erasure coding. Multi-replica technology is one of the earliest and most widely adopted data protection schemes in distributed storage systems. Its typical implementation involves replicating data into two, three, or even more copies, storing them on different nodes or disks, such as the three-replica scheme. In this scheme, when a client writes data, the data is first written to the primary replica and then synchronized to other secondary replicas. Only after all replicas have been successfully written will a write success response be returned to the client. When reading data, the client can read data from any available replica, and the system will select the optimal reading node based on the load of the physical nodes where each replica resides.

[0023] like Figure 2As shown, a typical application of erasure coding technology in distributed storage is to divide the original data into k data blocks, and then generate m parity blocks through encoding calculation. These k+m blocks are stored on different physical nodes or disks. When no more than m blocks (data blocks or parity blocks) are lost, the original data can be recovered from the remaining parity blocks. Common erasure coding schemes include Reed-Solomon (RS) codes. For example, RS(4,2) means dividing the data into 4 data blocks and generating 2 parity blocks, allowing a maximum of 2 blocks to be lost (whether data blocks or parity blocks).

[0024] However, as the scale of distributed storage systems continues to expand and application scenarios become increasingly complex, both multi-replica technology and erasure coding technology have gradually revealed their respective limitations. How to balance disk utilization and data read / write performance while ensuring data reliability has become a key issue that urgently needs to be addressed in the field of distributed storage.

[0025] Although the two current data protection mechanisms have different implementation schemes, both technical solutions have certain drawbacks: While multi-replica technology is simple to implement and offers good read and write performance, it has significant drawbacks: First, disk utilization is extremely low. Taking a three-replica scheme as an example, storing one copy of data requires three times the disk space, resulting in a large waste of storage space. As data volume surges, this waste of storage space leads to enormous cost pressures. Second, write performance is limited in large-scale data scenarios. Although the write logic for a single copy of data is simple, when the data volume is huge, writing to multiple replicas consumes a large amount of network bandwidth and disk I / O resources, leading to a decline in overall write performance. In addition, maintaining data consistency is complex. During replica synchronization, abnormal situations such as network failures and node crashes may occur, requiring complex consistency protocols to ensure data consistency between replicas, increasing the design and implementation difficulty of the system.

[0026] While erasure coding improves disk utilization, it also has two major drawbacks: First, it incurs high computational overhead. The encoding and decoding processes require extensive mathematical operations, especially during data writing and recovery, consuming significant CPU resources and impacting overall system performance. Second, it suffers from poor write performance for small files. Erasure coding requires dividing the file into small blocks for encoding, and the computational overhead of this process is relatively large compared to the file size, resulting in low writing efficiency for small files. Third, it incurs high data update costs. When a data block in the original data is updated, the relevant check block needs to be recalculated. For erasure coding schemes containing multiple check blocks, this update operation incurs significant computational and I / O overhead. Furthermore, recovery is slow. When data loss requires recovery, multiple remaining blocks need to be read for decoding calculations, consuming significant CPU resources and network bandwidth, leading to a lengthy recovery process. If another data block is lost during the recovery process, it may result in permanent data loss.

[0027] Understandably, existing multi-copy technology has low disk utilization, resulting in serious waste of storage space and increased storage costs, especially in scenarios involving massive data storage, where this cost pressure is even more pronounced. When writing large amounts of data, multi-replica technology requires writing to multiple copies, which consumes a lot of network bandwidth and disk I / O resources, resulting in a decrease in write performance and failing to meet the needs of high-performance storage. Erasure coding technology has high computational overhead, and the encoding and decoding processes consume a lot of CPU resources, affecting the overall performance of the system, especially in the data writing and recovery stages, where performance bottlenecks are obvious. Erasure coding technology is inefficient for processing small files, as the encoding overhead for small files accounts for a large proportion, resulting in poor I / O read and write performance for small files; Erasure coding technology has high costs for data updates and recovery. When updating data blocks, multiple check blocks need to be recalculated, and when recovering data, multiple blocks need to be read and decoded, which is time-consuming and resource-intensive. Therefore, the relevant technical solutions cannot dynamically adjust the data protection strategy according to the characteristics of data access frequency, importance, size, etc., making it difficult to simultaneously take into account disk utilization and data read / write I / O performance in different scenarios.

[0028] Embodiments of this application provide a data processing method, such as... Figure 3 As shown, the method includes: Retrieve the data to be stored; The data to be stored is encoded according to the size of the data file; The target data level is determined based on the data attributes of the data to be stored. The data attributes include one or more of the following: data access frequency, data file size, and data read / write performance requirements. Determine the target data protection strategy corresponding to the target data level based on the data attributes of the target data level; Determine the number of data blocks and check blocks that need to be written to the data to be stored based on the target data protection strategy; The target storage node corresponding to the data block and the check block is determined based on the number of data blocks and check blocks that need to be written to the data to be stored. The encoded data to be stored is stored to the target storage node according to the target data hierarchy and the target data protection strategy.

[0029] Understandably, this application proposes a data processing method that balances disk utilization and I / O read / write performance based on the existing problems. It introduces a dynamic hierarchical storage architecture, an adaptive encoding switching mechanism, a pre-computed parity block pool, intelligent I / O read / write scheduling, and a data recovery mechanism to make up for the shortcomings of multi-copy and erasure coding schemes.

[0030] Storing data information: Inputting the data information that needs to be saved into the system. The data information includes various types of data, such as text, images, videos, etc. Adaptive encoding switching: Before data is stored, the system selects the most suitable encoding method based on the characteristics of the data. This adaptive mechanism can improve data compression efficiency, reduce storage space occupation, and at the same time ensure data integrity and readability. Dynamic tiered storage: Encoded data is allocated to different storage tiers. This tiered storage strategy can store data on the most suitable medium based on factors such as data access frequency and importance. For example, frequently accessed data may be stored on faster solid-state drives (SSDs), while infrequently accessed data may be stored on lower-cost hard disk drives (HDDs). Pre-computed check block pool: To ensure data integrity, the system pre-computes the stored data, generates check blocks, and stores them in the pre-computed check block pool. These check blocks can be used to verify the correctness of the data during data recovery, ensuring that the data has not been damaged during transmission or storage. Intelligent I / O scheduling strategy: Through intelligent I / O scheduling algorithms, the system can effectively manage data read and write operations, avoid I / O bottlenecks, and improve the system's response speed and overall performance; Data recovery mechanism: When data is lost or damaged, the system can recover the data through checksum blocks and other copies, ensuring that data can be recovered in a timely manner even in the event of a failure, thus guaranteeing data security and availability.

[0031] The entire process is a closed-loop system, starting with data storage and going through multiple stages such as encoding, hierarchical storage, verification, and scheduling, ultimately achieving secure data storage and efficient management.

[0032] Embodiments of this application provide a data processing method, such as... Figure 4 As shown, the method includes: Step S01: Obtain the data to be stored; The data to be stored is encoded according to the size of the data file.

[0033] Here, file size has a significant impact on processing efficiency during data storage, especially for small files, which are not only time-consuming but also consume a lot of computing resources. To further optimize the processing efficiency of files of different sizes, this application introduces an adaptive encoding switching mechanism that automatically selects the appropriate encoding method based on the file size.

[0034] Step S011: When the memory space size of the data file is less than the first preset value (32MB), the data to be stored is encoded using an XOR checksum encoding algorithm. When the memory space size of the data file is greater than the first preset value (32MB) and less than the second preset value (1GB), the data to be stored is encoded using the second forward error correction code encoding algorithm. When the memory space size of the data file is greater than the second preset value (1GB), the data to be stored is encoded using the first forward error correction code encoding algorithm.

[0035] Specifically, for files smaller than 32MB, lightweight encoding algorithms such as XOR encoding are used. XOR encoding is simple to calculate and has low overhead, making it suitable for fast encoding and decoding of small files and effectively improving the I / O read and write performance of small files.

[0036] For files ranging from 32MB to 1GB, an improved RS encoding algorithm (second forward error correction code encoding algorithm) is adopted. This algorithm optimizes the generation method of the encoding matrix based on the traditional RS encoding, reduces the number of multiplication operations in the encoding process, and reduces computational overhead.

[0037] For files larger than 1GB, the standard RS encoding algorithm (first forward error correction code encoding algorithm) is used to ensure encoding reliability and efficient disk utilization.

[0038] Step S012: When reading or restoring stored data, decoding is performed according to the encoding method corresponding to the stored data. Create a data encoding mapping table; Map the data file to the corresponding file encoding method based on the size of the data file; In response to a change in the memory space size of the data file, the encoding method corresponding to the data file and the data encoding method mapping table are updated.

[0039] Specifically, during file storage and writing, the file size is first determined, and then the corresponding encoding method is selected for encoding processing according to the above criteria. During file reading or recovery, the corresponding decoding processing is performed according to the encoding method recorded in the file metadata. In order to achieve smooth switching of encoding methods, the system maintains an encoding method mapping table to record the encoding method used by each file. When the file size changes (such as increasing through append operations), the system will automatically trigger the reselection and switching of encoding methods and update the encoding method mapping table.

[0040] Step S02: Determine the target data level based on the data attributes of the data to be stored. The data attributes include one or more of the following: data access frequency, data file size, and data read / write performance requirements.

[0041] Specifically, dynamic tiering means dividing the stored data into different tiers based on the attributes of the data (including performance indicators such as access frequency, file size, and importance), and formulating different data protection strategies for each tier.

[0042] Step S021: Store data that has a data access frequency greater than the third preset value (10 times) and / or a data file size less than the fifth preset value (64MB) and / or meets the first level of read / write performance requirements in the hot data layer. The number of data accesses within the first preset interval (24 hours) is between the fourth preset value (3 times) and the third preset value (10 times) and / or the data file size is between the fifth preset value (64MB) and the second preset value (1GB) and / or the data to be stored that meets the second level of read and write performance requirements is stored in the warm data layer; Data access frequency within the first preset interval is less than the fourth preset value (3 times) and / or data file size is greater than the second preset value (1GB) and / or data that meets the third level of read and write performance requirements is stored in the cold data layer; The data read / write performance requirements are categorized into three levels: Level 1, Level 2, and Level 3.

[0043] The first level of read / write performance requirements is high-level, the second level of read / write performance requirements is medium-level, and the first level of read / write performance requirements is low-level.

[0044] Step S03: Determine the target data protection strategy corresponding to the target data level based on the data attributes of the target data level; Step S031: In response to the data to be stored being stored in the hot data layer, data protection is performed on the data in the hot data layer using a hybrid multi-replica strategy. Data protection for the hot data layer is achieved through a hybrid multi-replica strategy, including: The stored data in the hot data layer is divided into several hot data blocks. Several replicas are set for any hot data block, and several corresponding check blocks are calculated based on the hot data block. In response to the data to be stored being stored in the warm data layer, a hybrid erasure coding strategy is used to protect the data in the warm data layer. Data protection for the warm data layer is achieved through a hybrid erasure coding strategy, including: The stored data in the warm data layer is divided into several warm data blocks. Data protection is carried out on some warm data blocks through erasure coding strategy and data protection is carried out on some warm data blocks through multi-replica strategy. In response to the data to be stored being stored in the cold data layer, data protection is implemented on the data in the cold data layer using erasure coding strategies. This data protection includes: The stored data in the cold data layer is divided into several cold data blocks, and several corresponding check blocks are calculated based on these cold data blocks.

[0045] Specifically, this application defines three data levels based on three indicators: data access frequency, file size, and importance. The first layer is the hot data layer: data with high access frequency (e.g., accessed more than 10 times in the last 24 hours), files smaller than 64MB, and high I / O performance requirements are stored in the hot data layer. This layer uses an improved multi-replica strategy to achieve data protection; that is, a 2-replica + check block approach. For each hot data block, two complete replicas are stored, and a small check block is calculated at the same time. This approach not only ensures high read and write performance of hot data (reading can choose from two replicas, and writing only requires writing to two replicas and one small check block), but also improves disk utilization to a certain extent, while saving about 33% of space compared to the 3-replica scheme.

[0046] The second layer is the warm data layer: data with moderate access frequency (e.g., accessed 3-10 times in the last 24 hours), moderate size (e.g., 64MB-1GB), and low I / O performance requirements are stored in the warm data layer. This layer adopts a hybrid erasure coding strategy, that is, after the data is divided into blocks, some data blocks are protected by erasure coding strategy, and some data blocks are protected by replication strategy. For example, the data is divided into 8 data blocks, of which 6 data blocks are protected by RS(6,2) erasure coding (i.e., 6 data blocks, calculate the corresponding 2 check blocks), and the other 2 data blocks are protected by 2 replicas. This takes advantage of the high disk utilization of erasure coding and improves the read and write performance of data through partial replication protection.

[0047] The third layer is the cold data layer: large files with low access frequency (e.g., accessed less than 3 times in the last 24 hours), high storage capacity requirements (e.g., greater than 1GB), and low I / O performance requirements are stored in the cold data layer; this layer adopts an efficient erasure coding strategy, such as RS(16,4) erasure coding, which divides the data into 16 data blocks and calculates and generates 4 corresponding parity blocks. This method can maximize disk utilization and is suitable for storing cold data with low access frequency.

[0048] Here, the data at all three layers is not static and changes with the number of accesses. The data monitoring module in this application periodically (e.g., hourly) performs statistical analysis on data access. When data attributes change, it automatically migrates the data from the current layer to the corresponding target layer and updates the corresponding data protection strategy. For example, when the access frequency of hot data drops to the standard of warm data within 24 hours, the system migrates it from the hot data layer to the warm data layer and changes its data protection strategy from 2 copies + checksum blocks to a hybrid erasure coding strategy. The data migration process uses an incremental migration method, migrating only the changed data parts to reduce network bandwidth and I / O resource consumption during the migration process. At the same time, during the migration process, data availability and consistency are ensured by using a read-write locking mechanism to prevent data inconsistencies caused by modifications during the migration process.

[0049] Step S04: Create a pre-computed check block pool; The data to be stored is pre-calculated to generate several corresponding check blocks for the data to be stored, and these check blocks are stored in the pre-calculated check block pool.

[0050] Step S041: Pre-calculate the data to be stored in each data layer and divide the data to be stored in each data layer into several data blocks; Based on the historical data access frequency and update status, determine the data blocks that need to be updated in each data layer. Calculate the data blocks that need to be updated in each data layer using an incremental calculation method at a second preset interval (e.g., every 10 minutes) to obtain several corresponding verification blocks for the data blocks that need to be updated. Store these verification blocks in a pre-calculated verification block pool.

[0051] Specifically, to reduce the computational overhead of checksum blocks during data writing and updating, this application designs a pre-calculated checksum block pool mechanism. The system predicts data blocks that may need updating based on historical data access frequency and update patterns, and pre-calculates the corresponding checksum blocks for these data blocks, storing them in the pre-calculated checksum block pool. For example, for frequently updated hot data blocks, the system periodically (e.g., every 10 minutes) recalculates their checksum blocks and updates them to the pool. When pre-calculating checksum blocks, an incremental calculation method is used, calculating only the checksum blocks corresponding to the data portions that have changed compared to the last calculation, thus reducing the computational load. Simultaneously, idle CPU resources are utilized for pre-calculation to avoid affecting normal read and write operations.

[0052] Step S05: Update the pre-calculated check block pool; Update the pre-computed checksum pool, including: In response to the pre-computed parity block pool reaching a first threshold (set according to specific application scenarios, system resources and performance requirements, such as 512MB), the least frequently used parity blocks are deleted. The validity of the verification blocks in the pre-calculated verification block pool is verified according to the third preset interval time (e.g., 30s ~ 5min); The validity of the check blocks in the pre-computed check block pool is verified, including: If the original data block corresponding to the check block changes and the check block is not updated, the check block is marked as invalid, and the check block corresponding to the changed data block is recalculated. In response to writing data to be stored, the data to be stored is divided into several data blocks, and it is determined whether there is a valid check block corresponding to the data block in the pre-calculated check block pool. If a valid check block corresponding to a data block exists in the pre-computed check block pool, then that valid check block is obtained. If there is no valid check block corresponding to the data block in the pre-calculated check block pool, the corresponding valid check block is recalculated based on the data block and stored in the pre-calculated check block pool.

[0053] Specifically, the pre-computed check block pool adopts the LRU (Least Recently Used) eviction policy. When the pool capacity reaches the upper limit, the least recently used check block is deleted to make room for newly computed check blocks.

[0054] Meanwhile, the system will periodically check the validity of the check blocks in the pool. When the corresponding original data block is updated but the check block is not updated in time, the check block is marked as invalid and a recalculation is triggered.

[0055] When writing or updating data, the system first checks whether there is a corresponding valid check block in the pre-calculated check block pool. If it exists, the check block is used directly without recalculation. If it does not exist, the check block is calculated in real time and added to the pre-calculated check block pool.

[0056] Step S06: Dynamically schedule read and write requests for stored data based on the data layer type, data protection strategy, and load information of the data storage node.

[0057] Specifically, in order to improve the overall I / O read and write performance of the system, this application proposes an intelligent I / O scheduling strategy, which rationally schedules I / O requests based on factors such as data storage hierarchy, data protection method, and data file node load.

[0058] Step S061: In response to a read request for stored data, several data sources for the stored data are determined based on the data layer type and data protection policy type of the stored data. The data sources are the nodes where the data blocks and / or check blocks and / or replicas of the stored data are stored. Obtain the current load information for each data source node, including processor utilization and memory usage. Calculate the predicted response time value of the corresponding data source based on the current load information of the node where each data source is located; The data source with the shortest predicted response time will be used as the target node for reading data in the read request. Schedule read requests for stored data to the target node for reading.

[0059] Specifically, for read requests, the system first determines possible data sources (such as replica nodes, nodes storing data blocks, and parity blocks) based on the storage hierarchy and data protection policies. Then, it collects current load information (such as CPU utilization, memory usage, and disk I / O activity) of each data source node and calculates the predicted response time for each data source. Finally, it selects the data source with the shortest response time as the read target and schedules the read request to that node. For read requests of frequently accessed data, since multiple replicas exist, the system can perform parallel reading, simultaneously reading different parts of the data from multiple replicas and then merging them into complete data to return to the client, further improving read speed.

[0060] In step S062, in response to a write request for stored data, the number of data blocks and check blocks to be written to the data to be stored is determined according to the target data protection policy. The target storage node corresponding to the data block and the check block is determined based on the number of data blocks and check blocks that need to be written to the data to be stored. The encoded data to be stored is stored in the target storage node according to the target data hierarchy and the target data protection strategy. Obtain the current load information for each target storage node; Write requests for data to be stored are dynamically scheduled based on the current load information of each target storage node. Based on the target data hierarchy and target data protection strategy, the encoded data to be stored is stored to the target storage node, including: In response to storing hot data to the target storage node, the hot data is written to the primary replica, asynchronously written to the secondary replica, and the check block corresponding to the data block of the hot data is written. The hot data includes several replicas. In response to storing warm and cold data to the target storage node, the warm and cold data are written using a batch write strategy.

[0061] Specifically, for write requests, the system determines the number of data blocks and checksum blocks to be written, as well as the target nodes, based on the data storage hierarchy and data protection policies. Then, it dynamically allocates write tasks according to the current load of each target node, avoiding concentrating too many write requests on nodes with high load. For write requests of hot data, an asynchronous write to the slave replica is adopted, that is, the data is first written to the primary replica, and then a write success response is immediately returned to the client. At the same time, the data is asynchronously synchronized to the slave replica and the checksum blocks are written in the background, reducing the client's waiting time. For write requests of warm and cold data, a batch write strategy is adopted, merging multiple small write requests into a large write request for processing, reducing disk seek time and network transmission overhead.

[0062] Step S07: In response to the loss or damage of stored data, the stored data is recovered according to the data layer type of the data storage and the data protection policy corresponding to the data layer.

[0063] Step S071: Obtain the data layer type where the lost or damaged data is stored and the data protection policy type corresponding to the data layer; Determine the data block copies and / or verification blocks for data recovery based on the data protection policy type corresponding to the data layer where the lost or damaged data is stored; Data is recovered by using data block copies and / or check blocks for data recovery.

[0064] Specifically, the data recovery mechanism of this application combines a dynamic hierarchical storage architecture and a pre-computed check block pool to improve the speed and efficiency of data recovery.

[0065] When a data block is detected as missing (e.g., due to disk failure causing the stored data block to become inaccessible), the system first determines the data storage level where the missing data block is located and the corresponding data protection policy. Then, based on the data protection policy, the system recovers the missing data block from the remaining valid data blocks and check blocks (including check blocks in the pre-computed check block pool).

[0066] In step S072, in response to the fact that the lost or damaged data is data of the hot data layer, the data of the hot data layer is recovered first through the data block master copy; When the primary copy of a data block is also lost or damaged, the data in the hot data layer is recovered from the copy using the corresponding check block and data block. If the lost or corrupted data is from the warm data layer or the cold data layer, the data from the warm data layer and the cold data layer is recovered by pre-compiling the check blocks corresponding to the data from the check block pool.

[0067] Specifically, for lost data blocks in the hot data layer, since there are multiple replicas and check blocks, the system can prioritize recovering data from the replicas. If the replicas are also lost, the system can use the check blocks and other replicas to recover the data, resulting in a fast recovery speed. For lost data blocks in the warm and cold data layers, the corresponding erasure coding scheme is used in conjunction with the check blocks in the pre-computed check block pool to reduce the computational overhead during the recovery process and improve recovery efficiency.

[0068] Step S073: Obtain the load information of all storage nodes; The storage node with the lowest load rate is determined based on the load information of all storage nodes; Lost or damaged data can be recovered using incremental recovery methods; Data that has been lost or recovered will be stored on the storage node with the lowest load.

[0069] Specifically, during the data recovery process, the system will assign the recovery task to the lightest node based on the load of the storage nodes to avoid affecting the normal read and write operations of the system; at the same time, an incremental recovery method is adopted to recover only the lost data blocks, reducing the amount of data transmission.

[0070] like Figure 5 As shown, the key technical point of this application's technical solution lies in achieving an effective balance between disk utilization and I / O read / write performance in distributed storage through the coordinated work of multiple mechanisms, specifically including the following aspects: 1. Intelligent partitioning and migration of dynamic tiered storage architecture: Precise tiering is performed based on attributes such as data access frequency and size, and dynamic and efficient migration of data between tiers is achieved. This tiering is not fixed, but can be adjusted in real time according to the actual use of data to ensure that data is always under the optimal protection strategy. The quantitative indicators of the tiering standard (such as the specific numerical range of access frequency and the division boundary of file size) and the incremental migration and consistency guarantee mechanism during the migration process are the core of the effective operation of this storage architecture.

[0071] 2. Flexible adaptation of the adaptive encoding switching mechanism: It automatically selects different encoding methods according to the file size, realizing the best match between the encoding algorithm and the file characteristics; the reasonable application scope of lightweight encoding algorithms, improved RS encoding algorithms and standard RS encoding algorithms, as well as the smooth transition and mapping table management during the encoding method switching process, ensure that files of different sizes can obtain efficient encoding and decoding performance.

[0072] 3. Precise prediction and management of the pre-computed check block pool: By analyzing historical data, the data blocks that may need to be updated are predicted, and the check blocks are pre-computed and stored in the pre-computed check block pool, which greatly reduces the real-time computing overhead. The judgment of update frequency, the application of incremental calculation method in the pre-computation strategy, as well as the LRU eviction policy and validity check mechanism of the check block pool, ensure the efficient utilization and accuracy of check blocks in the pool.

[0073] 4. Optimized allocation of intelligent I / O scheduling strategy: Taking into account factors such as data storage hierarchy, data protection strategy and node load, read and write requests are reasonably scheduled; the construction of response time prediction model and implementation of parallel reading in read requests, and the application of strategies such as asynchronous writing and batch writing in write requests, effectively improve the overall I / O read and write performance of the system.

[0074] 5. Highly efficient data recovery mechanism: Combining a dynamic tiered storage architecture and a pre-computed check block pool, the optimal recovery method is selected based on the storage tier and data protection strategy of the data. During the recovery process, the storage node load is considered for task allocation, and an incremental recovery method is adopted to reduce the amount of data transmission, thereby improving the speed and efficiency of data recovery.

[0075] In addition, by using incremental calculation to calculate the data blocks that need to be updated in each data layer, several corresponding check blocks for the data blocks that need to be updated are obtained, including: The check block corresponding to several data blocks that need to be updated is calculated using the XOR check method. Retrieve multiple data blocks D1, D2, ..., Dn that need to be updated; The check block corresponding to multiple data blocks is calculated using the formula: P=D1⊕D2⊕⋯⊕Dn; When any data block Di is modified to Di′, the check block is updated incrementally. The check block is updated incrementally, including: The updated check block P′ is calculated using the formula: P′=P⊕(Di⊕Di′); The check block corresponding to the data block that needs to be updated is calculated using a linear encoding check method. Obtain data blocks D=[D1, D2, ..., Dk], and generate the parity part Gp of the data block matrix; The check block of the data block is calculated using the formula: P=D⋅Gp; When any Di becomes Di′, the change in data block ΔDi is calculated using the formula: ΔDi = Di′-Di; Obtain the verification coefficient vector gi corresponding to the i-th column in the generated data block matrix; The change ΔP of the check block can be calculated using the formula: ΔP=ΔD i ⋅gi; The new check block P′ is calculated using the formula: P′=P⊕(ΔDi⋅gi).

[0076] Here, by calculating the corresponding check block based on several data blocks using incremental calculation, we can avoid reading all data blocks in full, which greatly reduces I / O latency; only the changed parts are processed, saving CPU and network resources; reducing disk / SSD read / write operations and extending device lifespan; and quickly synchronizing check information to reduce the risk of silent errors and ensure data consistency.

[0077] The data processing method provided in the embodiments of this application can be improved and optimized in several ways without departing from the technical solution of this application, and these improvements and optimizations should also be considered within the scope of protection of this application.

[0078] The beneficial effects of the technical solutions provided in this application are: This application can balance disk utilization and data read / write performance, reduce resource computing overhead, and improve file processing efficiency.

[0079] This application's technical solution employs a dynamic hierarchical storage architecture, using different data protection strategies for data with different attributes. While ensuring data reliability, it significantly improves disk utilization: the hot data layer uses two replicas plus a checksum block, increasing disk utilization by approximately 33% compared to the traditional three-replica scheme; the warm data layer uses a hybrid erasure coding strategy, increasing disk utilization by 40%-50% compared to the multi-replica scheme; and the cold data layer uses an efficient erasure coding strategy, achieving disk utilization of over 80%, more than 60% higher than the multi-replica scheme. Therefore, this application improves overall disk utilization by 30%-50%, significantly reducing storage costs.

[0080] The intelligent I / O scheduling strategy of this application can select the optimal data source for read requests. The parallel reading of multiple replicas of hot data further improves the read speed. Experimental data shows that the read performance of hot data is 15%-20% higher than that of traditional multi-replica schemes, and the read performance of warm and cold data is 20%-30% higher than that of traditional erasure coding schemes. The adaptive encoding switching mechanism uses a lightweight encoding algorithm for small files, reducing the encoding overhead of small files. The pre-computation check block pool reduces the real-time calculation time of check blocks. The asynchronous writing of hot data from replicas reduces the client waiting time. The batch writing strategy of warm and cold data reduces disk seek and network transmission overhead. The I / O write performance of this application is 10%-20% higher than that of traditional multi-replica schemes and 30%-40% higher than that of traditional erasure coding schemes.

[0081] Embodiments of this application also provide a distributed storage system, which includes a data processing module and several physical nodes (disks). The data processing module is used for: Retrieve the data to be stored; The data to be stored is encoded according to the size of the data file; The target data level is determined based on the data attributes of the data to be stored. The data attributes include one or more of the following: data access frequency, data file size, and data read / write performance requirements. Determine the target data protection strategy corresponding to the target data level based on the data attributes of the target data level; Determine the number of data blocks and check blocks that need to be written to the data to be stored based on the target data protection strategy; The target storage node corresponding to the data block and the check block is determined based on the number of data blocks and check blocks that need to be written to the data to be stored. The encoded data to be stored is stored to the target storage node according to the target data hierarchy and the target data protection strategy.

[0082] In one embodiment, the data processing module is used to encode the data to be stored using an XOR checksum encoding algorithm when the memory space size of the data file is less than a first preset value. When the memory space size of the data file is greater than the first preset value and less than the second preset value, the data to be stored is encoded using the second forward error correction code encoding algorithm. When the memory space size of the data file is greater than the second preset value, the data to be stored is encoded using the first forward error correction code encoding algorithm.

[0083] In one embodiment, the data processing module is used to store data to be stored in the hot data layer if the number of times the data accesses within a first preset interval is greater than a third preset value and / or the data file size is less than a fifth preset value and / or the data meets the first level of read and write performance requirements. Data access frequency within the first preset interval is between the fourth preset value and the third preset value, and / or data file size is between the fifth preset value and the second preset value, and / or data to be stored that meets the second level of read and write performance requirements is stored in the warm data layer. Data access frequency less than the fourth preset value within the first preset interval and / or data file size greater than the second preset value and / or data that meets the third level of read and write performance requirements will be stored in the cold data layer. The data read / write performance requirements are categorized into three levels: Level 1, Level 2, and Level 3.

[0084] In one embodiment, the data processing module is configured to protect the data in the hot data layer by means of a hybrid multi-replica strategy in response to the data to be stored being stored in the hot data layer. Data protection for the hot data layer is achieved through a hybrid multi-replica strategy, including: The stored data in the hot data layer is divided into several hot data blocks. Several replicas are set for any hot data block, and several corresponding check blocks are calculated based on the hot data block. In response to the data to be stored being stored in the warm data layer, a hybrid erasure coding strategy is used to protect the data in the warm data layer. Data protection for the warm data layer is achieved through a hybrid erasure coding strategy, including: The stored data in the warm data layer is divided into several warm data blocks. Data protection is carried out on some warm data blocks through erasure coding strategy and data protection is carried out on some warm data blocks through multi-replica strategy. If the data to be stored is stored in the cold data layer, then erasure coding strategy is used to protect the data in the cold data layer. Data protection for the cold data layer is achieved through erasure coding strategies, including: The stored data in the cold data layer is divided into several cold data blocks, and several corresponding check blocks are calculated based on these cold data blocks.

[0085] In one embodiment, a data processing module is used to create a pre-computed check block pool; The data to be stored is pre-calculated to generate several corresponding verification blocks for the data to be stored, and these verification blocks are stored in the pre-calculated verification block pool. The data to be stored is pre-computed to generate several corresponding check blocks, and these check blocks are stored in a pre-computed check block pool, including: Pre-calculate the data to be stored in each data layer and divide the data to be stored in each data layer into several data blocks; Based on the historical data access frequency and update status, determine the data blocks that need to be updated in each data layer. Calculate the data blocks that need to be updated in each data layer using an incremental calculation method according to the second preset interval, and obtain several corresponding verification blocks for the data blocks that need to be updated. Store several corresponding check blocks for the data blocks that need to be updated in a pre-computed check block pool.

[0086] In one embodiment, a data processing module is used to update the pre-computed check block pool; Update the pre-computed checksum pool, including: In response to the pre-computed check block pool reaching a first threshold, the least frequently used check block is deleted. The validity of the verification blocks in the pre-calculated verification block pool is verified according to the third preset interval time. The validity of the check blocks in the pre-computed check block pool is verified, including: If the original data block corresponding to the check block changes and the check block is not updated, the check block is marked as invalid, and the check block corresponding to the changed data block is recalculated. In response to writing data to be stored, the data to be stored is divided into several data blocks, and it is determined whether there is a valid check block corresponding to the data block in the pre-calculated check block pool. If a valid check block corresponding to a data block exists in the pre-computed check block pool, then that valid check block is obtained. If there is no valid check block corresponding to the data block in the pre-calculated check block pool, the corresponding valid check block is recalculated based on the data block and stored in the pre-calculated check block pool.

[0087] In one embodiment, the data processing module is used to dynamically schedule read requests for stored data based on the data level type of the data storage, the data protection strategy corresponding to the data level, and the load information of the data storage node. Dynamic scheduling of read requests for stored data is performed based on the data level type, the corresponding data protection policy, and the load information of the data storage nodes, including: In response to a read request for stored data, several data sources for the stored data are determined based on the data level type and data protection policy type of the stored data. The data sources are the storage nodes where the data blocks and / or check blocks and / or replicas of the stored data are stored. Obtain the current load information for each storage node where the data source is located. The load information includes processor utilization and memory utilization. Calculate the predicted response time value for each data source based on the current load information of the storage node where each data source is located; The data source with the shortest predicted response time will be used as the target storage node for the read request data. The read request for the stored data is scheduled to the target storage node.

[0088] In one embodiment, the data processing module is used to obtain the current load information corresponding to each target storage node; Write requests for data to be stored are dynamically scheduled based on the current load information of each target storage node. In response to storing hot data to the target storage node, the hot data is written to the primary replica, asynchronously written to the secondary replica, and the check block corresponding to the data block of the hot data is written. The hot data includes several replicas. In response to storing warm and cold data to the target storage node, the warm and cold data are written using a batch write strategy.

[0089] In one embodiment, the data processing module is configured to recover the stored data according to the data level type of the data storage and the data protection strategy corresponding to the data level in response to the loss or damage of the stored data. Data recovery is performed based on the data hierarchy type and the corresponding data protection policy, including: Retrieve the data level type where the lost or corrupted data is stored and the corresponding data protection policy type for that data level; The data block copies and / or verification blocks for data recovery are determined based on the data protection policy type corresponding to the data level at which the lost or damaged data is stored; Data is recovered by using data block copies and / or check blocks for data recovery.

[0090] In one embodiment, the data processing module is configured to, in response to the loss or damage of data being data of the hot data layer, prioritize the recovery of the data of the hot data layer through the data block master copy; When the primary copy of the data block is also lost or damaged, the data of the hot data layer is recovered from the replica through the check block and the hot data block corresponding to the hot data block. If the lost or corrupted data is from the warm data layer or the cold data layer, the corresponding check blocks are calculated from the pre-calculated check block pool to recover the data from the warm data layer and the cold data layer.

[0091] In one embodiment, the data processing module is used to obtain load information of all storage nodes; The storage node with the lowest load rate is determined based on the load information of all storage nodes; Lost or damaged data can be recovered using incremental recovery methods; Data that has been lost or recovered will be stored on the storage node with the lowest load.

[0092] In one embodiment, the data processing module is configured to perform decoding processing according to the encoding method corresponding to the stored data when reading or restoring the stored data; Create a data encoding mapping table; Map the data file to the corresponding file encoding method based on the size of the data file; In response to a change in the memory space size of the data file, the encoding method corresponding to the data file and the data encoding method mapping table are updated.

[0093] Specifically, dynamic tiered storage includes specific criteria for dividing data into hot data, warm data, and cold data layers based on attributes such as data access frequency and size; data protection strategies adopted by each layer, such as 2 copies + check blocks, hybrid erasure coding, and efficient erasure coding; and mechanisms for dynamic data migration between layers, including migration triggering conditions, incremental migration methods, and data consistency assurance measures.

[0094] Adaptive encoding switching: The specific selection criteria for choosing a lightweight encoding algorithm, an improved RS encoding algorithm, or a standard RS encoding algorithm based on file size; the mechanism for achieving smooth switching of encoding methods, including the maintenance and updating method of the encoding method mapping table; and the process for reselecting and switching encoding methods when the file size changes.

[0095] Construction and management of the pre-computation check block pool: a pre-computation strategy based on historical data access and update data to predict the data blocks that need to be updated and pre-computation check blocks, including the pre-computation cycle, incremental calculation method, and implementation method of utilizing idle system resources; a method of managing the check block pool using the LRU eviction policy, including setting the upper limit of the pool capacity, marking invalid check blocks, and recalculation triggering mechanism; and the process of querying and using pre-computation check blocks when writing or updating data.

[0096] Intelligent I / O scheduling: In read request scheduling, the data source is determined, node load information is collected, response time prediction is calculated, and the optimal data source is selected, as well as the parallel reading implementation method for hot data read requests; in write request scheduling, the method of determining the target node for writing and dynamically allocating tasks according to the node load is used; for hot data write requests, the asynchronous write from replica mode is used, and for warm and cold data write requests, the batch write strategy is used.

[0097] Data recovery: When data blocks are lost, the method for recovering lost data blocks from the remaining valid blocks and the pre-computed check block pool according to the storage level where the data is located and the data protection policy; the mechanism for allocating recovery tasks according to node load during the recovery process; and the implementation method of reducing data transmission volume by using incremental recovery.

[0098] The beneficial effects of the technical solutions provided in this application are: This application can balance disk utilization and data read / write performance, reduce resource computing overhead, and improve file processing efficiency.

[0099] This application's technical solution employs a dynamic hierarchical storage architecture, using different data protection strategies for data with different attributes. While ensuring data reliability, it significantly improves disk utilization: the hot data layer uses two replicas plus a checksum block, increasing disk utilization by approximately 33% compared to the traditional three-replica scheme; the warm data layer uses a hybrid erasure coding strategy, increasing disk utilization by 40%-50% compared to the multi-replica scheme; and the cold data layer uses an efficient erasure coding strategy, achieving disk utilization of over 80%, more than 60% higher than the multi-replica scheme. Therefore, this application improves overall disk utilization by 30%-50%, significantly reducing storage costs.

[0100] The technical solution of this application adopts a pre-computation check block pool mechanism, which utilizes idle system resources to pre-compute check blocks, reducing the real-time computation overhead during data writing and updating; the adaptive encoding switching mechanism uses a computationally simple lightweight encoding algorithm for small files, reducing the computational load of encoding and decoding for small files; the improved RS encoding algorithm reduces the number of multiplication operations in the encoding process, reducing the computational overhead of medium-sized files, making the overall computational overhead of this application 25%-35% lower than that of traditional erasure coding schemes.

[0101] This application's technical solution addresses the characteristics of small files by employing an optimized multi-copy strategy at the hot data layer and selecting lightweight XOR checksum encoding, thereby reducing the encoding and storage overhead of small files. The read / write performance of small files is improved by 40%-50% compared to traditional erasure coding schemes, solving the problem of low efficiency in small file processing by traditional erasure coding technology.

[0102] For a description of the features in the embodiment corresponding to the data processing module, please refer to the relevant description of the embodiment corresponding to the data processing method, which will not be repeated here.

[0103] Embodiments of this application also provide an electronic device, including a memory and a processor. The memory stores a computer program, and the processor is configured to run the computer program to perform the steps in the data processing method embodiments, the method including: Retrieve the data to be stored; The data to be stored is encoded according to the size of the data file; The target data level is determined based on the data attributes of the data to be stored. The data attributes include one or more of the following: data access frequency, data file size, and data read / write performance requirements. Determine the target data protection strategy corresponding to the target data level based on the data attributes of the target data level; Determine the number of data blocks and check blocks that need to be written to the data to be stored based on the target data protection strategy; The target storage node corresponding to the data block and the check block is determined based on the number of data blocks and check blocks that need to be written to the data to be stored. The encoded data to be stored is stored to the target storage node according to the target data hierarchy and the target data protection strategy.

[0104] like Figure 6 As shown, embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in the data processing method embodiments at runtime, the method including: Retrieve the data to be stored; The data to be stored is encoded according to the size of the data file; The target data level is determined based on the data attributes of the data to be stored. The data attributes include one or more of the following: data access frequency, data file size, and data read / write performance requirements. Determine the target data protection strategy corresponding to the target data level based on the data attributes of the target data level; Determine the number of data blocks and check blocks that need to be written to the data to be stored based on the target data protection strategy; The target storage node corresponding to the data block and the check block is determined based on the number of data blocks and check blocks that need to be written to the data to be stored. The encoded data to be stored is stored to the target storage node according to the target data hierarchy and the target data protection strategy.

[0105] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0106] Embodiments of this application also provide a computer program product, which includes a computer program. When executed by a processor, the computer program implements the steps in the data processing method embodiments, the method including: Retrieve the data to be stored; The data to be stored is encoded according to the size of the data file; The target data level is determined based on the data attributes of the data to be stored. The data attributes include one or more of the following: data access frequency, data file size, and data read / write performance requirements. Determine the target data protection strategy corresponding to the target data level based on the data attributes of the target data level; Determine the number of data blocks and check blocks that need to be written to the data to be stored based on the target data protection strategy; The target storage node corresponding to the data block and the check block is determined based on the number of data blocks and check blocks that need to be written to the data to be stored. The encoded data to be stored is stored to the target storage node according to the target data hierarchy and the target data protection strategy.

[0107] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the steps in the data processing method embodiments, the method including: Retrieve the data to be stored; The data to be stored is encoded according to the size of the data file; The target data level is determined based on the data attributes of the data to be stored, wherein the data attributes include one or more of the following: data access frequency, data file size, and data read / write performance requirements. Determine the target data protection strategy corresponding to the target data level based on the data attributes of the target data level; Determine the number of data blocks and check blocks that need to be written to the data to be stored based on the target data protection strategy; The target storage node corresponding to the data block and the check block is determined based on the number of data blocks and check blocks that need to be written to the data to be stored. The encoded data to be stored is stored to the target storage node according to the target data hierarchy and the target data protection strategy.

[0108] This application can balance disk utilization and data read / write performance, reduce resource computing overhead, and improve file processing efficiency.

[0109] This application's technical solution employs a dynamic hierarchical storage architecture, using different data protection strategies for data with different attributes. While ensuring data reliability, it significantly improves disk utilization: the hot data layer uses two replicas plus a checksum block, increasing disk utilization by approximately 33% compared to the traditional three-replica scheme; the warm data layer uses a hybrid erasure coding strategy, increasing disk utilization by 40%-50% compared to the multi-replica scheme; and the cold data layer uses an efficient erasure coding strategy, achieving disk utilization of over 80%, more than 60% higher than the multi-replica scheme. Therefore, this application improves overall disk utilization by 30%-50%, significantly reducing storage costs.

[0110] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0111] The foregoing has provided a detailed description of a data processing method, apparatus, medium, and program product provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to aid in understanding the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A data processing method, characterized in that, The method includes: Retrieve the data to be stored; The data to be stored is encoded according to the size of the data file; The target data level is determined based on the data attributes of the data to be stored, wherein the data attributes include one or more of the following: data access frequency, data file size, and data read / write performance requirements. Determine the target data protection strategy corresponding to the target data level based on the data attributes of the target data level; The number of data blocks and verification blocks that need to be written to the data to be stored is determined according to the target data protection strategy; The target storage node corresponding to the data block and the check block is determined based on the number of data blocks and check blocks that need to be written to the data to be stored. The encoded data to be stored is stored in the target storage node according to the target data hierarchy and the target data protection strategy.

2. The data processing method according to claim 1, characterized in that, The process of encoding the data to be stored according to the size of the data file includes: When the memory space size of the data file is less than the first preset value, the data to be stored is encoded using an XOR checksum encoding algorithm. When the memory space size of the data file is greater than a first preset value and less than a second preset value, the data to be stored is encoded using a second forward error correction code encoding algorithm. When the memory space size of the data file is greater than the second preset value, the data to be stored is encoded using the first forward error correction code encoding algorithm.

3. The data processing method according to claim 1, characterized in that, The step of determining the target data level based on the data attributes of the data to be stored includes: Data access frequency greater than the third preset value and / or data file size less than the fifth preset value within the first preset interval and / or data to be stored that meets the first level of read and write performance requirements will be stored in the hot data layer. Data access frequency within the first preset interval is between the fourth preset value and the third preset value, and / or data file size is between the fifth preset value and the second preset value, and / or data to be stored that meets the second level of read and write performance requirements is stored in the warm data layer. Data access frequency less than the fourth preset value within the first preset interval and / or data file size greater than the second preset value and / or data that meets the third level of read and write performance requirements will be stored in the cold data layer. The data read / write performance requirements are categorized into three levels: Level 1, Level 2, and Level 3.

4. The data processing method according to claim 1, characterized in that, The step of determining the target data protection strategy corresponding to the target data level based on the data attributes of the target data level includes: In response to the data to be stored being stored in the hot data layer, a hybrid multi-replica strategy is used to protect the data in the hot data layer. The method of protecting the data in the hot data layer through a hybrid multi-replica strategy includes: The stored data in the hot data layer is divided into several hot data blocks. Several replicas are set for any hot data block, and several corresponding check blocks are calculated based on the hot data block. In response to the data to be stored being stored in a warm data layer, a hybrid erasure coding strategy is used to protect the data in the warm data layer. The method of protecting the data in the warm data layer using a hybrid erasure coding strategy includes: The stored data in the warm data layer is divided into several warm data blocks. Data protection is carried out on some warm data blocks through erasure coding strategy and data protection is carried out on some warm data blocks through multi-replica strategy. In response to the fact that the data to be stored is stored in the cold data layer, the data in the cold data layer is protected by erasure coding strategy; The data protection of the cold data layer using erasure coding strategy includes: The stored data in the cold data layer is divided into several cold data blocks, and several corresponding check blocks are calculated based on these cold data blocks.

5. The data processing method according to claim 1, characterized in that, After determining the target data protection strategy corresponding to the target data level based on the data attributes of the target data level, the method further includes: Create a pre-computed check block pool; Pre-calculate the data to be stored to generate several corresponding verification blocks for the data to be stored, and store the several corresponding verification blocks for the data to be stored in the pre-calculated verification block pool. The step of pre-computing the data to be stored, generating several corresponding verification blocks for the data to be stored, and storing the several corresponding verification blocks for the data to be stored in the pre-computed verification block pool includes: Pre-calculate the data to be stored in each data layer and divide the data to be stored in each data layer into several data blocks; Based on the historical data access frequency and update status, determine the data blocks that need to be updated in each data layer. Calculate the data blocks that need to be updated in each data layer using an incremental calculation method according to the second preset interval, and obtain several corresponding verification blocks for the data blocks that need to be updated. The corresponding check blocks of the data block that needs to be updated are stored in the pre-calculated check block pool.

6. The data processing method according to claim 5, characterized in that, The method further includes: pre-computing the data to be stored to generate several corresponding verification blocks for the data to be stored, and storing the several corresponding verification blocks for the data to be stored in the pre-computed verification block pool. Update the pre-calculated check block pool; The update of the pre-calculated check block pool includes: In response to the pre-calculated check block pool reaching a first threshold, the check block with the least frequent use is deleted. The validity of the verification blocks in the pre-calculated verification block pool is verified according to the third preset interval time. The verification of the validity of the verification blocks in the pre-calculated verification block pool includes: If the original data block corresponding to the check block changes and the check block is not updated, the check block is marked as invalid, and the check block corresponding to the changed data block is recalculated. In response to writing data to be stored, the data to be stored is divided into several data blocks, and it is determined whether there is a valid check block corresponding to the data block in the pre-calculated check block pool. If a valid check block corresponding to the data block exists in the pre-calculated check block pool, then the valid check block is obtained. If there is no valid check block corresponding to the data block in the pre-calculated check block pool, then the corresponding valid check block is recalculated based on the data block, and the valid check block is stored in the pre-calculated check block pool.

7. The data processing method according to claim 1, characterized in that, The method further includes: The read requests for stored data are dynamically scheduled based on the data level type of the data storage, the data protection strategy corresponding to the data level, and the load information of the data storage node. The dynamic scheduling of read requests for stored data based on the data level type, the corresponding data protection strategy, and the load information of the data storage nodes includes: In response to a read request for the stored data, several data sources for the stored data are determined based on the data level type and data protection policy type of the stored data, wherein the data sources are storage nodes that store data blocks and / or check blocks and / or replicas corresponding to the stored data; Obtain the current load information corresponding to the storage node where each data source is located, wherein the load information includes processor utilization and memory utilization; Calculate the predicted response time value for each data source based on the current load information of the storage node where each data source is located; The data source with the shortest predicted response time will be used as the target storage node for the read request data. The read request for the stored data is scheduled to the target storage node.

8. The data processing method according to claim 1, characterized in that, The step of storing the encoded data to be stored to the target storage node according to the target data level and the target data protection strategy includes: Obtain the current load information for each target storage node; The write requests for the data to be stored are dynamically scheduled based on the current load information of each target storage node. In response to storing hot data to the target storage node, the hot data is written to the primary replica, asynchronously written to the secondary replica, and the check block corresponding to the data block of the hot data is written, wherein the hot data includes several replicas; In response to storing warm and cold data to the target storage node, the warm and cold data are written using a batch write strategy.

9. The data processing method according to claim 5, characterized in that, The method further includes: In response to the loss or corruption of stored data, the stored data is recovered according to the data level type of the data storage and the data protection policy corresponding to the data level; The step of restoring stored data according to the data level type of the data storage and the corresponding data protection policy includes: Retrieve the data level type where the lost or corrupted data is stored and the corresponding data protection policy type for that data level; The data block copies and / or verification blocks for data recovery are determined based on the data protection policy type corresponding to the data level at which the lost or damaged data is stored; Data is recovered using data block copies and / or verification blocks from the data recovery process.

10. The data processing method according to claim 9, characterized in that, The data recovery process using the data block copies and / or checksum blocks recovered from the data includes: If the lost or corrupted data is from the hot data layer, the hot data layer data will be recovered first through the primary copy of the data block; When the primary copy of the data block is also lost or damaged, the data of the hot data layer is recovered from the replica through the check block and the hot data block corresponding to the hot data block. If the lost or damaged data is from the warm data layer or the cold data layer, the corresponding check blocks are calculated from the pre-calculated check block pool to recover the data from the warm data layer and the cold data layer.

11. The data processing method according to claim 9, characterized in that, The step of restoring stored data according to the data level type of the data storage and the corresponding data protection strategy also includes: Obtain load information for all storage nodes; The storage node with the lowest load rate is determined based on the load information of all the storage nodes; Lost or damaged data can be recovered using incremental recovery methods; The data that has been lost or recovered will be stored on the storage node with the lowest load.

12. The data processing method according to claim 2, characterized in that, The method further includes: When reading or restoring stored data, decoding is performed according to the encoding method corresponding to the stored data; Create a data encoding mapping table; Map the data file to its corresponding file encoding method based on the size of the data file; In response to a change in the memory space size of the data file, the encoding method corresponding to the data file and the data encoding method mapping table are updated.

13. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the data processing method as described in any one of claims 1 to 12.

Images