Erasure code algorithm processing method for cold and hot fragmented data in distributed storage system in local area network or wide area network
By using the cold and cold sharded data erasure coding algorithm processing method in a distributed storage system, the erasure coding algorithm is dynamically adjusted according to the hot and cold properties of the data and nodes, and the performance and complexity problems of the erasure coding process in the prior art are solved, and efficient data storage and recovery are achieved.
Patent Information
- Application Number
- CN202510163373.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-14
- Publication Date
- 2025-05-30
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
When the prior art uses an erasure coding algorithm to process hot and cold data in distributed storage systems, there are problems such as write amplification, degradation of read performance, increased complexity, extended recovery time and excessive resource consumption.
A method of processing method for cold and cold sharded data erasure coding is proposed. Through the blocking of data blocks, the judgment of hot and cold data between data and nodes, the processing mechanism of the erasure coding algorithm for different types of data, dynamic adjustment of encoding processing, etc., different erasure coding algorithms are used to process according to the hot and cold properties of data and nodes.
This method can reduce the processing time of data storage, maintain normal read and write performance, improve the rationality of hot and cold data division, enhance the reliability and fault tolerance of the storage system, and is suitable for distributed storage systems in large-scale wide-area weak network environments.
Smart Images

Figure CN120066847A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of distributed storage, and particularly to a method for processing erasure coding algorithms for hot and cold sharded data in a distributed storage system in a local area network or a wide area network. Background Art
[0002] In the era of big data, the storage of massive amounts of data, especially in a distributed storage system in a local area network or a wide area network environment, effective protection of data is very important. For a distributed system with heterogeneous nodes, while performing hierarchical storage and processing according to the hot and cold degrees of data, it is also necessary to perform hierarchical redundancy protection of data.
[0003] Hierarchical storage of hot and cold data is a method of dividing data into hot data and cold data according to attributes such as data access frequency, data size, timeliness, etc., and storing them on different storage media respectively. Among them, hot data refers to data that is frequently accessed or requires quick response, while cold data refers to data that is rarely accessed or does not require quick response.
[0004] The core idea of hierarchical storage of hot and cold data is to store hot data on high-performance and high-cost storage media to meet the needs of quick access, reading and writing; while storing cold data on low-cost and low-power storage media to reduce storage costs and energy consumption. Through this hierarchical storage method, while ensuring data access efficiency, the optimal allocation of storage resources can be achieved.
[0005] Since in a distributed storage system, especially in a distributed storage system in a large-scale wide area weak network environment, there are a large number of non-homogeneous storage nodes, in a distributed storage system, it is necessary to make a reasonable and scientific distinction according to the characteristics of the storage nodes and distribute different storage tasks.
[0006] Simply considering the data access frequency or the latest access time cannot accurately reflect whether the data is hot or cold data, and the access to hot and cold data is completely dependent on and restricted by the network capabilities of the distributed nodes storing the data and the data read and write latency.
[0007] Erasure Coding in a distributed storage system is a data redundancy technology that divides data into multiple segments through a mathematical algorithm and generates additional check information for these segments. The purpose of doing this is to improve the reliability and fault tolerance of data, so that even if some data or storage nodes are lost or fail, the original data can be restored based on the remaining data and check information.
[0008] Cold data refers to data that is not frequently accessed or used, mainly for specific purposes such as backup, compliance with legal regulations, or offline analysis. This data is typically stored in systems with lower performance and lower cost to reduce the costs associated with maintaining archived data. Examples include enterprise backup data, operation log data, call records, and statistical data. When storing cold data, two key points need to be considered: low cost and low power consumption. Since cold data is rarely accessed, its real-time requirement is not as high as that of OLTP or OLAP. The processing method varies depending on the data usage. The following are the ways to handle cold data from several perspectives: The I / O path refers to the facilities and mechanisms in the host that can direct I / O requests to a storage device to an access path. It is the path for data transmission in a computer system. It consists of multiple levels, including user space, kernel space, device drivers, controllers / interfaces, and external devices.
[0009] Currently, the use of data erasure codes in distributed systems has both advantages and disadvantages, especially when there is a large amount of cold data and hot data in distributed systems.
[0010] Hot Data refers to data that is frequently accessed or modified. In a distributed storage system, using erasure coding (EC) for hot data may bring some specific problems and challenges: Write amplification: For frequently updated hot data, each update will trigger the entire EC encoding process. Since erasure coding generates parity blocks based on a set of data blocks, even if only a small part of the data is updated, all related data blocks and parity blocks need to be recalculated. This will result in a significant write amplification effect, that is, the actual amount of data written is much larger than the amount of data the user intends to write, increasing the I / O burden.
[0011] Read performance degradation: When reading hot data, if this data is scattered and stored and some nodes are unavailable or the network latency is high, then to reconstruct the original data, it may be necessary to obtain sufficient data fragments from multiple nodes, which will increase the read latency. In addition, the decoding process itself will also introduce additional computational overhead.
[0012] Increased complexity: Implementing and supporting an erasure coding scheme optimized for hot data will increase the complexity of the system. For example, it may be necessary to design a dedicated caching mechanism to reduce unnecessary encoding / decoding operations, or develop a more intelligent data layout strategy to improve access efficiency.
[0013] Extended recovery time: After a node failure, the recovery of hot data may be more urgent than that of cold data, but because erasure coding involves more cooperation between nodes, its recovery process is usually slower, which may lead to an extended service interruption time.
[0014] Resource consumption: Due to the high-frequency access characteristics of hot data, continuous encoding and decoding operations consume a large amount of computing resources such as CPU and memory, which poses a relatively large pressure on system resources.
[0015] In current market applications, when dealing with hot data, simple redundancy methods such as mirror replication are chosen, which provide better performance and lower complexity in the face of frequent read and write operations. However, it occupies more storage space, and replica synchronization requires a large amount of network and IO resources, especially in the case of multiple replicas.
[0016] Another approach is to make trade-offs at the system design level. For example, separate hot data from cold data for management, adopt a more efficient storage strategy for hot data, and apply erasure codes to cold data to save storage costs.
[0017] However, the above methods are relatively arbitrary and do not consider more dimensions of hot and cold data and node attributes, lacking flexibility and refinement.
[0018] There is an urgent need for a new processing method to solve the above problems. Summary of the Invention
[0019] The present invention proposes a method for processing erasure code algorithms for hot and cold sharded data in a distributed storage system in a local area network or a wide area network, which solves the problems brought by using erasure code algorithms to process hot and cold data in the prior art.
[0020] The technical solution of the present invention is implemented as follows: A method for processing erasure code algorithms for hot and cold sharded data in a distributed storage system in a local area network or a wide area network, including the following steps: (1) Data block partitioning: Divide the data to be stored into multiple data blocks of a fixed size, and group and store them according to the state and quantity of distributed nodes; each node is responsible for storing one or more groups of data blocks; (2) Parameter strategy for data hot and cold judgment: Through the metadata server of the distributed storage system, intelligent algorithm management is used to judge the hot and cold attributes of the data (including data access frequency $f$ (frequent access, frequent update), time window $T$ (heat timeliness within a certain period of time), access mode $M$ (high and low latency, concurrency), data size $S$ (size of data shards), data age $A$ (time length since the data was generated); According to the hot and cold attributes of the data at the metadata node, access frequency, time window, data size, and data age, the hot and cold data attribute values calculated by the data heat algorithm; The accessed data with hot and cold data attribute values lower than the set threshold is set as cold data, and the accessed data with hot and cold data attribute values higher than the set threshold is set as hot data; (3) Parameter strategy for judging node hot and cold: parameters such as the performance of the node, network bandwidth status, CPU occupancy, recent IO read and write quantity and latency, number of network data packet transmissions and receptions, significant differences in the number of file reads and writes and network traffic, and the region where the node is located are comprehensively considered to judge the hot and cold situation of the node; the hot and cold node attribute value of the node calculated through the node heat algorithm; nodes with hot and cold node attribute values lower than the set threshold are set as cold data, and nodes with hot and cold node attribute values higher than the set threshold are set as hot data; (4) Writing data: Write the data block to the node allocated by the distributed storage system, record the hot and cold attributes of the data, and use metadata to record the hot and cold attributes of the data when the system configuration policy takes effect; (5) Erasure code algorithm processing mechanism for different types of data: Different erasure code algorithms are used for processing hot data on hot nodes, hot data on cold nodes, and cold data on hot nodes; (6) Processing after erasure code algorithm processing; Stored on cold nodes, redundant data will be stored on the central computer room server of the wide area network distributed storage system, and migration and erasure code processing will be carried out according to the amount of cold data and the number of cold nodes (7) Processing of data hot and cold transformation: Carry out encoding processing; The encoding parameters are dynamically adjusted during encoding processing: For hot data, the encoding parameters of the erasure code, including data block size, redundancy, etc., can be dynamically adjusted according to actual needs to balance data reliability and performance.
[0021] Further, the erasure code algorithm processing mechanism for different types of data in step (5) is specifically: (5.1) Hot data on hot nodes: Set a certain number of copy backups according to the heat, and the nodes where the copies are located are allocated and decided by the heat of the data and the request origin, and no erasure code algorithm is used for the copy data; (5.2) Hot data on cold nodes: The performance, bandwidth capacity, and storage capacity of the cold node need to be used as the core decision-making basis, and the hot data will be gradually migrated to the hot node according to the quantity; (5.3) Cold data on hot nodes: Allocated by the management server of the distributed storage system according to factors such as the size and number of shards of the cold data and the number of cold nodes, and the cold data is moved to the cold node. After the cold data is written to disk, the encoding process of the erasure code is carried out; Further, the processing method of cold data in step (2) is: (2.1) EC encoding processing: Achieve data redundancy and recovery by adding redundant data blocks; (2.2) Read operation: The data block read is the original unencoded data block: Directly read the data block from the storage medium and return it to the user; The read data block is an encoded redundant data block: first, decode the relevant encoded data block to reconstruct the original data block; the decoding operation can use an erasure code decoding algorithm to calculate and recover the original data block; (2.3) Write operation: The written data block is an original data block: directly write the data into the storage medium, and perform erasure code encoding as needed to generate corresponding redundant data blocks; The written data block is a redundant data block: write the data block to the corresponding position in the storage medium according to the erasure code encoding rule.
[0022] (2.4) Read operation optimization: According to the hot and cold attributes of the data judged in step 2, decide whether to perform real-time decoding on cold data, or delay the decoding operation until the data is accessed, so as to reduce the overhead of the read operation; (2.5) Write operation optimization: Adopt an asynchronous method to reduce the requirement for real-time performance, thereby reducing the delay of the write operation; (2.6) Delete data grouping: Delete the data grouping of cold data, and add a group of parity check blocks in place; the parity check blocks are obtained by performing exclusive OR operations on other data blocks; (2.7) Node failure or data corruption: The erasure code algorithm can be used for data recovery and repair; when the data on a certain node is lost or damaged, the lost or damaged data block can be calculated through parity check blocks and other valid data blocks.
[0023] Preferably, through the node management and data sharding management of the central management server, comprehensively adjust the data hot and cold strategy, node hot and cold strategy, and data replication factor; A method for processing erasure code algorithms for hot and cold sharded data in a distributed storage system in a local area network or a wide area network disclosed by the present invention has the following beneficial effects: First, it can reduce the processing time of data storage, and at the same time keep the normal read and write performance unaffected by the erasure code processing during the EC encoding process, and improve the rationality of hot and cold data partitioning in the distributed storage system; Second, store and migrate hot and cold data in layers respectively, and use the node hot and cold management strategy to perform hierarchical redundant protection and replica backup on hot and cold data, so as to balance the read and write performance of the distributed storage and improve the capacity utilization rate; Third, the method of delayed erasure code can be implemented in a distributed storage system composed of non-homogeneous nodes in a wide area network. The data is divided into blocks and grouped and stored according to non-homogeneous nodes. The data is encoded according to the data and node hot and cold judgment time strategy, maintaining normal read and write performance, and at the same time realizing data redundancy and recovery; IV. Improve the reliability, fault tolerance and performance of the storage system, and are applicable to scenarios such as distributed storage systems and object storage systems in large-scale wide-area weak network environments. Description of the Drawings
[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.
[0025] Figure 1 : Flowchart of the processing method of the hot and cold shard data erasure code algorithm in the present invention. Detailed Embodiments
[0026] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.
[0027] A method for processing a hot and cold shard data erasure code algorithm in a distributed storage system in a local area network or a wide area network disclosed by the present invention includes the following steps: (1) Data block partitioning: Divide the data to be stored into multiple data blocks of a fixed size, and group and store them according to the status and quantity of distributed nodes; each node is responsible for storing one or more groups of data blocks; (2) Parameter strategy for data hot and cold judgment: Through the metadata server of the distributed storage system, intelligent algorithm management is used to judge the hot and cold attributes of the data (including data access frequency $f$ (frequently accessed, updated frequently), time window $T$ (heat timeliness within a certain time), access mode $M$ (high and low latency, concurrency), data size $S$ (size of data shards), data age $A$ (time length since the data was generated); according to the hot and cold attributes of the metadata node data, access frequency, time window, data size, data age, the hot and cold data attribute values calculated by the data heat algorithm; the accessed data with hot and cold data attribute values lower than the set threshold is set as cold data, and the accessed data with hot and cold data attribute values higher than the set threshold is set as hot data; The metadata is important management data stored and managed by the data center-level metadata server in the distributed storage system. It is a navigation map for accessing the business data stored in the system and is also the basis for the consistency, security, and manageability of business data. The metadata records key information such as the storage location, type, size, access permissions, and hot and cold attributes of user business data, and is the "index" for accessing business data. Through the metadata, the system can quickly locate the required business data and improve data access efficiency.
[0028] The intelligent algorithm is implemented by the metadata node based on the improved LRU-k algorithm plus additional dimensions of time window and data age for decision-making; (3) Parameter strategy for node hot and cold judgment: Parameters such as the performance of the node, network bandwidth status, CPU occupancy, recent IO read and write quantity and latency, number of network data packet transmissions and receptions, number of file reads and writes and network traffic, and the region where the node is located are comprehensively considered to judge the hot and cold situation of the node; The hot and cold node attribute value of the node calculated through the node heat algorithm; Nodes with hot and cold node attribute values lower than the set threshold are set as cold data, and nodes with hot and cold node attribute values higher than the set threshold are set as hot data; (4) Writing data: Write the data block to the node allocated by the distributed storage system, record the hot and cold attributes of the data, and use the metadata to record the hot and cold attributes of the data when the system configuration policy takes effect; This decision refers to the policy setting item for whether the storage and allocation of the file / data should be migrated and erasure-coded on hot and cold nodes according to the hot and cold degree of the data; (5) Erasure code algorithm processing mechanism for different types of data: Different erasure code algorithms are used for processing hot data on hot nodes, hot data on cold nodes, and cold data on hot nodes; (6) Processing after erasure code algorithm processing; Stored on cold nodes, redundant data will be stored on the central computer room server of the wide area network distributed storage system, and migration and erasure code processing will be carried out according to the amount of cold data and the number of cold nodes to ensure data recoverability and redundancy reliability; (7) Processing of data hot and cold transformation: Encoding processing is carried out to ensure the read and write performance of the data; The encoding parameters are dynamically adjusted during encoding processing: For hot data, the encoding parameters of the erasure code, including data block size, redundancy degree, etc., can be dynamically adjusted according to actual needs to balance data reliability and performance.
[0029] Furthermore, the erasure code algorithm processing mechanism for different types of data in step (5) is specifically: (5.1) Hot data on hot nodes: Set a certain number of copy backups according to the heat. The nodes where the copies are located are allocated and decided by the heat of the data and the request origin. The copy data is not processed by the erasure code algorithm; (5.2)Hot data on cold nodes: The performance, bandwidth capacity, and storage capacity of cold nodes should be used as the core decision-making basis. Hot nodes are high-value nodes, and hot data will not be migrated. At the same time, hot data copies of other hot nodes can also be stored. Cold nodes are low-value nodes and do not have the basic ability to provide hot data read and write services. Therefore, hot data will be migrated to hot nodes piece by piece according to the quantity. (5.3)Cold data on hot nodes: The management server of the distributed storage system allocates cold data to cold nodes based on the size and number of shards of the cold data, as well as the number of cold nodes. After the cold data is written to the disk, erasure coding processing is carried out. Furthermore, the processing method of cold data in step (2) is as follows: (2.1)EC coding processing: Data redundancy and recovery are achieved by adding redundant data blocks. During the cold data coding process, the read and write operations of the system will be hindered to a certain extent. (2.2)Read operation: The data block read is an uncoded original data block: The data block is directly read from the storage medium and returned to the user. The data block read is an encoded redundant data block: First, decode the relevant encoded data blocks to reconstruct the original data block. The decoding operation can use the erasure code decoding algorithm to calculate and recover the original data block. (2.3)Write operation: The data block written is an original data block: The data is directly written into the storage medium, and erasure coding is performed as needed to generate corresponding redundant data blocks. The data block written is a redundant data block: According to the erasure code coding rules, the data block is written to the corresponding position in the storage medium.
[0030] (2.4)Read operation optimization: According to the cold and hot attributes of the data judged in step 2, decide whether to perform real-time decoding on the cold data, or delay the decoding operation until the data is accessed, so as to reduce the overhead of the read operation. (2.5)Write operation optimization: It is carried out asynchronously to reduce the requirement for real-time performance, thereby reducing the delay of the write operation. (2.6)Delete data grouping: Delete the data grouping of cold data and add a group of parity check blocks in place. The parity check blocks are obtained by performing exclusive OR operations on other data blocks and are used to achieve data redundancy and recovery. (2.7)Node failure or data corruption: The erasure code algorithm can be used for data recovery and repair. When the data on a certain node is lost or damaged, the lost or damaged data block can be calculated through the parity check blocks and other valid data blocks to recover the data.
[0031] Preferably, performance optimization is carried out according to actual requirements and the performance bottleneck of the storage system; through node management and data sharding management of the central management server, the data hot and cold policy, node hot and cold policy, and data replication factor are comprehensively adjusted; to improve the read and write performance and storage efficiency of the system.
[0032] The method for processing the erasure code algorithm of hot and cold sharded data in a distributed storage system in a local area network or a wide area network disclosed by the present invention has the following beneficial effects: First, it can reduce the processing time of data storage, and at the same time keep the normal read and write performance unaffected by the erasure code processing during the EC encoding process, and improve the rationality of the hot and cold data division in the distributed storage system; Second, the hot and cold data are stored and migrated in layers according to the hot and cold data levels respectively, and the node hot and cold management strategy is used to perform hierarchical redundancy protection and replica backup on the hot and cold data, so as to balance the read and write performance of the distributed storage and improve the capacity utilization rate; Third, the method of delayed erasure code can be implemented in a distributed storage system composed of non-homogeneous nodes in a wide area network. The data is divided into blocks and stored in groups according to non-homogeneous nodes. The data encoding process is carried out according to the time strategy of data and node hot and cold judgment, keeping the normal read and write performance, and at the same time realizing data redundancy and recovery; Fourth, it improves the reliability, fault tolerance and performance of the storage system, and is applicable to scenarios such as distributed storage systems and object storage systems in large-scale wide area weak network environments.
[0033] Of course, without departing from the spirit and essence of the present invention, those skilled in the art should be able to make various corresponding changes and deformations according to the present invention, but these corresponding changes and deformations should all fall within the protection scope of the appended claims of the present invention.
Claims
1. A method for processing cold and hot shard data erasure coding algorithms in a distributed storage system in a local area network or a wide area network, characterized in that: The following steps are involved: (1) Data block segmentation: The data to be stored is divided into multiple fixed-size data blocks, and stored in groups according to the status and number of distributed nodes; each node is responsible for storing one or more groups of data blocks; (2) Parameter strategy for judging data hotness and coldness: Intelligent algorithm management is performed through the metadata server of the distributed storage system to judge the hotness and coldness attributes of the data (including data access frequency $f (frequent access, frequent updates), time window $T (timeliness of heat within a certain period of time), access mode $M (high and low latency, concurrency), data size $S (size of data shards), and data age $A (length of time since data was generated); based on the hotness and coldness attributes of metadata node data, access frequency, time window, data size, and data age, the hotness and coldness data attribute values are calculated through the data heat algorithm; the accessed data whose hotness and coldness data attribute values are lower than the set threshold are set as cold data, and the accessed data whose hotness and coldness data attribute values are higher than the set threshold are set as hot data; (3) Parameter strategy for judging whether a node is hot or cold: Comprehensively judge the hot or cold status of a node based on the node's performance, network bandwidth status and CPU usage, recent IO read / write quantity and latency, number of network data packets sent and received, number of file read / writes and network traffic, and the region where the node is located. Calculate the hot and cold node attribute values of the node using the node heat algorithm. Nodes with hot and cold node attribute values lower than the set threshold are set as cold data, and nodes with hot and cold node attribute values higher than the set threshold are set as hot data. (4) Writing data: writing data blocks to nodes assigned by the distributed storage system, recording the hot and cold properties of the data, and using metadata to record the hot and cold properties of the data when the system configuration policy takes effect; (5) Erasure code algorithm processing mechanism for different types of data: Different erasure code algorithms are used to process hot data on hot nodes, hot data on cold nodes, and cold data on hot nodes; (6) Processing after the erasure code algorithm is used; Stored on cold nodes, redundant data will be stored on the central computer room server of the WAN distributed storage system, and migration and erasure coding will be performed based on the amount of cold data and the number of cold nodes. (7) Processing of data hot and cold conversion: Encoding processing; Dynamic adjustment of encoding parameters during encoding processing: For hot data, the encoding parameters of the erasure code can be dynamically adjusted according to actual needs, including data block size, redundancy, etc., to balance data reliability and performance.
2. According to claim 1, a method for processing cold and hot shard data erasure coding algorithm in a distributed storage system in a local area network or a wide area network, characterized in that: The specific processing mechanism of the erasure code algorithm for different types of data in step (5) is: (5.1) Hot data on hot nodes: a set number of replicas are backed up according to the popularity. The nodes where the replicas are located are allocated and decided based on the popularity of the data and the location where the request is initiated. The replica data is not processed by the erasure coding algorithm; (5.2) Hot data on cold nodes: The performance, bandwidth and storage capacity of cold nodes are the core decision-making basis. Hot data will be migrated to hot nodes piece by piece according to the quantity. (5.3) Cold data on hot nodes: The management server of the distributed storage system allocates cold data based on the size of the cold data, the number of shards, and the number of cold nodes, and migrates the cold data to the cold nodes. After the cold data is stored on the disk, the erasure code encoding process is carried out.
3. According to claim 2, a method for processing cold and hot shard data erasure coding algorithm in a distributed storage system in a local area network or a wide area network, characterized in that: The method for processing cold data in step (2): (2.1) EC encoding processing: data redundancy and recovery are achieved by adding redundant data blocks (2.2) Read operation: The data block read is the original data block without encoding: the data block is directly read from the storage medium and returned to the user; The read data block is an encoded redundant data block: firstly, a decoding operation is performed on the relevant encoded data block to reconstruct the original data block; the decoding operation may use an erasure code decoding algorithm to calculate and restore the original data block; (2.3) Write operation: The data blocks written are original data blocks: the data are directly written into the storage medium and erasure coded as needed to generate corresponding redundant data blocks; The data blocks written are redundant data blocks: according to the erasure code encoding rules, the data blocks are written to the corresponding positions in the storage medium.
4. According to claim 3, a method for processing cold and hot shard data erasure coding algorithm in a distributed storage system in a local area network or a wide area network, characterized in that: The cold data processing method further includes: (2.4) Read operation optimization: Based on the hot and cold properties of the data determined in step 2, decide whether to decode the cold data in real time, or delay the decoding operation until the data is accessed to reduce the overhead of the read operation; (2.5) Write operation optimization: Use asynchronous mode to reduce the real-time requirements and thus reduce the delay of write operations; (2.6) Deleting data groups: deleting data groups of cold data and adding a set of parity blocks in place; the parity blocks are obtained by performing an XOR operation on other data blocks; (2.7) Node failure or data corruption: Erasure coding algorithms can be used to recover and repair data. When data on a node is lost or corrupted, the parity blocks and other valid data blocks can be used to calculate and recover the lost or corrupted data blocks.
5. According to claim 4, a method for processing cold and hot shard data erasure coding algorithm in a distributed storage system in a local area network or a wide area network, characterized in that: Through the node management and data sharding management of the central management server, the data hot and cold strategies, node hot and cold strategies, and the number of data copies are comprehensively adjusted.
Citation Information
Patent Citations
An elastic multi-dimensional redundancy method in a distributed storage system
CN109783016A
Reliable data storage method based on erasure code adaptive coding
CN117435387A
Data redundancy method and device, program product and medium
CN118860295A
Cited By
Multi-terminal-oriented distributed audio and video adaptive production system
CN120301991A
Data processing method and device, medium and program product
CN120872264A
Erasure code strategy conversion method and device, equipment, medium and product
CN121116206A
An erasure coding strategy conversion method, apparatus, device, medium, and product
CN121116206B
Intelligent fire-fighting data management system based on Internet of Things
CN121524138A