Erasure correction storage pool data processing method and system based on distributed storage system

Abstract

The invention discloses an erasure storage pool data processing method and system based on a distributed storage system, and relates to the technical field of distributed storage data redundancy. According to the method, the main object storage device only undertakes the tasks of updating local data and synchronizing copies and does not need to undertake the whole task of'reading-modifying-encoding-writing ', so that the burden of the main object storage device is reduced, the main object storage device does not become a system bottleneck any more in a high-concurrency scene, and the user experience is improved. And the performance of the storage system is relatively improved. Moreover, the odd-even object storage device locally holds a complete data block copy, so that after receiving the target data block sent by the main object storage device, the odd-even object storage device can locally obtain the original data block corresponding to the target data block, and then can perform autonomous coding; the method does not need to depend on main object storage equipment or strip data of a full placement group, and avoids causing a large amount of redundant read-write and delay increase.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed storage data redundancy, and in particular to a method and system for processing erasure storage pool data based on a distributed storage system. Background Art

[0002] In distributed storage systems, erasure coding (EC) is widely used in large-scale data storage scenarios due to its higher storage efficiency than the replication mode.

[0003] The EC storage pool uses a centralized encoding model with a primary Object Storage Device (OSD). In this model, client write requests are received by the primary OSD, which is responsible for data sharding, encoding to generate parity blocks, and syncing them to the secondary OSDs. Overwrite operations require reading the entire PG stripe, executing the full "read-modify-encode-write" process. Because the primary OSD bears the heavy responsibility of encoding and synchronization, it becomes a system bottleneck in high-concurrency scenarios. Furthermore, overwrite operations require accessing all placement group (PG) stripe data, resulting in a large number of redundant reads and writes, significantly increasing latency.

[0004] It can be seen that how to improve the performance of the system in high-concurrency distributed storage scenarios, reduce input / output amplification, and reduce latency are technical problems that people in this field urgently need to solve. Summary of the Invention

[0005] The purpose of the present invention is to provide an erasure storage pool data processing method and system based on a distributed storage system to solve the problem that the master OSD bears the heavy responsibility of encoding and synchronization, and becomes a system bottleneck in high concurrency scenarios; and the overwrite operation requires access to the stripe data of all placement groups, causing a large amount of redundant reading and writing, and significantly increasing the delay.

[0006] To solve the above technical problems, the present invention provides a data processing method for an erasure storage pool based on a distributed storage system, which is applied to a parity object storage device in the erasure storage pool. Any data block in the erasure storage pool is stored in both a primary object storage device and a parity object storage device. The method comprises: Receiving a target data block sent by a primary object storage device; wherein the target data block is obtained by the primary object storage device updating the locally stored preset data block according to a write request sent by a client for representing an update of the preset data block; Obtaining a difference between the target data block and an original data block corresponding to the target data block stored locally; obtaining a coding coefficient determined based on a failure domain configuration of a placement group, and determining a parity block increment based on the coding coefficient and the difference; The locally stored parity block is incrementally updated using the parity block.

[0007] On the one hand, before the primary object storage device updates the locally stored preset data block to the target data block, the method further includes: The primary object storage device records a log and writes it into the persistent memory space in multiple times through the preset byte atomic write feature of the persistent memory; wherein the log includes at least the unique code of the preset data block, the version number prefix, the old value hash and the new value pointer.

[0008] On the other hand, the primary object storage device synchronizes the target data block to the parity object storage device, including: Locating multiple parity object storage devices of the placement group stripe to which the target data block belongs based on metadata of the placement group; Obtaining load values ​​of the plurality of parity object storage devices; The target data blocks are synchronized to the parity object storage devices in sequence according to the load value from low to high.

[0009] On the other hand, the object storage device synchronizes the target data block to the parity object storage device, including: Get the access frequency of the target data block; In a case where the target data block is determined to be a hot data block according to the access frequency, synchronizing the target data block to a parity object storage device, and waiting for confirmation information returned by the parity object storage device; When the target data block is determined to be a cold data block according to the access frequency, multiple operations on the cold data block are accumulated, and the target data block obtained by merging the multiple operations is synchronized to the parity object storage device.

[0010] On the other hand, after the primary object storage device synchronizes the target data block to the parity object storage device, the following steps are also performed: After synchronizing the target data block to the parity object storage device, a first confirmation message is received from the parity object storage device within a first preset time period; wherein the first confirmation message includes at least information indicating that the parity object storage device has successfully received the target data block and information indicating that the parity object storage device has stored the target data block in a cache; receiving second confirmation information sent by the parity object storage device; wherein the second confirmation information at least includes a log record indicating that the parity object storage device has written the target data block into the local persistent memory space; Information indicating that the update is successful is returned to the client, and the data state of the preset data block is marked as a state to be encoded.

[0011] On the other hand, before the primary object storage device updates the locally stored preset data block to the target data block, the method further includes: Maintain a version vector for each placement group and record the latest version number of each data block; When concurrent write requests sent by the client for representing updates to preset data blocks are detected, version numbers of the data blocks corresponding to all write requests are obtained; the highest version number is determined based on all version numbers; and the write request corresponding to the highest version number is responded to.

[0012] On the other hand, before obtaining the difference between the target data block and the locally stored original data block corresponding to the target data block, the method further includes: Reading the local log to extract the unique code of the target data block, the old value hash, the target data block, the old version number, and the new version number; Locating the corresponding original data block using the unique code of the target data block and obtaining a positioning result; When it is detected that the value of the new version number is greater than the value of the old version number, and / or when it is detected that the positioning result indicates that the original data block exists locally, entering the step of obtaining a difference between the target data block and the original data block corresponding to the target data block stored locally; When it is detected that the positioning result is that the original data block does not exist locally, a request for representing hash verification is sent to the primary object storage device; if the hash received from the primary object storage device and used to represent the original data block matches the old value hash in the log, the original data block sent by the primary object storage device is obtained, and the step of obtaining the difference between the target data block and the original data block corresponding to the target data block stored locally is entered; if there is no match, a prompt message representing the rejection code is output.

[0013] On the other hand, determining the encoding coefficient based on the failure domain configuration of the placement group includes: Obtaining a relationship between itself and a fault domain of a data block object storage device; wherein the fault domain is a cabinet or a node; If it is detected that the relationship is the same fault domain, the same fault domain coefficient is set to 1; If it is detected that the relationship is across fault domains, setting the cross-fault domain coefficients to coprime coefficients; Generate a coding matrix according to the value of the same-fault domain coefficient and the value of the cross-fault domain coefficient; A coding coefficient is determined according to the elements in the coding matrix.

[0014] On the other hand, determining the coprime coefficients includes: Get the fault domain distance value and the pre-created prime number library; Prime numbers that are positively correlated with the fault domain distance value are selected from the prime number library to obtain coprime coefficients.

[0015] On the other hand, after generating the coding matrix according to the value of the same-fault domain coefficient and the value of the cross-fault domain coefficient, the method further includes: Use metadata servers to monitor the fault domain distribution of object storage devices within a placement group in real time; In the case where a change in the fault domain topology is detected, the process returns to the step of obtaining the relationship between the fault domain of the data block object storage device and the fault domain of the data block object storage device.

[0016] On the other hand, before using the parity block to incrementally update the locally stored parity block, the method further includes: Obtaining the parity block increment, the original data block, and the target data block, performing an XOR calculation on them, and obtaining an XOR calculation result; In the case where it is detected that the XOR calculation result is equal to an element in the encoding matrix, entering the step of incrementally updating the locally stored parity block using the parity block; In the case where it is detected that the XOR calculation result is not equal to the element in the encoding matrix, the method returns to the step of receiving the target data block sent by the primary object storage device.

[0017] On the other hand, incrementally updating the locally stored parity block using the parity block includes: Write the updated parity block into the new storage address; After a successful write is detected, the metadata pointer is updated to point to the new address through an atomic write operation in the persistent memory space.

[0018] On the other hand, after the locally stored parity block is updated with the parity block increment, the method further includes: Updating the version number of the target data block to obtain a new version number; Generate an encoding completion log; wherein the encoding completion log includes at least the unique code of the target data block, the hash value of the new parity block obtained after the update, and the new version number.

[0019] On the other hand, the storage area of ​​the persistent memory space includes an area for storing logs of unfinished updates and an area for storing logs of completed encoding; The unfinished logs stored in the area for storing unfinished update logs are retained until encoding is completed; and the encoding completion logs stored in the area for storing encoding completion logs are retained for a second preset time period.

[0020] On the other hand, after the locally stored parity block is incrementally updated using the parity block, the method further includes: After detecting that the encoding is completed, deleting the original data blocks and logs stored locally; Release the local storage space occupied by the original data blocks and logs.

[0021] On the other hand, after the locally stored parity block is incrementally updated using the parity block, the method further includes: Sending information indicating that encoding is complete to the primary object storage device; wherein the information indicating that encoding is complete includes a hash value of the new parity block; so that the primary object storage device changes the data state from the pending encoding state to the completed state, and synchronizes the data state to the metadata server.

[0022] In order to solve the above technical problems, the present invention further provides an erasure storage pool data processing method based on a distributed storage system, which is applied to a primary object storage device in the erasure storage pool, wherein any data block in the erasure storage pool is stored in the primary object storage device and the parity object storage device; the method comprises: Obtain a write request sent by a client to indicate an update of a preset data block; updating the preset data block stored locally according to the write request to obtain a target data block; Synchronizing the target data block to a parity object storage device; allowing the parity object storage device to obtain a difference between the target data block and an original data block corresponding to the target data block stored locally; obtaining a coding coefficient determined based on a failure domain configuration of a placement group, and determining a parity block increment based on the coding coefficient and the difference; and updating the locally stored parity block using the parity block increment.

[0023] In order to solve the above technical problems, the present invention further provides a distributed storage system, comprising a primary object storage device, a parity object storage device, and a metadata server; wherein the primary object storage device is used to obtain a write request sent by a client for indicating an update of a preset data block; update the locally stored preset data block according to the write request to obtain a target data block; and synchronize the target data block to the parity object storage device; The metadata server is configured to determine a coding coefficient based on a failure domain configuration of a placement group; The parity object storage device is used to receive a target data block sent by a primary object storage device; obtain a difference between the target data block and an original data block corresponding to the target data block stored locally; obtain a coding coefficient determined based on a failure domain configuration of a placement group, and determine a parity block increment based on the coding coefficient and the difference; and update the locally stored parity block using the parity block increment.

[0024] In order to solve the above technical problems, the present invention further provides a data processing device, comprising: memory for storing computer programs; A processor is configured to implement the steps of the above-mentioned method for processing erasure storage pool data based on a distributed storage system when executing the computer program.

[0025] In order to solve the above technical problems, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned erasure storage pool data processing method based on a distributed storage system are implemented.

[0026] The beneficial effect of the present invention lies in that, in this method, the primary object storage device updates the locally stored preset data block based on a write request sent by a client representing an update to the preset data block, obtaining a target data block, and then sending the target data block to the parity object storage device. That is, in the method provided by the present invention, the primary object storage device is responsible for updating local data and synchronizing replicas. Because any data block in the erasure storage pool is stored on both the primary object storage device and the parity object storage device, i.e., the parity object storage device locally holds a complete copy of the data block, after receiving the target data block sent by the primary object storage device, the parity object storage device can locally obtain the original data block corresponding to the target data block and then perform autonomous encoding. During autonomous encoding, the parity object storage device obtains the difference between the target data block and the original data block corresponding to the locally stored target data block; obtains a coding coefficient determined based on the fault domain configuration of the placement group, determines a parity block increment based on the coding coefficient and the difference, and finally uses the parity block increment to update the locally stored parity block. Because in this method, the parity object storage device uses incremental encoding to update the locally stored parity blocks during autonomous encoding, there is no need to rely on the primary object storage device or the stripe data of the full placement group, avoiding the occurrence of a large number of redundant reads and writes and increased latency. In this method, the parity object storage device takes on the function of autonomous encoding and updating parity blocks, realizing the decoupling of encoding and updating. Moreover, the primary object storage device only takes on the task of updating local data and synchronizing replicas, instead of the previous primary object storage device having to perform the entire "read-modify-encode-write" operation. This reduces the burden on the primary object storage device, so that the primary object storage device no longer becomes a system bottleneck in high-concurrency scenarios, and relatively improves the performance of the storage system.

[0027] Furthermore, before the primary object storage device updates its own stored preset data block to the target data block, it logs the data. This introduces a "hashed old value" in the log, replacing the full old value storage. This reduces persistent memory usage (down to 1 / 8th to 1 / 16th of the original full old value storage). Furthermore, hash verification ensures the consistency of old values, eliminating the space redundancy caused by traditional log storage of complete old values.

[0028] When synchronizing target data blocks to parity object storage devices, the primary object storage device obtains the load value of the parity object storage device and dynamically selects the synchronization order based on the load balancing status (giving priority to sending to the low-load parity object storage device) to prevent a single parity object storage device from becoming a bottleneck.

[0029] When the primary object storage device synchronizes the target data blocks to the parity object storage device, it uses the "synchronous + asynchronous confirmation" mode (sending data first and then waiting for confirmation) for hot data blocks (highly accessed), and the "batch synchronization" mode (accumulating multiple updates and then merging and sending) for cold data blocks, which can reduce the number of network interactions.

[0030] After the primary object storage device synchronizes the target data block to the parity object storage device, it receives the first confirmation information and the second confirmation information sent by the parity object storage device. Through the "double confirmation" mechanism, it balances performance and reliability and avoids the consistency risk caused by cache loss in the traditional single confirmation mode.

[0031] Before the primary object storage device updates the preset data block stored in itself to the target data block, only when concurrent update requests are detected will only the request with the highest version number be allowed to enter the data update and synchronization, and the remaining requests will be delayed and retried to ensure consistency.

[0032] Before performing autonomous encoding, the parity object storage device introduces hash verification instead of transmitting the complete old value, reducing cross-node data interaction (only transmitting 32-byte hash values ​​instead of complete data blocks) and preventing encoding errors caused by inconsistent data between the primary object storage device and the parity object storage device.

[0033] A coding matrix is ​​dynamically generated based on the placement group's fault domains (such as cabinets and nodes). Setting the same-fault-domain coefficient to 1 simplifies calculations. Setting the cross-fault-domain coefficient to mutually prime numbers ensures linear independence and improves fault recovery efficiency. When selecting prime numbers, larger prime numbers are recommended for greater distances, improving cross-domain fault tolerance.

[0034] The metadata server monitors the fault domain distribution of object storage devices within the placement group in real time. When a change in the fault domain topology is detected (such as the addition of a new cabinet), the coefficients are automatically recalculated and synchronized to the relevant object storage devices to ensure that the matrix adapts to the current cluster status.

[0035] Before using the parity block increment to update the locally stored parity block, an XOR check is performed on the calculated parity block increment to ensure the accuracy of the increment calculation.

[0036] A copy-on-write mechanism is used to write the updated parity block to a new storage address. After detecting a successful write, an atomic write operation in the persistent memory space is used to update the metadata pointer to the new address, thus avoiding parity block damage caused by power outages during the update process.

[0037] In addition, the present invention also provides an erasure storage pool data processing method based on a distributed storage system, a distributed storage system, a data processing device, and a computer-readable storage medium applied to a primary object storage device in an erasure storage pool. These methods have the same or corresponding technical features as the above-mentioned erasure storage pool data processing method based on a distributed storage system, applied to a parity object storage device in an erasure storage pool, and have the same effects as above. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0039] Figure 1 An overall architecture diagram of a distributed storage system provided by an embodiment of the present invention; Figure 2 A flowchart of a method for processing data in an erasure storage pool based on a distributed storage system and applied to a parity object storage device in an erasure storage pool provided by an embodiment of the present invention; Figure 3 A schematic diagram of the entire process of a method for processing data in an erasure storage pool based on a distributed storage system provided by an embodiment of the present invention; Figure 4 A structural diagram of a data processing device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0041] The core of the present invention is to provide an erasure storage pool data processing method and system based on a distributed storage system to solve the problem that the master OSD bears the heavy responsibility of encoding and synchronization, and becomes a system bottleneck in high-concurrency scenarios; and the overwrite operation requires access to the stripe data of all placement groups, causing a large amount of redundant reading and writing, and significantly increasing the delay.

[0042] In order to enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific implementation methods. Figure 1 The overall architecture diagram of a distributed storage system provided by an embodiment of the present invention is as follows: Figure 1 As shown, it includes distributed storage clients, distributed storage systems, and a unified, self-controlled, and scalable distributed storage consistency management system. In the distributed storage client, the client layer provides three interfaces: block, object, and file. Correspondingly, objects correspond to object network management services, blocks correspond to block device services, and files correspond to file system services. The object gateway service is connected to the storage pool, and the storage pool is connected to the file system service through the metadata server cluster and the monitoring service cluster. The storage pool includes the storage pool To storage pool , using a controllable, scalable, distributed replica data placement algorithm mapped to groups 1 to n, each group is mapped to a node To Node Each node corresponds to multiple OSDs. Specifically, the distributed storage system adopts a decentralized and distributed architecture. The core components include clients, control planes (Monitors), metadata servers (MDSs), and object storage devices (OSDs). These modules work together through high-speed networks (such as Ethernet and InfiniBand) to achieve data storage and management. The architecture layer is described as follows: 1. Client layer: The client layer is the entry point for users to interact with distributed storage, providing three interfaces: block, object, and file. The core components include: librados library: encapsulates the underlying Application Programming Interface (API) and is responsible for interacting with the cluster.

[0043] CRUSH Client Module: Locally caches the CRUSH map and calculates data storage locations. Input: object name and storage pool rules (such as number of replicas and fault domain hierarchy). Processing steps: Hash the object name to generate a random number. Traverse the CRUSH bucket hierarchy (e.g., host -> cabinet -> data center) according to the storage pool rules and select OSDs based on weights. This ensures balanced data distribution and meets redundancy policies (e.g., deploying replicas across cabinets).

[0044] 2. Control plane (MoKitor / MDS): MoKitor: Maintains cluster metadata (OSD Map, CRUSH Map, authentication information) and ensures multi-node consistency through the Paxos algorithm.

[0045] Interaction with the client: The client pulls the latest cluster map when it first connects, and then updates it through heartbeat synchronization.

[0046] MDS (File Scenario): Manages file system metadata (directory tree, permissions), and caches hot metadata to accelerate access.

[0047] 3. Data plane (OSD cluster): OSD nodes: Hardware components include CPU, memory, disk (HDD / SSD), and network interface (Gigabit / 10G network card). They are responsible for data storage, replication, erasure coding, and consistency maintenance, and perceive cluster topology through OSDMap.

[0048] The present invention provides a data processing method (also known as a write optimization method) for an erasure storage pool based on a distributed storage system. This method is applied to parity object storage devices (i.e., Parity OSDs) in an erasure storage pool. Parity OSDs are core nodes in the distributed storage system responsible for storing parity blocks and performing autonomous encoding. They correspond to primary OSDs (i.e., Primary OSDs) that store original data blocks. The core function of parity OSDs is to ensure data reliability through redundant parity blocks. Traditional distributed storage systems only store parity blocks and do not have data block replicas. In the present invention, ensuring that the parity OSDs locally hold complete copies of data blocks is a prerequisite for enabling autonomous encoding by the parity OSDs. That is, in the present invention, any data block in the erasure storage pool is stored in both the primary OSD and the parity OSD. Figure 2 A flowchart of a distributed storage system-based erasure storage pool data processing method for a parity object storage device in an erasure storage pool provided by an embodiment of the present invention is shown in FIG. Figure 2 As shown, the method includes: S10: receiving a target data block sent by the primary object storage device; wherein the target data block is obtained by the primary object storage device updating a locally stored preset data block according to a write request sent by the client for representing an update of the preset data block; S11: Obtaining the difference between the target data block and the original data block corresponding to the locally stored target data block; S12: Obtain a coding coefficient determined based on the failure domain configuration of the placement group, and determine a parity block increment based on the coding coefficient and the difference; S13: Update the locally stored parity block using the parity block increment.

[0049] It should be noted that the client sends a write request to the primary object storage device to update a preset data block. The preset data block is not limited and is determined based on actual circumstances. The preset data block in the write request is the same as the original data block corresponding to the target data block in the parity object storage device.

[0050] To record the value before the update (i.e., the old value), the primary object storage device also includes the following before updating the locally stored preset data block to the target data block: The primary object storage device records the log and writes it to the persistent memory space in multiple times through the preset byte atomic write feature of the persistent memory; wherein the log includes at least the unique code of the preset data block, the version number prefix, the old value hash and the new value pointer.

[0051] For example, the preset byte size is 8 bytes. Before the primary object storage device updates the preset data block it stores with the target data block, it logs the data. This reduces persistent memory usage by using a "hashed old value" instead of storing the entire old value in the log (reducing it to 1 / 8 to 1 / 16 of the original full old value storage). Furthermore, hash verification ensures the consistency of old values, resolving the space redundancy issue caused by traditional log storage of complete old values.

[0052] After recording the old value information, the primary object storage device updates the locally stored preset data block to obtain the target data block. The target data block is then synchronized to the parity object storage device. To prevent a single parity object storage device from becoming a bottleneck, the primary object storage device synchronizes the target data block to the parity object storage device, including: Locating multiple parity object storage devices of the placement group stripe to which the target data block belongs based on metadata of the placement group; Get the load values ​​of multiple parity object storage devices; The target data blocks are synchronized to each parity object storage device in order from low to high load value.

[0053] That is, when the primary object storage device synchronizes the target data block to the parity object storage device, it obtains the load value of the parity object storage device and dynamically selects the synchronization order based on the load balancing status (giving priority to sending to the low-load parity object storage device) to prevent a single parity object storage device from becoming a bottleneck.

[0054] In practice, some data blocks are accessed less frequently, while others are accessed more frequently. To reduce the number of network interactions, the primary object storage device synchronizes the target data blocks to the parity object storage device in the following steps: Get the access frequency of the target data block; When the target data block is determined to be a hot data block according to the access frequency, the target data block is synchronized to the parity object storage device, and confirmation information returned by the parity object storage device is waited for; When the target data block is determined to be a cold data block according to the access frequency, multiple operations on the cold data block are accumulated, and the target data block obtained by merging the multiple operations is synchronized to the parity object storage device.

[0055] By adopting the "synchronous + asynchronous confirmation" mode (sending data first and then waiting for confirmation) for hot data blocks (with high access frequency) and the "batch synchronization" mode (accumulating multiple updates and then merging and sending) for cold data blocks, the number of network interactions is reduced.

[0056] After the primary object storage device synchronizes the target data block to the parity object storage device, for the reliability of data transmission, the primary object storage device further includes: After synchronization of the target data block to the parity object storage device begins, a first confirmation message is received from the parity object storage device within a first preset time period; wherein the first confirmation message includes at least information indicating that the parity object storage device has successfully received the target data block and information indicating that the parity object storage device has stored the target data block in a cache; receiving second confirmation information sent by the parity object storage device; wherein the second confirmation information at least includes a log record indicating that the parity object storage device has written the target data block into the local persistent memory space; Returns information indicating that the update was successful to the client, and marks the data state of the preset data block as a state to be encoded.

[0057] In this method, after the primary object storage device synchronizes the target data block to the parity object storage device, it receives the first confirmation information and the second confirmation information sent by the parity object storage device. Through the "double confirmation" mechanism, it balances performance and reliability and avoids the consistency risk caused by cache loss in the traditional single confirmation mode.

[0058] To maintain consistency, before the primary object storage device updates the locally stored preset data block to the target data block, it also includes: Maintain a version vector for each placement group and record the latest version number of each data block; When concurrent write requests sent by the client for representing updates to preset data blocks are detected, version numbers of the data blocks corresponding to all write requests are obtained; the highest version number is determined based on all version numbers; and the write request corresponding to the highest version number is responded to.

[0059] In this method, when the primary object storage device detects concurrent update requests, it only allows the request with the highest version number to enter the data update and synchronization, and delays the retry of the remaining requests to ensure consistency.

[0060] The above process describes the processing process of the primary object storage device in the erasure storage pool data processing method based on the distributed storage system, that is, the primary object storage device is responsible for updating local data and synchronizing replicas.

[0061] After receiving the target data block from the primary object storage device, the parity object storage device ensures that both the original data block and the target data block exist locally so that autonomous encoding can be performed. In implementation, before obtaining the difference between the target data block and the original data block corresponding to the locally stored target data block, the parity object storage device further includes: Read the local log to extract the unique code of the target data block, the old value hash, the target data block, the old version number, and the new version number; Locate the corresponding original data block through the unique code of the target data block and obtain the positioning result; When it is detected that the value of the new version number is greater than the value of the old version number, and / or when it is detected that the positioning result indicates that the original data block exists locally, entering the step of obtaining a difference between the target data block and the original data block corresponding to the locally stored target data block; When it is detected that the positioning result is that the original data block does not exist locally, a request for representing the hash verification is sent to the primary object storage device; if the hash representing the original data block received from the primary object storage device matches the old value hash in the log, the original data block sent by the primary object storage device is obtained, and the step of obtaining the difference between the target data block and the original data block corresponding to the target data block stored locally is entered; if there is no match, a prompt message representing the rejection code is output.

[0062] In this method, before performing autonomous encoding, the parity object storage device introduces hash verification instead of transmitting the complete old value, reducing cross-node data interaction (only transmitting 32-byte hash values ​​instead of complete data blocks) and preventing encoding errors caused by data inconsistency between the primary object storage device and the parity object storage device.

[0063] When performing autonomous encoding, the parity object storage device obtains the difference between the target data block and the original data block corresponding to the locally stored target data block; obtains the encoding coefficient determined based on the failure domain configuration of the placement group, and determines the parity block increment based on the encoding coefficient and the difference.

[0064] Specifically, determining the coding coefficient based on the failure domain configuration of the placement group includes: Get the relationship between itself and the fault domain of the block object storage device; where the fault domain is a cabinet or node; If the relationship is detected as the same fault domain, the same fault domain coefficient is set to 1; If the relationship is detected to be across fault domains, the cross-fault domain coefficients are set to be coprime coefficients; Generate a coding matrix according to the value of the same-fault domain coefficient and the value of the cross-fault domain coefficient; The coding coefficients are determined according to the elements in the coding matrix.

[0065] To improve the fault tolerance of cross-domain failures, the coprime coefficients are determined as follows: Get the fault domain distance value and the pre-created prime number library; Prime numbers that are positively correlated with the fault domain distance value are selected from the prime number library to obtain coprime coefficients.

[0066] This method dynamically generates a coding matrix based on the placement group's fault domains (such as cabinets and nodes). Setting the same-fault-domain coefficient to 1 simplifies calculations. Setting the cross-fault-domain coefficient to mutually prime numbers ensures linear independence and improves fault recovery efficiency. When selecting prime numbers, larger prime numbers are chosen for greater distances, improving cross-domain fault tolerance.

[0067] In practice, the fault domain topology may change (e.g., adding a new cabinet). In order to adapt the obtained coding coefficient to the current cluster state, after generating the coding matrix based on the values ​​of the same fault domain coefficient and the values ​​of the cross-fault domain coefficient, the following steps are also included: Use metadata servers to monitor the fault domain distribution of object storage devices within a placement group in real time; When a change in the fault domain topology is detected, the process returns to the step of obtaining the relationship between the process itself and the fault domain of the data block object storage device.

[0068] In this method, the metadata server monitors the fault domain distribution of object storage devices within the placement group in real time. When a change in the fault domain topology (such as the addition of a new cabinet) is detected, the coefficients are automatically recalculated and synchronized to the relevant object storage devices to ensure that the matrix adapts to the current cluster status.

[0069] Determine the parity block increment based on the coding coefficients and differences The expression is: ; in, Represents the original data block, Indicates the target data block (that is, the data block obtained after the primary object storage device updates). Indicates the The parity block corresponds to Data block coding coefficients, Indicates The remainder of the modulus.

[0070] After obtaining the parity increment, in order to ensure the accuracy of the increment calculation, in implementation, before using the parity block increment to update the locally stored parity block, the following steps are further included: Obtain the parity block increment, the original data block and the target data block, perform XOR calculation on them, and obtain the XOR calculation result; When it is detected that the XOR calculation result is equal to the element in the encoding matrix, entering the step of using the parity block increment to update the locally stored parity block; In the case where it is detected that the XOR calculation result is not equal to the element in the encoding matrix, the process returns to the step of receiving the target data block sent by the primary object storage device.

[0071] Before using the parity block increment to update the locally stored parity block, an XOR check is performed on the calculated parity block increment to ensure the accuracy of the increment calculation.

[0072] If the verification passes, the locally stored parity block is updated using the parity block increment. To avoid parity block damage caused by power outage during the update process, in practice, the locally stored parity block is updated using the parity block increment including: Write the updated parity block into the new storage address; After a successful write is detected, the metadata pointer is updated to point to the new address through an atomic write operation in the persistent memory space.

[0073] That is, a copy-on-write mechanism is used to write the updated parity block to a new storage address. After detecting a successful write, an atomic write operation in the persistent memory space is used to update the metadata pointer to the new address, thus avoiding damage to the parity block caused by power outages during the update process.

[0074] In order to update the log in time, after using the parity block to incrementally update the locally stored parity block, it also includes: Update the version number of the target data block to obtain a new version number; Generate an encoding completion log; wherein the encoding completion log includes at least the unique code of the target data block, the hash value of the new parity block obtained after the update, and the new version number.

[0075] The storage area of ​​the persistent memory space includes an area for storing unfinished update logs and an area for storing completed encoding logs. The unfinished logs stored in the area for storing unfinished update logs are retained until encoding is completed. The completed encoding logs stored in the area for storing completed encoding logs are retained for a second preset duration. The second preset duration is not limited and is determined based on actual circumstances.

[0076] To reduce the storage space occupied by data blocks and logs, after incrementally updating the locally stored parity blocks with the parity blocks, the following steps are also performed: After detecting that the encoding is complete, delete the original data blocks and logs stored locally; Release the local storage space occupied by the original data blocks and logs.

[0077] Furthermore, in order to enable the primary object storage device to record a correct data state, in implementation, after incrementally updating the locally stored parity block using the parity block, the following steps are further included: Send information indicating that encoding is complete to the primary object storage device; wherein the information indicating that encoding is complete includes a hash value of the new parity block; so that the primary object storage device changes the data state from the pending encoding state to the completed state and synchronizes the data state to the metadata server.

[0078] Based on the above-described method for processing erasure storage pool data in a distributed storage system, the present invention adds an "erasure write optimization module" to the existing IO process of the distributed storage system. The optimization module is specifically embedded in the following locations in the distributed storage system data IO: Master OSD side: Located between the data receiving module and the synchronization module, responsible for data update triggering and log management; Parity OSD side: Located between the data receiving module and the storage module, it is responsible for data copy storage, incremental encoding, and parity block updates; Metadata layer: interacts with MDS (metadata service) to obtain PG stripe distribution information and fault domain configuration (reference).

[0079] Comparison of distributed storage system data IO process before and after optimization: Traditional process: client → master OSD (sharding + encoding) → slave OSD (storage); Optimization process: Client → Primary OSD (Optimization module: Update data + Synchronize replicas) → Parity OSD (Optimization module: Autonomous encoding + Update parity blocks).

[0080] The primary OSD is the core node responsible for data block management and update coordination in the erasure code (EC) storage pool. It is directly associated with the PG (Placement Group) in the distributed storage system architecture. As the coordinator of data operations within the PG, it receives client write requests and manages the initial storage and replica synchronization of data blocks. In traditional EC storage pools, the primary OSD is responsible for data sharding, full PG stripe encoding (generating parity blocks), and cross-node synchronization, and is the main source of performance bottlenecks.

[0081] In the present invention, the responsibilities of the primary OSD are optimized to "data update trigger + replica synchronization". It no longer participates in the encoding process, but is only responsible for synchronizing the updated data blocks to the parity OSD, and recording the update status (block ID + version number) through the PMem log to ensure the traceability of the update process.

[0082] Parity OSD refers to a node that stores parity blocks and performs encoding operations autonomously, and is deeply bound to the PG's fault domain design: it stores parity blocks and data block copies within the corresponding PG (according to the "1 master M slaves" placement strategy of the present invention, each data block has 1 copy in each of the M parity OSDs); in a traditional EC storage pool, the parity OSD only passively receives the parity blocks generated by the master OSD and has no autonomous encoding capability; in the present invention, the parity OSD is given autonomous encoding capability: based on the locally stored data block copies, the parity block increments are calculated using the incremental encoding algorithm proposed below by the present invention, without relying on the master OSD or the full PG stripe data, thereby achieving decoupling of encoding and updating.

[0083] In order to enable those skilled in the art to better understand the above-mentioned erasure storage pool data processing method based on a distributed storage system, the following further describes it with reference to the accompanying drawings and specific embodiments. Figure 3 The schematic diagram of the entire process of a method for processing erasure storage pool data based on a distributed storage system provided by an embodiment of the present invention. Figure 3 As shown, a client sends a data write I / O request to the primary object storage device. The primary object storage device records the persistent memory tiering log, updates the local data blocks, and synchronizes the data to the parity object storage device. The primary object storage device then sends a write request to the parity object storage device, which then parses the log, performs hash verification, synchronizes the data blocks, performs incremental encoding, and updates the copy-on-write mechanism. After the parity object storage device completes the parity block update, the primary object storage device receives an encoding completion notification from the parity object storage device.

[0084] Specifically, the steps include: The first step is data placement strategy (for a (K,M) EC storage pool, that is, a storage pool containing K data blocks and M parity OSDs): 1. Data block distribution: K data blocks Storage is done in a "1 master, M slaves" mode: Each data block is stored in one copy on the primary OSD and one copy on each of the M parity OSDs (forming K groups of M+1 copies), ensuring that the parity OSD has a local copy of the complete data block.

[0085] 2. Parity block distribution: M parity blocks Stored in M ​​parity OSDs respectively, each parity OSD only stores the parity blocks it is responsible for And the corresponding data block (i.e. data block copies), without relying on the primary OSD to obtain full PG stripe data.

[0086] Step 2: Optimize the two-phase write method based on the erasure data pool of the distributed storage system: Phase 1: Data update (synchronous replica mode, layered log and multi-dimensional confirmation mechanism), the client sends a write request to the primary OSD (update for ), the master OSD optimizes the module through the "data update module": 1. Recording PMem (persistent memory) logs (including a 64-bit block ID, a 32-bit version number prefix, a 64-bit old value hash, and a 64-bit new value pointer), totaling 224 bits (28 bytes), is atomically written in three passes (the first two 8-byte passes, followed by a final 4-byte fill) using PMem's 8-byte atomic write feature to ensure log integrity. The introduction of an "old value hash" replaces the storage of complete old values, reducing PMem space usage (reducing it to 1 / 8 to 1 / 16 of the original full storage of old values). Hash verification also ensures old value consistency (resolving the space redundancy issue caused by traditional log storage of complete old values). Due to PMem's atomic write feature, logs require only 16 bytes (block ID + version number), improving performance by over 80% compared to traditional logs, providing a lightweight log.

[0087] 2. Data update and intelligent synchronization. Update local data blocks to , and synchronize to the corresponding replicas of M parity OSDs, locate the M parity OSDs of the PG stripe to which the data block belongs based on the PG metadata (to ensure distribution across fault domains), and give priority to OSDs with loads below the threshold (such as 70%) when synchronizing replicas to avoid single point overload. for ) and locates the M parity OSDs of the PG stripe to which the data block belongs based on the PG (Placement Group) metadata (obtaining the current PG's OSD mapping table through the MDS). Combined with the PG's load balancing status (obtaining the OSD load value through the Monitor), the synchronization order is dynamically selected (prioritizing sending to the parity OSD with the lowest load) to prevent a single parity OSD from becoming a bottleneck.

[0088] In addition, during synchronization, the synchronization strategy is: for hot data blocks (with high access frequency), the "synchronous + asynchronous confirmation" mode is adopted (data is sent first, then waits for confirmation), and for cold data blocks, the "batch synchronization" mode is adopted (accumulated N updates and then merged and sent), reducing the number of network interactions.

[0089] 3. Confirmation and response mechanism. The master OSD needs to receive two types of confirmations: Basic confirmation: parity OSD successfully received And write it into the temporary cache (must return within 10ms, otherwise retransmission is triggered); Log confirmation: Parity OSD will The log records are written to the local PMem (ensuring that they are not lost during power failure and return within 20ms).

[0090] Only after all parity OSDs return two types of confirmation, the primary OSD returns "update successful" to the client (without waiting for encoding to complete) and marks the data block status as "pending encoding".

[0091] The "double confirmation" mechanism balances performance and reliability, avoiding the consistency risk caused by cache loss in the traditional single confirmation mode.

[0092] Phase 2: Autonomous encoding (performed asynchronously by both parity and even OSDs). After both parity and even OSDs detect an update to their local data block replicas (based on version number increments or log triggers) through the autonomous encoding module, they asynchronously execute the following process: 1. Log parsing and version verification: Read local logs to obtain 、 and version number .

[0093] Read the local PMem log and extract the block ID, old value hash, and new value and version number , locate the corresponding original data block through the block ID (If there is no local , then send a hash check request to the primary OSD. Only when the primary OSD returns Only when the hash matches the old value hash in the log, it is obtained , otherwise the encoding is rejected and an exception is flagged).

[0094] In this method, hash check is introduced to replace the transmission of complete old values, reducing cross-node data interaction (only 32-byte hash values ​​are transmitted instead of complete data blocks), while preventing encoding errors caused by inconsistent data between the primary OSD and parity OSD.

[0095] 2. Incremental coding and matrix dynamic adaptation: Based on the coding matrix proposed in this invention Calculate the parity block increment (see the following description for a detailed embodiment); Based on the coding matrix ( represents the field of real numbers, The elements in are available Indicates that, For the The parity block corresponds to The encoding coefficients of each data block are dynamically generated by MDS based on the PG's fault domain configuration (for example, the coefficient weight is increased when deployed across cabinets). Calculate the parity block increment: ; Perform "XOR check" on the calculation result ( ), to ensure that the increment calculation is correct.

[0096] In this method, the matrix Binding to the PG fault domain (e.g., the coefficient for OSDs within the same cabinet is 1, and the coefficient across cabinets is a prime number) improves recovery efficiency in fault scenarios, which is different from the fixed matrix design in existing related technologies.

[0097] The following describes the erasure coding algorithm provided by the present invention. Based on a dynamically generated matrix and an incremental computation model, this algorithm is designed for (K,M) EC storage pools. It supports both initial full encoding and avoiding full PG striping operations through local incremental updates. Its core lies in the dynamic matrix coefficients adapted to fault domains and the incremental computation of hash checks, as detailed below: First, the initial encoding formula (full encoding): For the (K,M) EC system, let the data block be , the generated matrix is , initial parity block calculation: ; in, For the parity blocks, generated by linear combination of all data blocks.

[0098] 3. Parity Block Update and Atomic Commit: Update Local Parity Block with Incremental for , the synchronous update version number is , and generate an "encoding completion log" (block ID + new parity block hash + version number).

[0099] Adopt the "Copy on Write (COW)" mechanism: first Write to the new address and atomically switch the pointer after success (update metadata via PMem's 8-byte atomic write), avoiding parity block corruption caused by power outage during the update process.

[0100] "Hash check + COW" is used to ensure the atomicity of parity block updates and solve the partial write problem in the traditional in-place update mode.

[0101] 4. Space recovery and state synchronization: After encoding is completed, delete the local Replicas and logs (retain the "encoding completion log" for 10 minutes for consistency verification on the primary OSD) are used to free up storage space. An encoding completion notification (including the new parity block hash) is sent to the primary OSD. The primary OSD updates the block status from "pending encoding" to "completed" and synchronizes it to the MDS (for PG status monitoring).

[0102] The difference between the present invention and related technologies lies in the layered log design: using old value hash + pointer to replace the complete data storage, combined with PMem atomic write characteristics, to achieve log lightweight and high reliability. Double confirmation synchronization mechanism: based on the PG load, the synchronization strategy is dynamically adjusted, and basic confirmation + log confirmation is used to ensure that data is not lost, which is better than the traditional single confirmation mode. Fault domain adaptation matrix: encoding matrix Binding to PG fault domains (such as cabinets and nodes) improves fault recovery efficiency and overcomes the limitations of fixed matrices. Hash checksums and COW: Hash checksums are introduced throughout the entire process to prevent data tampering. Combined with the COW mechanism, they ensure code atomicity and resolve consistency conflicts caused by concurrent updates.

[0103] Through the above optimizations, the two-phase write method reduces write latency to less than 10% of traditional methods while maintaining compatibility with the distributed storage erasure pool architecture, increases the number of concurrent input / output operations per second (IOPS) by more than three times, and avoids the space redundancy problem caused by the reliance on a fixed number of replicas (such as a fixed number of replicas of M+1) in related technologies.

[0104] The following takes the (3,2) EC storage pool (K=3, M=2) as an example to illustrate the encoding matrix of the present invention. By adopting a fault-domain-aware dynamic coefficient allocation strategy and combining linear algebra with prime number properties, we achieve the following innovative designs in the (3,2) EC scenario: 1. Fault domain topology analysis and matrix initialization. MDS first analyzes the PG topology: Data block distribution: 3 OSDs in cabinet A. Parity block distribution: Located in cabinet A, Located in cabinet B. MDS performs the following steps to generate the matrix : Same fault domain coefficient allocation, for the parity block in cabinet A , the coefficients are set to all 1 vectors [1,1,1]. Mathematical basis: OSD failures in the same cabinet are highly correlated, and all 1 coefficients can simplify incremental calculations. , and only local data is needed to complete decoding during fault recovery. Cross-fault domain coefficient allocation: For the parity block of cabinet B , the coefficients are selected from a sequence of mutually prime numbers [3,5,7]. Mathematical basis: there is no common divisor between prime numbers, ensuring that the matrix row vectors are linearly independent (avoiding the failure of cabinet B). and The modular arithmetic properties of prime numbers can reduce coding conflicts (e.g., 3 mod 2 = 1, 5 mod 2 = 1, ensuring independence in the binary domain).

[0105] 2. Therefore, in this embodiment of the present invention, the MDS configuration coding matrix is: ; If the data block value is ; but .

[0106] Second, the incremental encoding formula: When the data block Updated to When the parity block increments Relying only on single-block changes, without reading the entire PG stripe: , avoiding the traditional "read all PG strips - re-encode" process, reducing I / O amplification. Perform "XOR check" on the calculation results ( ), to ensure that the increment calculation is correct.

[0107] Following the above (3,2) EC example, if Updated from 20 to 25 (i.e. ), then the difference value is , incremental calculation: ; Updated parity block: ; XOR check: The goal of simplified verification is to quickly eliminate obvious errors, rather than strict mathematical equivalence. In simplified verification, when When is an odd number, The result should be 0 (as in the example ), verify the parity of the coefficients; in actual projects, combined with CRC32 to check the complete data, the total time consumption is controlled within 5μs.

[0108] This paper proposes the following erasure coding algorithm process: The algorithm process revolves around the three core steps of "log parsing - incremental calculation - atomic update", combining hash verification and copy-on-write (COW) mechanisms to ensure the consistency and reliability of the encoding process. The specific steps are as follows: Step 1: Log parsing and version verification (parity OSD side): 1. Parity OSD reads the local PMem log and extracts the field: Block ID = 2 (corresponding to )、Old value hash = hash(20) = 0xABC、New value =25, version number .

[0109] 2. Locate the original data blocks stored locally If it does not exist, a "hash verification request" is sent to the primary OSD.

[0110] 3. The primary OSD returns The hash of the value = 0xABC matches the hash of the old value in the log, confirming data consistency and allowing encoding.

[0111] Step 2: Dynamic matrix acquisition and incremental calculation: 1. Odd-even OSD obtains the encoding matrix of the current PG through MDS (as described above).

[0112] 2. Calculate the difference between data blocks: .

[0113] 3. Calculate the increment: .

[0114] 4. Perform XOR check: To simplify the checksum, CRC32 is actually used to confirm that the calculation is correct.

[0115] Step 3: Parity block atomic update and COW mechanism: 1. Parity OSD uses copy-on-write (COW): and Allocate a new address and write the values ​​65 and 365. Update the metadata pointer via an 8-byte atomic write from PMem, switching from the old address to the new one.

[0116] 2. Generate "Encoding Completion Log": Block ID = 2, New Parity Block Hash = hash(65,365) = 0xDEF, Version Number .

[0117] Step 4: Space recovery and status synchronization: 1. Delete local The old copy (value is 20) and the update log are retained, and the "encoding completion log" is retained for 10 minutes.

[0118] 2. Send "Encoding Completion Notification" to the master OSD, and the master OSD will The status is marked from "To be Encoded" to "Completed" and synchronized to MDS.

[0119] In addition, the present invention also provides the following reliable guarantees: Version control: Each data block / parity block is associated with a global version number, which is incremented when updated. This ensures that read and write operations are based on the latest version and avoids concurrent update conflicts. Logging mechanism: PMem logs only record incomplete updates (8-byte block ID + 8-byte version number). In the event of a node failure, data is synchronized through the log to ensure strong consistency (reference). Fault tolerance: Based on the (K,M) EC feature, it can tolerate up to M node failures. The old and new data block copies stored by the parity OSD can recover the lost data through encoding.

[0120] Furthermore, the erasure storage pool data processing method based on the distributed storage system and the distributed storage system collaborative processing process provided by the present invention are described: 1. Combination with PG technology: PG is a logical unit for data distribution and redundancy management in a distributed storage system. The present invention implements the collaboration between the primary OSD, parity OSD, and PG in the following ways: 1. Role division within PG: Each PG contains the primary OSD corresponding to K data blocks and the parity OSD corresponding to M parity blocks (corresponding to the (K,M) EC configuration). For example, in a (3,2) EC PG, 3 primary OSDs manage each block and 2 parity OSDs manage each block. The primary OSD is mapped to the PG through the CRUSH algorithm and is responsible for updating and synchronizing the data blocks within the PG. The parity OSD is also distributed in different fault domains (such as different hosts / cabinets) through the CRUSH algorithm to avoid the loss of multiple parity blocks due to a single point failure.

[0121] 2. Concurrency control at the PG level: The primary OSD records the update status (block ID + version number) of the data blocks in the PG through the PMem log to ensure that concurrent updates in the same PG are executed in order according to the version number; Parity OSD triggers encoding only when there are no unfinished logs in PG (that is, no concurrent updates) to avoid encoding conflicts.

[0122] 3. Adaptation of PG and fault domain: The distribution of primary and parity OSDs within the PG follows the principle of fault domain isolation (e.g., cross-host deployment), ensuring the fault tolerance of (K,M) EC (tolerating up to M node failures). The incremental encoding mechanism of the present invention retains the old and new version copies of the data block at the PG level. Even if some nodes in the PG fail, the data can still be recovered through the remaining copies or parity block decoding.

[0123] 2. Technical Integration with the Erasure Pool (EC Pool): The erasure pool is a storage pool that uses the EC redundancy strategy. The design of the primary OSD and parity OSD directly serves to optimize the performance and reliability of the erasure pool: 1. Data placement and erasure pool architecture: The erasure pool is divided into PG stripes according to the (K,M) EC configuration. Each PG stripe corresponds to one or more PGs. The primary OSD stores the original data blocks in the PG stripe, and the parity OSD stores the corresponding parity blocks and data block replicas (the hybrid placement strategy of the present invention). Compared with the traditional erasure pool (centralized encoding on the master OSD), the present invention uses autonomous encoding by parity OSDs to separate the encoding operation from the master OSD and distribute it to multiple nodes, thereby reducing the overall load of the erasure pool.

[0124] 2. Optimization of overwrite operations (solving the "read-modify-encode-write" bottleneck): The overwrite operation of the traditional erasure pool requires the primary OSD to read the entire PG stripe data (involving multiple PG OSDs), re-encode it, and then write it, resulting in I / O amplification; In the present invention, the primary OSD only synchronizes the updated data blocks to the parity OSD (Phase 1), and the parity OSD updates the parity blocks through incremental encoding based on the local replica (Phase 2), without reading the entire PG stripe, reducing the I / O overhead of the overwrite operation to 0.

[0125] 3. Collaboration with erasure pool metadata: The primary OSD obtains the PG stripe distribution information of the erasure pool (such as the OSD list corresponding to each PG) through the MDS to ensure the accuracy of data block synchronization; Parity OSD identifies the parity block location and encoding matrix it is responsible for through PG metadata The coefficient of , updates the metadata version of the erasure pool after autonomously completing the encoding, and is compatible with the OSD Map synchronization mechanism of the distributed storage system.

[0126] In implementation, specifically: 1. System Configuration: Based on the (3,2) RS code (3 data blocks + 2 parity blocks), a 6-node distributed storage system is deployed. Each node is configured with: 2 CPUs, 256GB DRAM memory, 2 256GB PMems (for log storage), and 24 HDDs (for data storage).

[0127] 2. Optimize module deployment: Primary OSD (3): Deploy the "data update module" to synchronize data to the parity OSD; Parity OSD (2): Deploys the "autonomous encoding module", stores data copies and parity blocks, and performs incremental encoding; Metadata layer: Obtain PG stripe distribution information through MDS to ensure correct mapping of data blocks / parity blocks.

[0128] Test results from the embodiment show that, in a (3,2) EC configuration, 4KB random write IOPS reached 210k (compared to the native 52k for the distributed storage system), and 64KB sequential write throughput reached 1.5GB / s (compared to the native 360MB / s for the distributed storage system). After hot and cold tiering, the storage redundancy rate dropped to 40%. Table 1 shows some of the test results.

[0129] Table 1 .

[0130] This invention embeds an optimization module within the erasure storage pool of a distributed storage system, decoupling data updates from encoding. Combining incremental encoding with a lightweight logging mechanism, it effectively addresses the performance bottlenecks of traditional EC storage pools. Its modular design is compatible with existing architectures and can be widely applied in high-concurrency distributed storage scenarios, demonstrating significant practical value.

[0131] Specifically, the beneficial effects brought about by the technical solution of the present invention are mainly reflected in four dimensions: performance, reliability, compatibility and resource efficiency, as follows: In terms of performance improvements, by decoupling the data update and encoding processes, encoding tasks are automatically transferred from the primary OSD to the parity OSD for execution, avoiding the I / O amplification of the traditional "read-modify-encode-write" process. Tested data shows that 4KB random write IOPS reaches 210k (4.04 times that of the native EC storage pool), and 64KB sequential write throughput reaches 1.5GB / s (4.17 times that of the native one). Overwrite operation latency is reduced by 75% to 90%, and 4KB random write latency is only 8.2μs (7.8% of the native one), significantly improving responsiveness in high-concurrency scenarios.

[0132] To ensure reliability and consistency, PMem's lightweight log (block ID + version number) and two-phase write "double confirmation" mechanism ensure traceability of data updates. Even node failures can be recovered through log synchronization. Furthermore, it is compatible with the fault tolerance of (K,M) EC, tolerating up to M node failures. The parity OSD stores copies of old and new data blocks, combined with a dynamic encoding matrix, further reducing the risk of data loss and eliminating consistency conflicts during concurrent updates.

[0133] In terms of compatibility, the optimization module is implemented by embedding it in the OSD layer, without the need to reconstruct the underlying code. It supports existing PG management, fault domain configuration, and CRUSH algorithm, adapts to the existing architecture of enterprise-level distributed storage, and reduces migration costs.

[0134] In terms of resource efficiency, incremental encoding only transmits data block differences, reducing cross-node data transmission by 90%; PMem logs use hash + pointers instead of full data storage, reducing metadata overhead by 60%; the hot and cold data tiering strategy (hot data replicas, cold data EC) reduces the storage redundancy rate by 60% compared to the fixed copy solution, and approaches the storage efficiency of traditional EC in the (3,2) EC configuration, balancing performance and space occupancy.

[0135] In summary, the present invention achieves the coordinated optimization of performance, reliability and resource efficiency in high-concurrency distributed storage scenarios, and has significant practical value.

[0136] Furthermore, it's important to note that compared to optimizations based on encoding algorithms (regeneration codes, local repair codes), data layout (strip merging and layering strategies), and system architecture, the write-focused optimization provided by this invention centers on "incremental encoding + dynamic matrix." This approach doesn't rely on complex encoding algorithms, but instead achieves efficient updates through local computation of even and odd OSDs. These two optimizations have different objectives: the former optimizes fault recovery, while the method provided by this invention optimizes real-time write performance, representing distinct technical approaches.

[0137] The above describes a method for processing erasure storage pool data based on a distributed storage system and applied to a parity object storage device in an erasure storage pool. Embodiments of the present invention also provide a method for processing erasure storage pool data based on a distributed storage system and applied to a primary object storage device in an erasure storage pool. The method includes: Obtain a write request sent by the client to indicate an update of a preset data block; According to the write request, the preset data block stored locally is updated to obtain the target data block; Synchronize the target data block to the parity object storage device; so that the parity object storage device obtains the difference between the target data block and the original data block corresponding to the locally stored target data block; obtain the encoding coefficient determined based on the failure domain configuration of the placement group, and determine the parity block increment based on the encoding coefficient and the difference; and update the locally stored parity block using the parity block increment.

[0138] The erasure storage pool data processing method based on a distributed storage system and applied to the primary object storage device in the erasure storage pool provided in this embodiment has the same or corresponding technical features as the erasure storage pool data processing method based on a distributed storage system and applied to the parity object storage device in the erasure storage pool described above. The embodiment of the erasure storage pool data processing method based on a distributed storage system and applied to the parity object storage device in the erasure storage pool has been described in detail above. The embodiment of the erasure storage pool data processing method based on a distributed storage system and applied to the primary object storage device in the erasure storage pool will not be repeated here, and the effect is the same as above.

[0139] In addition, an embodiment of the present invention further provides a distributed storage system, comprising a primary object storage device, a parity object storage device, and a metadata server; wherein the primary object storage device is configured to obtain a write request sent by a client for updating a preset data block; update the locally stored preset data block according to the write request to obtain a target data block; and synchronize the target data block to the parity object storage device; The metadata server is used to determine the encoding coefficient based on the failure domain configuration of the placement group; The parity object storage device is used to receive the target data block sent by the primary object storage device; obtain the difference between the target data block and the original data block corresponding to the locally stored target data block; obtain the encoding coefficient determined based on the failure domain configuration of the placement group, and determine the parity block increment based on the encoding coefficient and the difference; and use the parity block increment to update the locally stored parity block.

[0140] Specifically, the primary object storage device: deploys a data update module to perform the write request processing, PMem logging, and data synchronization described in Phase 1 above; Parity object storage device: Deploys an autonomous encoding module to perform the log parsing, incremental encoding, parity block updates, and space reclamation described in Phase 2 above; Metadata server: Deployment matrix generation module, dynamically generate encoding matrix based on fault domain , and maintain the stripe distribution and version information of PG; Client: Initiates write requests to the primary object storage device and receives responses, interacting with the cluster through the distributed storage interface.

[0141] Among them, the data update module of the main object storage device and the autonomous encoding module of the parity object storage device interact through the network protocol to synchronize data blocks and confirmation information. The interactive data only contains the block ID, version number and hash value, and the complete data block is not transmitted.

[0142] The matrix generation module of the metadata server has a built-in prime number sequence library, and the prime numbers that are mutually prime across the fault domain coefficients are selected from the library to ensure that the encoding matrix The row vectors are linearly independent.

[0143] In addition, the distributed storage system is also equipped with: Storage unit: includes PMem (storage tiered log) and persistent storage media (storage data blocks and parity blocks); Processing unit: executes the data update logic of the primary object storage device or the autonomous encoding logic of the parity object storage device; Network unit: realizes communication with other object storage devices and MDS, and transmits data blocks, confirmation information and matrix coefficients.

[0144] PMem's log storage area adopts a partitioned design to independently store unfinished update logs and encoding completion logs. Unfinished logs are retained until encoding is completed, and encoding completion logs are retained for 10 minutes for consistency verification.

[0145] The processing unit integrates incremental encoding acceleration instructions, performs hardware-level optimization on the incremental calculation process, and reduces encoding delay.

[0146] When generating the encoding matrix, MDS obtains the fault domain labels (such as cabinet ID and node ID) of all object storage devices in the PG; For each parity object storage device, configure a coefficient based on its fault domain relationship with the data block object storage device: the same fault domain coefficient is 1, and the cross-fault domain coefficient is selected from the preset prime number library. Generate an encoding matrix of (M×K) dimensions ,After verifying the linear independence of matrix row vectors, synchronize to the relevant OSD and update the metadata version.

[0147] The prime number library contains prime numbers within 1000, and when selecting coefficients, prime numbers that are positively correlated with the distance to the fault domain are given priority (the longer the distance, the larger the prime number), improving the fault tolerance capability of cross-domain faults.

[0148] Furthermore, through version control, each data block and parity block is associated with a globally incremented version number, allowing updates to only be performed based on the latest version. Log recovery: After a node fails and restarts, unfinished updates are reconstructed using the PMem log, comparing the hash value of the local data with the log record. If there is a discrepancy, data synchronization is requested from the primary object storage device. Conflict arbitration: The primary object storage device detects concurrent update conflicts using the PG version vector and only processes requests with the highest version number, delaying all other requests by 1ms before retrying. This ensures consistency in the erasure storage pool.

[0149] In summary, the data processing method for an erasure code storage pool based on a distributed storage system provided by the present invention solves the performance bottleneck problem caused by the dependence of the erasure code (EC) storage pool in traditional distributed storage systems on centralized encoding by the master object storage device and the need to execute the full "read-modify-encode-write" process for overwrite operations. By embedding an optimization module in the IO process, it adopts a "1 master M slaves" hybrid data placement strategy, a two-stage write method (master object storage device synchronizes data + parity OSD autonomous incremental encoding), a fault domain-adaptive dynamic encoding matrix, and a PMem lightweight log mechanism to achieve decoupling of data updates and encoding, while optimizing storage redundancy by combining hot and cold data tiering. This method is compatible with existing architectures, can increase write throughput by 2.5 to 4.2 times, and reduce overwrite operation latency to 10% to 25% of traditional methods. It is suitable for high-concurrency distributed storage scenarios and balances performance and storage efficiency.

[0150] In the above embodiment, the erasure storage pool data processing method based on a distributed storage system applied to the parity object storage device in the erasure storage pool is described in detail. The present invention also provides a corresponding embodiment of the base data processing device.

[0151] Figure 4 This is a structural diagram of a data processing device provided by an embodiment of the present invention. This embodiment is based on the hardware perspective, such as Figure 4 As shown, the data processing device includes: Memory 20, for storing computer programs; The processor 21 is configured to implement the steps of the method for processing erasure storage pool data based on a distributed storage system as described in the above embodiment when executing a computer program.

[0152] The processor 21 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 21 may be implemented in at least one of the following hardware forms: a digital signal processor (DSP), a field-programmable gate array (FPGA), or a programmable logic array. The processor 21 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 21 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing the content required to be displayed on the display screen. In some embodiments, the processor 21 may also include an artificial intelligence (AI) processor, which is responsible for processing computing operations related to machine learning.

[0153] The memory 20 may include one or more computer-readable storage media, which may be non-transitory. The memory 20 may also include high-speed random access memory, and non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 20 is at least used to store the following computer program 201, wherein, after the computer program is loaded and executed by the processor 21, it can implement the relevant steps of the erasure storage pool data processing method based on the distributed storage system disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 20 may also include an operating system 202 and data 203, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 202 may include Windows, Unix, Linux, etc. The data 203 may include but is not limited to the data involved in the above-mentioned erasure storage pool data processing method based on the distributed storage system, etc.

[0154] In some embodiments, the data processing device may further include a display screen 22 , an input / output interface 23 , a communication interface 24 , a power supply 25 , and a communication bus 26 .

[0155] Those skilled in the art will understand that Figure 4 The structure shown in the figure does not constitute a limitation on the data processing device, and may include more or fewer components than shown in the figure.

[0156] The data processing device provided by an embodiment of the present invention includes a memory and a processor. When the processor executes the program stored in the memory, it can implement the following method: a data processing method for an erasure storage pool based on a distributed storage system, with the same effect as above.

[0157] An embodiment of the present invention further provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned method for processing erasure storage pool data based on a distributed storage system.

[0158] Finally, the present invention also provides an embodiment corresponding to a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps described in the aforementioned method embodiments (which may be methods applied to a primary object storage device in an erasure storage pool, methods applied to a parity object storage device in an erasure storage pool, or methods applied to both a primary object storage device and a parity object storage device).

[0159] It is understood that if the methods in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage media include various media that can store program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0160] The computer-readable storage medium provided by the present invention includes the above-mentioned erasure storage pool data processing method based on the distributed storage system, and the effect is the same as above.

[0161] The above is a detailed introduction to the erasure storage pool data processing method and system based on a distributed storage system provided by the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, the present invention can also be improved and modified in several ways, and these improvements and modifications also fall within the scope of protection of the present invention.

[0162] It should also be noted that, in this specification, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

Claims

1. A method for processing erasure storage pool data based on a distributed storage system, characterized in that: A parity object storage device is applied to an erasure storage pool, wherein any data block in the erasure storage pool is stored in a primary object storage device and a parity object storage device; the method comprises: Receiving a target data block sent by a primary object storage device; wherein the target data block is obtained by the primary object storage device updating the locally stored preset data block according to a write request sent by a client for representing an update of the preset data block; Obtaining a difference between the target data block and an original data block corresponding to the target data block stored locally; obtaining a coding coefficient determined based on a failure domain configuration of a placement group, and determining a parity block increment based on the coding coefficient and the difference; The locally stored parity block is incrementally updated using the parity block.

2. The method for processing erasure storage pool data based on a distributed storage system according to claim 1, characterized in that: Before the primary object storage device updates the locally stored preset data block to the target data block, the method further includes: The primary object storage device records a log and writes it into the persistent memory space in multiple times through the preset byte atomic write feature of the persistent memory; wherein the log includes at least the unique code of the preset data block, the version number prefix, the old value hash and the new value pointer.

3. The method for processing erasure storage pool data based on a distributed storage system according to claim 2, characterized in that: The primary object storage device synchronizes the target data blocks to the parity object storage device, including: Locating multiple parity object storage devices of the placement group stripe to which the target data block belongs based on metadata of the placement group; Obtaining load values ​​of the plurality of parity object storage devices; The target data blocks are synchronized to the parity object storage devices in sequence according to the load value from low to high.

4. The method for processing erasure storage pool data based on a distributed storage system according to claim 2, characterized in that: The primary object storage device synchronizes the target data blocks to the parity object storage device, including: Get the access frequency of the target data block; In a case where the target data block is determined to be a hot data block according to the access frequency, synchronizing the target data block to a parity object storage device, and waiting for confirmation information returned by the parity object storage device; When the target data block is determined to be a cold data block according to the access frequency, multiple operations on the cold data block are accumulated, and the target data block obtained by merging the multiple operations is synchronized to the parity object storage device.

5. The method for processing erasure storage pool data based on a distributed storage system according to claim 3, characterized in that: After the primary object storage device synchronizes the target data blocks to the parity object storage device, the following steps are also performed: After synchronizing the target data block to the parity object storage device, a first confirmation message is received from the parity object storage device within a first preset time period; wherein the first confirmation message includes at least information indicating that the parity object storage device has successfully received the target data block and information indicating that the parity object storage device has stored the target data block in a cache; receiving second confirmation information sent by the parity object storage device; wherein the second confirmation information at least includes a log record indicating that the parity object storage device has written the target data block into the local persistent memory space; Information indicating that the update is successful is returned to the client, and the data state of the preset data block is marked as a state to be encoded.

6. The method for processing erasure storage pool data based on a distributed storage system according to any one of claims 1 to 5, characterized in that: Before the primary object storage device updates the locally stored preset data block to the target data block, the method further includes: Maintain a version vector for each placement group and record the latest version number of each data block; When detecting concurrent write requests sent by the client for representing updates to preset data blocks, obtain the version numbers of the data blocks corresponding to all write requests; determine the highest version number based on all version numbers; and respond to the write request corresponding to the highest version number.

7. The method for processing erasure storage pool data based on a distributed storage system according to claim 2, characterized in that: Before obtaining the difference between the target data block and the locally stored original data block corresponding to the target data block, the method further includes: Reading the local log to extract the unique code of the target data block, the old value hash, the target data block, the old version number, and the new version number; Locating the corresponding original data block using the unique code of the target data block and obtaining a positioning result; When it is detected that the value of the new version number is greater than the value of the old version number, and / or when it is detected that the positioning result indicates that the original data block exists locally, entering the step of obtaining a difference between the target data block and the original data block corresponding to the target data block stored locally; When it is detected that the positioning result is that the original data block does not exist locally, a request for representing hash verification is sent to the primary object storage device; if the hash representing the original data block received from the primary object storage device matches the old value hash in the log, the original data block sent by the primary object storage device is obtained, and the step of obtaining the difference between the target data block and the original data block corresponding to the target data block stored locally is entered; if there is no match, a prompt message representing the rejection code is output.

8. The method for processing erasure storage pool data based on a distributed storage system according to claim 2, characterized in that: Determining the encoding coefficient based on the placement group's fault domain configuration includes: Obtaining a relationship between itself and a fault domain of a data block object storage device; wherein the fault domain is a cabinet or a node; If it is detected that the relationship is the same fault domain, the same fault domain coefficient is set to 1; If it is detected that the relationship is across fault domains, setting the cross-fault domain coefficients to coprime coefficients; Generate a coding matrix according to the value of the same-fault domain coefficient and the value of the cross-fault domain coefficient; A coding coefficient is determined according to the elements in the coding matrix.

9. The method for processing erasure storage pool data based on a distributed storage system according to claim 8, characterized in that: Determining the coprime coefficients includes: Get the fault domain distance value and the pre-created prime number library; Prime numbers that are positively correlated with the fault domain distance value are selected from the prime number library to obtain coprime coefficients.

10. The method for processing erasure storage pool data based on a distributed storage system according to claim 8, characterized in that: After generating the coding matrix according to the value of the same-fault domain coefficient and the value of the cross-fault domain coefficient, the method further includes: Use metadata servers to monitor the fault domain distribution of object storage devices within a placement group in real time; In the case where a change in the fault domain topology is detected, the process returns to the step of obtaining the relationship between the fault domain of the data block object storage device and the fault domain of the data block object storage device.

11. The method for processing erasure storage pool data based on a distributed storage system according to claim 1, characterized in that: Before incrementally updating the locally stored parity block using the parity block, the method further includes: Obtaining the parity block increment, the original data block, and the target data block, performing an XOR calculation on them, and obtaining an XOR calculation result; In the case where it is detected that the XOR calculation result is equal to an element in the encoding matrix, entering the step of incrementally updating the locally stored parity block using the parity block; In the case where it is detected that the XOR calculation result is not equal to the element in the encoding matrix, the method returns to the step of receiving the target data block sent by the primary object storage device.

12. The method for processing erasure storage pool data based on a distributed storage system according to claim 7, characterized in that: Incrementally updating the locally stored parity block using the parity block includes: Write the updated parity block into the new storage address; After a successful write is detected, the metadata pointer is updated to point to the new address through an atomic write operation in the persistent memory space.

13. The method for processing erasure storage pool data based on a distributed storage system according to claim 12, characterized in that: After the locally stored parity block is incrementally updated using the parity block, the method further includes: Updating the version number of the target data block to obtain a new version number; Generate an encoding completion log; wherein the encoding completion log includes at least the unique code of the target data block, the hash value of the new parity block obtained after the update, and the new version number.

14. The method for processing erasure storage pool data based on a distributed storage system according to claim 13, characterized in that: The storage area of ​​the persistent memory space includes an area for storing logs of unfinished updates and an area for storing logs of completed encodings; The unfinished logs stored in the area for storing unfinished update logs are retained until encoding is completed; and the encoding completion logs stored in the area for storing encoding completion logs are retained for a second preset time period.

15. The method for processing erasure storage pool data based on a distributed storage system according to claim 14, characterized in that: After the locally stored parity block is incrementally updated using the parity block, the method further includes: After detecting that the encoding is completed, deleting the original data blocks and logs stored locally; Release the local storage space occupied by the original data blocks and logs.

16. The method for processing erasure storage pool data based on a distributed storage system according to claim 5, characterized in that: After the locally stored parity block is incrementally updated using the parity block, the method further includes: Sending information indicating that encoding is complete to the primary object storage device; wherein the information indicating that encoding is complete includes a hash value of the new parity block; so that the primary object storage device changes the data state from the pending encoding state to the completed state, and synchronizes the data state to the metadata server.

17. A method for processing erasure storage pool data based on a distributed storage system, characterized in that: A primary object storage device is applied to an erasure storage pool, wherein any data block in the erasure storage pool is stored in the primary object storage device and the parity object storage device; the method comprises: Obtain a write request sent by the client to indicate an update of a preset data block; updating the preset data block stored locally according to the write request to obtain a target data block; Synchronizing the target data block to a parity object storage device; allowing the parity object storage device to obtain a difference between the target data block and an original data block corresponding to the target data block stored locally; obtaining a coding coefficient determined based on a failure domain configuration of a placement group, and determining a parity block increment based on the coding coefficient and the difference; and updating the locally stored parity block using the parity block increment.

18. A distributed storage system, characterized in that: The system comprises a primary object storage device, a parity object storage device and a metadata server; wherein the primary object storage device is used to obtain a write request sent by a client for representing an update of a preset data block; according to the write request, the locally stored preset data block is updated to obtain a target data block; and the target data block is synchronized to the parity object storage device; The metadata server is configured to determine a coding coefficient based on a failure domain configuration of a placement group; The parity object storage device is used to receive a target data block sent by a primary object storage device; obtain a difference between the target data block and an original data block corresponding to the target data block stored locally; obtain a coding coefficient determined based on a failure domain configuration of a placement group, and determine a parity block increment based on the coding coefficient and the difference; and update the locally stored parity block using the parity block increment.

19. A data processing device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the method for processing erasure storage pool data based on a distributed storage system according to any one of claims 1 to 17 when executing the computer program.

20. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps of the erasure storage pool data processing method based on the distributed storage system are implemented as described in any one of claims 1 to 17.

Images