A data processing method and related equipment

Abstract

A data processing method and related equipment relate to the field of storage technology. The method, executed by a master node in a distributed storage system, includes obtaining a first request for updating a data block in an erasure code (EC) stripe, determining a first data block based on the first request, and sending a processing request to a set of slave nodes to instruct the master node to offload the data block update operation to one or more slave nodes in the set. This method offloads the master node's data block update operation to the slave nodes, reducing the number of read operations during the EC stripe update process, avoiding the occupation of a large amount of network resources, and ensuring the system performance of the distributed storage system.

Description

[0001] This application is a divisional application. The original application has application number 202211017671.X and an original filing date of August 23, 2022. The entire contents of the original application are incorporated herein by reference. This application and the Chinese patent application with application number 202211017671.X both claim priority to Chinese patent application number 202210740423.1, filed with the State Intellectual Property Office of China on June 27, 2022, entitled “A Method for Data Processing,” the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present application relates to the field of storage technology, and in particular to a data processing method, a data processing apparatus, a computing device cluster, a computer-readable storage medium, and a computer program product. Background Art

[0003] With the continuous development of information technology, more and more industrial applications are adopting information-based deployment methods, generating a large amount of data. To reduce data storage costs and ensure data reliability, the industry has proposed erasure code (EC) technology. EC specifically divides data into multiple groups of data blocks, then calculates a check block based on each group of data blocks. The data blocks and check blocks are then distributed and stored on different nodes in a distributed storage system.

[0004] A set of data blocks and the parity blocks generated by verifying these blocks is called an EC stripe. To ensure data consistency, data read and update operations are typically performed by the node where a data block (for example, the first data block) in the EC stripe resides. The node where the first data block in the EC stripe resides is also called the master node, and the nodes where the other data blocks in the EC stripe and the parity blocks reside are called slave nodes.

[0005] However, when updating an EC stripe, for example, updating a data block in an EC stripe, the master node is usually required to perform multiple read and write operations. Multiple read and write operations can occupy a large amount of network resources and reduce the system performance of the distributed storage system. Summary of the Invention

[0006] This application provides a data processing method that offloads data block updates from a master node to slave nodes, reducing the number of read operations during EC stripe updates, avoiding the use of significant network resources, and ensuring the performance of a distributed storage system. This application also provides a data processing device, computing device cluster, computer-readable storage medium, and computer program product corresponding to the aforementioned method.

[0007] In a first aspect, the present application provides a data processing method. The method can be executed by a master node in a distributed storage system. Specifically, the master node obtains a first request, the first request is used to update a data block in an EC stripe, and then determines a first data block based on the first request, the first data block being a data block associated with the first request, and then the master node sends a processing request to a slave node set including at least one slave node in the distributed storage system to instruct the master node to update the data block to be offloaded to one or more slave nodes in the slave node set. In this way, the number of read operations in the process of updating the data block in the EC stripe can be reduced, the network transmission overhead can be reduced, and the system performance can be guaranteed.

[0008] In some possible implementations, the master node may send a second request including the second data block to the first slave node, then receive the first data block returned by the first slave node after updating the first data block to the second data block, then determine the check block update information based on the first data block and the second data block, and then send a third request including the check block update information to the second slave node, wherein the check block update information is used to update the check block.

[0009] This method pushes down the operator part of calculating the new check block to the slave node, avoiding the master node reading the check block from the second slave node where the check block is located, reducing the number of read operations, lowering network transmission overhead, and ensuring system performance.

[0010] In some possible implementations, the master node may send a second request including the second data block to the first slave node, where the second request is used to instruct the first slave node to update the first data block to the second data block and to determine parity block update information based on the first data block and the second data block. The master node may then send a third request including the parity block update information to the second slave node via the first slave node, where the parity block update information is used to update the parity block.

[0011] This method pushes down all operators for calculating new check blocks to slave nodes, specifically to the first slave node (also called the update node) where the first data block is located and the second slave node where the check block is located, so as to avoid the first slave node reading the check block from the second slave node, reduce the number of read operations, reduce network transmission overhead, and ensure system performance.

[0012] In some possible implementations, the second request sent by the master node to the first slave node is an update request, and the return value of the update request is the first data block, which instructs the first slave node to update the first data block to the second data block and return the first data block. In this way, the master node only needs to perform a single update operation to replace the one read operation and one write operation in the related art, reducing the number of operations, lowering network transmission overhead, and ensuring system performance.

[0013] In some possible implementations, the first data block may be stored on a first slave node, and the master node may be the same node as the first slave node. Similarly, in other embodiments, the check block may be stored on a second master node, and the master node may be the same node as the second slave node.

[0014] In this way, the master node can read the first data block or check block locally, reducing the number of remote read operations, thereby reducing the occupied network resources and ensuring system performance.

[0015] In some possible implementations, before the master node obtains the first request, it may also obtain a fourth request including a data stream. The master node then divides the data in the data stream into blocks to obtain multiple data blocks, and writes the multiple data blocks into columns in a data block storage node in the distributed storage system. The data block storage node includes the master node and a first slave node. The master node then calculates a check block based on each group of data blocks in the multiple data blocks, and writes the check block into a check block storage node in the distributed storage system. The check block storage node includes a second slave node.

[0016] Storing data blocks by column in a distributed storage system can reduce the number of cross-disk operations during subsequent data reading and reduce read overhead.

[0017] In some possible implementations, when the multiple data blocks obtained by data stream segmentation cannot fill at least one EC stripe, the master node can also perform a null operation on the shards without data in at least one EC stripe without performing a fill operation, thereby reducing write amplification.

[0018] In some possible implementations, the master node can also receive a fifth request including a starting address, determine a target node based on the starting address, and read the target data block from the target node column by column. This allows the required data to be read from the hard disk only once, at a single node, reducing read amplification.

[0019] In a second aspect, the present application provides a data processing method. The method is applied to a distributed storage system and includes:

[0020] The master node obtains a first request, where the first request is used to update a first data block in an erasure code (EC) stripe, determines the first data block according to the first request, where the first data block is the data block associated with the first request, and sends a processing request to a set of slave nodes, where the set of slave nodes includes at least one slave node in the distributed storage system, where the processing request is used to instruct the master node to offload an operation of updating the data block to one or more slave nodes in the set of slave nodes.

[0021] The slave node set updates the first data block and check block according to the processing request.

[0022] This method offloads the master node's data block update operations to a set of slave nodes, reducing the number of read operations during the update of EC stripe data blocks, lowering network transmission overhead and ensuring system performance.

[0023] In some possible implementations, the master node sends a processing request to the set of slave nodes, including:

[0024] The master node sends a second request including the second data block to the first slave node;

[0025] The first slave node updates the first data block according to the processing request, including:

[0026] The first slave node updates the first data block to a second data block and returns the first data block;

[0027] The method further comprises:

[0028] The master node determines check block update information according to the first data block and the second data block;

[0029] The master node sends a processing request to the slave node set, including:

[0030] Sending a third request including check block update information to the second slave node;

[0031] The second slave node updates the check block according to the processing request, including:

[0032] The second slave node updates the check block according to the check block update information.

[0033] This method pushes down the operator part of calculating the new check block in the process of the master node updating the data block to the slave node, reducing the operation of reading the check block from the slave node where the check block is located, reducing network transmission overhead, and ensuring system performance.

[0034] In some possible implementations, the master node sends a processing request to the set of slave nodes, including:

[0035] The master node sends a second request including the second data block to the first slave node;

[0036] The first slave node updates the first data block according to the processing request, including:

[0037] The first slave node updates the first data block to a second data block;

[0038] The method further comprises:

[0039] The first slave node determines check block update information according to the first data block and the second data block;

[0040] The master node sends a processing request to the slave node set, including:

[0041] Sending a third request including the check block update information to the second slave node through the first slave node;

[0042] The second slave node updates the check block according to the processing request, including:

[0043] The second slave node updates the check block according to the check block update information.

[0044] This method pushes down all operators used to calculate new check blocks when the master node updates the data block to the slave node, reducing the number of operations required to read the check blocks from the slave node where the check blocks are located, reducing network transmission overhead, and ensuring system performance.

[0045] In some possible implementations, the second request is an update request, which instructs the first slave node to update the first data block with the second data block and return the first data block. In this way, the master node only needs to perform a single update operation to replace the one read operation and one write operation in the related art, thereby reducing the number of operations, lowering network transmission overhead, and ensuring system performance.

[0046] In a third aspect, the present application provides a data processing device, which includes various modules for executing the data processing method in the first aspect or any possible implementation of the first aspect.

[0047] In a third aspect, the present application provides a data processing device, which includes various units for executing the data processing method in the second aspect or any possible implementation of the second aspect.

[0048] In a fourth aspect, the present application provides a data processing system, which includes various devices for executing the data processing method in the second aspect or any possible implementation of the second aspect.

[0049] In a fifth aspect, the present application provides a computing device cluster. The computing device cluster includes at least one computing device, each of which includes at least one processor and at least one memory. The at least one processor and the at least one memory communicate with each other. The at least one processor is configured to execute instructions stored in the at least one memory, causing the computing device or computing device cluster to perform the data processing method described in any implementation of the first or second aspects.

[0050] In a sixth aspect, the present application provides a computer-readable storage medium, in which instructions are stored, and the instructions instruct a computing device or a computing device cluster to execute the data processing method described in the first aspect or any implementation of the first aspect.

[0051] In a seventh aspect, the present application provides a computer program product comprising instructions, which, when executed on a computing device or a computing device cluster, enables the computing device or computing device cluster to execute the data processing method described in the first aspect or any one of the implementations of the first aspect.

[0052] Based on the implementation methods provided in the above aspects, this application can also be further combined to provide more implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A schematic diagram of the EC strip update process provided in this application;

[0054] Figure 2 A system architecture diagram of a distributed storage system provided in this application;

[0055] Figure 3 A system architecture diagram of a distributed storage system provided in this application;

[0056] Figure 4 A schematic diagram of an application scenario of a distributed storage system provided in this application;

[0057] Figure 5 A flowchart of a data processing method provided in this application;

[0058] Figure 6 A schematic diagram of row storage and column storage provided by this application;

[0059] Figure 7 A schematic diagram of a data writing process provided by this application;

[0060] Figure 8 A schematic diagram of the EC strip update process provided in this application;

[0061] Figure 9 A schematic diagram of the EC strip update process provided in this application;

[0062] Figure 10 A schematic diagram of the structure of a data processing device provided in this application;

[0063] Figure 11 A schematic diagram of the structure of a data processing system provided in this application;

[0064] Figure 12A schematic diagram of the structure of a computing device cluster provided in this application;

[0065] Figure 13 A schematic diagram of the structure of a computing device cluster provided in this application. DETAILED DESCRIPTION

[0066] To facilitate understanding, some technical terms involved in the embodiments of this application are first introduced.

[0067] EC stripe update, which can also be referred to as EC stripe overwrite, specifically uses several new data blocks to replace several data blocks in the EC stripe, and based on the update of the data blocks, updates the check blocks in the EC stripe accordingly. According to the different ways of generating new check blocks, EC stripe overwrite can also be divided into EC lowercase and EC uppercase. EC lowercase refers to reading the check block, the modified data block, and the modified data block to determine the new check block. EC uppercase refers to reading other data blocks in the EC stripe, and determining the new check block based on the modified data block and other data blocks in the EC stripe. When the overwrite data block is small, the amount of data read by the EC lowercase method is small and the efficiency is higher. When the overwrite data block is large, the amount of data read by the EC uppercase method is small and the efficiency is higher.

[0068] The following example illustrates the EC stripe update process. Figure 1 The EC stripe update process diagram shown in FIG. 1 is a flow chart of an EC stripe update process. In this example, the EC stripe includes data blocks D0 to D1. k-1 And check block P, check block Q. The node where data block D0 is located is the master node, data blocks D1 to D k-1 The node is the first slave node, storing data blocks D0 to D k-1 The node where the check block P and the check block Q are located is the second slave node, also called the check block storage node.

[0069] like Figure 1 As shown, the client requests data block D1 ′ To overwrite the data block D1 in the EC stripe, in order to ensure strong consistency of the data, it is usually necessary to read the data to be modified, such as data block D1, to the master node first, and read the check block P and check block Q to the master node. The master node reads the check block P and check block Q according to the data block D1 and data block D1. ′ Sum check block P and calculate new check block P ′ , according to data block D1, data block D1 ′ Sum check block Q to calculate new check block Q ′ , as shown below:

[0070] P ′ =α0(D1′ -D1)+P (1)

[0071] Q ′ =β0(D1 ′ -D1)+Q (2)

[0072] Among them, α0 and β0 are different calibration coefficients.

[0073] Then, the master node sends data block D1 ′ , new check block P ′ , new check block Q ′ Write to the node where data block D1, check block P, and check block Q are located. This results in updating a data block in the EC stripe requiring three read operations and three write operations, which consumes a large amount of network resources and reduces system performance.

[0074] In order to solve the problem in traditional technologies that multiple read and write operations occupy a large amount of network resources, resulting in a decrease in system performance, the present application provides a data processing method applied to a distributed storage system. Specifically, a master node in the distributed storage system obtains a first request, where the first request is used to update a data block in an EC stripe. The master node can determine a first data block based on the first request, where the first data block is the data block associated with the first request. Then, the master node sends a processing request to a slave node set including at least one slave node in the distributed storage system to instruct the master node to offload the operation of updating the data block to one or more slave nodes in the slave node set.

[0075] This method offloads the master node's data block update operations to a set of slave nodes. For example, it pushes the operator for calculating a new check block down to the second slave node where the check block resides. This prevents the master node or the first slave node (also called the update node) where the first data block resides from reading the check block from the second slave node. This reduces the number of read operations, lowers network transmission overhead, and ensures system performance. Unlike other EC optimization techniques, this application focuses on changing the EC data transmission process and data distribution, enhancing data transmission and disk access efficiency. This makes it applicable to a variety of storage scenarios and offers high availability.

[0076] Furthermore, this method supports optimizing the data block update process. For example, it can combine a read and a write operation into a single read-write operation. This allows EC stripe updates to be completed with just one read-write operation plus two write operations, halving network transmission overhead, significantly reducing network resource utilization, and improving system performance.

[0077] The system architecture of the embodiment of the present application is introduced below with reference to the accompanying drawings.

[0078] See also Figure 2The system architecture diagram of the distributed storage system shown in FIG. 1 is a storage-computing separation structure, which includes a computing node cluster and a storage node cluster. The computing node cluster includes one or more computing nodes 110 ( Figure 2 Two computing nodes 110 are shown in FIG, but are not limited to two computing nodes 110. Each computing node 110 can communicate with each other. The computing node 110 is a computing device, such as a server, a desktop computer, or a controller of a storage array. In terms of hardware, Figure 2 As shown, the computing node 110 includes at least a processor 112, a memory 113 and a network card 114. The processor 112 is a central processing unit (CPU) for processing data access requests from outside the computing node 110 or requests generated inside the computing node 110. Exemplarily, when the processor 112 receives write data requests sent by users, it temporarily saves the data in these write data requests in the memory 113. When the total amount of data in the memory 113 reaches a certain threshold, the processor 112 sends the data stored in the memory 113 to the storage node 100 for persistent storage. In addition, the processor 112 is also used for data calculation or processing, such as metadata management, data deduplication, data compression, virtualized storage space, and address translation. Figure 2 Only one CPU 112 is shown. In actual applications, there are often multiple CPUs 112, wherein one CPU 112 has one or more CPU cores. This embodiment does not limit the number of CPUs or CPU cores.

[0079] Memory 113 refers to internal memory that directly exchanges data with the processor. It can read and write data at any time and at a high speed, and serves as temporary data storage for the operating system or other running programs. Memory includes at least two types of memory. For example, memory can be either random access memory (RAM) or read-only memory (ROM). For example, random access memory is dynamic random access memory (DRAM) or storage class memory (SCM). DRAM is a type of semiconductor memory that, like most random access memory (RAM), is a volatile memory device. SCM is a hybrid storage technology that combines the characteristics of traditional storage devices and memory. Storage class memory can provide faster read and write speeds than hard drives, but has slower access speeds than DRAM and is also cheaper than DRAM. However, DRAM and SCM are merely exemplary in this embodiment. Memory can also include other random access memories, such as static random access memory (SRAM). As for the read-only memory, for example, it can be a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), etc. In addition, the memory 113 can also be a dual in-line memory module or a dual-line memory module (DIMM), that is, a module composed of dynamic random access memory (DRAM), or a solid state disk (SSD). In actual applications, multiple memories 113 and different types of memories 113 can be configured in the computing node 110. This embodiment does not limit the number and type of memory 113. In addition, the memory 113 can be configured to have a power-saving function. The power-saving function means that when the system loses power and then powers on again, the data stored in the memory 113 will not be lost. A memory with a power-saving function is called a non-volatile memory.

[0080] The network card 114 is used to communicate with the storage node 100. For example, when the total amount of data in the memory 113 reaches a certain threshold, the computing node 110 can send a request to the storage node 100 through the network card 114 to store the data persistently. In addition, the computing node 110 can also include a bus for communication between the various components within the computing node 110. Functionally, due to Figure 2 The primary function of compute nodes 110 in this system is computing. When storing data, they can leverage remote storage for persistent storage. Therefore, they have less local storage than conventional servers, saving both cost and space. However, this does not mean that compute nodes 110 cannot have local storage. In actual implementations, compute nodes 110 may also have a small number of internal hard drives or external hard drives.

[0081] Any computing node 110 can access any storage node 100 in the storage node cluster through the network. The storage node cluster includes multiple storage nodes 100 ( Figure 2 Three storage nodes 100 are shown in the figure, but are not limited to three storage nodes 100). A storage node 100 includes one or more controllers 101, a network card 104 and multiple hard disks 105. The network card 104 is used to communicate with the computing node 110. The hard disk 105 is used to store data and can be a disk or other type of storage medium, such as a solid-state drive or a shingled magnetic recording hard disk. The controller 101 is used to write data to the hard disk 105 or read data from the hard disk 105 according to the read / write data request sent by the computing node 110. In the process of reading and writing data, the controller 101 needs to convert the address carried in the read / write data request into an address that can be recognized by the hard disk. It can be seen that the controller 101 also has some simple computing functions.

[0082] Figure 3 This is a system architecture diagram of another distributed storage system used in the embodiment of the present application. The system is a storage-computing integrated architecture, and the system includes a storage cluster. The storage cluster includes one or more servers 110 ( Figure 3 Three servers 110 are shown, but not limited to, three servers 110. Each server 110 can communicate with each other. Server 110 is a device with both computing and storage capabilities, such as a server or desktop computer. For example, either an ARM server or an X86 server can be used as server 110.

[0083] In hardware, such as Figure 3As shown, server 110 includes at least a processor 112, memory 113, a network interface card 114, and a hard disk 105. Processor 112, memory 113, network interface card 114, and hard disk 105 are connected via a bus. Processor 112 and memory 113 provide computing resources. Specifically, processor 112 is a central processing unit (CPU) that processes data access requests from outside server 110 (application servers or other servers 110), as well as requests generated within server 110. Memory 113 refers to internal storage that directly exchanges data with the processor. It can read and write data at any time and at high speed, and serves as temporary data storage for the operating system or other running programs. In actual applications, server 110 can be configured with multiple memories 113, as well as different types of memories 113. This embodiment does not limit the number and type of memories 113. In addition, memory 113 can be configured to have a power conservation function. Hard disk 105 provides storage resources, such as storing data. It can be a magnetic disk or other type of storage medium, such as a solid-state drive or a shingled magnetic recording hard disk. The network card 114 is used to communicate with other servers 110 .

[0084] It should be noted that the above Figure 2 、 Figure 3 This is only a schematic architecture of a distributed storage system. In other possible implementations of the embodiments of the present application, the distributed storage system may also use other architectures. For example, the distributed storage system may also adopt a fully integrated architecture or a Memory Fabric architecture.

[0085] Furthermore, the above-mentioned distributed storage system can provide storage services, for example, providing users with storage servers in the form of storage interfaces, so that users can use the storage resources of the distributed storage system through the above-mentioned storage interfaces. Figure 4 The distributed storage system shown in the application scenario diagram is as follows: a user can access the distributed storage system through a client (such as an application client), and the distributed storage system can be the above-mentioned Figure 2 or Figure 3 The architecture shown.

[0086] For example, distributed storage systems use Figure 2In the architecture shown, the client can call the storage interface provided by the storage service to generate a storage request and send the storage request to the distributed storage system. The computing node in the distributed storage system that receives the storage request can divide the data into multiple groups of data blocks, and then calculate the check blocks based on each group of data blocks, and write each group of data blocks and the check blocks determined by the group of data blocks to different storage nodes 100 in a dispersed manner, for example, to the hard disks 105 of different storage nodes 100 to form an EC stripe. The storage node 100 that stores the first data block (for example, data block D0) in the EC stripe can be regarded as the master node, and the storage node 100 that stores other data blocks in the EC stripe (for example, data block D1, ... data block D1) can be regarded as the master node. k-1 ) can be regarded as the first slave node, and the storage node 100 storing the check blocks (for example, the check block P and the check block Q) can be regarded as the second slave node.

[0087] For example, distributed storage systems use Figure 3 In the architecture shown, the client can call the storage interface provided by the storage service to generate a storage request and send the storage request to the distributed storage system. The server 110 in the distributed storage system can divide the data into multiple groups of data blocks, and then calculate the parity block based on each group of data blocks, and write each group of data blocks and the parity block determined by the group of data blocks to different servers 110 in a dispersed manner, for example, to the hard disks 105 of different servers 100. The server 110 that stores the first data block (for example, data block D0) in the EC stripe can be regarded as the master node, and the server 110 that stores other data blocks in the EC stripe (for example, data block D1, ..., data block D0) can be regarded as the master node. k-1 ) can be regarded as the first slave node, and the server 110 storing the check blocks (for example, check block P and check block Q) can be regarded as the second slave node.

[0088] The client accesses the distributed storage system through the storage service. The distributed storage system responds to the user's access to the distributed storage system through the storage service and returns an access result. The access result can vary depending on the access operation. For example, if the access operation is a write operation, the access result may indicate a successful write. For another example, if the access operation is a read operation, the access result may indicate a read data block.

[0089] In the EC stripe overwrite scenario, the master node can obtain a first request for updating a data block in the EC stripe, determine the first data block based on the first request, and then send a processing request to the set of slave nodes to instruct the master node to offload the data block update operation to one or more slave nodes in the set of slave nodes. For example, the processing request can instruct the master node to offload the data block update operation to the first slave node where the first data block is located and the second slave node where the parity block is located.

[0090] It should be noted that the first data block and check block described above may be stored on different nodes other than the master node in the distributed storage system. In some embodiments, the first data block may also be stored on the master node, in which case the master node and the first slave node are the same node. In other embodiments, the check block may also be stored on the master node, in which case the master node and the second slave node may be the same node.

[0091] Next, the data processing method of the embodiment of the present application is introduced using the scenarios of the master node, the first slave node, and the second slave node as different nodes.

[0092] See also Figure 5 The data processing method shown is a flowchart, the method comprising:

[0093] S502: The master node obtains request 1.

[0094] Request 1 includes a data stream. Request 1 is used to request that the data in the data stream be written to the distributed storage system for persistent storage. Request 1 may be generated by the application client based on business needs. Request 1 may be a write request or other request that requires writing data. Depending on the business needs of the application client, Request 1 may include different types of data streams. For example, when the application client is a short video application or a long video application, Request 1 may include a video data stream; for another example, when the application client is a file management application or a text editing application, Request 1 may include a text data stream. The master node may receive Request 1 issued by the application client to persistently store the data in the data stream carried by Request 1.

[0095] S504: The master node divides the data in the data stream included in request 1 into blocks to obtain multiple data blocks.

[0096] A data stream can be an ordered sequence of bytes with a start and end point. Specifically, the master node can use fixed-length chunking or variable-length chunking to chunk the data in the data stream carried by request 1, thereby obtaining multiple data chunks. Fixed-length chunking refers to chunking the data in the data stream according to a set chunking granularity. Variable-length chunking divides the data in the data stream into data chunks of varying sizes. Variable-length chunking can include sliding-window-based variable-length chunking and content-defined chunking (CDC).

[0097] For ease of understanding, the following example uses fixed-length blocks. Specifically, when the size of the data in the data stream is an integer multiple of the block granularity, the master node can evenly divide the data in the data stream into multiple data blocks. When the size of the data in the data stream is not an integer multiple of the block granularity, the master node can pad the data in the data stream, for example, by padding the end of the data stream with zeros, so that the size of the data in the padded data stream is an integer multiple of the block granularity, and then the master node evenly divides the data in the data stream into multiple data blocks according to the block granularity. For example, if the size of the data in the data stream is 20KB, the master node can block it according to the block granularity of 4KB and obtain 5 data blocks of 4KB in size.

[0098] In some embodiments, when the size of the data in the data stream is not an integer multiple of the block granularity, the master node may not fill the data in the data stream, but instead divide it into K-1 data blocks with a size equal to the block granularity and one data block with a size not equal to the block granularity according to the block granularity.

[0099] It should be noted that the data processing method of the embodiment of the present application may not execute the above S504. For example, if the data in the data stream is too small to be divided into blocks, or if the data in the data stream has been divided into blocks in advance, the above S504 may not be executed.

[0100] S506: The master node writes the multiple data blocks into the data block storage nodes including the master node and the first slave node in columns.

[0101] Assuming that the data block storage node can store L data blocks in each column, where L is a positive integer, the master node can first write multiple data blocks to the master node by column. When the column in the master node is full, the remaining data blocks are written to the first slave node by column.

[0102] When there are multiple first slave nodes, the master node can write the remaining data blocks to the first first slave node in columns. Similarly, when the first slave node is full, if there are still data blocks left, the master node will write the remaining data blocks to the next first slave node in columns.

[0103] For easier understanding, the following is an example. Figure 6The schematic diagram of row storage and column storage is shown, where row storage refers to storage by row and column storage refers to storage by column. In this example, the size of each data block is 4K, the data block storage node can store 256 data blocks in each column, the number of data block storage nodes is 4, and the number of check block storage nodes is 2. If the master node writes to the data block storage node by row, then data blocks D0 to D3 are written to 4 data block storage nodes respectively, specifically one master node and 3 first slave nodes, and P0 and Q0 are written to different check block storage nodes respectively. Similarly, data blocks D4 to D7 are written to 4 data block storage nodes respectively, and P1 and Q1 are written to different check block storage nodes respectively. If the master node writes to the data block storage node by column, then data blocks D0 to D 255 Write to the master node, data block D 256 to D 511 Write to the first slave node, and so on, data block D 512 to D 767 Write to the second first slave node, data block D 768 to D 1023 Write to the third first slave node.

[0104] In some possible implementations, multiple data blocks may not be able to fill an EC stripe. For example, if each column stores 256 data blocks and the number of data block storage nodes is 4, if the number of data blocks in the data stream is less than 769 (256*3+1), then at least one data block in the data stream is insufficient to fill an EC stripe. Write amplification can be reduced by omitting writing the empty portion. In a specific implementation, the master node can perform a no-op (zero op) on the chunks (chunks) without data in at least one EC stripe, rather than performing a fill operation, thereby reducing write amplification.

[0105] like Figure 7 As shown in the figure, when the master node writes a data block in chunk 1, it can add a zero Op to the master node's corresponding transaction 1 (denoted as transaction 1). The write Ops in transactions 2 through 4 (denoted as transactions 2-4) are all replaced with zero Ops. Consequently, no space is allocated for chunks 2-4, and no data is actually written to disk. When reading chunks 2-4, the correct data is returned to the upper layer without actually reading the disk. This reduces write amplification.

[0106] Based on this, the master node can first determine the size of the data stream. If the data in the data stream is not large enough to fill the stripe, it can only write the chunks that need to be filled, and leave the idle chunks unfilled. This not only improves write performance but also reduces space waste.

[0107] S508: The master node calculates a check block according to each group of data blocks in the multiple data blocks.

[0108] Specifically, the master node can group multiple data blocks. For example, the master node can group multiple data blocks according to the rows where each data block is located. The rows where the same group of data blocks are located have the same row number. Then, the master node can calculate each group of data blocks according to the verification algorithm to generate a verification block. The master node can use different verification algorithms to generate different verification blocks. For ease of understanding, we still use Figure 6 An example is given to illustrate the process of calculating the check block.

[0109] In this example, when the master node writes data blocks by column, the master node can 256 , data block D 512 , data block D 768 Calculate the check block P0 and check block Q0. Similarly, the master node can calculate the check block P0 and check block Q0 based on the data block D2 and data block D 257 , data block D 513 , data block D 769 The check block P1 and the check block Q1 are obtained by calculation.

[0110] In the embodiment of the present application, the data distribution method can be adjusted from row storage to column storage. In this way, data blocks with adjacent addresses can be concentrated on the same disk, such as data block D0 and data block D1 on the same disk. Accordingly, an EC stripe can include data blocks in different data segments, rather than continuous data blocks in one data segment. Figure 6 As shown, an EC stripe may include data block D0, data block D 256 , data block D 512 , data block D 768 When the disk or node where the data block D0 is located fails and the data block D0 is lost, the data block D0 can be 256 , data block D 512 , data block D 768 The above data block D0 is restored by summing the check blocks P0 and Q0.

[0111] S510: The master node writes the check block to the check block storage nodes including the second slave node.

[0112] When there are multiple check block storage nodes, that is, when there are multiple second slave nodes, the master node may write the check blocks into the corresponding second slave nodes respectively.

[0113] It should be noted that the above S506 and S508 can be executed sequentially in a set order, and then S510 is executed. In some embodiments, the above S506 and S508 can be executed in parallel, and then S510 is executed. In other embodiments, S506 and S510 can also be executed in parallel. For example, after S508 is executed to obtain the check block, the data block and the check block can be written to the corresponding node in parallel. The embodiment of the present application does not limit the order of the above S506, S508, and S510.

[0114] It should also be noted that the above steps S502 to S510 are optional steps in the embodiment of the present application, and the data processing method of the embodiment of the present application does not need to perform the above steps. For example, the data processing method of the embodiment of the present application can directly perform the following steps to update the EC stripe. Detailed description is given below.

[0115] S511: The master node obtains request 2.

[0116] Request 2 is used to update a data block in the EC stripe. For example, Request 2 is used to update the first data block in the EC stripe to the second data block. Request 2 includes the second data block. In some embodiments, Request 2 may also include the logical address of the first data block for fast addressing of the first data block.

[0117] S512: The master node determines the first data block according to request 2.

[0118] The first data block is specifically the data block associated with request 2. Specifically, the master node may parse request 2, obtain the logical address of the data block to be updated in request 2, and determine the first data block according to the logical address.

[0119] S514: The master node sends a request 3 to the first slave node where the first data block is located.

[0120] S516: The first slave node where the first data block is located updates the first data block to the second data block.

[0121] S518: The master node receives the first data block returned by the first slave node where the first data block is located.

[0122] S520: The master node determines the check block update information according to the first data block and the second data block.

[0123] S522: The master node sends request 4 to the second slave node where the check block is located.

[0124] S524: The second slave node updates the check block according to the check block update information in request 4.

[0125] In the EC stripe update scenario, request 2 can also be called the first request, and requests 3 and 4 can also be collectively referred to as processing requests. A processing request is a request sent by the master node to a set of slave nodes. Request 3 can be called the second request, and request 4 can be called the third request. In the EC stripe construction scenario, request 1 can also be called the fourth request.

[0126] exist Figure 5 In the example, requests 3 and 4 are used to instruct the master node to partially offload the operation of updating the data block to the first slave node and the second slave node. The offloading process is described in detail below.

[0127] Request 3, sent by the master node to the first slave node, includes the second data block. Request 3 specifically instructs the first slave node to update the first data block with the second data block. Since updating the first data block in the EC stripe also changes the parity block, the master node can read the first data block based on Request 3 to calculate the new parity block.

[0128] It should be noted that request 3 can be an update request, and the return value of the update request is the first data block. In this way, the first slave node where the first data block is located can read the first data block and then write the second data block when updating the first data block. In addition, the first slave node can also return the first data block to the master node. In this way, the second data block is written and the first data block is read through a single update operation (specifically a read-write operation). In some possible implementations, the master node can also send an additional request to read the first data block for calculating the check block update information.

[0129] The master node receives the first data block returned by the first slave node where the first data block is located, and can determine the parity block update information based on the first data block and the second data block using the EC algorithm. For example, the master node can determine the parity block update information based on the first data block and the second data block using formula (1) or formula (2).

[0130] Request 4 sent by the master node to the second slave node includes check block update information. Request 4 is specifically used to update the check block. The second slave node can update the check block based on the check block update information in request 4. For example, the second slave node can read the check block, determine a new check block based on the check block and the check block update information, and then store the new check block, thereby updating the check block.

[0131] Different from the traditional method in which the check block is read to the master node, the master node calculates a new check block based on the first data block, the second data block and the check block, and then sends the new check block to the check block storage node for updating, the embodiment of the present application offloads the operation part of updating the data block to the first slave node and the second slave node. Specifically, the process of updating the check block in the operation of updating the data block is decomposed into two steps, which are completed by different nodes.

[0132] Specifically, the master node can complete the previous step, which is to calculate the check block update information based on the first data block and the second data block, and then send the check block update information to the check block storage node. The check block storage node completes the next step, which is to update the check block according to the check block update information.

[0133] To facilitate understanding, a specific example is provided below for explanation.

[0134] like Figure 8 As shown, the master node completes the check block update information P″=α0(D2 ′ 56 -D 256 ) and check block update information Q″=β0(D2 ′ 56 -D 256 ) calculation, and then the master node pushes the check block update information P″ and the check block update information Q″ to the check block storage node. The check block storage node completes the new check block P ′ =P″+P and new check block Q ′ =Q″+Q calculation. And D2 ′ 56 and D 256 The read-then-write operation of the new and old data is transformed into a single update operation. Specifically, the update request includes the data to be written, but before writing, the data at the original address is read as the return value of the request. After the data is written to disk, the read data is returned to the master node.

[0135] It should also be noted that the data processing method according to the embodiment of the present application may also not execute S516 to S518. For example, Request 3 and Request 4 may be used to instruct the master node to offload all operations for updating the data block to the first slave node where the first data block is located and the second slave node where the check block is located.

[0136] For example, the first slave node (i.e., the update node) where the first data block is located can directly calculate the parity block update information based on the read first data block and second data block. The master node can send request 4 through the first slave node. The first slave node carries the parity block update information in request 4 and pushes it down to the second slave node where the parity block is located. The second slave node then calculates a new parity block based on the parity block update information in request 4, thereby implementing the parity block update.

[0137] For ease of understanding, the following still updates the data block D in the EC stripe. 256 Provide an example.

[0138] like Figure 9 As shown, the first slave node completes the check block update information P″=α0(D2 ′ 56 -D 256 ) and check block update information Q″=β0(D2 ′ 56 -D 256 ), and then the first slave node pushes the check block update information P″ and the check block update information Q″ down to the check block storage node. The second slave node completes the new check block P ′ =P″+P and new check block Q ′ =Q″+Q. And only the result of the operation is needed at the master node, and the unused data does not need to be read to the master node, thus reducing data transmission. ′ 56 and D 256 The read-then-write operation of the new and old data is transformed into a single update operation. Specifically, the update request includes the data to be written, but before writing, the data at the original address is read as the return value of the request. After the data is written to disk, the read data is returned to the master node.

[0139] The above are some specific implementation methods of the master node sending a processing request to the slave node set in an embodiment of the present application, and the slave node set updating the first data block and the check block according to the processing request. In other possible implementation methods of the embodiment of the present application, the master node and the slave node can also update the first data block and the check block through other method steps.

[0140] In some possible implementations, the master node may also receive request 5, which may be a read request, and then the master node may read the target data block according to the read request. It should be noted that in the EC stripe query scenario, request 5 may also be referred to as the fifth request. When the data block is stored in columns, the master node may read the target data block by column. Specifically, the read request may include a starting address, and further, the read request may also include the length of the read data. The master node may determine the target node from the data block storage node according to the above starting address, and then the master node may read the target data block from the target node by column.

[0141] In this way, when reading data, the required data can be read by reading the hard disk only once at one node, reducing read amplification. Figure 6 To illustrate, if you want to read 8KB or 16KB of data from the starting address, although it is still data block D0, data block D1, data block D2, and data block D3, it is only completed on the same disk of one machine, reducing the number of times data is read across disks.

[0142] Based on the above description, an embodiment of the present application provides a data processing method. This method divides the update process of the check block when updating the EC stripe into local verification and remote update, and optimizes the process in which the master node calculates the new check block and sends it to the check block storage node where the check block is located for updating, so that the master node or the update node calculates the check block update information, and the check block storage node where the check block is located generates a new check block according to the check block update information and writes the new check block. In this way, the master node or the update node is avoided from reading the check block from the check block storage node, the number of read operations is reduced, the network transmission overhead is reduced, and the system performance is guaranteed. Furthermore, the method supports converting row storage to column storage when writing data, so that when reading data, it can be completed in the same disk of a machine, reducing the number of cross-disk data reads and improving read performance.

[0143] Combination of the above Figures 1 to 9 The data processing method provided in this application is introduced. Next, the functions of the data processing device and data processing system provided in this application and the computing device or computing device cluster that implements the data processing device and data processing system are introduced in conjunction with the accompanying drawings.

[0144] First, see Figure 10 , shows a schematic structural diagram of a data processing device. The data processing device 1000 can be deployed in a master node in a distributed storage system. The device 1000 includes:

[0145] An acquiring unit 1002 is configured to acquire a first request for updating a data block in an erasure code (EC) stripe.

[0146] A determining unit 1004 is configured to determine the first data block according to the first request, where the first data block is a data block associated with the first request;

[0147] Communication unit 1006 is used to send a processing request to a slave node set, where the slave node set includes at least one slave node in the distributed storage system, and the processing request is used to instruct the master node to offload the operation of updating the data block to one or more slave nodes in the slave node set.

[0148] It should be understood that the device 1000 of the embodiment of the present invention can be implemented by a central processing unit (CPU), or by an application-specific integrated circuit (ASIC), or by a programmable logic device (PLD), wherein the PLD can be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a data processing unit (DPU), a system on chip (SoC) or any combination thereof. It can also be implemented by software. Figures 5 to 9 When the data processing method is shown, the apparatus 1000 and its modules may also be software modules.

[0149] In some possible implementations, the communication unit 1006 is specifically configured to:

[0150] sending a second request including the second data block to the first slave node;

[0151] receiving the first data block returned by the first slave node after updating the first data block to a second data block;

[0152] The determining unit 1004 is further configured to:

[0153] Determine check block update information according to the first data block and the second data block;

[0154] The communication unit 1006 is specifically configured to:

[0155] A third request including the check block update information is sent to the second slave node, where the check block update information is used to update the check block.

[0156] In some possible implementations, the communication unit 1006 is specifically configured to:

[0157] Sending a second request including the second data block to the first slave node, where the second request is used to instruct the first slave node to update the first data block to the second data block, and to determine parity block update information according to the first data block and the second data block;

[0158] A third request including the check block update information is sent to the second slave node through the first slave node, where the check block update information is used to update the check block.

[0159] In some possible implementations, the first data block is stored in a first slave node, and the master node and the first slave node are the same node; or, the check block in the EC stripe is stored in a second slave node, and the master node and the second slave node are the same node.

[0160] In some possible implementations, the acquiring unit 1002 is further configured to:

[0161] Before obtaining the first request, obtaining a fourth request including the data stream;

[0162] The apparatus 1000 further includes:

[0163] A read / write unit 1008 is configured to divide the data in the data stream into multiple data blocks, and write the multiple data blocks into data block storage nodes in the distributed storage system in columns, wherein the data block storage nodes include the master node and the first slave node;

[0164] The read-write unit 1008 is further configured to calculate a check block according to each group of data blocks in the plurality of data blocks, and write the check block into a check block storage node in the distributed storage system, wherein the check block storage node includes the second slave node;

[0165] When the multiple data blocks cannot fully write at least one EC stripe, the read-write unit is specifically configured to perform a null operation on slices without data in the at least one EC stripe.

[0166] In some possible implementations, the acquiring unit 1002 is further configured to:

[0167] Obtaining a fifth request including a starting address;

[0168] The read-write unit 1008 is further configured to:

[0169] The target node is determined according to the starting address, and the target data block is read column by column.

[0170] because Figure 10 The data processing device 1000 shown corresponds to Figure 5 、 Figure 8 、 Figure 9 The method shown, Figure 10 The specific implementation of the data processing device 1000 and its technical effects can be found in the relevant descriptions of the aforementioned embodiments and will not be elaborated here.

[0171] Then, see Figure 11 , shows a structural diagram of a data processing system, the data processing system 1100 includes a first data processing device 1000A and a second data processing device 1000B, the first data processing device 1000A is deployed in a master node in a distributed storage system, and the second data processing device 1000B is deployed in a slave node in the distributed storage system.

[0172] A first data processing device 1000A is configured to: obtain a first request, where the first request is used to update a first data block in an erasure code (EC) stripe; determine the first data block according to the first request, where the first data block is the data block associated with the first request; and send a processing request to a set of slave nodes, where the set of slave nodes includes at least one slave node in the distributed storage system; wherein the processing request is used to instruct to offload an operation of updating the data block by the master node to one or more slave nodes in the set of slave nodes;

[0173] The second data processing device 1000B is configured to update the first data block and check block according to the processing request.

[0174] In some possible implementations, the first data processing device 1000A is specifically configured to:

[0175] sending a second request including the second data block to the first slave node;

[0176] The second data processing device 1000B on the first slave node is specifically configured to:

[0177] Updating the first data block to the second data block, and returning the first data block;

[0178] The first data processing device 1000A is further configured to:

[0179] Determine check block update information according to the first data block and the second data block;

[0180] The first data processing device 1000A is specifically configured to:

[0181] Sending a third request including the check block update information to the second slave node;

[0182] The second data processing device 1000B on the second slave node is specifically configured to:

[0183] The check block is updated according to the check block update information.

[0184] In some possible implementations, the first data processing device 1000A is specifically configured to:

[0185] sending a second request including the second data block to the first slave node;

[0186] The second data processing device 1000B on the first slave node is specifically configured to:

[0187] Updating the first data block to a second data block;

[0188] The second data processing device 1000B on the first slave node is further configured to:

[0189] Determine check block update information according to the first data block and the second data block;

[0190] The first data processing device 1000A is specifically configured to:

[0191] Sending a third request including the check block update information to the second slave node through the first slave node;

[0192] The second data processing device 1000B on the second slave node is specifically configured to:

[0193] The check block is updated according to the check block update information.

[0194] In some possible implementations, the second request is an update request, and the update request is used to instruct the first slave node to update the first data block to the second data block and return the first data block.

[0195] because Figure 11 The data processing system 1100 shown corresponds to Figure 5 、 Figure 8 、 Figure 9 The method shown, Figure 11 The specific implementation of the data processing system 1100 and its technical effects can be found in the relevant descriptions of the aforementioned embodiments and will not be repeated here.

[0196] Figure 12This is a hardware structure diagram of a computing device 1200 provided in this application. The computing device 1200 may be the aforementioned master node, configured to implement the functions of the data processing apparatus 1000. The computing device 1200 may be a server or a terminal device. Terminal devices include, but are not limited to, desktop computers, laptops, tablet computers, or smartphones.

[0197] like Figure 12 As shown, computing device 1200 includes a bus 1202, a processor 1204, a memory 1206, and a communication interface 1208. Processor 1204, memory 1206, and communication interface 1208 communicate with each other via bus 1202. It should be understood that the present application does not limit the number of processors and memories in computing device 1200.

[0198] The bus 1202 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 12 The bus 1204 may include a path for transmitting information between various components of the computing device 1200 (eg, the memory 1206, the processor 1204, and the communication interface 1208).

[0199] The processor 1204 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0200] The memory 1206 may include a volatile memory, such as a random access memory (RAM). The processor 1204 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD). The memory 1206 stores executable program code, and the processor 1204 executes the executable program code to implement the aforementioned data processing method. Specifically, the memory 1206 stores instructions for the data processing device 1000 to execute the data processing method.

[0201] The communication interface 1208 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 1200 and other devices or a communication network.

[0202] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0203] like Figure 13 As shown, the computing device cluster includes at least one computing device 1200. The memory 1206 of one or more computing devices 1200 in the computing device cluster may store instructions of the same data processing system 1100 for executing the data processing method.

[0204] In some possible implementations, one or more computing devices 1200 in the computing device cluster may also be used to execute some of the instructions of the data processing system 1100 for executing the data processing method. In other words, the combination of one or more computing devices 1200 may jointly execute the instructions of the data processing system 1100 for executing the data processing method.

[0205] It should be noted that the memories 1206 in different computing devices 1200 in the computing device cluster may store different instructions for executing partial functions of the data processing system 1100 .

[0206] The present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the above-mentioned execution data processing method.

[0207] The present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored on any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the aforementioned data processing method.

[0208] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method, characterized in that: The method is executed by a master node in a distributed storage system, and includes: receiving a first request from a client, where the first request is used to update a data block in an erasure code (EC) stripe, and the master node is used to store the data block or check block in the EC stripe; determining a first data block according to the first request, where the first data block is a data block associated with the first request; A processing request is sent to a set of slave nodes, where the set of slave nodes includes one or more slave nodes, and the slave nodes are slave nodes in the distributed storage system. The processing request is used to instruct to offload the operation of updating the data block of the master node to one or more slave nodes in the set of slave nodes, wherein offloading the operation of updating the data block of the master node to one or more slave nodes in the set of slave nodes includes: partially or completely pushing down to one or more slave nodes the operators for calculating the new check block in the process of updating the data block of the master node.

2. The method according to claim 1, characterized in that The sending of the processing request to the set of slave nodes includes: sending a second request including a second data block to the first slave node; receiving the first data block returned by the first slave node after updating the first data block to a second data block; Determine check block update information according to the first data block and the second data block; A third request including the check block update information is sent to the second slave node, where the check block update information is used to update the check block.

3. The method according to claim 1, characterized in that The sending of the processing request to the set of slave nodes includes: Sending a second request including a second data block to the first slave node, where the second request is used to instruct the first slave node to update the first data block to the second data block, and to determine parity block update information based on the first data block and the second data block; A third request including the check block update information is sent to the second slave node through the first slave node, where the check block update information is used to update the check block.

4. The method according to any one of claims 1 to 3, characterized in that The first data block is stored in a first slave node, and the master node and the first slave node are the same node; or, The check blocks in the EC stripe are stored in the second slave node, and the master node and the second slave node are the same node.

5. The method according to any one of claims 1 to 3, characterized in that Before receiving the first request from the client, the method further includes: obtaining a fourth request including a data stream; Dividing the data in the data stream into multiple data blocks, and writing the multiple data blocks into data block storage nodes in the distributed storage system in columns, wherein the data block storage nodes include the master node and the first slave node; Calculating a check block according to each group of data blocks in the plurality of data blocks, and writing the check block into a check block storage node in the distributed storage system, wherein the check block storage node includes a second slave node; When the plurality of data blocks cannot fully write at least one EC stripe, a no-op is performed on slices without data in the at least one EC stripe.

6. The method according to claim 5, characterized in that The method further comprises: Obtaining a fifth request including a starting address; The target node is determined according to the starting address, and the target data block is read column by column.

7. A data processing device, characterized in that: The device is deployed in a master node in a distributed storage system, and includes: an acquiring unit, configured to receive a first request from a client, wherein the first request is used to update a data block in an erasure code (EC) stripe, and the master node is configured to store the data block or check block in the EC stripe; a determining unit, configured to determine a first data block according to the first request, where the first data block is a data block associated with the first request; A communication unit is used to send a processing request to a set of slave nodes, wherein the set of slave nodes includes one or more slave nodes, and the slave nodes are slave nodes in the distributed storage system. The processing request is used to instruct to offload the operation of the master node to update the data block to one or more slave nodes in the slave node set, wherein the operation of the master node to update the data block to one or more slave nodes in the slave node set includes: partially or completely pushing down to one or more slave nodes the operator for calculating the new check block in the process of the master node updating the data block.

8. The device according to claim 7, characterized in that The communication unit is specifically used for: sending a second request including a second data block to the first slave node; receiving the first data block returned by the first slave node after updating the first data block to a second data block; The determining unit is further configured to: Determine check block update information according to the first data block and the second data block; The communication unit is specifically used for: A third request including the check block update information is sent to the second slave node, where the check block update information is used to update the check block.

9. The device according to claim 7, characterized in that The communication unit is specifically used for: Sending a second request including a second data block to the first slave node, where the second request is used to instruct the first slave node to update the first data block to the second data block, and to determine parity block update information based on the first data block and the second data block; A third request including the check block update information is sent to the second slave node through the first slave node, where the check block update information is used to update the check block.

10. The device according to any one of claims 7 to 9, characterized in that The first data block is stored in a first slave node, and the master node and the first slave node are the same node; or, The check blocks in the EC stripe are stored in the second slave node, and the master node and the second slave node are the same node.

11. The device according to any one of claims 7 to 9, characterized in that The acquisition unit is further configured to: Before receiving the first request from the client, obtaining a fourth request including the data stream; The device further comprises: a read / write unit, configured to divide the data in the data stream into multiple data blocks, and write the multiple data blocks into data block storage nodes in the distributed storage system in columns, wherein the data block storage nodes include the master node and the first slave node; The read-write unit is further configured to calculate a check block according to each group of data blocks in the plurality of data blocks, and write the check block into a check block storage node in the distributed storage system, wherein the check block storage node includes the second slave node; When the multiple data blocks cannot fully write at least one EC stripe, the read-write unit is specifically configured to perform a null operation on slices without data in the at least one EC stripe.

12. The device according to claim 11, characterized in that The acquisition unit is further configured to: Obtaining a fifth request including a starting address; The read-write unit is further used for: The target node is determined according to the starting address, and the target data block is read column by column.

13. A computing device cluster, characterized in that: The computing device cluster includes at least one computing device, and the at least one computing device includes at least one processor and at least one memory, wherein the at least one memory stores computer-readable instructions; the at least one processor executes the computer-readable instructions so that the computing device cluster performs the method according to any one of claims 1 to 6.

14. A computer-readable storage medium, characterized in that The method comprises computer-readable instructions; the computer-readable instructions are used to implement the method according to any one of claims 1 to 6.

15. A computer program product, characterized in that The method comprises computer-readable instructions; the computer-readable instructions are used to implement the method according to any one of claims 1 to 6.

Images