Data processing method and related equipment

Abstract

The invention discloses a data processing method and related equipment, and relates to the technical field of storage. The method is executed by a master node in a distributed storage system, and comprises the following steps: acquiring a first request which is used for updating data blocks in an erasure code EC strip, then determining a first data block according to the first request, and sending a processing request to a slave node set, the operation of updating the data block by the master node is unloaded to one or more slave nodes in the slave node set. According to the method, the operation of updating the data block by the master node is unloaded to the slave node, so that the number of times of reading operation in the process of updating the EC strip is reduced, a large number of network resources are prevented from being occupied, and the system performance of the distributed storage system is guaranteed.

Description

[0001] This application is a divisional application. The application number of the original application is 202211017671.X, and the original application date is August 23, 2022. The entire contents of the original application are incorporated by reference into this application. This application and the Chinese application with application number 202211017671.X both claim the priority of the Chinese patent application with application number 202210740423.1, filed with the State Intellectual Property Office of China on June 27, 2022, and the invention name is “A method for data processing”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of storage technology, and in particular to a data processing method, a data processing apparatus, a computing device cluster, a computer-readable storage medium, and a computer program product. Background Art

[0003] With the continuous development of information technology, more and more industrial applications adopt information deployment methods, which generates a large amount of data. In order to reduce the storage cost of data and ensure the reliability of data, the industry has proposed the erasure code (EC) technology. EC specifically divides the data into multiple groups of data blocks, and then calculates the check blocks based on each group of data blocks, and stores the group of data blocks and check blocks in different nodes of the distributed storage system.

[0004] A set of data blocks and check blocks generated by checking based on this set of data blocks is called an EC stripe. To ensure data consistency, data reading and updating operations are usually completed by the node where a data block (for example, the first data block) in the EC stripe is located. Among them, the node where the first data block in the EC stripe is located can also be called the master node, and the nodes where other data blocks in the EC stripe and the nodes where the check blocks are located are called slave nodes.

[0005] However, when updating an EC stripe, for example, updating a data block in the EC stripe, the master node is usually required to perform multiple read and write operations. Multiple read and write operations can occupy a large amount of network resources and reduce the system performance of the distributed storage system. Summary of the invention

[0006] The present application provides a data processing method, which offloads the operation of updating a data block from a master node to a slave node, reduces the number of read operations during the update of an EC stripe, avoids occupying a large amount of network resources, and ensures the system performance of a distributed storage system. The present application also provides a data processing device, a computing device cluster, a computer-readable storage medium, and a computer program product corresponding to the above method.

[0007] In a first aspect, the present application provides a data processing method. The method can be executed by a master node in a distributed storage system. Specifically, the master node obtains a first request, the first request is used to update a data block in an EC stripe, and then determines a first data block based on the first request, the first data block is a data block associated with the first request, and then the master node sends a processing request to a slave node set including at least one slave node in the distributed storage system to instruct the master node to update the data block. The operation is unloaded to one or more slave nodes in the slave node set. In this way, the number of read operations in the process of updating the data block in the EC stripe can be reduced, the network transmission overhead can be reduced, and the system performance can be guaranteed.

[0008] In some possible implementations, the master node may send a second request including a second data block to the first slave node, then receive the first data block returned by the first slave node when the first data block is updated to the second data block, then determine the check block update information based on the first data block and the second data block, and then send a third request including the check block update information to the second slave node, wherein the check block update information is used to update the check block.

[0009] The method pushes down the operator part of calculating the new check block to the slave node, avoiding the master node from reading the check block from the second slave node where the check block is located, reducing the number of read operations, reducing network transmission overhead, and ensuring system performance.

[0010] In some possible implementations, the master node may send a second request including the second data block to the first slave node, the second request being used to instruct the first slave node to update the first data block to the second data block, and to determine the check block update information according to the first data block and the second data block. Then, the master node sends a third request including the check block update information to the second slave node through the first slave node, the check block update information being used to update the check block.

[0011] The method pushes down all operators for calculating new check blocks to slave nodes, specifically to the first slave node (also called update node) where the first data block is located and the second slave node where the check block is located, so as to avoid the first slave node reading the check block from the second slave node, reduce the number of read operations, reduce network transmission overhead, and ensure system performance.

[0012] In some possible implementations, the second request sent by the master node to the first slave node is an update request, and the return value of the update request is the first data block, which is used to instruct the first slave node to update the first data block to the second data block and return the first data block. In this way, the master node only needs one update operation to replace one read operation and one write operation in the related art, which reduces the number of operations, reduces network transmission overhead, and ensures system performance.

[0013] In some possible implementations, the first data block may be stored in a first slave node, and the master node may be the same node as the first slave node. Similarly, in other embodiments, the check block may be stored in a second master node, and the master node may be the same node as the second slave node.

[0014] In this way, the master node can read the first data block or check block locally, reducing the number of remote read operations, thereby reducing the occupied network resources and ensuring system performance.

[0015] In some possible implementations, before the master node obtains the first request, it may also obtain a fourth request including a data stream, and then the master node divides the data in the data stream into blocks to obtain multiple data blocks, and writes the multiple data blocks into columns in a data block storage node in the distributed storage system, the data block storage node including the master node and a first slave node, and then the master node calculates a check block based on each group of data blocks in the multiple data blocks, and writes the check block into a check block storage node in the distributed storage system, the check block storage node including a second slave node.

[0016] Among them, storing data blocks in a distributed storage system by column can reduce the number of times the disk is crossed during the subsequent data reading process and reduce the reading overhead.

[0017] In some possible implementations, when multiple data blocks obtained by segmenting the data stream cannot fill at least one EC stripe, the master node can also perform a null operation on the shards without data in at least one EC stripe without performing a fill operation, thereby reducing write amplification.

[0018] In some possible implementations, the master node may also obtain a fifth request including a start address, and then determine a target node according to the start address, and read the target data block from the target node by column. In this way, when reading data, the required data can be read by reading the hard disk only once at one node, thereby reducing read amplification.

[0019] In a second aspect, the present application provides a data processing method. The method is applied to a distributed storage system, comprising:

[0020] The master node obtains a first request, where the first request is used to update a first data block in an erasure code (EC) stripe, determines the first data block according to the first request, where the first data block is a data block associated with the first request, and sends a processing request to a slave node set, where the slave node set includes at least one slave node in the distributed storage system, and the processing request is used to instruct to offload an operation of updating the data block by the master node to one or more slave nodes in the slave node set;

[0021] The slave node set updates the first data block and the check block according to the processing request.

[0022] This method offloads the operation of updating data blocks of the master node to the set of slave nodes, reduces the number of read operations in the process of updating the data blocks of the EC stripe, reduces network transmission overhead, and ensures system performance.

[0023] In some possible implementations, the master node sends a processing request to the set of slave nodes, including:

[0024] The master node sends a second request including the second data block to the first slave node;

[0025] The first slave node updates the first data block according to the processing request, including:

[0026] The first slave node updates the first data block to a second data block, and returns the first data block;

[0027] The method further comprises:

[0028] The master node determines check block update information according to the first data block and the second data block;

[0029] The master node sends a processing request to the slave node set, including:

[0030] Sending a third request including check block update information to the second slave node;

[0031] The second slave node updates the check block according to the processing request, including:

[0032] The second slave node updates the check block according to the check block update information.

[0033] This method pushes down the operator part of calculating the new check block in the process of the master node updating the data block to the slave node, reducing the operation of reading the check block from the slave node where the check block is located, reducing network transmission overhead, and ensuring system performance.

[0034] In some possible implementations, the master node sends a processing request to the set of slave nodes, including:

[0035] The master node sends a second request including the second data block to the first slave node;

[0036] The first slave node updates the first data block according to the processing request, including:

[0037] The first slave node updates the first data block to a second data block;

[0038] The method further comprises:

[0039] The first slave node determines check block update information according to the first data block and the second data block;

[0040] The master node sends a processing request to the slave node set, including:

[0041] Sending a third request including the check block update information to the second slave node through the first slave node;

[0042] The second slave node updates the check block according to the processing request, including:

[0043] The second slave node updates the check block according to the check block update information.

[0044] This method pushes down all operators for calculating new check blocks in the process of updating data blocks by the master node to the slave nodes, reducing the operation of reading check blocks from the slave nodes where the check blocks are located, reducing network transmission overhead, and ensuring system performance.

[0045] In some possible implementations, the second request is an update request, and the update request is used to instruct the first slave node to update the first data block to the second data block and return the first data block. In this way, the master node only needs one update operation to replace one read operation and one write operation in the related art, thereby reducing the number of operations, reducing network transmission overhead, and ensuring system performance.

[0046] In a third aspect, the present application provides a data processing device, which includes various modules for executing the data processing method in the first aspect or any possible implementation of the first aspect.

[0047] In a third aspect, the present application provides a data processing device, which includes various units for executing the data processing method in the second aspect or any possible implementation of the second aspect.

[0048] In a fourth aspect, the present application provides a data processing system, which includes various devices for executing the data processing method in the second aspect or any possible implementation of the second aspect.

[0049] In a fifth aspect, the present application provides a computing device cluster. The computing device cluster includes at least one computing device, and the at least one computing device includes at least one processor and at least one memory. The at least one processor and the at least one memory communicate with each other. The at least one processor is used to execute instructions stored in the at least one memory, so that the computing device or the computing device cluster performs the data processing method described in any implementation of the first aspect or the second aspect.

[0050] In a sixth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, wherein the instructions instruct a computing device or a computing device cluster to execute the data processing method described in the first aspect or any one of the implementations of the first aspect.

[0051] In a seventh aspect, the present application provides a computer program product comprising instructions, which, when executed on a computing device or a computing device cluster, enables the computing device or the computing device cluster to execute the data processing method described in the first aspect or any one of the implementations of the first aspect.

[0052] Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A schematic diagram of a flow chart of EC strip update provided in this application;

[0054] Figure 2 A system architecture diagram of a distributed storage system provided for this application;

[0055] Figure 3 A system architecture diagram of a distributed storage system provided for this application;

[0056] Figure 4 A schematic diagram of an application scenario of a distributed storage system provided in this application;

[0057] Figure 5 A flowchart of a data processing method provided for this application;

[0058] Figure 6 A schematic diagram of row storage and column storage provided by this application;

[0059] Figure 7 A schematic diagram of a data writing process provided for this application;

[0060] Figure 8 A schematic diagram of a flow chart of EC strip update provided in this application;

[0061] Fig. 9 A schematic diagram of a flow chart of EC strip update provided in this application;

[0062] Fig.10 A schematic diagram of the structure of a data processing device provided in this application;

[0063] Fig.11 A schematic diagram of the structure of a data processing system provided for this application;

[0064] Fig.12A schematic diagram of the structure of a computing device cluster provided for this application;

[0065] Fig.13 A schematic diagram of the structure of a computing device cluster provided in this application. DETAILED DESCRIPTION

[0066] To facilitate understanding, some technical terms involved in the embodiments of the present application are first introduced.

[0067] EC stripe update, also referred to as EC stripe overwrite, specifically uses several new data blocks to replace several data blocks in the EC stripe, and based on the update of the data blocks, the check blocks in the EC stripe are updated accordingly. According to the different ways of generating new check blocks, EC stripe overwrite can also be divided into EC lowercase and EC uppercase. EC lowercase refers to reading the check block, the modified data block, and the modified data block to determine the new check block. EC uppercase refers to reading other data blocks in the EC stripe, and determining the new check block based on the modified data block and other data blocks in the EC stripe. When the overwrite data block is small, the amount of data read by the EC lowercase method is small and the efficiency is higher. When the overwrite data block is large, the amount of data read by the EC uppercase method is small and the efficiency is higher.

[0068] The following is an example of the EC stripe update process. Figure 1 The EC stripe update process diagram shown in FIG. 1 is a flow chart of an EC stripe update process. In this example, the EC stripe includes data blocks D0 to D1. k-1 And check block P, check block Q. The node where data block D0 is located is the master node, data blocks D1 to D k-1 The node is the first slave node, storing data blocks D0 to D k-1 The node is also called the data block storage node, and the node where the check block P and the check block Q are located is the second slave node, also called the check block storage node.

[0069] like Figure 1 As shown, the client requests data block D1 ′ Overwrite the data block D1 in the EC stripe. To ensure strong data consistency, it is usually necessary to read the data to be modified, such as data block D1, to the master node first, and read the check block P and check block Q to the master node. The master node reads the check block P and check block Q according to the data block D1 and data block D1. ′ Sum check block P to calculate the new check block P ′ , according to data block D1, data block D1 ′ Sum check block Q to calculate new check block Q ′ , as shown below:

[0070] P ′ =α0(D1′ -D1)+P (1)

[0071] Q ′ =β0(D1 ′ -D1)+Q (2)

[0072] Among them, α0 and β0 are different calibration coefficients.

[0073] Then, the master node sends data block D1 ′ , new check block P ′ , new check block Q ′ Write to the node where the data block D1, check block P, and check block Q are located. This results in that updating a data block in the EC stripe requires 3 read operations and 3 write operations, which occupies a large amount of network resources and reduces system performance.

[0074] In order to solve the problem that multiple read operations and write operations in traditional technologies occupy a large amount of network resources and cause system performance degradation, the present application provides a data processing method applied to a distributed storage system. Specifically, a master node in the distributed storage system obtains a first request, and the first request is used to update a data block in an EC stripe. The master node can determine a first data block based on the first request, and the first data block is a data block associated with the first request. Then the master node sends a processing request to a slave node set including at least one slave node in the distributed storage system to instruct the master node to update the data block. The operation is unloaded to one or more slave nodes in the slave node set.

[0075] This method offloads the operation of updating the data block of the master node to the set of slave nodes, for example, pushing down the operator for calculating the new check block to the second slave node where the check block is located, avoiding the master node or the first slave node (also called the update node) where the first data block is located from reading the check block from the second slave node, reducing the number of read operations, reducing network transmission overhead, and ensuring system performance. Different from other EC optimization technologies, this application focuses on changing the EC data transmission process and data distribution, enhancing data transmission and disk access efficiency, and is therefore applicable to a variety of storage scenarios with high availability.

[0076] Furthermore, the method also supports the optimization of the data block update process, for example, a read operation and a write operation can be combined into one read and write operation. In this way, only one read and write operation plus two write operations are needed to complete the update of the EC stripe, and the network transmission overhead is reduced by half, which greatly reduces the network resource occupancy rate and improves the system performance.

[0077] The system architecture of the embodiment of the present application is introduced below with reference to the accompanying drawings.

[0078] See also Figure 2The system architecture diagram of the distributed storage system shown in FIG. 1 is a storage-computing separation structure, and the system includes a computing node cluster and a storage node cluster. The computing node cluster includes one or more computing nodes 110 ( Figure 2 Two computing nodes 110 are shown in the figure, but are not limited to two computing nodes 110), and each computing node 110 can communicate with each other. The computing node 110 is a computing device, such as a server, a desktop computer, or a controller of a storage array. In terms of hardware, Figure 2 As shown, the computing node 110 includes at least a processor 112, a memory 113 and a network card 114. Among them, the processor 112 is a central processing unit (CPU), which is used to process data access requests from outside the computing node 110, or requests generated inside the computing node 110. Exemplarily, when the processor 112 receives write data requests sent by users, it temporarily saves the data in these write data requests in the memory 113. When the total amount of data in the memory 113 reaches a certain threshold, the processor 112 sends the data stored in the memory 113 to the storage node 100 for persistent storage. In addition, the processor 112 is also used for data calculation or processing, such as metadata management, deduplication, data compression, virtualized storage space, and address translation. Figure 2 Only one CPU 112 is shown in the figure. In actual applications, there are often multiple CPUs 112, wherein one CPU 112 has one or more CPU cores. This embodiment does not limit the number of CPUs and the number of CPU cores.

[0079] Memory 113 refers to an internal memory that directly exchanges data with the processor. It can read and write data at any time, and the speed is very fast. It serves as a temporary data storage for the operating system or other running programs. Memory includes at least two types of memory. For example, memory can be either random access memory or read-only memory (ROM). For example, random access memory is dynamic random access memory (DRAM) or storage class memory (SCM). DRAM is a semiconductor memory, and like most random access memories (RAM), it is a volatile memory device. SCM is a composite storage technology that combines the characteristics of traditional storage devices and memory. Storage class memory can provide faster read and write speeds than hard disks, but the access speed is slower than DRAM, and the cost is cheaper than DRAM. However, DRAM and SCM are only exemplary descriptions in this embodiment, and memory can also include other random access memories, such as static random access memory (SRAM). As for the read-only memory, for example, it can be a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), etc. In addition, the memory 113 can also be a dual in-line memory module or a dual-line memory module (DIMM), that is, a module composed of a dynamic random access memory (DRAM), or a solid state disk (SSD). In actual applications, multiple memories 113 and different types of memories 113 can be configured in the computing node 110. This embodiment does not limit the number and type of memory 113. In addition, the memory 113 can be configured to have a power-saving function. The power-saving function means that when the system loses power and then powers on again, the data stored in the memory 113 will not be lost. The memory with a power-saving function is called a non-volatile memory.

[0080] The network card 114 is used to communicate with the storage node 100. For example, when the total amount of data in the memory 113 reaches a certain threshold, the computing node 110 can send a request to the storage node 100 through the network card 114 to store the data persistently. In addition, the computing node 110 can also include a bus for communication between the components inside the computing node 110. Figure 2 The main function of the computing node 110 in the system is computing services. When storing data, remote storage can be used to achieve persistent storage. Therefore, it has less local storage than a conventional server, thereby saving costs and space. However, this does not mean that the computing node 110 cannot have local storage. In actual implementation, the computing node 110 can also have a small number of hard disks built in or externally connected.

[0081] Any computing node 110 can access any storage node 100 in the storage node cluster through the network. The storage node cluster includes multiple storage nodes 100 ( Figure 2 Three storage nodes 100 are shown in the figure, but are not limited to three storage nodes 100). A storage node 100 includes one or more controllers 101, a network card 104 and multiple hard disks 105. The network card 104 is used to communicate with the computing node 110. The hard disk 105 is used to store data and can be a disk or other types of storage media, such as a solid-state hard disk or a shingled magnetic recording hard disk. The controller 101 is used to write data to the hard disk 105 or read data from the hard disk 105 according to the read / write data request sent by the computing node 110. In the process of reading and writing data, the controller 101 needs to convert the address carried in the read / write data request into an address that can be recognized by the hard disk. It can be seen that the controller 101 also has some simple computing functions.

[0082] Figure 3 This is a system architecture diagram of another distributed storage system used in the embodiment of the present application. The system is a storage-computing integrated architecture, and the system includes a storage cluster. The storage cluster includes one or more servers 110 ( Figure 3 Three servers 110 are shown in the figure, but not limited to three servers 110), and each server 110 can communicate with each other. The server 110 is a device with both computing and storage capabilities, such as a server, a desktop computer, etc. For example, an ARM server or an X86 server can be used as the server 110 here.

[0083] In hardware, such as Figure 3As shown, the server 110 includes at least a processor 112, a memory 113, a network card 114 and a hard disk 105. The processor 112, the memory 113, the network card 114 and the hard disk 105 are connected by a bus. Among them, the processor 112 and the memory 113 are used to provide computing resources. Specifically, the processor 112 is a central processing unit CPU, which is used to process data access requests from the outside of the server 110 (application server or other server 110), and is also used to process requests generated inside the server 110. The memory 113 refers to an internal memory that directly exchanges data with the processor. It can read and write data at any time, and the speed is very fast, as a temporary data storage for the operating system or other running programs. In actual applications, multiple memories 113 and different types of memories 113 can be configured in the server 110. This embodiment does not limit the number and type of memory 113. In addition, the memory 113 can be configured to have a power conservation function. The hard disk 105 is used to provide storage resources, such as storing data. It can be a disk or other types of storage media, such as a solid state drive or a shingled magnetic recording hard disk. The network card 114 is used to communicate with other servers 110 .

[0084] It should be noted that the above Figure 2 , Figure 3 This is only a schematic architecture of a distributed storage system. In other possible implementations of the embodiments of the present application, the distributed storage system may also use other architectures. For example, the distributed storage system may also adopt a fully integrated architecture or a Memory Fabric architecture.

[0085] Furthermore, the above-mentioned distributed storage system can provide storage services, for example, providing a storage server to users in the form of a storage interface, so that users can use the storage resources of the distributed storage system through the above-mentioned storage interface. Figure 4 The distributed storage system shown in the figure is an application scenario diagram. A user can access the distributed storage system through a client (such as an application client). The distributed storage system can be the above-mentioned Figure 2 or Figure 3 The architecture shown.

[0086] For example, distributed storage systems use Figure 2In the architecture shown, the client can call the storage interface provided by the storage service to generate a storage request, and send the storage request to the distributed storage system. The computing node that receives the storage request in the distributed storage system can divide the data into multiple groups of data blocks, and then calculate the check blocks based on each group of data blocks, and write each group of data blocks and the check blocks determined by the group of data blocks to different storage nodes 100 in a dispersed manner, for example, to the hard disks 105 of different storage nodes 100, to form an EC stripe. The storage node 100 that stores the first data block (for example, data block D0) in the EC stripe can be regarded as the master node, and the storage node 100 that stores other data blocks in the EC stripe (for example, data blocks D1, ... data blocks D2) can be regarded as the master node. k-1 ) can be regarded as the first slave node, and the storage node 100 storing the check blocks (for example, the check block P and the check block Q) can be regarded as the second slave node.

[0087] For example, distributed storage systems use Figure 3 In the architecture shown, the client can call the storage interface provided by the storage service to generate a storage request, and send the storage request to the distributed storage system. The server 110 in the distributed storage system can divide the data into multiple groups of data blocks, and then calculate the check blocks according to each group of data blocks, and write each group of data blocks and the check blocks determined by the group of data blocks to different servers 110 in a dispersed manner, for example, to the hard disks 105 of different servers 100. The server 110 storing the first data block (for example, data block D0) in the EC stripe can be regarded as the master node, and the server 110 storing other data blocks (for example, data blocks D1, ... data blocks D2) in the EC stripe can be regarded as the master node. k-1 ) can be regarded as the first slave node, and the server 110 storing the check blocks (for example, the check block P and the check block Q) can be regarded as the second slave node.

[0088] The client is used to access the distributed storage system through the storage service, and the distributed storage system can respond to the user's access to the distributed storage system through the storage service and return the access result. The access result can be different depending on the access operation. For example, when the access operation is a write operation, the access result can represent a notification of successful write, and when the access operation is a read operation, the access result can be a read data block.

[0089] For the scenario of overwriting the EC stripe, the master node may obtain a first request for updating a data block of the EC stripe, determine the first data block according to the first request, and then send a processing request to the slave node set to instruct the master node to offload the operation of updating the data block to one or more slave nodes in the slave node set. For example, the processing request may instruct the master node to offload the operation of updating the data block to the first slave node where the first data block is located and the second slave node where the check block is located.

[0090] It should be noted that the first data block and the check block may be stored in different nodes other than the master node in the distributed storage system. In some embodiments, the first data block may also be stored in the master node, in which case the master node and the first slave node are the same node. In other embodiments, the check block may also be stored in the master node, in which case the master node and the second slave node may be the same node.

[0091] Next, the data processing method of an embodiment of the present application is introduced using a scenario in which a master node, a first slave node, and a second slave node are different nodes.

[0092] See also Figure 5 The data processing method shown is a flowchart, the method comprising:

[0093] S502: The master node obtains request 1.

[0094] Request 1 includes a data stream. Request 1 is used to request that the data in the data stream be written to the distributed storage system for persistent storage. Request 1 may be generated by an application client based on business needs. Request 1 may be a write request or other request that requires writing data. Depending on the business needs of the application client, Request 1 may include different types of data streams. For example, when the application client is a short video application or a long video application, Request 1 may include a video data stream; for another example, when the application client is a file management application or a text editing application, Request 1 may include a text data stream. The master node may receive Request 1 sent by the application client to persistently store the data in the data stream carried by Request 1.

[0095] S504: The master node divides the data in the data stream included in request 1 into blocks to obtain multiple data blocks.

[0096] The data stream may be an ordered sequence of bytes with a start point and an end point. Specifically, the master node may adopt fixed-length chunking or variable-length chunking to chunk the data in the data stream carried by request 1, thereby obtaining a plurality of data chunks. Among them, fixed-length chunking refers to chunking the data in the data stream according to a set chunking granularity. Variable-length chunking is to divide the data in the data stream into data chunks of unfixed size, and variable-length chunking may include variable-length chunking based on sliding windows and variable-length chunking based on content (content-defined chunking, CDC).

[0097] For ease of understanding, the following example is explained using fixed-length blocks. Specifically, when the size of the data in the data stream is an integer multiple of the block granularity, the master node can evenly divide the data in the data stream into multiple data blocks. When the size of the data in the data stream is not an integer multiple of the block granularity, the master node can pad the data in the data stream, for example, by padding the end of the data stream with zeros, so that the size of the data in the padded data stream is an integer multiple of the block granularity, and then the master node evenly divides the data in the data stream into multiple data blocks according to the block granularity. For example, the size of the data in the data stream is 20KB, and the master node blocks it according to the block granularity of 4KB, and 5 data blocks of 4KB in size can be obtained.

[0098] In some embodiments, when the size of the data in the data stream is not an integer multiple of the block granularity, the master node may not fill the data in the data stream, but instead divide it into K-1 data blocks with a size equal to the block granularity and one data block with a size not equal to the block granularity.

[0099] It should be noted that the data processing method of the embodiment of the present application may not execute the above S504. For example, if the size of the data in the data stream is small and insufficient for block division, or the data in the data stream has been pre-blocked, the above S504 may not be executed.

[0100] S506: The master node writes the multiple data blocks into the data block storage nodes including the master node and the first slave node in columns.

[0101] Assuming that the data block storage node can store L data blocks in each column, where L is a positive integer, the master node can first write multiple data blocks to the master node by column, and when the column in the master node is full, the remaining data blocks are written to the first slave node by column.

[0102] When there are multiple first slave nodes, the master node can write the remaining data blocks to the first first slave node by column. Similarly, when the first slave node is full, if there are still data blocks left, the master node writes the remaining data blocks to the next first slave node by column.

[0103] For ease of understanding, the following is an example. Figure 6The schematic diagram of row storage and column storage is shown, where row storage refers to storage by row and column storage refers to storage by column. In this example, the size of each data block is 4K, the data block storage node can store 256 data blocks in each column, the number of data block storage nodes is 4, and the number of check block storage nodes is 2. If the master node writes to the data block storage node by row, then data blocks D0 to D3 are written to 4 data block storage nodes respectively, specifically one master node and 3 first slave nodes, and P0 and Q0 are written to different check block storage nodes respectively. Similarly, data blocks D4 to D7 are written to 4 data block storage nodes respectively, and P1 and Q1 are written to different check block storage nodes respectively. If the master node writes to the data block storage node by column, then data blocks D0 to D 255 Write to the master node, data block D 256 To D 511 Write to the first slave node, and so on, data block D 512 To D 767 Write to the second first slave node, data block D 768 To D 1023 Write to the third first slave node.

[0104] In some possible implementations, multiple data blocks may not be able to fill up an EC stripe. For example, when each column stores 256 data blocks and the number of data block storage nodes is 4, if the number of data blocks in the data stream is less than 769 (256*3+1), then at least one data block in the data stream is not enough to fill up an EC stripe. Write amplification can be reduced by omitting writing the empty part. In specific implementations, the master node can perform a null operation (denoted as zero Op) on a chunk without data in at least one EC stripe, without having to perform a fill operation, thereby reducing write amplification.

[0105] like Figure 7 As shown in the figure, when the master node writes a data block in chunk1, it can add zero Op to the transaction 1 (recorded as transaction1) corresponding to the master node, and the write Op in transactions 2 to 4 (recorded as transaction2~4) are all replaced by zero Op. Accordingly, chunk2~4 will not actually allocate space, and no actual data will be written to the disk. When reading, it can request the upper layer to return the correct data, and the disk will not be actually read. In this way, write amplification can be reduced.

[0106] Based on this, the master node can first determine the size of the data stream. If the size of the data in the data stream is not enough to fill the stripe, only the chunks that need to be filled can be written, and the idle chunks are not filled. This can not only improve the write performance, but also reduce space waste.

[0107] S508: The master node calculates a check block according to each group of data blocks in the multiple data blocks.

[0108] Specifically, the master node can group multiple data blocks. For example, the master node can group multiple data blocks according to the rows where the data blocks are located. The rows where the same group of data blocks are located have the same row number. Then, the master node can calculate each group of data blocks according to the verification algorithm to generate a verification block. The master node can use different verification algorithms to generate different verification blocks. For ease of understanding, we will still use Figure 6 The process of calculating the check block is described with an example.

[0109] In this example, when the master node writes data blocks by column, the master node can 256 , data block D 512 , data block D 768 The check block P0 and the check block Q0 are calculated. Similarly, the master node can calculate the check block P0 and the check block Q0 based on the data block D2 and the data block D 257 , data block D 513 , data block D 769 The check block P1 and the check block Q1 are obtained by calculation.

[0110] In the embodiment of the present application, the data distribution method can be adjusted from row storage to column storage. In this way, data blocks with adjacent addresses can be concentrated on the same disk, such as data block D0 and data block D1 on the same disk. Accordingly, an EC stripe can include data blocks in different data segments, rather than continuous data blocks in one data segment. Figure 6 As shown, an EC stripe may include data blocks D0, D 256 , data block D 512 , data block D 768 When the disk or node where data block D0 is located fails and causes data block D0 to be lost, the data block D0 can be 256 , data block D 512 , data block D 768 The above data block D0 is restored by summing up the check blocks P0 and Q0.

[0111] S510: The master node writes the check block into the check block storage nodes including the second slave node.

[0112] When there are multiple check block storage nodes, that is, when there are multiple second slave nodes, the master node may write the check blocks into the second slave nodes corresponding to each of them.

[0113] It should be noted that the above S506 and S508 can be executed in sequence according to the set order, and then S510 is executed. In some embodiments, the above S506 and S508 can be executed in parallel, and then S510 is executed. In other embodiments, S506 and S510 can also be executed in parallel, for example, S508 can be executed first to obtain the check block, and then the data block and the check block can be written to the corresponding node in parallel. The embodiment of the present application does not limit the order of the above S506, S508, and S510.

[0114] It should also be noted that the above S502 to S510 are optional steps of the embodiment of the present application, and the data processing method of the embodiment of the present application may not perform the above steps. For example, the data processing method of the embodiment of the present application may directly perform the following steps to update the EC stripe. The following is a detailed description.

[0115] S511: The master node obtains request 2.

[0116] Request 2 is used to update the data block in the EC stripe, for example, request 2 is used to update the first data block in the EC stripe to the second data block. Request 2 includes the second data block. In some embodiments, request 2 may also include the logical address of the first data block for fast addressing of the first data block.

[0117] S512: The master node determines the first data block according to request 2.

[0118] The first data block is specifically a data block associated with request 2. Specifically, the master node may parse request 2, obtain a logical address of a data block to be updated in request 2, and determine the first data block according to the logical address.

[0119] S514: The master node sends a request 3 to the first slave node where the first data block is located.

[0120] S516: The first slave node where the first data block is located updates the first data block to the second data block.

[0121] S518: The master node receives the first data block returned by the first slave node where the first data block is located.

[0122] S520: The master node determines the check block update information according to the first data block and the second data block.

[0123] S522: The master node sends request 4 to the second slave node where the check block is located.

[0124] S524: The second slave node updates the check block according to the check block update information in request 4.

[0125] In the scenario of updating the EC stripe, request 2 can also be called the first request, and requests 3 and 4 can also be collectively referred to as processing requests. A processing request is a request sent by the master node to a set of slave nodes, wherein request 3 can be called the second request and request 4 can be called the third request. In the scenario of building an EC stripe, request 1 can also be called the fourth request.

[0126] exist Figure 5 In the example of FIG. 1 , request 3 and request 4 are used to instruct to partially offload the operation of updating the data block of the master node to the first slave node and the second slave node. The offloading process is described in detail below.

[0127] The request 3 sent by the master node to the first slave node includes the second data block, and the request 3 is specifically used to instruct the first slave node to update the first data block to the second data block. Considering that when the first data block in the EC stripe is updated, the check block will also change accordingly, the master node can read the first data block according to request 3 to calculate the new check block.

[0128] It should be noted that request 3 can be an update request, and the return value of the update request is the first data block. In this way, the first slave node where the first data block is located can read the first data block and then write the second data block when updating the first data block. In addition, the first slave node can also return the first data block to the master node. In this way, the second data block is written and the first data block is read through an update operation (specifically a read-write operation). In some possible implementations, the master node can also send an additional request to read the first data block for calculating the check block update information.

[0129] The master node receives the first data block returned by the first slave node where the first data block is located, and can determine the check block update information based on the first data block and the second data block by using the EC algorithm. For example, the master node can determine the check block update information based on the first data block and the second data block by using formula (1) or formula (2).

[0130] The request 4 sent by the master node to the second slave node includes the check block update information. The request 4 is specifically used to update the check block. The second slave node can update the check block according to the check block update information in the request 4. For example, the second slave node can read the check block, determine a new check block according to the check block and the check block update information, and then store the new check block, thereby updating the check block.

[0131] Different from the traditional method in which the check block is read to the master node, the master node calculates a new check block based on the first data block, the second data block and the check block, and then sends the new check block to the check block storage node for updating, the embodiment of the present application offloads part of the operation of updating the data block to the first slave node and the second slave node. Specifically, the process of updating the check block in the operation of updating the data block is decomposed into two steps, which are completed by different nodes.

[0132] Specifically, the master node can complete the previous step, which is to calculate the check block update information based on the first data block and the second data block, and then send the check block update information to the check block storage node. The check block storage node completes the next step, which is to update the check block according to the check block update information.

[0133] To facilitate understanding, a specific example is provided below for explanation.

[0134] like Figure 8 As shown, the master node completes the check block update information P″=α0(D2 ′ 56 -D 256 ) and check block update information Q″=β0(D2 ′ 56 -D 256 ), and then the master node pushes the check block update information P″ and the check block update information Q″ down to the check block storage node. The check block storage node completes the new check block P ′ =P″+P and new check block Q ′ =Q″+Q. And D2 ′ 56 and D 256 The read-then-write operation of the new and old data becomes an update operation. Specifically, the update request carries the data to be written, but before writing, the data at the original address is read as the return value of the request. Then, after the data is written to the disk, the read data is returned to the master node.

[0135] It should also be noted that the data processing method of the embodiment of the present application may not execute the above S516 to S518. For example, request 3 and request 4 may be used to indicate that all operations of updating the data block of the master node are offloaded to the first slave node where the first data block is located and the second slave node where the check block is located.

[0136] For example, the first slave node (i.e., the update node) where the first data block is located can directly calculate the check block update information based on the read first data block and the second data block. The master node can send request 4 through the first slave node, and the first slave node carries the check block update information in request 4 and pushes it down to the second slave node where the check block is located, so that the second slave node calculates a new check block based on the check block update information in request 4, thereby implementing the check block update.

[0137] For ease of understanding, the following still updates the data block D in the EC stripe. 256 Provide an example.

[0138] like Fig. 9 As shown, the first slave node completes the check block update information P″=α0(D2 ′ 56 -D 256 ) and check block update information Q″=β0(D2 ′ 56 -D 256 ), and then the first slave node pushes the check block update information P″ and the check block update information Q″ down to the check block storage node. The second slave node completes the new check block P ′ =P″+P and new check block Q ′ =Q″+Q. In addition, only the result of the operation is needed at the master node, and the unused data does not need to be read to the master node, thus reducing data transmission. ′ 56 and D 256 The read-then-write operation of the new and old data becomes an update operation. Specifically, the update request carries the data to be written, but before writing, the data at the original address is read as the return value of the request. Then, after the data is written to the disk, the read data is returned to the master node.

[0139] The above are some specific implementation methods in which the master node sends a processing request to a set of slave nodes in an embodiment of the present application, and the set of slave nodes updates the first data block and the check block according to the processing request. In other possible implementation methods of the embodiment of the present application, the master node and the slave node may also update the first data block and the check block through other method steps.

[0140] In some possible implementations, the master node may also receive request 5, which may be a read request, and then the master node may read the target data block according to the read request. It should be noted that in the EC stripe query scenario, request 5 may also be referred to as the fifth request. When the data block is stored in columns, the master node may read the target data block by column. Specifically, the read request may include a starting address, and further, the read request may also include the length of the read data. The master node may determine the target node from the data block storage node according to the above starting address, and then the master node may read the target data block by column from the target node.

[0141] In this way, when reading data, the required data can be read by reading the hard disk only once at one node, reducing read amplification. Figure 6 To illustrate, if you want to read 8KB or 16KB of data from the starting address, although it is still data block D0, data block D1, data block D2, and data block D3, it is only completed on the same disk of one machine, reducing the number of times data is read across disks.

[0142] Based on the above description, an embodiment of the present application provides a data processing method. This method divides the update process of the check block when updating the EC stripe into local verification and remote update, and optimizes the process in which the master node calculates the new check block and sends it to the check block storage node where the check block is located for updating, so that the master node or the update node calculates the check block update information, and the check block storage node where the check block is located generates a new check block according to the check block update information and writes the new check block. In this way, the master node or the update node is avoided from reading the check block from the check block storage node, the number of read operations is reduced, the network transmission overhead is reduced, and the system performance is guaranteed. Furthermore, the method supports converting row storage to column storage when writing data, so that when reading data, it can be completed in the same disk of a machine, reducing the number of cross-disk data readings and improving read performance.

[0143] Combination of the above Figures 1 to 9 The data processing method provided in the present application is introduced, and then the functions of the data processing device and data processing system provided in the present application and the computing device or computing device cluster that implements the data processing device and data processing system are introduced in conjunction with the accompanying drawings.

[0144] First, see Fig.10 , shows a schematic diagram of the structure of a data processing device, the data processing device 1000 can be deployed in a master node in a distributed storage system, and the device 1000 includes:

[0145] An acquiring unit 1002 is configured to acquire a first request, where the first request is used to update a data block in an erasure code (EC) stripe;

[0146] A determining unit 1004 is configured to determine the first data block according to the first request, where the first data block is a data block associated with the first request;

[0147] The communication unit 1006 is used to send a processing request to a slave node set, wherein the slave node set includes at least one slave node in the distributed storage system, and the processing request is used to indicate that the operation of updating the data block of the master node is offloaded to one or more slave nodes in the slave node set.

[0148] It should be understood that the device 1000 of the embodiment of the present invention can be implemented by a central processing unit (CPU), an application-specific integrated circuit (ASIC), or a programmable logic device (PLD), and the above PLD can be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a data processing unit (DPU), a system on chip (SoC) or any combination thereof. It can also be implemented by software Figures 5 to 9 When the data processing method is shown, the device 1000 and its various modules may also be software modules.

[0149] In some possible implementations, the communication unit 1006 is specifically configured to:

[0150] Sending a second request including the second data block to the first slave node;

[0151] receiving the first data block returned by the first slave node after updating the first data block to a second data block;

[0152] The determining unit 1004 is further configured to:

[0153] Determine check block update information according to the first data block and the second data block;

[0154] The communication unit 1006 is specifically used for:

[0155] A third request including the check block update information is sent to the second slave node, where the check block update information is used to update the check block.

[0156] In some possible implementations, the communication unit 1006 is specifically configured to:

[0157] Sending a second request including the second data block to the first slave node, where the second request is used to instruct the first slave node to update the first data block to the second data block, and to determine check block update information according to the first data block and the second data block;

[0158] A third request including the check block update information is sent to the second slave node through the first slave node, where the check block update information is used to update the check block.

[0159] In some possible implementations, the first data block is stored in a first slave node, and the master node and the first slave node are the same node; or, the check block in the EC stripe is stored in a second slave node, and the master node and the second slave node are the same node.

[0160] In some possible implementations, the acquiring unit 1002 is further configured to:

[0161] Before obtaining the first request, obtaining a fourth request including the data stream;

[0162] The device 1000 further includes:

[0163] A read-write unit 1008, configured to divide the data in the data stream into blocks to obtain a plurality of data blocks, and write the plurality of data blocks into data block storage nodes in the distributed storage system in columns, wherein the data block storage nodes include the master node and the first slave node;

[0164] The read-write unit 1008 is further configured to calculate a check block according to each group of data blocks in the plurality of data blocks, and write the check block into a check block storage node in the distributed storage system, wherein the check block storage node includes the second slave node;

[0165] When the multiple data blocks cannot fully write at least one EC stripe, the read-write unit is specifically configured to perform a null operation on the slices without data in the at least one EC stripe.

[0166] In some possible implementations, the acquiring unit 1002 is further configured to:

[0167] obtaining a fifth request including a start address;

[0168] The read-write unit 1008 is also used for:

[0169] The target node is determined according to the start address, and the target data block is read column by column.

[0170] because Fig.10 The data processing device 1000 shown corresponds to Figure 5 , Figure 8 , Fig. 9 The method shown, therefore Fig.10 The specific implementation method of the data processing device 1000 shown and its technical effects can be found in the relevant descriptions of the aforementioned embodiments, and will not be elaborated here.

[0171] Then, see Fig.11 , shows a structural diagram of a data processing system, the data processing system 1100 includes a first data processing device 1000A and a second data processing device 1000B, the first data processing device 1000A is deployed in a master node in a distributed storage system, and the second data processing device 1000B is deployed in a slave node in the distributed storage system.

[0172] A first data processing device 1000A is configured to: obtain a first request, the first request being used to update a first data block in an erasure code (EC) stripe, determine the first data block according to the first request, the first data block being a data block associated with the first request, send a processing request to a slave node set, the slave node set including at least one slave node in the distributed storage system, the processing request being used to instruct to offload an operation of the master node to update the data block to one or more slave nodes in the slave node set;

[0173] The second data processing device 1000B is used to update the first data block and the check block according to the processing request.

[0174] In some possible implementations, the first data processing device 1000A is specifically configured to:

[0175] Sending a second request including the second data block to the first slave node;

[0176] The second data processing device 1000B on the first slave node is specifically configured to:

[0177] Updating the first data block to the second data block, and returning the first data block;

[0178] The first data processing device 1000A is further configured to:

[0179] Determine check block update information according to the first data block and the second data block;

[0180] The first data processing device 1000A is specifically used for:

[0181] Sending a third request including the check block update information to the second slave node;

[0182] The second data processing device 1000B on the second slave node is specifically configured to:

[0183] The check block is updated according to the check block update information.

[0184] In some possible implementations, the first data processing device 1000A is specifically configured to:

[0185] Sending a second request including the second data block to the first slave node;

[0186] The second data processing device 1000B on the first slave node is specifically configured to:

[0187] Updating the first data block to a second data block;

[0188] The second data processing device 1000B on the first slave node is further configured to:

[0189] Determine check block update information according to the first data block and the second data block;

[0190] The first data processing device 1000A is specifically configured to:

[0191] Sending a third request including the check block update information to the second slave node through the first slave node;

[0192] The second data processing device 1000B on the second slave node is specifically configured to:

[0193] The check block is updated according to the check block update information.

[0194] In some possible implementations, the second request is an update request, and the update request is used to instruct the first slave node to update the first data block to the second data block and return the first data block.

[0195] because Fig.11 The data processing system 1100 shown corresponds to Figure 5 , Figure 8 , Fig. 9 The method shown, therefore Fig.11 The specific implementation method of the data processing system 1100 shown and its technical effects can be found in the relevant descriptions of the aforementioned embodiments, and will not be repeated here.

[0196] Fig.12The hardware structure diagram of a computing device 1200 provided in the present application, the computing device 1200 may be the aforementioned master node, and is used to implement the functions of the data processing device 1000. The computing device 1200 may be a server or a terminal device. The terminal device includes but is not limited to a desktop computer, a laptop computer, a tablet computer or a smart phone.

[0197] like Fig.12 As shown, computing device 1200 includes: bus 1202, processor 1204, memory 1206 and communication interface 1208. Processor 1204, memory 1206 and communication interface 1208 communicate through bus 1202. It should be understood that the present application does not limit the number of processors and memories in computing device 1200.

[0198] The bus 1202 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.12 The bus 1204 may include a path for transmitting information between various components of the computing device 1200 (eg, the memory 1206, the processor 1204, and the communication interface 1208).

[0199] The processor 1204 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0200] The memory 1206 may include a volatile memory, such as a random access memory (RAM). The processor 1204 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid state drive (SSD). The memory 1206 stores executable program code, and the processor 1204 executes the executable program code to implement the aforementioned data processing method. Specifically, the memory 1206 stores instructions for the data processing device 1000 to execute the data processing method.

[0201] The communication interface 1208 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 1200 and other devices or communication networks.

[0202] The present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0203] like Fig.13 As shown, the computing device cluster includes at least one computing device 1200. The memory 1206 in one or more computing devices 1200 in the computing device cluster may store the same data processing system 1100 for executing instructions of the data processing method.

[0204] In some possible implementations, one or more computing devices 1200 in the computing device cluster may also be used to execute some instructions of the data processing system 1100 for executing the data processing method. In other words, a combination of one or more computing devices 1200 may jointly execute instructions of the data processing system 1100 for executing the data processing method.

[0205] It should be noted that the memory 1206 in different computing devices 1200 in the computing device cluster may store different instructions for executing partial functions of the data processing system 1100 .

[0206] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the above-mentioned execution data processing method.

[0207] The embodiment of the present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the above data processing method.

[0208] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method, characterized in that: The method is executed by a master node in a distributed storage system, and includes: Obtain a first request, where the first request is used to update a data block in an erasure code (EC) stripe; determining a first data block according to the first request, where the first data block is a data block associated with the first request; A processing request is sent to a slave node set, wherein the slave node set includes one or more slave nodes, and the slave node is a slave node in the distributed storage system. The processing request is used to instruct the master node to offload the operation of updating the data block to one or more slave nodes in the slave node set.

2. The method according to claim 1, characterized in that The sending of the processing request to the slave node set includes: Sending a second request including the second data block to the first slave node; receiving the first data block returned by the first slave node after updating the first data block to a second data block; Determine check block update information according to the first data block and the second data block; A third request including the check block update information is sent to the second slave node, where the check block update information is used to update the check block.

3. The method according to claim 1, characterized in that The sending of the processing request to the slave node set includes: Sending a second request including the second data block to the first slave node, where the second request is used to instruct the first slave node to update the first data block to the second data block, and to determine check block update information according to the first data block and the second data block; A third request including the check block update information is sent to the second slave node through the first slave node, where the check block update information is used to update the check block.

4. The method according to any one of claims 1 to 3, characterized in that The first data block is stored in a first slave node, and the master node and the first slave node are the same node; or, The check blocks in the EC stripe are stored in the second slave node, and the master node and the second slave node are the same node.

5. The method according to any one of claims 1 to 4, characterized in that Before obtaining the first request, the method further includes: obtaining a fourth request including a data stream; Divide the data in the data stream into multiple data blocks, and write the multiple data blocks into data block storage nodes in the distributed storage system in columns, wherein the data block storage nodes include the master node and the first slave node; Calculate a check block according to each group of data blocks in the plurality of data blocks, and write the check block into a check block storage node in the distributed storage system, wherein the check block storage node includes a second slave node; When the multiple data blocks cannot fully write at least one EC stripe, a no-operation is performed on slices without data in the at least one EC stripe.

6. The method according to claim 5, characterized in that The method further comprises: obtaining a fifth request including a start address; The target node is determined according to the start address, and the target data block is read column by column.

7. A data processing device, characterized in that: The device is deployed in a master node in a distributed storage system, and the device includes: An acquiring unit, configured to acquire a first request, wherein the first request is used to update a data block in an erasure code (EC) stripe; a determining unit, configured to determine a first data block according to the first request, wherein the first data block is a data block associated with the first request; A communication unit is used to send a processing request to a slave node set, wherein the slave node set includes one or more slave nodes, and the slave node is a slave node in the distributed storage system. The processing request is used to instruct the master node to offload the operation of updating the data block to one or more slave nodes in the slave node set.

8. The device according to claim 7, characterized in that The communication unit is specifically used for: Sending a second request including the second data block to the first slave node; receiving the first data block returned by the first slave node after updating the first data block to a second data block; The determining unit is further configured to: Determine check block update information according to the first data block and the second data block; The communication unit is specifically used for: A third request including the check block update information is sent to the second slave node, where the check block update information is used to update the check block.

9. The device according to claim 7, characterized in that The communication unit is specifically used for: Sending a second request including the second data block to the first slave node, where the second request is used to instruct the first slave node to update the first data block to the second data block, and to determine check block update information according to the first data block and the second data block; A third request including the check block update information is sent to the second slave node through the first slave node, where the check block update information is used to update the check block.

10. The device according to any one of claims 7 to 9, characterized in that The first data block is stored in a first slave node, and the master node and the first slave node are the same node; or, The check blocks in the EC stripe are stored in the second slave node, and the master node and the second slave node are the same node.

11. The device according to any one of claims 7 to 10, characterized in that The acquisition unit is also used for: Before obtaining the first request, obtaining a fourth request including the data stream; The device also includes: a read-write unit, configured to divide the data in the data stream into blocks to obtain a plurality of data blocks, and write the plurality of data blocks into data block storage nodes in the distributed storage system in columns, wherein the data block storage nodes include the master node and the first slave node; The read-write unit is further configured to calculate a check block according to each group of data blocks in the plurality of data blocks, and write the check block into a check block storage node in the distributed storage system, wherein the check block storage node includes the second slave node; When the multiple data blocks cannot fully write at least one EC stripe, the read-write unit is specifically configured to perform a null operation on the slices without data in the at least one EC stripe.

12. The device according to claim 11, characterized in that The acquisition unit is also used for: obtaining a fifth request including a start address; The read-write unit is also used for: The target node is determined according to the start address, and the target data block is read column by column.

13. A computing device cluster, characterized in that: The computing device cluster includes at least one computing device, and the at least one computing device includes at least one processor and at least one memory, wherein the at least one memory stores computer-readable instructions; the at least one processor executes the computer-readable instructions so that the computing device cluster executes the method as described in any one of claims 1 to 6.

14. A computer-readable storage medium, characterized in that: The method comprises computer-readable instructions; the computer-readable instructions are used to implement the method according to any one of claims 1 to 6.

15. A computer program product, characterized in that The method comprises computer-readable instructions; the computer-readable instructions are used to implement the method according to any one of claims 1 to 6.

Images