Method and apparatus for writing erasure coding data in a multi-AZ environment

Abstract

This specification provides a method and apparatus for writing erasure coding data in a multi-AZ environment. The method deploys relay nodes within the AZ of a distributed system. The relay nodes calculate and generate local check blocks locally based on the data blocks received from the client, and write the local check blocks into the data nodes of the same AZ. Therefore, during the erasure coding data writing process, the local check blocks in the AZ are calculated and generated locally in the AZ, which effectively reduces the amount of data transmission and improves the stability and operating efficiency of the system.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a method for writing erasure coding data in a multi-AZ environment. One or more embodiments of this specification also relate to an apparatus for writing erasure coding data in a multi-AZ environment, a computing device, and a computer-readable storage medium. Background Technology

[0002] As the storage scale of distributed systems increases significantly, data storage cost and reliability become crucial factors to consider in distributed system design. Multi-AZ (Availability Zone) solutions distribute data across multiple individual data centers, ensuring data availability even when a single AZ experiences a data center or network equipment failure. Because multi-AZ provides enhanced data security and system stability, and erasure coding can reduce system storage overhead while maintaining data reliability, erasure coding is currently employed in multi-AZ environments to ensure data availability in response to increasingly complex business scenarios and system environments.

[0003] However, writing erasure coding data in a multi-AZ environment may result in a large amount of data transmission, which has an adverse impact on the stability and operating efficiency of the system. Summary of the Invention

[0004] In view of this, embodiments of this specification provide a method for writing erasure coding data in a multi-AZ environment. One or more embodiments of this specification also relate to an apparatus for writing erasure coding data in a multi-AZ environment, a computing device, and a computer-readable storage medium, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, a method for writing erasure coding data in a multi-AZ environment is provided, applied to a relay node within an AZ. The method includes: receiving a data block from a client of a multi-AZ distributed system; generating a local check block using the received data block through an LRC erasure coding algorithm; and writing the local check block into a data node of the AZ to which the relay node belongs.

[0006] Optionally, the method further includes writing the received data block into the data node of the AZ to which it belongs.

[0007] Optionally, the method further includes: distributing the received data blocks and the local check blocks equally among the data nodes belonging to the AZ according to the number of blocks;

[0008] Optionally, writing the local parity block to the data node of the AZ to which the relay node belongs includes: writing the local parity block to the assigned data node. Writing the received data block to the data node of the AZ to which the relay node belongs includes: writing the received data block to the assigned data node.

[0009] Optionally, generating a local verification block using the received data block via the LRC erasure coding algorithm includes: receiving configuration information of the LRC erasure coding algorithm from the client; setting the LRC erasure coding algorithm using the configuration information; and generating a local verification block using the received data block via the set LRC erasure coding algorithm.

[0010] According to a second aspect of the embodiments of this specification, a device for writing erasure coding data in a multi-AZ environment is provided, configured in a relay node within an AZ. The device includes: a data receiving module configured to receive data blocks from clients of a distributed system with multiple AZs; a local checksum calculation module configured to generate local checksum blocks using the received data blocks via an LRC erasure coding algorithm; and a local checksum writing module configured to write the local checksum blocks into data nodes within the AZ to which the relay node belongs.

[0011] According to a third aspect of the embodiments of this specification, a method for writing erasure coding data in a multi-AZ environment is provided, applied to a client of a multi-AZ distributed system. The method includes: generating a global check block using a plurality of data blocks through an LRC erasure coding algorithm; writing the global check block into data nodes of the AZs; and sending data blocks corresponding to the AZs to relay nodes of the multiple AZs respectively, so that the relay nodes use the received data blocks to generate local check blocks through an LRC erasure coding algorithm, and write the local check blocks into the data nodes of their respective AZs.

[0012] Optionally, writing the global check block into the data nodes of the AZ includes: directly sending the global check block corresponding to the AZ to the data nodes of the plurality of AZs respectively.

[0013] Optionally, writing the global check block into the data nodes of the Availability Zone (AZ) includes: if there are multiple global check blocks, distributing the global check blocks equally among the AZs according to the number of blocks; and writing the global check block into the data nodes of the assigned AZ. Sending the corresponding AZ data blocks to the relay nodes of the multiple AZs includes: distributing the multiple data blocks equally among the AZs according to the number of blocks; and sending the corresponding AZ data blocks to the relay nodes of the multiple AZs.

[0014] Optionally, the method further includes: receiving configuration information of the LRC erasure coding algorithm from a user; and sending the configuration information to the relay nodes of the plurality of AZs, so that the relay nodes can use the configuration information to set the LRC erasure coding algorithm.

[0015] According to a fourth aspect of the embodiments of this specification, a device for writing erasure coding data in a multi-AZ environment is provided, configured on a client side of a distributed system with multiple AZs. The device includes: a global verification calculation module configured to generate a global verification block using a plurality of data blocks via an LRC erasure coding algorithm; a global verification writing module configured to write the global verification block into data nodes of the AZs; and a data sending module configured to send data blocks corresponding to the AZs to relay nodes of the multiple AZs, so that the relay nodes use the received data blocks to generate local verification blocks using an LRC erasure coding algorithm and write the local verification blocks into the data nodes of their respective AZs.

[0016] According to a fifth aspect of the embodiments of this specification, a multi-AZ distributed system is provided, comprising: a client and multiple AZs, each AZ having a relay node and several data nodes communicating with the relay nodes, wherein: the client is configured to generate a global check block using several data blocks through an LRC erasure coding algorithm, write the global check block into the data nodes of the assigned AZ, and send the corresponding AZ's data blocks to the respective relay nodes within the multiple AZs. The relay nodes are configured to receive data blocks from the client, generate local check blocks using the received data blocks through an LRC erasure coding algorithm, and write the local check blocks into the data nodes of their respective AZs.

[0017] According to a sixth aspect of the embodiments of this specification, a computing device is provided, comprising: a memory and a processor; the memory being configured to store computer-executable instructions, and the processor being configured to execute the computer-executable instructions: receiving a data block from a client of a multi-AZ distributed system; generating a local check block using the received data block via an LRC erasure coding algorithm; and writing the local check block into a data node of the AZ to which the relay node belongs.

[0018] According to a seventh aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer instructions, which, when executed by a processor, implement the steps of the method for writing erasure coding data in a multi-AZ environment applied to a relay node as described in any embodiment of this specification.

[0019] According to an eighth aspect of the embodiments of this specification, a computing device is provided, comprising: a memory and a processor; the memory being used to store computer-executable instructions, and the processor being used to execute the computer-executable instructions: generating a global parity block using a plurality of data blocks through an LRC erasure coding algorithm; writing the global parity block into data nodes of an Availability Zone (AZ); and sending data blocks corresponding to a AZ to relay nodes of a plurality of AZs, such that the relay nodes use the received data blocks to generate local parity blocks using an LRC erasure coding algorithm, and write the local parity blocks into data nodes of their respective AZs.

[0020] According to a ninth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer instructions, which, when executed by a processor, implement the steps of the method for writing erasure coding data in a multi-AZ environment applied to a client as described in any embodiment of this specification.

[0021] This specification provides an embodiment of a method for writing erasure coding data in a multi-AZ environment. Because this method deploys relay nodes within the AZs of the distributed system, the relay nodes calculate and generate local check blocks locally based on the received data blocks, and then write these local check blocks into the data nodes of their respective AZs. Thus, during the erasure coding data writing process, the local check blocks in the AZs are calculated and generated locally, avoiding the data transmission of local check blocks from the client to the data nodes. This effectively reduces the amount of data transmission during the erasure coding data writing process in a multi-AZ environment, improving system stability and operating efficiency. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of a distributed system architecture provided in one embodiment of this specification;

[0023] Figure 2 This is a flowchart illustrating a method for writing erasure coding data in a multi-AZ environment, as provided in one embodiment of this specification.

[0024] Figure 3 This is a schematic diagram illustrating the processing steps of a method for writing erasure coding data in a multi-AZ environment, provided in one embodiment of this specification.

[0025] Figure 4 This is a schematic diagram of data distribution provided in one embodiment of this specification;

[0026] Figure 5 This is a flowchart illustrating a method for writing erasure coding data in a multi-AZ environment, provided in another embodiment of this specification.

[0027] Figure 6 This is a block diagram of a multi-AZ distributed system provided in one embodiment of this specification;

[0028] Figure 7 This is a schematic diagram of a device for writing erasure coding data in a multi-AZ environment, provided in one embodiment of this specification.

[0029] Figure 8 This is a schematic diagram of the structure of a device for writing erasure coding data in a multi-AZ environment, provided in another embodiment of this specification;

[0030] Figure 9 This is a schematic diagram of the structure of a device for writing erasure coding data in a multi-AZ environment according to one embodiment of this specification;

[0031] Figure 10 This is a schematic diagram of the structure of a device for writing erasure coding data in a multi-AZ environment, provided in another embodiment of this specification;

[0032] Figure 11 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0033] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0034] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0035] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0036] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0037] An Availability Zone (AZ) is a physical area within the same geographic region where power and network infrastructure are independent. Instances within the same Availability Zone experience lower network latency.

[0038] Erasure coding is a fault-tolerant coding technique. Its basic principle is to divide the stored data into fragments, and then use a certain checksum calculation method to generate k+m fragments of data from k original data. The original data can be restored from any k+m fragments. This way, even if some data is lost, the system can still recover the original data.

[0039] Locally Repairable Codes (LRC) erasure coding algorithm is a local parity coding method. Its core idea is to divide parity blocks into global parity blocks and local reconstruction parity blocks, which can be calculated in groups during fault recovery. LRC encoding is generally represented as LRC(k, z, g), where k represents the total number of data blocks, z represents the number of partitions, and g represents the number of global parity blocks.

[0040] A data block is a block created when data objects are distributed and stored on the disk according to the system's data distribution rules; it is the basic unit of data.

[0041] A local parity block is redundant data calculated using erasure coding algorithms within a group of data blocks after the data blocks have been grouped.

[0042] A global parity is redundant data calculated using erasure coding algorithms for all data blocks on a stripe.

[0043] This specification provides a method for writing erasure coding data in a multi-AZ environment. This specification also relates to apparatus, computing devices, and computer-readable storage media for erasure coding data in a multi-AZ environment, which will be described in detail in the following embodiments.

[0044] To make the methods provided in the embodiments of this specification easier to understand, the system architecture of the multi-AZ distributed system provided in the embodiments of this specification will first be illustrated. For example, as Figure 1 The distributed system architecture diagram shown includes: a client, multiple Availability Zones (AZs), a relay node proxy deployed in each AZ, and several data nodes that communicate with the relay node.

[0045] Figure 2A flowchart is shown of a method for writing erasure coding data in a multi-AZ environment according to an embodiment of this specification. Figure 2 The method shown can be applied to relay nodes within an AZ. The method includes steps 202 to 206.

[0046] Step 202: Receive data blocks from the client of the multi-AZ distributed system.

[0047] The client can use several data blocks to generate a global check block using the LRC erasure coding algorithm. After generating the global check block, it writes it to the data nodes. In one or more instances, the relay node can receive the global check block from the client and write it to the data node of the corresponding Availability Zone (AZ). In another one or more embodiments, the client can directly send the global check block of the corresponding AZ to the data nodes of multiple AZs. It is understood that having the client directly send the global check block to the data nodes can effectively reduce data forwarding and reduce data transmission volume.

[0048] Step 204: Using the received data block, generate a local check block using the LRC erasure coding algorithm.

[0049] It should be noted that the LRC erasure coding algorithm function is not limited in the embodiments of this specification and can be set according to the needs of the implementation scenario. The encoding function and related parameters of the LRC erasure coding algorithm can be set to fixed according to the needs of the implementation scenario, or they can be flexibly configured according to the information provided by the user.

[0050] For example, the relay node can receive configuration information from the client regarding the LRC erasure coding algorithm; use this configuration information to set the LRC erasure coding algorithm; and use the received data blocks to generate local parity blocks using the set LRC erasure coding algorithm. The configuration information may include, for example, an erasure coding scheme that includes information such as the encoding function, the number of data blocks, the number of local parity blocks, and the number of global parity blocks. For example, the configuration information could be LRC(6, 3, 3), where 6 represents the number of data blocks, 3 represents the number of local parity blocks, and 3 represents the number of global parity blocks. By receiving the configuration information, the relay node can generate one or more local parity blocks according to the user's configuration, meeting the user's flexible configuration needs.

[0051] In one or more embodiments, the relay node may also write the received data blocks into the data nodes of the same Availability Zone (AZ). In another one or more embodiments, the client may also directly send the corresponding AZ data blocks to the data nodes of multiple AZs. It is understood that having the relay node send data blocks to the data nodes can reduce the amount of data transmission caused by the client repeatedly sending data blocks, thereby improving the stability and operating efficiency of the system.

[0052] Step 206: Write the local verification block into each data node on the AZ.

[0053] To ensure data security and system stability, in one or more embodiments of this specification, data nodes are evenly distributed according to the number of blocks. Specifically, the method can also evenly distribute the received data blocks and the local check blocks to the data nodes of their respective Availability Zones (AZs) according to the number of blocks. When writing a local check block to a data node, the local check block is written to the assigned data node; when writing a received data block to a data node, the received data block is written to the assigned data node. It should be noted that the even distribution in the embodiments of this specification can be an approximate even distribution. For example, when there are five data blocks and three AZs, two data blocks can be sent to two of the AZs respectively, and one data block can be sent to the other AZ. One block is written to one data node.

[0054] Because this method deploys relay nodes within the Availability Zones (AZs) of the distributed system, the relay nodes calculate and generate local check blocks locally based on the received data blocks, and write the local check blocks into the data nodes of their respective AZs. Thus, during the erasure coding data writing process, the local check blocks in the AZ are calculated and generated locally, avoiding the data transmission of local check blocks from the client to the data nodes. This effectively reduces the amount of data transmission during the erasure coding data writing process in a multi-AZ environment, improving the stability and operating efficiency of the system.

[0055] The following is a detailed description of the method for writing erasure coding data in a multi-AZ environment that combines the above-described embodiments. Figure 3 A schematic diagram illustrating the processing steps of a method for writing erasure coding data in a multi-AZ environment, according to another embodiment of this specification, is shown. Figure 3 As shown, the method includes steps 302 to 316.

[0056] Step 302: The client uses all data blocks and generates a global check block using the LRC erasure coding algorithm according to the user-configured erasure coding algorithm.

[0057] Step 304: The client distributes the global checksum blocks equally among the various Availability Zones (AZs) according to the number of global checksum blocks.

[0058] Step 306: The client, following the allocation in step 304, directly sends the corresponding global check block of each AZ to the data nodes of each AZ.

[0059] Step 308: The client distributes all data blocks equally among the various Availability Zones (AZs) according to the number of blocks.

[0060] Step 310: The client sends the corresponding AZ data blocks to the respective relay nodes within each AZ.

[0061] Step 312: The relay nodes of the AZ use the received data blocks to generate local parity blocks using the LRC erasure coding algorithm.

[0062] Step 314: The relay node distributes the received data blocks and the local check blocks equally among the data nodes of its respective AZ according to the number of blocks.

[0063] Step 316: The relay node writes the received data block and local check block into the corresponding data node according to the allocation in step 314.

[0064] For example, in one implementation scenario of this specification, suppose a user needs to send an instruction to... Figure 1 In the storage system shown, which has three availability zones AZ0, AZ1, and AZ2, data A1 to A6 are written, and a global parity block is generated using LRC(6,3,3) erasure coding. Then, according to... Figure 3 The processing procedure shown and Figure 1 The system architecture shown, in this implementation scenario, the process of writing erasure coding data from A1 to A6 can include:

[0065] 1. Figure 1 The client shown uses data blocks A1-A6 locally to generate global check blocks P1, P2, and P3 using the LRC erasure coding algorithm.

[0066] 2. According to Figure 3 In the step of evenly distributing blocks shown in the process, the client directly sends a global parity block P1 to a data node in availability zone AZ0, sends P2 to a data node in availability zone AZ1, and sends P3 to a data node in availability zone AZ2. The global parity block is directly stored without processing.

[0067] 3. The client sends two data blocks (a total of 6 data blocks from A1 to A6 are evenly distributed among the three AZs, with each AZ corresponding to two data blocks) and the configuration information of the erasure coding algorithm to the respective relay node proxy of the three AZs.

[0068] 4. Each of the three Availability Zone (AZ) proxies uses the received data blocks and the LRC erasure coding algorithm to calculate and generate a local partial check block. For example, ... Figure 1 As shown, the proxy of AZ0 uses A1-A2 to generate local check block L1, the proxy of AZ1 uses A3-A4 to generate local check block L2, and the proxy of AZ2 uses A5-A6 to generate local check block L2.

[0069] 5. The proxy writes the generated local checksum blocks into the data nodes of this AZ. For example, the proxy of AZ0 writes data blocks A1, A2 and local checksum block L1 into the three data nodes of this AZ, one block per data node. The processing logic of other proxies is similar and will not be described in detail here.

[0070] As can be seen from the above writing process, the user writes data A1 to A6 to a storage system with three availability zones, using LRC(6,3,3) erasure coding. The final data distribution is as follows. Figure 4 As shown in the data distribution diagram, each Availability Zone (AZ) stores two data blocks: a local check block and a global check block. The local check block is generated locally by the proxy in the AZ using the data blocks, while the global check block is generated by the client using all the data blocks and then sent directly to the data nodes. Therefore, during the erasure coding data writing process, the client only needs to send the user data block and the global check block to each AZ. The local check block in each AZ is actively calculated and generated during local data writing, reducing the network traffic from the client to the data nodes for the local check block.

[0071] Figure 5 A flowchart is shown of a method for writing erasure coding data in a multi-AZ environment according to another embodiment of this specification. Figure 5 The method shown can be applied to clients of a multi-AZ distributed system. The method includes steps 502 to 506.

[0072] Step 502: Using several data blocks, generate a global check block using the LRC erasure coding algorithm.

[0073] The encoding function and related parameters of the LRC erasure coding algorithm can be flexibly configured based on information provided by the user. For example, the client can receive the user's configuration information for the LRC erasure coding algorithm; using this configuration information, it can set the LRC erasure coding algorithm and generate a global parity block using the configured LRC erasure coding algorithm. Alternatively, the client can send the configuration information to the relay nodes of the multiple Availability Zones (AZs) so that the relay nodes can use the configuration information to set the LRC erasure coding algorithm.

[0074] Step 504: Write the global check block into the data nodes of the AZ.

[0075] In one or more instances, the client can send a global checksum block to a relay node, which then writes the global checksum block into the data node of the corresponding Availability Zone (AZ). In another one or more embodiments, the client can directly send the global checksum block for each AZ to the data nodes of multiple AZs. It is understood that having the client directly send the global checksum block to the data nodes effectively reduces data forwarding and data transmission volume.

[0076] Step 506: Send the corresponding AZ data blocks to the relay nodes of multiple AZs respectively, so that the relay nodes can use the received data blocks to generate local check blocks through the LRC erasure coding algorithm, and write the local check blocks into the data nodes of their respective AZs.

[0077] To ensure data security and system stability, in one or more embodiments of this specification, data nodes can be evenly distributed according to the number of blocks. Specifically, for example, writing the global check block into the data nodes of an Availability Zone (AZ) may include: if there are multiple global check blocks, distributing the global check blocks evenly among the AZs according to the number of blocks; and writing the global check block into the data nodes of the assigned AZ. As another example, sending the corresponding AZ data blocks to relay nodes of multiple AZs may include: distributing the plurality of data blocks evenly among the AZs according to the number of blocks; and sending the corresponding AZ data blocks to the relay nodes of the multiple AZs.

[0078] In this method, the client sends the data block to the relay node deployed within the AZ. The relay node calculates and generates a local check block locally based on the received data block and writes the local check block into the data node of its own AZ. Thus, during the erasure coding data writing process, the local check block in the AZ is calculated and generated locally, avoiding the data transmission of the local check block from the client to the data node. This effectively reduces the amount of data transmission during the erasure coding data writing process in a multi-AZ environment, improving the stability and operating efficiency of the system.

[0079] Figure 6 A block diagram of a multi-AZ distributed system according to one embodiment of this specification is shown. Figure 6 As shown, the distributed system may include: a client 602 and multiple Availability Zones (AZs) 604, wherein each AZ 604 is equipped with a relay node 606 and several data nodes communicating with the relay node 606.

[0080] The client 602 is configured to use several data blocks to generate a global check block using the LRC erasure coding algorithm, write the global check block into the data nodes of the assigned AZ, and send the corresponding AZ data blocks to the relay nodes of the multiple AZs respectively.

[0081] The relay node 606 is configured to receive data blocks from the client, generate local check blocks using the received data blocks through the LRC erasure coding algorithm, and write the local check blocks into the data nodes of the AZ to which they belong.

[0082] In this system, the client sends data blocks to relay nodes deployed within the Availability Zone (AZ). The relay nodes then calculate and generate local check blocks locally based on the received data blocks and write these local check blocks into the data nodes of their respective AZs. As a result, during the erasure coding data writing process, the local check blocks in the AZ are calculated and generated locally, avoiding the data transmission of local check blocks from the client to the data nodes. This effectively reduces the amount of data transmission during the erasure coding data writing process in a multi-AZ environment, improving the system's stability and operating efficiency.

[0083] Corresponding to the above method embodiments, this specification also provides an embodiment of a device for writing erasure coding data in a multi-AZ environment with relay nodes configured within an AZ. Figure 7 This specification illustrates a schematic diagram of a device for writing erasure coding data in a multi-AZ environment, according to one embodiment of the present specification. Figure 7 As shown, the device includes: a data receiving module 702, a local verification calculation module 704, and a local verification writing module 706.

[0084] The data receiving module 702 can be configured to receive data blocks from clients of a multi-AZ distributed system.

[0085] The local check calculation module 704 can be configured to generate a local check block using the received data block and the LRC erasure coding algorithm.

[0086] The local verification writing module 706 can be configured to write the local verification block into the data node of the AZ to which the relay node belongs.

[0087] Because the device deploys relay nodes within the Availability Zones (AZs) of the distributed system, the relay nodes calculate and generate local check blocks locally based on the received data blocks, and write the local check blocks into the data nodes of the same AZ. Thus, during the erasure coding data writing process, the local check blocks in the AZ are calculated and generated locally, avoiding the data transmission of local check blocks from the client to the data nodes. This effectively reduces the amount of data transmission during the erasure coding data writing process in a multi-AZ environment, and improves the stability and operating efficiency of the system.

[0088] Figure 8 A schematic diagram of a device for writing erasure coding data in a multi-AZ environment, according to another embodiment of this specification, is shown. Figure 8As shown, the device may further include a data writing module 708, which can be configured to write the received data blocks into the data nodes of the respective AZ.

[0089] To ensure data security and system stability, in one or more embodiments of this specification, such as Figure 8 As shown, the device may further include: an average allocation module 710, configured to distribute the received data blocks and the local check blocks equally among the data nodes of the respective Availability Zone (AZ) according to the number of blocks. Correspondingly, the local check block writing module 706 may be configured to write the local check blocks into the allocated data nodes. The data writing module 708 may be configured to write the received data blocks into the allocated data nodes.

[0090] It should be noted that the encoding function and related parameters of the LRC erasure coding algorithm can be set to a fixed value according to the needs of the implementation scenario, or they can be flexibly configured according to the information provided by the user. For example, in one or more embodiments of this specification, such as Figure 8 As shown, the local verification calculation module 704 may include: a configuration information receiving submodule 7042, a configuration information setting submodule 7044, and a local verification calculation submodule 7046.

[0091] The configuration information receiving submodule 7042 can be configured to receive user configuration information for the LRC erasure coding algorithm from the client.

[0092] The configuration information setting submodule 7044 can be configured to use the configuration information to set the LRC erasure coding algorithm.

[0093] The local check calculation submodule 7046 can be configured to generate a local check block using the received data block and a pre-set LRC erasure coding algorithm.

[0094] The above is a schematic scheme of the erasure coding data writing device configured in a multi-AZ environment of a relay node according to this embodiment. It should be noted that the technical solution of the erasure coding data writing device in the multi-AZ environment is of the same concept as the technical solution of the erasure coding data writing method applied to a relay node in a multi-AZ environment described above. For details not described in detail in the technical solution of the erasure coding data writing device configured in a relay node in a multi-AZ environment, please refer to the description of the technical solution of the erasure coding data writing method applied to a relay node in a multi-AZ environment described above.

[0095] Corresponding to the above method embodiments, this specification also provides an embodiment of a device for writing erasure coding data in a multi-AZ environment configured on the client side. Figure 9This specification illustrates a schematic diagram of a device for writing erasure coding data in a multi-AZ environment, according to one embodiment of the present specification. Figure 9 As shown, the device includes: a global verification calculation module 902, a global verification writing module 904, and a data sending module 906.

[0096] The global verification calculation module 902 can be configured to generate a global verification block using several data blocks through the LRC erasure coding algorithm.

[0097] The global verification writing module 904 can be configured to write the global verification block into the data nodes of the AZ.

[0098] The data sending module 906 can be configured to send data blocks corresponding to multiple AZs to relay nodes respectively, so that the relay nodes can use the received data blocks to generate local check blocks through the LRC erasure coding algorithm and write the local check blocks into the data nodes of their respective AZs.

[0099] In one or more embodiments of this specification, the global verification writing module 904 can be configured to directly send the corresponding global verification block of each of the multiple Availability Zones (AZs) to the data nodes. It is understood that having the client directly send the global verification block to the data nodes can effectively reduce data forwarding and data transmission volume.

[0100] Figure 10 This diagram illustrates a structural schematic of a device for writing erasure coding data in a multi-AZ environment, according to yet another embodiment of this specification. Figure 10 As shown, the global verification writing module 904 includes: a verification block allocation submodule 9042 and a global verification writing submodule 9044.

[0101] The verification block allocation submodule 9042 can be configured to distribute the global verification blocks equally among the various Availability Zones (AZs) according to the number of blocks, when there are multiple global verification blocks.

[0102] The global verification write submodule 9044 can be configured to write the global verification block into the data nodes assigned to the AZ.

[0103] The data sending module 906 includes a data block allocation submodule 9062 and a data block sending submodule 9064.

[0104] The data block allocation submodule 9062 can be configured to distribute the plurality of data blocks equally among the various AZs according to the number of blocks.

[0105] The data block sending submodule 9064 can be configured to send data blocks corresponding to the AZs to the relay nodes of the multiple AZs respectively.

[0106] In one or more embodiments of this specification, such as Figure 10 As shown, the device may further include a user configuration receiving module 908 and a user configuration sending module 910.

[0107] The user configuration receiving module 908 can be configured to receive user configuration information for the LRC erasure coding algorithm.

[0108] The user configuration sending module 910 can be configured to send the configuration information to the relay nodes of the plurality of AZs, so that the relay nodes can use the configuration information to set the LRC erasure coding algorithm.

[0109] The client can generate one or more local parity blocks according to the user configuration by sending the erasure coding algorithm configuration information to the relay node, thus meeting the user's flexible configuration needs.

[0110] The above is a schematic scheme of the erasure coding data writing device configured on the client in a multi-AZ environment according to this embodiment. It should be noted that the technical solution of the erasure coding data writing device in the multi-AZ environment is of the same concept as the technical solution of the erasure coding data writing method applied to the client in a multi-AZ environment. For details not described in detail in the technical solution of the erasure coding data writing device configured on the client in a multi-AZ environment, please refer to the description of the technical solution of the erasure coding data writing method applied to the client in a multi-AZ environment.

[0111] Figure 11 A structural block diagram of a computing device 1100 according to one embodiment of this specification is shown. The components of the computing device 1100 include, but are not limited to, a memory 1110 and a processor 1120. The processor 1120 is connected to the memory 1110 via a bus 1130, and a database 1150 is used to store data.

[0112] The computing device 1100 also includes an access device 1140, which enables the computing device 1100 to communicate via one or more networks 1160. Examples of these networks include a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the Internet. The access device 1140 may include one or more of any type of wired or wireless network interface (e.g., a Network Interface Card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) interface, a Wi-MAX interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0113] In one embodiment of this specification, the aforementioned components of the computing device 1100 and Figure 11 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 11 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0114] The computing device 1100 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or PCs. The computing device 1100 can also be a mobile or stationary server.

[0115] In one aspect, processor 1120 is used to execute the following computer-executable instructions:

[0116] Receive data blocks from clients in a multi-AZ distributed system;

[0117] Using the received data block, a local check block is generated through the LRC erasure coding algorithm;

[0118] The local verification block is written into the data node of the AZ to which the relay node belongs.

[0119] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the above-described method for writing erasure coding data in a multi-AZ environment applied to relay nodes belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the above-described method for writing erasure coding data in a multi-AZ environment applied to relay nodes.

[0120] On the other hand, processor 1120 is used to execute the following computer-executable instructions:

[0121] A global check block is generated using several data blocks and the LRC erasure coding algorithm.

[0122] Write the global verification block into the data nodes of the AZ;

[0123] Data blocks corresponding to the respective AZs are sent to relay nodes of multiple AZs, so that the relay nodes can use the received data blocks to generate local check blocks using the LRC erasure coding algorithm, and write the local check blocks into the data nodes of their respective AZs.

[0124] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the above-described method for writing erasure coding data in a multi-AZ environment applied to a client belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the above-described method for writing erasure coding data in a multi-AZ environment applied to a client.

[0125] This specification also provides a computer-readable storage medium storing computer instructions, wherein, when executed by a processor, the instructions are used for:

[0126] Receive data blocks from clients in a multi-AZ distributed system;

[0127] Using the received data block, a local check block is generated through the LRC erasure coding algorithm;

[0128] The local verification block is written into the data node of the AZ to which the relay node belongs.

[0129] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the technical solution of the method for writing erasure coding data in a multi-AZ environment applied to relay nodes described above. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the method for writing erasure coding data in a multi-AZ environment applied to relay nodes described above.

[0130] On the other hand, when this instruction is executed by the processor, it is used for:

[0131] A global check block is generated using several data blocks and the LRC erasure coding algorithm.

[0132] Write the global verification block into the data nodes of the AZ;

[0133] Data blocks corresponding to the respective AZs are sent to relay nodes of multiple AZs, so that the relay nodes can use the received data blocks to generate local check blocks using the LRC erasure coding algorithm, and write the local check blocks into the data nodes of their respective AZs.

[0134] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the technical solution of the above-described method for writing erasure coding data in a multi-AZ environment applied to a client. Details not described in detail in the technical solution of the storage medium can be found in the description of the technical solution of the above-described method for writing erasure coding data in a multi-AZ environment applied to a client.

[0135] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0136] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0137] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0138] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0139] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A method for writing erasure coding data in a multi-AZ environment, applied to relay nodes within an AZ, the method comprising: The client of the multi-AZ distributed system receives the data blocks corresponding to the AZ, wherein the data block is a block generated after the data object is distributed and stored on the disk according to the data distribution rules, and is the basic unit of data; Using the configuration information of the LRC erasure coding algorithm sent by the client, the number of local parity blocks of the LRC erasure coding algorithm is set; Using the received data block, the relay node generates a local check block through the LRC erasure coding algorithm; The received data blocks and local check blocks are allocated and written to the data nodes of the AZ to which the relay node belongs, according to the number of data blocks and the number of local check blocks.

2. The method according to claim 1, further comprising: The received data blocks are written to the data nodes of the respective AZ.

3. The method according to claim 2, further comprising: The received data blocks and the local verification blocks are distributed equally among the data nodes of the respective Availability Zones (AZs) according to the number of blocks. Writing the local parity block to the data node of the AZ to which the relay node belongs includes: writing the local parity block to the assigned data node; Writing the received data block to the data node of the AZ includes writing the received data block to the assigned data node.

4. The method according to claim 1, wherein the step of generating a local check block by the relay node using the LRC erasure coding algorithm based on the received data block comprises: Receive user configuration information for the LRC erasure coding algorithm from the client; Using the configuration information, the LRC erasure coding algorithm is set; Using the received data block, a local check block is generated through the configured LRC erasure coding algorithm.

5. A device for writing erasure coding data in a multi-AZ environment, configured in a relay node within an AZ, the device comprising: The data receiving module is configured to receive data blocks corresponding to the AZ from the client of the multi-AZ distributed system, wherein the data block is a block generated after the data object is distributed and stored on the disk according to the data distribution rules, and is the basic unit of data; The local check calculation module is configured to use the configuration information of the LRC erasure coding algorithm sent by the client to set the number of local check blocks of the LRC erasure coding algorithm; and to generate local check blocks by the relay node using the LRC erasure coding algorithm using the received data blocks. The local checksum writing module is configured to allocate and write the received data blocks and local checksum blocks to the data nodes of the AZ to which the relay node belongs, according to the number of data blocks and the number of local checksum blocks.

6. A method for writing erasure coding data in a multi-AZ environment, applied to a client of a multi-AZ distributed system, the method comprising: Using several data blocks, a global verification block is generated through the LRC erasure coding algorithm. The data block is a block generated after the data object is distributed and stored on the disk according to the data distribution rules, and it is the basic unit of data. Write the global verification block into the data nodes of the AZ; Data blocks corresponding to multiple Availability Zones (AZs) are sent to relay nodes respectively, so that the relay nodes can use the configuration information of the LRC erasure coding algorithm sent by the client to set the number of local check blocks of the LRC erasure coding algorithm; and use the received data blocks to generate local check blocks through the LRC erasure coding algorithm, and allocate and write the received data blocks and the local check blocks to the data nodes of their respective AZs according to the number of data blocks and the number of local check blocks.

7. The method according to claim 6, wherein writing the global check block into the data node of the AZ comprises: The global verification block corresponding to each of the multiple AZs is sent directly to the data nodes of those AZs.

8. The method according to claim 6, wherein writing the global check block into the data node of the AZ comprises: When there are multiple global check blocks, the global check blocks are evenly distributed among the various Availability Zones (AZs) according to the number of blocks. Write the global verification block into the data node assigned to the AZ; Sending data blocks corresponding to the respective AZs to relay nodes of multiple AZs includes: The data blocks are distributed equally among the AZs according to the number of blocks. The data blocks corresponding to the AZs are sent to the relay nodes of the multiple AZs respectively.

9. The method according to claim 6, further comprising: Receive user configuration information for the LRC erasure coding algorithm; The configuration information is sent to the relay nodes of the multiple Availability Zones (AZs) so that the relay nodes can use the configuration information to set the LRC erasure coding algorithm.

10. A device for writing erasure coding data in a multi-AZ environment, configured on a client side of a multi-AZ distributed system, the device comprising: The global verification calculation module is configured to generate a global verification block using several data blocks and the LRC erasure coding algorithm. The data block is a block generated after the data object is distributed and stored on the disk according to the data distribution rules, and is the basic unit of data. The global verification write module is configured to write the global verification block into the data nodes of the AZ; The data sending module is configured to send data blocks corresponding to multiple Availability Zones (AZs) to relay nodes respectively, so that the relay nodes can use the configuration information of the LRC erasure coding algorithm sent by the client to set the number of local check blocks of the LRC erasure coding algorithm; and use the received data blocks to generate local check blocks through the LRC erasure coding algorithm, and allocate and write the received data blocks and the local check blocks to the data nodes of their respective AZs according to the number of data blocks and the number of local check blocks.

11. A multi-AZ distributed system, comprising: The client and multiple Availability Zones (AZs), wherein each AZ contains a relay node and several data nodes that communicate with the relay node, wherein: The client is configured to use several data blocks to generate a global check block using the LRC erasure coding algorithm, write the global check block into the data nodes of the assigned AZ, and send the corresponding AZ data blocks to the relay nodes in the multiple AZs respectively. The data block is a block generated after the data object is distributed and stored on the disk according to the data distribution rules, and is the basic unit of data. The relay node is configured to receive data blocks from the client, set the number of local check blocks for the LRC erasure coding algorithm using the configuration information of the LRC erasure coding algorithm sent by the client, generate local check blocks using the received data blocks through the LRC erasure coding algorithm, and allocate and write the received data blocks and the local check blocks to the data nodes of the respective AZ according to the number of data blocks and the number of local check blocks.

12. A computing device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions: Receive data blocks corresponding to the AZ from the client of the multi-AZ distributed system; Using the configuration information of the LRC erasure coding algorithm sent by the client, the number of local parity blocks of the LRC erasure coding algorithm is set; Using the received data block, the relay node generates a local check block through the LRC erasure coding algorithm; The received data blocks and local check blocks are allocated and written to the data nodes of the AZ to which the relay node belongs, according to the number of data blocks and the number of local check blocks.

13. A computer-readable storage medium storing computer instructions that, when executed by a processor, implement the steps of the method for writing erasure coding data in a multi-AZ environment as described in any one of claims 1 to 4.

14. A computing device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions: Using several data blocks, a global verification block is generated through the LRC erasure coding algorithm. The data block is a block generated after the data object is distributed and stored on the disk according to the data distribution rules, and it is the basic unit of data. Write the global verification block into the data nodes of the AZ; Data blocks corresponding to multiple Availability Zones (AZs) are sent to relay nodes respectively, so that the relay nodes can use the configuration information of the LRC erasure coding algorithm sent by the client to set the number of local check blocks of the LRC erasure coding algorithm; and using the received data blocks, local check blocks are generated by the LRC erasure coding algorithm, and the received data blocks and local check blocks are allocated and written to the data nodes of the respective AZs according to the number of data blocks and the number of local check blocks.

15. A computer-readable storage medium storing computer instructions that, when executed by a processor, implement the steps of the method for writing erasure coding data in a multi-AZ environment as described in any one of claims 6 to 9.

16. A computer program product, characterized in that, Includes computer instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 4 or 6 to 9.

Images