Data processing method, system, device, medium and product

Abstract

The embodiment of the invention provides a data processing method, system and device, a medium and a product, and belongs to the field of data processing. The method comprises the following steps: a first service node determines a first data block group corresponding to a data writing request of a first data source, and performs verification processing on the first data block group to obtain a first verification data group, and the original copy and the mirror copy of the first verification data set are respectively stored in a first available area where the first service node is located and a second available area where the second service node is located. And the second service node generates a first verification data group and a first verification block group corresponding to the first verification data group according to the mirror image copy of the first verification data group and the mirror image copy of the second verification data group, and stores metadata information of the first verification block group to the metadata node. And the first service node stores the metadata information corresponding to the original copy and the mirror copy of the first verification data group to the metadata node. Therefore, the bandwidth overhead for reading the data can be reduced, and the data redundancy ratio in the system can be reduced.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data processing method, system, medium, and product. Background Technology

[0002] A distributed system is a system composed of multiple independent computers that can communicate and collaborate through a network to jointly complete various tasks and functions within the system. Distributed systems have been widely applied in various fields such as cloud computing systems, big data processing systems, social networks, and e-commerce.

[0003] As users increasingly demand higher data reliability, user data is typically stored using Availability Zones (AZs). Therefore, a distributed system can include multiple AZs, each of which can be a data center (or simply a server room). Each AZ can contain several compute nodes (which may be cloud servers). Based on the roles of the compute nodes, each AZ can include service nodes that respond to user-triggered data read / write requests, and multiple data nodes that store data. The distributed system can also include at least one metadata node, which can be distributed across different AZs or set up independently of each AZ, and is used to manage metadata information within the distributed system.

[0004] When user data is stored in a multi-AZ configuration, after any service node receives user data and divides it into multiple data blocks, it performs verification processing (such as erasure coding calculation) and then distributes the multiple data blocks and multiple verification blocks to multiple AZs. Correspondingly, when a user reads the above multiple data blocks, the service node determines the AZ where each data block is located based on the data read request and reads the corresponding data block from the data nodes in multiple AZs respectively, which will lead to a large bandwidth overhead. Summary of the Invention

[0005] This invention provides a data processing method, system, device, medium, and product to reduce bandwidth overhead for reading data and to reduce data redundancy in distributed systems.

[0006] In a first aspect, embodiments of the present invention provide a data processing method applied to a distributed system, the distributed system including multiple service nodes located in different availability zones, the method comprising:

[0007] The first service node responds to the data write request from the first data source and determines the first group of data blocks to be written.

[0008] The first service node performs verification processing on the first data block group to obtain the first verification data group, which includes the first data block group and multiple verification blocks.

[0009] The first service node stores the original copy of the first verification data group in the first availability zone where the first service node is located, and stores the mirror copy of the first verification data group in the second availability zone where the second service node is located. The first service node and the second service node are different service nodes among multiple service nodes.

[0010] Secondly, embodiments of the present invention provide a distributed system, including: multiple service nodes located in different availability zones;

[0011] The first service node is used to obtain the first data block group to be written from the first data source, perform verification processing on the first data block group to obtain the first verification data group, store the original copy of the first verification data group in the first availability zone where the first service node is located, and store the mirror copy of the first verification data group in the second availability zone where the second service node is located. The first verification data group includes the first data block group and multiple verification blocks.

[0012] The third service node is used to obtain the second data block group to be written from the second data source, perform verification processing on the second data block group to obtain the second verification data group, store the original copy of the second verification data group in the third availability zone where the third service node is located, and store the mirror copy of the second verification data group in the second availability zone where the second service node is located. The second verification data group includes the second data block group and multiple verification blocks. The first service node, the second service node, and the third service node are different service nodes among the multiple service nodes.

[0013] Thirdly, embodiments of the present invention provide a first service node, including:

[0014] The response module is used to respond to the data write request from the first data source and determine the first group of data blocks to be written.

[0015] The verification module is used to perform verification processing on the first data block group to obtain a first verification data group, which includes the first data block group and multiple verification blocks.

[0016] The first storage module is used to store the original copy of the first verification data group in the first availability zone where the first service node is located, and to store the mirror copy of the first verification data group in the second availability zone where the second service node is located. The first service node and the second service node are different service nodes among a plurality of service nodes.

[0017] Fourthly, embodiments of the present invention provide an electronic device, including: a memory, a processor, and a communication interface; wherein, the memory stores executable code, and when the executable code is executed by the processor, the processor can at least implement the data processing methods described above.

[0018] Fifthly, embodiments of the present invention provide a non-transitory machine-readable storage medium storing executable code, which, when executed by a processor of an electronic device, enables the processor to at least implement the data processing methods described above.

[0019] Sixthly, embodiments of the present invention provide a computer program product containing a computer program that, when executed by a processor of an electronic device, enables the processor to at least implement the data processing methods described above.

[0020] In the solution provided by the embodiments of the present invention, the first service node determines the first data block group corresponding to the data write request of the first data source, performs verification processing on the first data block group, such as performing erasure coding calculation processing on the first data block group to obtain the first verification data group, and stores the original copy of the first verification data group in the first availability zone where the first service node is located. Since the blocks in the first verification data group are all stored in the first availability zone, cross-availability zone data read traffic will not be caused when reading data. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram of a data storage scheme;

[0023] Figure 2 A schematic diagram of the structure of a distributed system provided in an embodiment of the present invention;

[0024] Figure 3 A flowchart of a data processing method provided in an embodiment of the present invention;

[0025] Figure 4 A schematic diagram of a data storage scheme in a data processing method provided in an embodiment of the present invention;

[0026] Figure 5 A flowchart of another data processing method provided in an embodiment of the present invention;

[0027] Figure 6 A flowchart illustrating yet another data processing method provided in an embodiment of the present invention;

[0028] Figure 7 A flowchart illustrating a data recovery scheme in a data processing method provided in an embodiment of the present invention;

[0029] Figure 8 This is a schematic diagram of the structure of a first service node provided in an embodiment of the present invention;

[0030] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in the embodiments of the present invention are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0033] The following detailed description of some embodiments of the present invention is provided in conjunction with the accompanying drawings. Where there is no conflict between the embodiments, the following embodiments and features can be combined with each other. Furthermore, the timing of the steps in the following method embodiments is merely an example and not a strict limitation.

[0034] First, some terms involved in the embodiments of this invention will be explained.

[0035] Distributed systems: Systems composed of numerous computing nodes deployed in a distributed manner. Based on the different roles and attributes of the computing nodes, they can be divided into metadata nodes, data nodes, service nodes, etc. Computing nodes can be virtual machines or physical computers. One or more computing nodes of the same or different types can be deployed on the same physical computer. These computing nodes can communicate and cooperate through a network to jointly complete various tasks and functions within the system.

[0036] Service node: A computing node in a distributed system that responds to data read and write requests triggered by users.

[0037] Data node: A computing node in a distributed system used to store user data.

[0038] Metadata node: A computing node used to manage metadata information in a distributed system.

[0039] Availability Zones (AZs) are physical areas within the same geographical region where power and network access are independent of each other.

[0040] Erasure coding (EC) is a fault-tolerant coding technique used to add redundancy to data storage or transmission to provide fault tolerance. Its basic principle is to divide the original stored data into k parts, and then use a certain verification calculation method to generate a checksum set containing k+m parts of data. The original data can then be recovered from any k parts of the k+m checksum set.

[0041] The distributed system provided in this embodiment of the invention typically includes multiple Availability Zones (AZs), each of which can be a data center and may include several compute nodes. Based on the attributes of the compute nodes, each AZ may include service nodes for responding to user-triggered data read / write requests, multiple data nodes for storing data, and at least one metadata node for managing metadata information within the distributed system. These metadata nodes may be distributed across different AZs or set up independently of each AZ.

[0042] Currently, user data is typically stored using a multi-AZ (Availability Zone) approach. Specifically, after any service node receives user data and divides it into multiple data blocks, it performs verification processing on these data blocks to obtain corresponding verification blocks. These data blocks and their corresponding verification blocks are then distributed and stored across multiple AZs, ensuring that user data remains accessible even if a single AZ fails.

[0043] Specifically, such as Figure 1 As shown, taking three Availability Zones (AZs) as an example, the first service node located in AZ1 responds to the data write request from the first data source by dividing the user data to be written into A1 to A2. 10 Ten data blocks are generated, and parity block P is generated using data blocks A1 to A5. 1-1 and P 2-1 Use data blocks A6 to A 10 Generate check block P 1-3 and P 2-3Use data blocks A1 and A6 to generate check block X1, use data blocks A2 and A7 to generate check block X2, use data blocks A3 and A8 to generate check block X3, use data blocks A4 and A9 to generate check block X4, and use data blocks A5 and A6 to generate check block X2. 10 Generate check block X5, and use check block P. 1-1 and P 1-3 Generate check block X6, and use check block P 2-1 and P 2-3 Generate check block X7. This includes data blocks A1-A5 and check block P. 1-1 and P 2-1 Data block A6-A was written into AZ1. 10 and check block P 1-3 and P 2-3 The check blocks X1-X7 are written into AZ2, and the check blocks X1-X7 are written into AZ3.

[0044] The second service node located in AZ3 responds to the data write request from the second data source by splitting the user data to be written into B1 to B2. 10 Ten data blocks are generated, and parity block P is generated using data blocks B1 to B5. 1-2 and P 2-2 Use data blocks B6 to B 10 Generate check block P 1-4 and P 2-4 Use data blocks B1 and B6 to generate check block X8, use data blocks B2 and B7 to generate check block X9, and use data blocks B3 and B8 to generate check block X. 10 Use data blocks B4 and B9 to generate check block X 11 Using data blocks B5 and B 10 Generate check block X 12 Use check block P 1-2 and P 1-4 Generate check block X 13 and using check block P 2-2 and P 2-4 Generate check block X 14 Among them, data blocks B1 to B5 and check block P 1-2 and P 2-2 Data block B6-B was written into AZ1. 10 and check block P 1-4 and P 2-4 Written into AZ2, check block X8-X 14It is written into AZ3. It should be noted that the example above, generating 2 check blocks from 5 data blocks, is merely an example and not a limitation. Furthermore, this embodiment provides erasure coding mechanisms in both horizontal and vertical directions. The horizontal erasure coding mechanism refers to erasure coding of the same group of data blocks (e.g., data blocks A1-A5 as one group, data blocks A6-A...). 10 An erasure coding calculation is performed on a set of data blocks to obtain a corresponding parity data group (a group consisting of data blocks and parity blocks). Vertical erasure coding refers to performing erasure coding calculations on two consecutive parity data groups from the same data source. Specifically, it involves performing erasure coding calculations on two blocks at the same corresponding position within these two parity data groups. For example, in the above example, parity block X1 is generated using data blocks A1 and A6, parity block X2 is generated using data blocks A2 and A7, parity block X3 is generated using data blocks A3 and A8, parity block X4 is generated using data blocks A4 and A9, and parity block X4 is generated using data blocks A5 and A6. 10 Generate check block X5, and use check block P. 1-1 and P 1-3 Generate check block X6, and use check block P. 2-1 and P 2-3 Generate check block X7.

[0045] In AZ1, if any data block or parity block belonging to the same data source is corrupted, it can be recovered using other data blocks and parity blocks belonging to the same data source within AZ1. Similarly, if any data block or parity block belonging to the same data source in AZ2 is corrupted, it can be recovered using other data blocks and parity blocks belonging to the same data source within AZ2. Likewise, if any data block or parity block belonging to the same data source in AZ3 is corrupted, it can be recovered using other data blocks and parity blocks belonging to the same data source within AZ3.

[0046] For example, assuming that data block A4 belonging to the first data source in AZ1 is corrupted, it can be recovered by using data blocks A1-A3, data block A5, and check block P from the first data source. 1-1 and check block P 2-1 Recovery can then proceed. Similarly, if data block A6 in AZ2, belonging to the first data source, is corrupted, it can be recovered via data blocks A7-A of the first data source. 10 and check block P 1-2 and check block P 2-2 To perform recovery, if checksum block X8 belonging to the second data source in AZ3 is corrupted, it can be recovered using checksum blocks X9-X from the second data source. 14 To restore.

[0047] If a corrupted data block in an Availability Zone (AZ) cannot be recovered using data blocks or check blocks stored within that AZ, it can be recovered using data blocks or check blocks from the other two AZs. For example, if data blocks A1 to A4 from the first data source in AZ1 are corrupted, data block A1 can be recovered using data block A6 from the first data source in AZ2 and check block X1 from the first data source in AZ3. Similarly, data block A2 can be recovered using data block A7 from AZ2 and check block X2 from AZ3, data block A3 can be recovered using data block A8 from AZ2 and check block X3 from AZ3, and data block A4 can be recovered using data block A9 from AZ2 and check block X4 from AZ3.

[0048] Therefore, the multi-AZ erasure coding storage scheme described above can achieve reliable data storage and enable data recovery in the event of data corruption.

[0049] When a user needs to read multiple data blocks, the service node receiving the data read request, after determining the Availability Zone (AZ) where each data block is located and the data node storing it within that AZ, reads the corresponding data block from the corresponding data node in the multiple AZs. Combined with... Figure 1 In the example below, assuming all data blocks are intact, if the first service node receives a data read request corresponding to the first data source, and assuming the data block to be read corresponding to this data read request is A1-A... 10 Then the first service node needs to read data blocks A1-A5 from the data node of AZ1 and data blocks A6-A5 from the data node of AZ2. 10 Similarly, if the second service node receives a data read request corresponding to the second data source, and assumes that the data block to be read corresponding to this data read request is B1-B... 10 Then the second service node needs to read data blocks B1-B5 from the data node of AZ1 and data blocks B6-B5 from the data node of AZ2. 10 This type of cross-AZ data reading operation results in significant bandwidth overhead.

[0050] Based on this, this invention provides a data processing method. Under the premise of data storage via multiple Availability Zones (AZs), during data writing in a distributed system, for each data source corresponding to a service node within an AZ, original copies of the data blocks corresponding to that data source, after verification processing (such as erasure coding calculation), are written into the same AZ. Similarly, identical data is written to another AZ using mirror replication. Furthermore, the distributed system periodically scans the data already written to each AZ, merging mirror copies of verification data groups from different data sources whose mirror copies are in the same AZ but whose original copies are in different AZs, to generate verification block groups. The core idea of ​​this data processing method is to store data block groups corresponding to the same data source in the same availability zone, thus avoiding cross-AZ data read traffic and reducing bandwidth overhead. Additionally, merging mirror copies of two verification data groups whose mirror copies are in the same availability zone but whose original copies are in different availability zones to generate corresponding verification block groups reduces the data redundancy ratio in the distributed system.

[0051] The data processing method and distributed system provided in the embodiments of the present invention will be described below.

[0052] Figure 2 This is a schematic diagram of the structure of a distributed system provided in an embodiment of the present invention. Figure 2 As shown, the distributed system 20 includes: a metadata node 21 and multiple service nodes located in different availability zones, such as a first service node 22 located in AZ1, a second service node 23 located in AZ2, and a third service node 24 located in AZ3.

[0053] Although Figure 2 The diagram only illustrates that each Availability Zone (AZ) has only one service node deployed, but in reality, an AZ can contain multiple service nodes. Different service nodes within the same AZ can correspond to the same or different data sources, and service nodes in different AZs can correspond to different data sources. Furthermore, Figure 2 In the illustrated embodiment, for ease of description, it is assumed that the distributed system has a unified metadata node 21. However, in reality, metadata nodes can also be set in different Availability Zones (AZs), and data is synchronized between different metadata nodes.

[0054] In this embodiment of the invention, the data source can be a data generator such as an application or a database. Different data sources can be different applications of different users or different applications of the same user. Different service nodes can be pre-assigned to different data sources, so that data read / write requests triggered by a certain data source are handled by that service node. The data that needs to be written to a certain data source will be stored in the Availability Zone (AZ) where the corresponding service node is located.

[0055] by Figure 2 Taking the service nodes illustrated in the diagram as an example, assuming that the first service node 22 corresponds to the first data source and the third service node 22 corresponds to the second data source, the data writing process of the service nodes is as follows:

[0056] The first service node 22 is used to obtain the first data block group to be written from the first data source, perform verification processing on the first data block group to obtain the first verification data group, store the original copy of the first verification data group in the first availability zone AZ1 where the first service node 22 is located, store the mirror copy of the first verification data group in the second availability zone AZ2 where the second service node 23 is located, and store the metadata information corresponding to the original copy of the first verification data group and the metadata information corresponding to the mirror copy of the first verification data group in the metadata node 21. The first verification data group includes the first data block group and multiple verification blocks.

[0057] The third service node 24 is used to obtain the second data block group to be written from the second data source, perform verification processing on the second data block group to obtain the second verification data group, store the original copy of the second verification data group in the third availability zone AZ3 where the third service node 24 is located, store the mirror copy of the second verification data group in the second availability zone AZ2 where the second service node 23 is located, and store the metadata information corresponding to the original copy of the second verification data group and the metadata information corresponding to the mirror copy of the second verification data group in the metadata node 21. The second verification data group includes the second data block group and multiple verification blocks. The first service node 22, the second service node 23 and the third service node 24 are different service nodes among multiple service nodes.

[0058] Metadata node 21 is used to store metadata information corresponding to the original copy of the first verification data group, metadata information corresponding to the mirror copy of the first verification data group, metadata information corresponding to the original copy of the second verification data group, and metadata information corresponding to the mirror copy of the second verification data group.

[0059] The second service node 23 is used to query the metadata node 21 to determine the mirror copy of the first verification data group and the mirror copy of the second verification data group stored in the second availability zone AZ2 in the form of mirror copies. Based on the mirror copy of the first verification data group and the mirror copy of the second verification data group, it generates the verification block group corresponding to the first verification data group and the second verification data group, and stores the metadata information corresponding to the verification block group to the metadata node 21.

[0060] Metadata node 21 is also used to update the metadata information corresponding to the mirror copy of the first verification data group and the metadata information corresponding to the mirror copy of the second verification data group based on the metadata information corresponding to the verification block group.

[0061] In the above scheme, both the first service node 22 and the third service node 24, upon receiving a data write request triggered by their respective data sources, perform verification processing on the corresponding data block groups to obtain a verification data group. They then store the original copy of the verification data group within their respective Availability Zone (AZ) and its mirror copy in the AZ2 corresponding to the second service node 23, thus completing the data write process. The distributed system periodically performs a scan and merge task of the written data: at this time, the second service node 23 discovers that two mirror copies of the verification data group are stored in its AZ2, and that the original copies of these two verification data groups are located in two different AZs (AZ1 and AZ3) different from its own AZ2. It merges these two verification data groups to reduce the amount of data stored. During the above data write and merge process, the service nodes need to access the metadata node 21 to write or read the required metadata information.

[0062] The working process of the above-described distributed system will be described in detail below with reference to the accompanying drawings and the following embodiments.

[0063] Figure 3 A flowchart of a data processing method provided in an embodiment of the present invention, such as... Figure 3 As shown, the method may include the following steps:

[0064] 301. The first service node responds to the data write request from the first data source and determines the first data block group that needs to be written.

[0065] 302. The first service node performs verification processing on the first data block group to obtain the first verification data group, which includes the first data block group and multiple verification blocks.

[0066] 303. The first service node stores the original copy of the first verification data group in the first availability zone where the first service node is located, and stores the mirror copy of the first verification data group in the second availability zone where the second service node is located.

[0067] The data write request received by the first service node may include the identifier of the first data source and the data to be written, which may be one or more of the following: text data, image data, video data, or audio data. In response to the data write request from the first data source, the first service node segments the data to be written to obtain a first data block group. Then, the first service node performs verification processing on the first data block group, for example, by performing erasure coding calculations on the first data block group to generate multiple verification blocks, and the first data block group and the multiple verification blocks are identified as the first verification data group.

[0068] When the first service node performs a data write operation, it stores an original copy of the first verification data group in the first availability zone where it is located, and then stores a mirror copy of the first verification data group in the second availability zone where the second service node is located. The first and second service nodes are different service nodes among multiple service nodes in the distributed system. Simply put, mirroring is copying the data.

[0069] The mirror copy of the first verification data group is used to enable the second service node to generate the first verification block group corresponding to the first verification data group and the second verification data group based on the mirror copy of the first verification data group and the mirror copy of the second verification data group, and to store the metadata information of the first verification block group in the metadata node. The mirror copy of the second verification data group is stored in the second availability zone by the third service node located in the third availability zone that serves the second data source.

[0070] Similarly, in response to the data write request from the second data source, the third service node segments the data to be written into a second data block group. Then, the third service node performs verification processing on the second data block group; for example, it performs erasure coding calculations on the second data block group to generate multiple check blocks, and the second data block group and the multiple check blocks are then identified as the second check data group.

[0071] When performing a data write operation, the third service node stores the original copy of the second verification data group in the third availability zone where the third service node is located, and stores a mirror copy of the second verification data group in the second availability zone where the second service node is located. The third service node is any service node in the distributed system that is different from the first and second service nodes.

[0072] It is understood that the time when the first service node and the third service node respond to the data write request and perform the data write operation may be the same or different; this embodiment does not specifically limit this. For example, refer to... Figure 4 Taking three Availability Zones (AZs) as an example, assuming the first service node is located in AZ1, the second service node is located in AZ2, and the third service node is located in AZ3. The first service node segments the user data received from the first data source into A1 to A2. 10 Ten data blocks are generated, and parity block P is generated using data blocks A1 to A5. 1-1 and P 2-1 Use data blocks A6 to A 10 Generate check block P 1-3 and P 2-3 This yields data blocks A1 to A5 and check block P. 1-1 and P 2-1The first check data group 41, and the data blocks A6 to A 10 and check block P 1-3 and P 2-3 The first check data group 42 is formed, where A1 to A5 form a data block group, and A6 to A 10 This is another data block group. Similarly, the third service node will divide the received user data from the second data source into B1 to B2. 10 Ten data blocks are generated, and parity block P is generated using data blocks B1 to B5. 1-2 and P 2-2 Use data blocks B6 to B 10 Generate check block P 1-4 and P 2-4 This yields data blocks B1 to B5 and check block P. 1-2 and P 2-2 The second check data group 43, and the data blocks B6 to B 10 and check block P 1-4 and P 2-4 The second verification data group 44 is formed.

[0073] The first service node stores original copies of the first check data group 41 and the first check data group 42 in the data node of AZ1, and stores mirror copies in the data node of AZ2. The third service node stores original copies of the second check data group 43 and the second check data group 44 in the data node of AZ3, and stores mirror copies in the data node of AZ2.

[0074] Furthermore, after storing the original copies of the first verification data group 41 and the first verification data group 42 to the data nodes in AZ1 and the mirror copy to the data nodes in AZ2, the first service node will also store the metadata information corresponding to the original copy of the first verification data group and the metadata information corresponding to the mirror copy of the first verification data group to the metadata node.

[0075] After storing the original copies of the first check data group 41 and the first check data group 42 in AZ1 and the mirror copies of the first check data group 41 and the first check data group 42 in AZ2, the first service node stores the metadata information corresponding to the original copies of the first check data group 41 and the first check data group 42, as well as the metadata information corresponding to the mirror copies of the first check data group 41 and the first check data group 42, in the metadata node. Specifically, the metadata information corresponding to the original copies of the first check data group 41 and the first check data group 42 is used to record the storage location of each block contained in the original copies of the first check data group 41 and the first check data group 42. The metadata information corresponding to the mirror copies of the first check data group 41 and the first check data group 42 is used to record the storage location of each block (including data blocks and check blocks) contained in the mirror copies of the first check data group 41 and the first check data group 42.

[0076] Similarly, the third service node stores the original copies of the second verification data group 43 and the second verification data group 44 in AZ3 and the mirror copies in AZ2. Then, it stores the metadata information corresponding to the original copies of the second verification data group 43 and the second verification data group 44, as well as the metadata information corresponding to the mirror copies of the second verification data group 43 and the second verification data group 44, in the metadata node. Specifically, the metadata information corresponding to the original copies of the second verification data group 43 and the second verification data group 44 records the storage location of each block contained in the original copies of the second verification data group 43 and the second verification data group 44. The metadata information corresponding to the mirror copies of the second verification data group 43 and the second verification data group 44 records the storage location of each block contained in the mirror copies of the first verification data group 41 and the first verification data group 42.

[0077] In practical applications, the first service node can determine the data node for storing each block (including data blocks and parity blocks) in the original replicas of the first check data group 41 and the first check data group 42, based on the load of each data node in its own AZ1 or through random sampling. For example, taking the seven blocks in the first check data group 41 as an example, these seven blocks can be stored in seven different data nodes in AZ1, or several blocks can be stored in the same data node; there are no specific restrictions. Then, metadata information corresponding to the first check data group 41 is generated. This metadata information can include not only the storage location of each of the seven blocks (i.e., which data node in which AZ it is stored in), but also the category label of the first check data group 41—original replica—as well as the tag of the service node corresponding to the first check data group 41 and the identifier of the AZ where that service node is located. Finally, the first service node stores the metadata information corresponding to the original replica of the first check data group 41 in the metadata node. Similarly, the metadata information corresponding to the original copy of the first verification data group 42, the original copy of the second verification data group 43, and the original copy of the second verification data group 43 each contains their respective similar metadata information, which will not be elaborated here.

[0078] Additionally, the first service node determines to store mirror copies of the first verification data group 41 and the first verification data group 42 in AZ2, which can be achieved through the following optional method: Each service node pre-stores access address information and distance information of other service nodes in AZ. Thus, the first service node can select the closest service node from the remaining service nodes based on the distance information, assuming it is the second service node. Based on the access address of the second service node, the first service node sends mirror copies of the first verification data group 41 and the first verification data group 42 to the second service node. After the second service node stores each block contained in the mirror copies of the first verification data group 41 and the first verification data group 42 in the data nodes of AZ2, it generates corresponding metadata information and feeds it back to the first service node. Thus, the first service node stores the metadata information corresponding to the mirror copies of the first verification data group 41 and the first verification data group 42 in the metadata node.

[0079] Optionally, the distributed system may also include a scheduling node. The first service node can send an allocation request to the scheduling node. The scheduling node allocates multiple data nodes in AZ2 to the first service node based on the load of the AZs where the other service nodes are located, for storing mirror copies of the first check data group 41 and the first check data group 42 of the first service node. At this time, the first service node can directly store each block in the mirror copy of the first check data group 41 and the first check data group 42 into the allocated multiple data nodes in AZ2 according to the allocation result of the scheduling node, generate the corresponding metadata information, and store it in the metadata node.

[0080] This embodiment of the invention does not limit how the first service node determines the AZ2 for storing the mirror copy and the storage process of the data nodes of each block in the mirror copy in AZ2.

[0081] Taking the mirror copy of the first verification data group 41 as an example, its corresponding metadata information may include the storage location of each block in the mirror copy, the category label of the mirror copy—mirror copy, the first service node corresponding to the original copy of the mirror copy, and the identifier of the AZ1 where the first service node is located. Other mirror copies are similar and will not be described in detail.

[0082] Through the above process, the first service node and the third service node completed the writing of their respective data.

[0083] Distributed systems periodically trigger tasks to merge written data. That is, each service node will periodically merge the data that has been written to its Availability Zone (AZ) and meets the set requirements in order to reduce storage redundancy.

[0084] Taking the second service node as an example, if the second service node determines that the mirror copies of the first verification data group 41 and the second verification data group 43 stored in AZ2 are both mirror copies stored in this AZ2, and determines that the original copies of the first verification data group 41 and the original copies of the second verification data group 43 are located in different availability zones, then the second service node determines that the mirror copies of the first verification data group 41 and the second verification data group 43 meet the merging conditions. Based on the mirror copies of the first verification data group 41 and the second verification data group 43, the second service node generates the first verification block group 45. After that, the second service node can delete the mirror copies of the first verification data group 41 and the second verification data group 43, thereby reducing the data redundancy ratio in the distributed system. Similarly, if the second service node determines that the mirror copies of the first verification data group 42 and the second verification data group 44 stored in AZ2 are both mirror copies stored in this AZ2, and determines that the original copies of the first verification data group 42 and the original copies of the second verification data group 44 are located in different availability zones, then the second service node determines that the mirror copies of the first verification data group 42 and the second verification data group 44 meet the merging conditions. Based on the mirror copies of the first verification data group 42 and the second verification data group 44, the second service node generates the first verification block group 46. After that, the second service node can delete the mirror copies of the first verification data group 42 and the second verification data group 44.

[0085] In practical applications, the second service node can query the metadata node to obtain the metadata information of each check data group already stored in AZ2. For example, based on the category tags in the metadata information of the first check data group 41 and the second check data group 43, it can be determined that the first check data group 41 and the second check data group 43 are stored in AZ2 as mirror copies, and based on the metadata information of the original copies of the first check data group 41 and the second check data group 43, it can be determined that the original copies of the two are stored in different AZs. Therefore, it can be determined that the first check data group 41 and the second check data group 43 can be merged.

[0086] The second service node performs erasure coding calculations on the i-th block in the mirror copy of the first check data group 41 and the i-th block in the mirror copy of the second check data group 43 to obtain the i-th check block in the first check block group 45. For example... Figure 4 As shown, erasure coding is performed on the first block A1 in the mirror copy of the first parity data group 41 and the first block B1 in the mirror copy of the second parity data group 43 to obtain the first parity block X1 in the first parity block group 45. Similarly, the seventh block P in the mirror copy of the first parity data group 41 is processed. 2-1 The 7th block P in the mirror copy of the second check data group 43 2-2Erasure coding is performed to obtain the 7th parity block X7 in the first parity block group 45, ultimately resulting in the first parity block group 45 composed of parity blocks X1-X7. The merging process of the mirror copy of the first parity data group 42 and the mirror copy of the second parity data group 44 is similar and will not be elaborated further. The merged result is composed of parity blocks X8-X7. 14 The first check block group 46 is formed.

[0087] Thus, the first check data group 41, the second check data group 43, and the first check block group 45 satisfy the erasure coding property. That is, if any target block in the first check data group 41 or the second check data group 43 is damaged, the damaged target block can be recovered from the target block corresponding to the damaged target block in the other check data group and the check block corresponding to the damaged target block in the first check block group 45. For example, assuming that data block B1 in the second check data group 43 is damaged, data block B1 can be recovered from data block A1 in the first check data group and check block X1 in the first check block group 45.

[0088] In addition, the second service node will store the metadata information of the first check block group 45 and the first check block group 46 to the metadata node. Taking the first check block group 45 as an example, the metadata information of the first check block group 45 is used to record the storage location information of each check block in the first check block group 45, and the identification information of the two check data groups (first check data group 41 and second check data group 43) related to the first check block group 45.

[0089] Taking the first check block group 45 as an example, after the metadata node receives the metadata information of the first check block group 45, it updates the metadata information corresponding to the mirror copy of the first check data group 41 and the mirror copy of the second check data group 43. This update process mainly involves: deleting the metadata information corresponding to the mirror copy of the first check data group 41 and the mirror copy of the second check data group 43, and storing the metadata information of the first check block group 45.

[0090] Based on the solution provided in the above embodiments of the present invention, a first verification data group is obtained by the first service node verifying the first data block group corresponding to the data write request of the first data source. The original copy of the first verification data group is then stored in the first availability zone where the first service node is located. Since the first verification data group is stored in the first availability zone, cross-availability zone data read traffic is avoided when reading data. Simultaneously, the first service node also stores a mirror copy of the first verification data group in the second availability zone where the second service node is located. This allows the second service node to generate the first verification data group and its corresponding first verification block group based on the mirror copies of the first and second verification data groups, and to store the metadata information of the first verification block group in the metadata node. Afterwards, the second service node deletes the mirror copies of the first and second verification data groups stored in the second availability zone where it is located, thereby reducing the data redundancy ratio in the distributed system. The data redundancy ratio can be defined as: (number of data blocks + number of verification blocks) / number of data blocks.

[0091] Figure 5 A flowchart of another data processing method provided in an embodiment of the present invention, such as Figure 5 As shown, the method may include the following steps:

[0092] 501. The first service node obtains a mirror copy of the third verification data group and a mirror copy of the fourth verification data group stored in the first availability zone; wherein, the mirror copy of the third verification data group is stored in the first availability zone by the second service node located in the second availability zone serving the third data source, and the mirror copy of the fourth verification data group is stored in the first availability zone by the third service node located in the third availability zone serving the second data source, and the original copy of the third verification data group is located in the second availability zone, and the original copy of the fourth verification data group is located in the third availability zone.

[0093] 502. The first service node performs a merge operation on the mirror copies of the third and fourth check data groups to obtain the second check block group corresponding to the third and fourth check data groups. The second check block group consists of multiple check blocks.

[0094] 503. The first service node stores the second check block group in the first availability zone and deletes the mirror copy of the third check data group and the mirror copy of the fourth check data group.

[0095] 504. The first service node stores the metadata information corresponding to the second check block group to the metadata node, so that the metadata node updates the metadata information corresponding to the mirror copy of the third check data group and the mirror copy of the fourth check data group according to the metadata information corresponding to the second check block group.

[0096] In this embodiment, from the perspective of the first service node, if it detects that the first availability zone where the first service node is located stores a mirror copy of the third verification data group and a mirror copy of the fourth verification data group, and the original copies of the third verification data group and the original copies of the fourth verification data group are stored in other different availability zones, the first service node can perform a merging operation on the mirror copies of the third verification data group and the mirror copies of the fourth verification data group to obtain a second verification block group including multiple verification blocks. The second verification block group is a verification block group corresponding to the third verification data group and the fourth verification data group.

[0097] Specifically, when performing the merge operation, the first service node can perform erasure coding calculations on the mirror copies of the third and fourth check data groups. In one possible implementation, the first service node performs erasure coding calculations on the i-th block in the mirror copy of the third check data group and the i-th block in the mirror copy of the fourth check data group to obtain the i-th check block in the second check block group.

[0098] For example, a mirror copy of the third check data group includes data blocks C1, C2, C3, C4, C5, and check block X. 3-1 and X 3-2 Among them, the check block X 3-1 and X 3-2 This is obtained by performing erasure coding calculations on data blocks C1, C2, C3, C4, and C5. The mirror copy of the fourth check data group includes data blocks D1, D2, D3, D4, D5, and check block X. 4-1 and X 4-2 Among them, the check block X 4-1 and X 4-2 It is obtained by performing erasure coding calculation on data blocks D1, D2, D3, D4, and D5.

[0099] When the first service node performs a merge operation on the mirror copies of the third and fourth check data groups, it performs erasure coding calculations on data blocks C1 and D1 to obtain check block T1 in the second check block group; it performs erasure coding calculations on data blocks C2 and D2 to obtain check block T2 in the second check block group; it performs erasure coding calculations on data blocks C3 and D3 to obtain check block T3 in the second check block group; it performs erasure coding calculations on data blocks C4 and D4 to obtain check block T4 in the second check block group; it performs erasure coding calculations on data blocks C5 and D5 to obtain check block T5 in the second check block group; and it performs erasure coding calculations on check block X... 3-1 and check block X 4-1 Perform erasure coding calculation to obtain parity block T6 in the second parity block group; for parity block X... 3-2and check block X 4-2 Perform erasure coding calculation to obtain parity block T7 in the second parity block group. This leads to the second parity block group consisting of T1, T2, T3, T4, T5, T6, and T7.

[0100] Subsequently, the first service node stores the second check block group in the data node of the first availability zone and deletes the mirror copies of the third and fourth check data groups to reduce the data redundancy ratio in the distributed system.

[0101] The first service node stores the metadata information corresponding to the second check block group to the metadata node. This metadata information records the storage location information of the second check block group, as well as the identification information of the third and fourth check data groups related to the second check block group. After receiving the metadata information corresponding to the second check block group, the metadata node can delete the metadata information corresponding to the previously stored mirror copies of the third and fourth check data groups. Optionally, the metadata node can also associate the metadata information of the second check block group with the metadata information corresponding to the original copies of the third and fourth check data groups, respectively, to facilitate subsequent direct queries based on the metadata information of the second check block group to retrieve the metadata information corresponding to the original copies of the third and fourth check data groups.

[0102] Based on the solution provided by the above embodiments of the present invention, the first service node performs a merging operation on the mirror copies of the third and fourth verification data groups to obtain the second verification block group, which is stored in the first availability zone. The mirror copies of the third and fourth verification data groups are deleted. By replacing the mirror copies of the third and fourth verification data groups stored in the first availability zone with the second verification block group, the data redundancy ratio of the distributed system is effectively reduced.

[0103] The process of writing data using the data processing method provided in the embodiments of the present invention has been described in detail above. The process of reading data using the data processing method provided in the embodiments of the present invention will now be described with reference to the accompanying drawings.

[0104] Figure 6 This is a flowchart illustrating yet another data processing method provided in an embodiment of the present invention. For example... Figure 6 As shown, the method may include the following steps:

[0105] 601. The first service node responds to the data read request corresponding to the first data source and determines the first data block group that needs to be read.

[0106] 602. The first service node obtains the metadata information corresponding to the original copy of the first verification data group from the metadata node;

[0107] 603. The first service node reads the original copy of the first verification data group from the first availability zone based on the metadata information corresponding to the original copy of the first verification data group to obtain the first data block group.

[0108] After receiving a data read request corresponding to the first data source, the first service node determines the first group of data blocks to be read based on the received data read request and undertakes the task. Figure 4 For example, suppose the first data block group contains data blocks A1-A5.

[0109] Subsequently, the first service node queries the metadata node for the metadata information corresponding to the first data block group. Specifically, the first service node can send a query request to the metadata node containing the identifiers of each data block in the first data block group. The metadata node then obtains the metadata information corresponding to the original copies of one or more checksum data groups corresponding to these data blocks. These data blocks may involve one or more checksum data groups. Figure 4 For example, suppose the metadata node receives the metadata information corresponding to the original copy of the first verification data group 41 and feeds it back to the first service node. Based on this metadata information, the first service node can determine the data node where each data block in the first data block group is stored in the first availability zone AZ1, so as to read the first data block group and return it to the client that initiated the data read request. Therefore, if the first data block group is not corrupted, based on the metadata information of the original copy of the first verification data group, the data blocks only need to be read in the AZ1 where the first service node is located, without needing to cross other AZs, thus reducing bandwidth overhead.

[0110] During the above data writing process, the first check data group is obtained by performing erasure coding calculation on the data blocks in the first data block group, so that the blocks (including data blocks and check blocks) in the first check data group can be mutually recovered. That is, if the data blocks or check blocks in the first check data group are damaged, they can be recovered based on other undamaged data blocks and / or check blocks.

[0111] Based on this, if the first service node determines that the first target block in the original copy of the first verification data group 41 is corrupted during the process of reading the original copy of the first verification data group 41 from the first availability zone AZ1, the first target block can be recovered based on the other blocks in the first verification data group 41, wherein the first target block is any data block contained in the first data block group.

[0112] For example, suppose the original copy of the first check data group 41 includes data blocks A1 to A5 and check block P. 1-1 P 2-1 If, during the process of reading the original copy of the first check data group 41 from AZ1, the first service node determines that the first target block A2 is corrupted, it can then use data blocks A1, A3, A4, and A5, as well as check block P, to... 1-1 P 2-1 Restore the first target block A2.

[0113] Therefore, based on the original copy of the first verification data group 41 stored in AZ1, even if only a small number of data blocks in the original copy are damaged, the damaged data blocks can still be recovered based on the original copy without the need for cross-AZ repair, thus reducing bandwidth overhead.

[0114] In cases where the first service node is unable to recover the first target block based on other blocks in the first verification data group, this embodiment of the invention also provides a data recovery scheme. Figure 7 This is a flowchart illustrating a data recovery scheme in a data processing method provided by an embodiment of the present invention. Figure 7 As shown, the method may include the following steps:

[0115] 701. If the first service node fails to recover the first target block from the other blocks in the first verification data group, it obtains the metadata information corresponding to the first verification block group and the metadata information corresponding to the original copy of the second verification data group from the metadata node.

[0116] 702. The first service node reads the first check block group from the second availability zone and the original copy of the second check data group from the third availability zone based on the metadata information corresponding to the first check block group and the metadata information corresponding to the original copy of the second check data group.

[0117] 703. The first service node recovers the first target block based on the check block in the first check block group corresponding to the first target block and the second target block in the original copy of the second check data group corresponding to the first target block.

[0118] Inherited from Figure 4For example, if the first service node fails to recover the first target block from other blocks in the original copy of the first check data group 41, it can obtain the metadata information corresponding to the first check block group 45 associated with the first check data group 41, and the metadata information corresponding to the original copy of the second check data group 43, from the metadata node. Specifically, the first service node can send a query request to the metadata node using the identification information of the first check data group 41 as the query keyword to query the metadata information of the check block group associated with the first check data group 41. The metadata node retrieves the metadata information of the first check block group 45 associated with the first check data group 41. The metadata information corresponding to the first check block group 45 records the storage location information of each check block in the first check block group 45, as well as the identification information of the two check data groups that generated the first check block group 45: the first check data group 41 and the second check data group 43. Therefore, based on the metadata information of the first check block group 45, it can be determined that the check block group associated with the first check data group 41 is the first check block group 45. The metadata node can also obtain the metadata information corresponding to the original copy of the associated second check data group 43 based on the metadata information of the first check block group 45. Then, the metadata node can send the metadata information corresponding to the original copy of the second check data group 43 and the metadata information of the first check block group 45 to the first service node.

[0119] Subsequently, the first service node reads the first check block group 45 from the second availability zone AZ2 based on the metadata information of the first check block group 45, and reads the original copy of the second check data group 43 from the third availability zone AZ3 based on the metadata information corresponding to the original copy of the second check data group 43. Thus, the first target block is recovered based on the check blocks in the first check block group 45 corresponding to the first target block and the second target block in the original copy of the second check data group 43 corresponding to the first target block.

[0120] Accept Figure 4 For example, assuming the first target block is data block A4 in the original copy of the first check data group 41, the first service node can recover data block A4 based on check block X4 in the first check block group 45 and data block B4 in the original copy of the second check data group 43.

[0121] Based on the solution provided by the above embodiments of the present invention, when the first service node cannot recover the first target block from other blocks in the first verification data group, the first service node reads the first verification block group from the second availability zone and the original copy of the second verification data group from the third availability zone based on the metadata information corresponding to the first verification block group and the metadata information corresponding to the original copy of the second verification data group. Based on the verification block in the first verification block group that corresponds to the first target block and the second target block in the original copy of the second verification data group that corresponds to the first target block, the first target block is recovered, thereby realizing data recovery.

[0122] The service nodes of one or more embodiments of the present invention will be described in detail below. Those skilled in the art will understand that these devices can be configured using commercially available hardware components through the steps taught in this solution.

[0123] Figure 8 This is a schematic diagram of the structure of a first service node provided in an embodiment of the present invention, as shown below. Figure 8 As shown, the first service node 80 includes: a response module 81, a verification module 82, and a first storage module 83.

[0124] The response module 81 is used to respond to the data write request from the first data source and determine the first data block group that needs to be written.

[0125] The verification module 82 is used to perform verification processing on the first data block group to obtain a first verification data group, which includes the first data block group and multiple verification blocks.

[0126] The first storage module 83 is used to store the original copy of the first verification data group in the first availability zone where the first service node is located, and to store the mirror copy of the first verification data group in the second availability zone where the second service node is located. The first service node and the second service node are different service nodes among multiple service nodes.

[0127] Figure 8 The first service device shown can perform the steps in the foregoing embodiments. For detailed execution process and technical effects, please refer to the description in the foregoing embodiments, which will not be repeated here.

[0128] This invention also provides an electronic device, such as... Figure 9 As shown, the electronic device may include: a processor 91, a memory 92, and a communication interface 93. The memory 92 stores executable code, which, when executed by the processor 91, enables the processor 91 to at least implement the data processing method provided in the foregoing embodiments.

[0129] In addition, embodiments of the present invention provide a non-transitory machine-readable storage medium on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor is able to implement at least the data processing method provided in the foregoing embodiments.

[0130] In addition, embodiments of the present invention provide a computer program product, which includes a computer program that, when executed by a processor of an electronic device, enables the processor to at least implement the data processing method provided in the foregoing embodiments.

[0131] The device embodiments described above are merely illustrative, and the network elements described as separate components may or may not be physically separate. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any inventive effort.

[0132] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of a necessary general-purpose hardware platform, or by a combination of hardware and software. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a computer product. The present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data processing method applied to a distributed system, the distributed system comprising multiple service nodes located in different availability zones, characterized in that, The method includes: The first service node responds to the data write request from the first data source and determines the first group of data blocks to be written. The first service node performs verification processing on the first data block group to obtain a first verification data group, which includes the first data block group and multiple verification blocks. The first service node stores an original copy of the first verification data group in the first availability zone where the first service node is located, and stores a mirror copy of the first verification data group in the second availability zone where the second service node is located. The first service node and the second service node are different service nodes among the plurality of service nodes.

2. The method according to claim 1, characterized in that, The mirror copy of the first verification data group is used to enable the second service node to generate a first verification block group corresponding to the first verification data group and the second verification data group based on the mirror copy of the first verification data group and the mirror copy of the second verification data group, and to store the metadata information of the first verification block group in the metadata node. The mirror copy of the second verification data group is stored in the second availability zone by the third service node located in the third availability zone serving the second data source. The original copy of the second verification data group is located in the third availability zone. The third service node is a service node among the plurality of service nodes that is different from the first service node and the second service node.

3. The method according to claim 1, characterized in that, The method further includes: The first service node stores the metadata information corresponding to the original copy of the first verification data group and the metadata information corresponding to the mirror copy of the first verification data group in the metadata node.

4. The method according to claim 1, characterized in that, The method further includes: The first service node obtains a mirror copy of the third verification data group and a mirror copy of the fourth verification data group stored in the first availability zone; wherein, the mirror copy of the third verification data group is stored in the first availability zone by the second service node located in the second availability zone serving the third data source, and the mirror copy of the fourth verification data group is stored in the first availability zone by the third service node located in the third availability zone serving the second data source; the original copy of the third verification data group is located in the second availability zone, and the original copy of the fourth verification data group is located in the third availability zone. The first service node performs a merging operation on the mirror copy of the third verification data group and the mirror copy of the fourth verification data group to obtain a second verification block group corresponding to the third verification data group and the fourth verification data group. The second verification block group consists of multiple verification blocks. The first service node stores the second check block group in the first availability zone and deletes the mirror copy of the third check data group and the mirror copy of the fourth check data group.

5. The method according to claim 4, characterized in that, The method further includes: The first service node stores the metadata information corresponding to the second check block group to the metadata node, so that the metadata node updates the metadata information corresponding to the mirror copy of the third check data group and the mirror copy of the fourth check data group according to the metadata information corresponding to the second check block group.

6. The method according to claim 4, characterized in that, The first service node performs a merge operation on the mirror copies of the third and fourth check data groups to obtain a second check block group corresponding to the third and fourth check data groups, including: The first service node performs erasure coding calculation on the mirror copy of the third check data group and the mirror copy of the fourth check data group to obtain the second check block group.

7. The method according to claim 6, characterized in that, The first service node performs erasure coding calculations on the mirror copies of the third and fourth check data groups to obtain the second check block group, including: The first service node performs erasure coding calculations on the i-th block in the mirror copy of the third check data group and the i-th block in the mirror copy of the fourth check data group to obtain the i-th check block in the second check block group.

8. The method according to any one of claims 1-7, characterized in that, The method further includes: The first service node responds to the data read request corresponding to the first data source and determines the first data block group that needs to be read. The first service node obtains the metadata information corresponding to the original copy of the first verification data group from the metadata node; The first service node reads the original copy of the first verification data group from the first availability zone based on the metadata information corresponding to the original copy of the first verification data group, so as to obtain the first data block group.

9. The method according to claim 8, characterized in that, The method further includes: If, during the process of reading the original copy of the first verification data group from the first availability zone, the first service node determines that the first target block in the original copy of the first verification data group is corrupted, then the first target block is recovered based on the other blocks in the first verification data group.

10. The method according to claim 9, characterized in that, The method further includes: If the first service node fails to recover the first target block based on other blocks in the first verification data group, it obtains the metadata information corresponding to the first verification block group and the metadata information corresponding to the original copy of the second verification data group from the metadata node. The first service node reads the first check block group from the second availability zone and the original copy of the second check data group from the third availability zone based on the metadata information corresponding to the first check block group and the metadata information corresponding to the original copy of the second check data group. The first service node recovers the first target block based on the check block in the first check block group corresponding to the first target block and the second target block in the original copy of the second check data group corresponding to the first target block.

11. A distributed system, characterized in that, include: Multiple service nodes located in different availability zones; The first service node is used to obtain the first data block group to be written from the first data source, perform verification processing on the first data block group to obtain the first verification data group, store the original copy of the first verification data group in the first availability zone where the first service node is located, and store the mirror copy of the first verification data group in the second availability zone where the second service node is located. The first verification data group includes the first data block group and multiple verification blocks. The third service node is used to obtain the second data block group to be written from the second data source, perform verification processing on the second data block group to obtain the second verification data group, store the original copy of the second verification data group in the third availability zone where the third service node is located, and store the mirror copy of the second verification data group in the second availability zone where the second service node is located. The second verification data group includes the second data block group and multiple verification blocks. The first service node, the second service node, and the third service node are different service nodes among the multiple service nodes.

12. An electronic device, characterized in that, include: The system includes a memory, a processor, and a communication interface; wherein the memory stores executable code, which, when executed by the processor, causes the processor to perform the method as described in any one of claims 1 to 10.

13. A non-transitory machine-readable storage medium, characterized in that, The non-transitory machine-readable storage medium stores executable code that, when executed by a processor of an electronic device, causes the processor to perform the method as described in any one of claims 1 to 10.

14. A computer program product, characterized in that, include: A computer program, when executed by a processor of an electronic device, causes the processor to perform the method as described in any one of claims 1 to 10.

Images