Erasure coding method, system and readable storage medium for cross-domain data centers

Abstract

The present invention provides an erasure code method, system and readable storage medium for cross-domain data centers, and the method includes the following steps: S1, using a two-layer coding method to divide k data blocks into p local check groups, and using MSR coding to generate l local check blocks in each local check group, and then using RS coding to generate g global check blocks, all global check blocks are combined into a global check group, and a total of p+1 groups are generated together with the local check groups; S2, the placement strategy of p+1 groups in N data centers is modeled, and the optimization problem of access delay and recovery transmission delay is solved to obtain the group placement probability matrix P; S3, according to the group placement probability matrix P, the p+1 block groups are placed in N data centers. The present invention effectively solves the problems of large cross-data center transmission traffic and high user access delay during data recovery by using MSR and RS double-layer coding and optimizing the placement strategy.

Description

Technical Field

[0001] The present invention relates to the field of data storage technology, and in particular to an erasure coding method, system and readable storage medium for cross-domain data centers. Background Art

[0002] In recent years, in order to further increase storage capacity, reduce storage costs, and provide high fault tolerance at the data center level, cross-data center erasure code storage systems have gradually attracted attention. Existing cross-data center erasure code technologies include XORBAS, MXOR, AZCode, etc.

[0003] XORBAS: It divides the data block into several local check groups. Each check group uses XOR coding to generate a local check block. The RS code is expanded by generating local check blocks and efficiently recovered.

[0004] MXOR: It redistributes ten data blocks into two rows, five in each row, and then calculates five vertical XOR check blocks and two horizontal XOR check blocks, for a total of seven check blocks. When dealing with a single-node error, MXOR downloads the data of the two surviving nodes on the vertical check to complete the recovery task. When multiple-node errors occur, both horizontal XOR check and vertical XOR check need to be used to recover the data.

[0005] AZCode: It uses two-layer encoding. The first layer uses MSR encoding as local verification, and the second layer uses RS encoding as global verification. It reduces the transmission overhead across availability zones through local verification within the availability zone.

[0006] In a cross-data center cloud storage environment, a large amount of cross-data center transmission traffic will be generated during the data recovery process. The network delay between different data centers is much higher than the network delay within a data center, and the bandwidth is smaller than the bandwidth within a data center. Therefore, the fault recovery performance and user access performance are poor, which affects the overall performance and reliability of the system. Summary of the invention

[0007] The purpose of the present invention is to provide an erasure coding method, system and readable storage medium for cross-domain data centers, which solve the problems of large cross-data center transmission traffic and high user access delay during data recovery.

[0008] To achieve the above object, the present invention provides an erasure coding method for cross-domain data centers, comprising the following steps:

[0009] S1, using a two-layer coding method, divide k data blocks into p local check groups, and use MSR coding to generate l local check blocks in each local check group, and then use RS coding to generate g global check blocks, and all global check blocks are combined into a global check group, and together with the local check groups, a total of p+1 groups are generated;

[0010] S2. Model the placement strategy of p+1 packets in N data centers, solve the optimization problem of access delay and recovery transmission delay, and obtain the packet placement probability matrix P;

[0011] S3. According to the group placement probability matrix P, p+1 blocks are grouped and placed in N data centers.

[0012] Optionally, the S2 specifically includes:

[0013] S21. Establish the objective function and constraints for optimizing access delay:

[0014]

[0015] S i , j X i,k X j,k =01≤i≠j≤n,1≤k≤N

[0016]

[0017]

[0018] Xi, k ∈{0,1}1≤i≤n,1≤k≤N

[0019] n=p+1

[0020] Where: i, j are the i-th block group and the j-th block group in the p+1 groups, k is the number of data centers, X i,k , X j,k are indicator variables for whether block group i and block group j are located in data center k, h i is the heat of block group i, C k S is the delay of a user accessing a packet in data center k, i,j is the indicator variable of whether block group i and block group j belong to the same band, and γ is the maximum storage coefficient;

[0021] S22, determine the objective function and constraints for optimizing the recovery transmission delay:

[0022]

[0023] S i,j X i,k X j,k =01≤i≠j≤n,1≤k≤N

[0024]

[0025]

[0026] Xi ,k ∈{0,1}1≤i≤n,1≤k≤N

[0027] Where: C k,l is the delay of a packet data from data center k to data center l, X j,l is an indicator variable for whether block group j is located in data center l;

[0028] S23, combining the objective functions and constraints of optimizing access delay and optimizing recovery transmission delay into one optimization problem:

[0029]

[0030] S i,j X i,k X j,k =01≤i≠j≤n,1≤k≤N

[0031]

[0032]

[0033] X i,k ∈{0,1}1≤i≤n,1≤k≤N

[0034] In the formula, η is the weight coefficient;

[0035] S24. Convert the above optimization problem into an approximate geometric programming problem and solve it:

[0036]

[0037]

[0038]

[0039] p i,k ≤1

[0040]

[0041] In the formula, p ,k and pj ,lare the probabilities of whether block group i and block group j are located in data center k and data center l, respectively.

[0042] Optionally, the S3 specifically includes:

[0043] S31, initialization i=1;

[0044] S32, taking out the submatrix corresponding to the i-th stripe in the block group placement probability matrix P;

[0045] S33, find the largest element in the submatrix, the row index of the submatrix corresponds to the block group index, the column index of the submatrix corresponds to the data center index, place the block group in the corresponding data center, and set the row and column where the largest element is located to 0;

[0046] S34, check whether the submatrix is ​​all zero, if not, return to S3.

[0047] Optionally, S3 further includes:

[0048] S35. If the submatrix is ​​all zero, check whether all stripes have been placed. If not, increment i and return to S32. If yes, it means that all block groups have been placed.

[0049] Based on the same inventive concept, the present application also proposes an erasure code system for cross-domain data centers, including:

[0050] A coding module, which is configured to use a two-layer coding method to divide k data blocks into p local check groups, and use MSR coding to generate l local check blocks in each local check group, and then use RS coding to generate g global check blocks, and all global check blocks are combined into a global check group, and together with the local check groups, a total of p+1 groups are generated;

[0051] An optimization module is configured to model the placement strategy of p+1 packets in N data centers, solve the optimization problem of access delay and recovery transmission delay, and obtain a packet placement probability matrix P;

[0052] A placement module is configured to place the p+1 block groups in the N data centers according to the group placement probability matrix P.

[0053] Based on the same inventive concept, the present application also proposes a readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it can implement the erasure coding method for cross-domain data centers as described above.

[0054] In an erasure coding method, system and readable storage medium for cross-domain data centers provided by the present invention, by using double-layer encoding of MSR and RS and optimizing the placement strategy for optimizing access delay and recovery performance, the problems of large cross-data center transmission traffic and high user access delay during data recovery are effectively solved, thereby improving the system's recovery performance and access performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Those skilled in the art will appreciate that the accompanying drawings are provided for a better understanding of the present invention and do not constitute any limitation on the scope of the present invention.

[0056] Figure 1 A flowchart of an erasure coding method for cross-domain data centers provided by an embodiment of the present invention;

[0057] Figure 2 A schematic diagram of an erasure coding device for cross-domain data centers provided by an embodiment of the present invention.

[0058] In the attached figure:

[0059] 100- encoding module; 200- optimization module; 300- placement module. DETAILED DESCRIPTION

[0060] In order to make the purpose, advantages and features of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the accompanying drawings are in a very simplified form and use non-precise proportions, which are only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention. In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, please refer to the accompanying drawings. It should be noted that the structure, proportion, size, etc. illustrated in the drawings of this specification are only used to match the content disclosed in the specification for people familiar with this technology to understand and read, and are not used to limit the limiting conditions for the implementation of the present invention. Any modification of the structure, change in the proportional relationship or adjustment of the size, under the same or similar conditions as the effects that can be produced by the present invention and the purposes that can be achieved, should still fall within the scope of the technical content disclosed by the present invention.

[0061] As used in the present invention, the singular forms "one", "an", and "the" include plural objects, unless the content clearly indicates otherwise. As used in the present invention, the term "or" is generally used in a sense that includes "and / or", unless the content clearly indicates otherwise. As used in the present invention, the term "several" is generally used in a sense that includes "at least one", unless the content clearly indicates otherwise. As used in the present invention, the term "at least two" is generally used in a sense that includes "two or more", unless the content clearly indicates otherwise. In addition, the terms "first", "second", and "third" are used only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first", "second", and "third" may explicitly or implicitly include one or at least two of the features.

[0062] In the description of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", and "fixed" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0063] Please refer to Figure 1 , Figure 1 A flowchart of an erasure coding method for a cross-domain data center provided by an embodiment of the present invention. This embodiment provides an erasure coding method for a cross-domain data center, including the following steps:

[0064] S1, using a two-layer coding method, divide k data blocks into p local check groups, and use MSR coding to generate l local check blocks in each local check group, and then use RS coding to generate g global check blocks, and all global check blocks are combined into a global check group, and together with the local check groups, a total of p+1 groups are generated;

[0065] S2. Model the placement strategy of p+1 packets in N data centers, solve the optimization problem of access delay and recovery transmission delay, and obtain the packet placement probability matrix P;

[0066] S3. According to the group placement probability matrix P, p+1 blocks are grouped and placed in N data centers.

[0067] The present invention uses double-layer encoding of MSR and RS and optimizes the placement strategy for optimizing access delay and recovery performance, effectively solving the problems of large cross-data center transmission traffic and high user access delay during the recovery process, thereby improving the system's recovery performance and access performance.

[0068] First, execute step S1, use the two-layer coding method to divide k data blocks into p local check groups, and use MSR coding to generate l local check blocks in each local check group, and then use RS coding to generate g global check blocks, and all global check blocks are combined into a global check group, and together with the local check groups, a total of p+1 groups are generated.

[0069] Then execute step S2 to solve the group probability matrix. After completing the construction of local check and global check in step S1, we get p local check groups and 1 global check group, and need to store p+1 groups in different data centers. In order to take into account both low access latency and high recovery performance, we need to model the group placement strategy and solve the optimization problem. The specific solution steps are as follows:

[0070] S21. Establish the objective function and constraints for optimizing access delay:

[0071]

[0072] S i , j X i,k X j,k =01≤i≠j≤n,1≤k≤N

[0073]

[0074]

[0075] X i,k ∈{0,1}1≤i≤n,1≤k≤N

[0076] n=p+1

[0077] Where: i, j are the i-th block group and the j-th block group in the p+1 groups, k is the number of data centers, X i,k , X j,k are indicator variables for whether block group i and block group j are located in data center k, h i is the heat of block group i, C k S is the delay of a user accessing a packet in data center k, i,j is the indicator variable of whether block group i and block group j belong to the same band, and γ is the maximum storage coefficient;

[0078] S22, determine the objective function and constraints for optimizing the recovery transmission delay:

[0079]

[0080] S i, JXj i,k X j,k =0 1≤i≠j≤n,1≤k≤N

[0081]

[0082]

[0083] X i ,k∈{0,1}1≤i≤n,1≤k≤N

[0084] Where: C k,l is the delay of a packet data from data center k to data center l, X j,l is an indicator variable for whether block group j is located in data center l;

[0085] S23, combining the objective functions and constraints of optimizing access delay and optimizing recovery transmission delay into one optimization problem:

[0086]

[0087] S i , j X i,k X j,k =01≤i≠j≤n,1≤k≤N

[0088]

[0089]

[0090] X i,k ∈{0,1}1≤i≤n,1≤τ≤N

[0091] In the formula, η is the weight coefficient.

[0092] S24. Convert the above optimization problem into an approximate geometric programming problem and solve it:

[0093]

[0094]

[0095]

[0096] p i,k ≤1

[0097]

[0098] In the formula, p i,k and p j,l are the probabilities of whether block group i and block group j are located in data center k and data center l, respectively.

[0099] Since this problem is an n×N dimensional 0-1 integer programming problem, and 0-1 integer programming is an NP-complete problem, the time complexity of the solution increases exponentially with n and N. If the total number of block groups n and the total number of data centers N are very large, the solution is very difficult. Therefore, this embodiment converts the problem into an approximate geometric programming problem and solves it.

[0100] Finally, step S3 is executed to place the p+1 block groups in N data centers according to the group placement probability matrix P.

[0101] The S3 specifically includes:

[0102] S31, initialization i=1;

[0103] S32, taking out the submatrix corresponding to the i-th stripe in the block group placement probability matrix P;

[0104] S33, find the largest element in the submatrix, the row index of the submatrix corresponds to the block group index, the column index of the submatrix corresponds to the data center index, place the block group in the corresponding data center, and set the row and column where the largest element is located to 0;

[0105] S34, check whether the submatrix is ​​all zero, if not, return to S3.

[0106] Further, S35, if the submatrix is ​​all zero, check whether all stripes are placed, if not, increment i and return to S32, if yes, it means that all block groups are placed.

[0107] Please refer to Figure 2 Based on the same inventive concept, an embodiment of the present invention further provides an erasure code system for cross-domain data centers, including:

[0108] The coding module 100 is configured to use a two-layer coding method to divide k data blocks into p local check groups, and use MSR coding to generate l local check blocks in each local check group, and then use RS coding to generate g global check blocks, and all global check blocks are combined into a global check group, and together with the local check groups, a total of p+1 groups are generated;

[0109] An optimization module 200 is configured to model the placement strategy of p+1 packets in N data centers, solve the optimization problem of access delay and recovery transmission delay, and obtain a packet placement probability matrix P;

[0110] The placement module 300 is configured to place the p+1 block groups in N data centers according to the group placement probability matrix P.

[0111] Based on the same inventive concept, the present application also proposes a readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it can implement any one of the erasure coding methods for cross-domain data centers described in the above-mentioned feature descriptions.

[0112] The readable storage medium can be a tangible device that can keep and store the instructions used by the instruction execution device, such as but not limited to an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device or any suitable combination of the above. The more specific example (non-exhaustive list) of the readable storage medium includes: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination of the above. The computer program described here can be downloaded from the readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device through a network, such as the Internet, a local area network, a wide area network and / or a wireless network. The network can include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer program from the network and forwards the computer program for storage in a readable storage medium in each computing / processing device. The computer program for performing the operation of the present invention may be an assembly instruction, an instruction set architecture (ISA) instruction, a machine instruction, a machine-dependent instruction, a microcode, a firmware instruction, a state setting data, or a source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as "C" language or similar programming languages. The computer program may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect through the Internet). In some embodiments, by utilizing the state information of a computer program to personalize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit can execute computer-readable program instructions to implement various aspects of the present invention.

[0113] Here, various aspects of the present invention are described with reference to the flowchart and / or block diagram of the method, system and computer program product according to the embodiment of the present invention. It should be understood that each square frame of the flowchart and / or block diagram and the combination of the square frames in the flowchart and / or block diagram can be realized by a computer program. These computer programs can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device, so as to produce a machine, so that these programs, when executed by the processor of a computer or other programmable data processing device, produce a device for realizing the function / action specified in one or more square frames in the flowchart and / or block diagram. These computer programs can also be stored in a readable storage medium, and these computer programs make the computer, programmable data processing device and / or other equipment work in a specific way, so that the readable storage medium storing the computer program includes a manufactured product, which includes instructions for realizing various aspects of the function / action specified in one or more square frames in the flowchart and / or block diagram.

[0114] The computer program may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are executed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the computer program executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0115] In summary, the embodiments of the present invention provide an erasure coding method, system and readable storage medium for cross-domain data centers. By using double-layer coding of MSR and RS and optimizing the placement strategy for optimizing access delay and recovery performance, the problems of large cross-data center transmission traffic and high user access delay during data recovery are effectively solved, thereby improving the system's recovery performance and access performance.

[0116] The above description is only a description of the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any changes and modifications made by a person skilled in the art in the field of the present invention based on the above disclosure are within the scope of protection of the present invention. Obviously, a person skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations fall within the scope of the present invention and its equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for erasure coding across cross-domain data centers, characterized in that: The following steps are involved: S1, using a two-layer coding method, divide k data blocks into p local check groups, and use MSR coding to generate l local check blocks in each local check group, and then use RS coding to generate g global check blocks, and all global check blocks are combined into a global check group, and together with the local check groups, a total of p+1 groups are generated; S2. Model the placement strategy of p+1 packets in N data centers, solve the optimization problem of access delay and recovery transmission delay, and obtain the packet placement probability matrix P; S3, placing p+1 blocks in groups in N data centers according to the group placement probability matrix P; Wherein, the S2 specifically includes: S21. Establish the objective function and constraints for optimizing access delay: s.t.S i,j X i,k X j,k =0 1≤i≠j≤n,1≤k≤N X i,k ∈{0,1} 1≤i≤n,1≤k≤N n=p+1 Where: i, j are the i-th block group and the j-th block group in the p+1 groups, k is the number of data centers, X i,k , X j,k are indicator variables for whether block group i and block group j are located in data center k, h i is the heat of block group i, C k S is the delay of a user accessing a packet in data center k, i,j is the indicator variable of whether block group i and block group j belong to the same band, and γ is the maximum storage coefficient; S22, determine the objective function and constraints for optimizing the recovery transmission delay: s.t.S i,j X i,k X j,k =0 1≤i≠j≤n,1≤k≤N X i,k ∈{0,1} 1≤i≤n,1≤k≤N Where: C k,l is the delay of a packet data from data center k to data center l, X j,l is an indicator variable for whether block group j is located in data center l; S23, combining the objective functions and constraints of optimizing access delay and optimizing recovery transmission delay into one optimization problem: s.t.S i,j X i,k X j,k =0 1≤i≠j≤n,1≤k≤N X i,k ∈{0,1} 1≤i≤n,1≤k≤N Where η is the weight coefficient; S24. Convert the above optimization problem into an approximate geometric programming problem and solve it: p i,k ≤1 In the formula, p i,k and p j,l are the probabilities of whether block group i and block group j are located in data center k and data center l respectively; The S3 specifically includes: S31, initialization i=1; S32, taking out the submatrix corresponding to the i-th stripe in the block group placement probability matrix P; S33, find the largest element in the submatrix, the row index of the submatrix corresponds to the block group index, the column index of the submatrix corresponds to the data center index, place the block group in the corresponding data center, and set the row and column where the largest element is located to 0; S34, check whether the submatrix is ​​all zero, if not, return to S3; S35. If the submatrix is ​​all zero, check whether all stripes have been placed. If not, increment i and return to S32. If yes, it means that all block groups have been placed.

2. An erasure code system for cross-domain data centers, characterized in that: include: A coding module, which is configured to use a two-layer coding method to divide k data blocks into p local check groups, and use MSR coding to generate l local check blocks in each local check group, and then use RS coding to generate g global check blocks, and all global check blocks are combined into a global check group, and together with the local check groups, a total of p+1 groups are generated; An optimization module is configured to model the placement strategy of p+1 packets in N data centers, solve the optimization problem of access delay and recovery transmission delay, and obtain a packet placement probability matrix P; A placement module configured to place the p+1 block groups in the N data centers according to the group placement probability matrix P; Wherein, the optimization module is specifically configured as follows: S21. Establish the objective function and constraints for optimizing access delay: s.t.S i,j X i,k X j,k =0 1≤i≠j≤n,1≤k≤N X i,k ∈{0,1} 1≤i≤n,1≤k≤N n=p+1 Where: i, j are the i-th block group and the j-th block group in the p+1 groups, k is the number of data centers, X i,k , X i,k are indicator variables for whether block group i and block group j are located in data center k, h i is the heat of block group i, C k S is the delay of a user accessing a packet in data center k, i,j is the indicator variable of whether block group i and block group j belong to the same band, and γ is the maximum storage coefficient; S22, determine the objective function and constraints for optimizing the recovery transmission delay: s.t.S i,j X i,k X j,k =0 1≤i≠j≤n,1≤k≤N X i,k ∈{0,1} 1≤i≤n,1≤k≤N Where: C k,l is the delay of a packet data from data center k to data center l, X j,l is an indicator variable for whether block group j is located in data center l; S23, combining the objective functions and constraints of optimizing access delay and optimizing recovery transmission delay into one optimization problem: s.t.S i,j X i,k X j,k =0 1≤i≠j≤n,1≤k≤N X i,k ∈{0,1} 1≤i≤n,1≤k≤N Where η is the weight coefficient; S24. Convert the above optimization problem into an approximate geometric programming problem and solve it: p i,k ≤1 In the formula, p i,k and p j,l are the probabilities of whether block group i and block group j are located in data center k and data center l respectively; The placement module is specifically configured as follows: S31, initialization i=1; S32, taking out the submatrix corresponding to the i-th stripe in the block group placement probability matrix P; S33, find the largest element in the submatrix, the row index of the submatrix corresponds to the block group index, the column index of the submatrix corresponds to the data center index, place the block group in the corresponding data center, and set the row and column where the largest element is located to 0; S34, check whether the submatrix is ​​all zero, if not, return to S3; S35. If the submatrix is ​​all zero, check whether all stripes have been placed. If not, increment i and return to S32. If yes, it means that all block groups have been placed.

3. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it can implement the erasure coding method for cross-domain data centers described in claim 1.

Images