Data backup and recovery method and system based on high-performance erasure code optimization

Abstract

The invention provides a data backup and recovery system based on high-performance erasure code optimization, and aims to solve the problems of data loss, low storage efficiency, slow data recovery speed and the like. The system realizes efficient data fragmentation and redundant information addition through the steps of data verification and slicing, twiddle factor pre-calculation, Fourier transform coding and the like. A set of complete data backup and recovery system is designed, the coding blocks are checked and stored, and data are backed up regularly; when the data is lost, the data is recovered through decoding calculation. Meanwhile, the system has monitoring and management functions, monitors states of storage nodes and coding blocks in real time, records backup and recovery logs, triggers alarm and performs data protection.

Description

Technical Field

[0001] The present invention relates to a data backup and recovery method and system based on high-performance erasure code optimization, and belongs to the field of database intelligent operation and maintenance. Background Art

[0002] With the rapid development of informatization today, database technology has made rapid progress and gradually occupied an important position. Databases not only carry the core data of enterprises and organizations, but also support the normal operation of various applications. The information technology field is paying more and more attention to how to effectively and securely manage and store data.

[0003] The data storage management method of the server directly determines the security, reliability and efficiency of the data. Traditional data storage methods often adopt simple backup and replication strategies, which not only takes up a lot of storage space, but also has certain risks in data security. Although traditional backup methods, such as full backup, incremental backup and differential backup, can provide a certain degree of data protection, they are often inefficient when processing massive amounts of data and require a high level of storage space.

[0004] To solve these problems, data centers and storage clusters usually adopt a multi-copy data redundancy strategy, that is, saving multiple copies of data in different storage devices or locations. This strategy has high access performance and can quickly switch to other copies when a copy is damaged or lost, ensuring high availability and reliability of data. However, the storage redundancy and storage cost of the multi-copy data redundancy strategy are high. For enterprises and organizations, this method is becoming increasingly uneconomical and unsustainable when facing the ever-increasing amount of data. Especially in the era of big data, how to improve the efficiency of backup and recovery and reduce storage costs while ensuring data security and reliability has become an urgent problem to be solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Figure 1 A flow chart is designed for the system of the present invention.

[0006] Figure 2 This is a flow chart of the data slicing and encoding module of the present invention. Summary of the invention

[0007] In order to solve the problems of data loss, low storage efficiency, slow data recovery, etc., this application proposes a data backup and recovery method based on high-performance erasure code optimization. Through data verification and slicing, erasure code technology, and Fourier transform coding, data is fragmented and redundant information is added. The method includes the following steps:

[0008] (1) Data slicing and encoding: Slice the original data and generate check blocks, and optimize the efficiency of erasure codes through improved Fourier transform and Cauchy matrix;

[0009] (2) Coded block storage: verify and store the coded blocks, and regularly back up and store the erasure coded data blocks;

[0010] (3) Data recovery: detect lost data, decode available blocks and recover the original data;

[0011] (4) Monitoring and management: Real-time monitoring of encoding blocks and storage status, and providing a management interface.

[0012] Furthermore, the specific steps of step (1) are as follows:

[0013] 1.1 Data verification and slicing: Generate a verification code before data slicing. At the same time, the database node divides the data D in the database into pieces with a length of N = 4 w data blocks, each data block is denoted as D i , where i = 1, 2, ..., k, k is the number of data blocks, and w is an integer;

[0014] 1.2 Determine the encoding parameters: Determine the parameters k and m, where m is the number of check blocks, and the total number of encoding blocks l = k + m.

[0015] 1.3 Precompute the rotation factors: Select a finite field GF(p), where p is a prime number or a power of a prime number, α i is a primitive root of the finite field GF(p), and the rotation factor is Where q is the parameter of the rotation factor corresponding to the output frequency point of the spectrum, s is the index range of the sub-problem, and h is the imaginary unit.

[0016] 1.4 Decomposing data blocks and calculating spectrum

[0017] 1.4.1 Data block D i Indicated as D i [4s+r], r is the subsequence index, r = 0, 1, 2, 3;

[0018] 1.4.2 Calculating the subproblem spectrum in is the specific representation of the frequency rotation factor;

[0019] 1.4.3 Combine the results of the sub-problems to get the final spectrum Where t is the index of the subproblem.

[0020] 1.5 Data Encoding

[0021] 1.5.1. Select x i and i :Select k different elements from the finite field GF(p) as x i, and m different elements as y j ; Calculate the elements of the Cauchy matrix C Select an appropriate submatrix from the Cauchy matrix C as the generator matrix G;

[0022] 1.5.2. Express the combined spectrum X[4q+t] as Where n is the spectrum index after re-indexing, n∈(0,N-1), and the final data block D=[D(α 0 ),D(α 1 ),…,D(α n-1 )]; use the generator matrix G to encode the data block to obtain the final encoded block E = G·D.

[0023] Based on the above method, the present application further proposes a data backup and recovery system based on high-performance erasure code optimization, the system comprising:

[0024] (1) Data slicing and encoding module: This module slices the original data and generates check blocks, and optimizes the efficiency of erasure codes through improved Fourier transform and Cauchy matrix;

[0025] (2) Coded block storage module: This module verifies and stores the coded blocks, and regularly backs up and stores the erasure-coded data blocks;

[0026] (3) Data recovery module: This module detects lost data, decodes available blocks and recovers the original data;

[0027] (4) Monitoring and management module: This module monitors the encoding blocks and storage status in real time and provides a management interface.

[0028] Furthermore, the specific steps of the data slicing and encoding module are as follows:

[0029] 1.1 Data verification and slicing: Generate a verification code before data slicing. At the same time, the database node divides the data D in the database into pieces with a length of N = 4 w data blocks, each data block is denoted as D i , where i = 1, 2, ..., k, k is the number of data blocks, and w is an integer;

[0030] 1.2 Determine the encoding parameters: Determine the parameters k and m, where m is the number of check blocks, and the total number of encoding blocks l = k + m.

[0031] 1.3 Precompute the rotation factors: Select a finite field GF(p), where p is a prime number or a power of a prime number, α i is a primitive root of the finite field GF(p), and the rotation factor is Where q is the parameter of the rotation factor corresponding to the output frequency point of the spectrum, s is the index range of the sub-problem, and h is the imaginary unit.

[0032] 1.4 Decomposing data blocks and calculating spectrum

[0033] 1.4.1 Data block D i Indicated as D i [4s+r], r is the subsequence index, r = 0, 1, 2, 3;

[0034] 1.4.2 Calculating the subproblem spectrum in is the specific representation of the frequency rotation factor;

[0035] 1.4.3 Combine the results of the sub-problems to get the final spectrum Where t is the index of the subproblem.

[0036] 1.5 Data Encoding

[0037] 1.5.1. Select x i and i :Select k different elements from the finite field GF(p) as x i , and m different elements as y j ; Calculate the elements of the Cauchy matrix C Select an appropriate submatrix from the Cauchy matrix C as the generator matrix G;

[0038] 1.5.2. Express the combined spectrum X[4q+t] as Where n is the spectrum index after re-indexing, n∈(0,N-1), and the final data block D=[D(α 0 ),D(α 1 ),…,D(α n-1 )]; use the generator matrix G to encode the data block to obtain the final encoded block E = G·D.

[0039] This application introduces erasure code technology. Even if some data blocks are lost due to hardware failure or network error, the system can still recover data through redundant check blocks to ensure data integrity and reliability. At the same time, the system optimizes the calculation process of erasure code through Fourier transform coding and Cauchy matrix, and improves the efficiency of data backup and recovery. The introduction of pre-calculation design reduces the computational complexity, makes the processing of large data blocks more efficient, and supports parallel computing. In addition, the generator matrix constructed using the Cauchy matrix improves the error resistance of the coding block, enhances the stability and reliability of the data block during transmission and storage, and reduces the risk of data loss. When the system detects that a data block is lost or damaged, it can quickly obtain the available coding block from the storage node, and restore the lost data block through decoding calculation to ensure the availability and continuity of the data. DETAILED DESCRIPTION

[0040] Example 1

[0041] This application proposes a data backup and recovery system based on high-performance erasure code optimization. The system fragments data and adds redundant information through steps such as data verification and slicing, erasure code technology, and Fourier transform coding. In addition, the system introduces a coding block storage and monitoring management mechanism, and improves overall performance through anomaly detection algorithms. The system design flow chart is shown below: Figure 1 shown.

[0042] 1. Data slicing and encoding module

[0043] In the database system storage and transmission environment, data blocks may be lost or damaged due to hardware failure, network errors or other reasons. A mechanism is needed to ensure the reliability of data so that complete data can be restored even if some data blocks are lost. Therefore, this module divides the original data into multiple data blocks and generates additional check blocks for recovering lost data blocks. It also uses the improved Fourier transform and Cauchy matrix technology to further optimize the erasure code and improve the efficiency and reliability of database data backup and recovery. Figure 2 The data slicing and encoding process is demonstrated.

[0044] (1) Data verification and slicing: A verification code is generated before data slicing. At the same time, the database node divides the data D in the database into pieces of length N = 4 w data blocks, each data block is denoted as D i , where i = 1, 2, ... k, k is the number of data blocks, and w is an integer that determines the length of the data block.

[0045] (2) Determine the coding parameters: Determine two parameters k and m, where k is the number of data blocks and m is the number of check blocks. The total number of coding blocks is l = k + m.

[0046] (3) Precompute the rotation factors: Select a finite field: Select a finite field GF(p), where p is usually a prime number or a power of a prime number, α i is a primitive root of the finite field GF(p), and the rotation factor is Where q is the parameter of the rotation factor corresponding to the output frequency point of the spectrum, s is the index range of the subproblem, and h is the imaginary unit.

[0047] This step calculates and stores these rotation factors in advance, which means that when performing subsequent calculations, there is no need to repeatedly calculate frequently occurring exponential terms, which can significantly improve the execution efficiency of the algorithm.

[0048] (4) Decompose the data block and calculate the spectrum

[0049] a. Decompose the input data block: i Indicated as Di [4s+r], i.e. data block D i The position of the rth data element in the sth subsequence, r is the subsequence index, which is used to distinguish each data element in the subsequence, r = 0, 1, 2, 3

[0050] b. Calculate the sub-problem spectrum: in The twiddle factors used in the frequency are specifically indicated.

[0051] c. Combined results: Combine the results of the sub-problems to get the final spectrum X[4q+t]: Where t represents the index of the subproblem.

[0052] Traditional Fourier transform processes the entire data block, which means that the amount of calculation is huge when processing large data blocks, which is difficult to optimize and parallelize. However, this application decomposes the data block into multiple subsequences through the above method, allowing the spectrum of each subsequence to be calculated in parallel. At the same time, the rotation factor is used for frequency rotation, making the spectrum calculation more efficient, and the accuracy and stability of the system are improved.

[0053] (5) Data encoding

[0054] In the data encoding stage, in order to improve the error resistance of the coding block, the Cauchy matrix is ​​used to construct the generator matrix G, and the data block is represented by a polynomial, making the encoding process more structured and optimized, which is conducive to improving the encoding and decoding efficiency. The data encoding steps are as follows:

[0055] a. Use Cauchy to construct the generator matrix

[0056] Select x i and i :Select k different elements from the finite field GF(p) as x i , and m different elements as y j ;

[0057] Compute the elements of the Cauchy matrix C:

[0058] Construct the generator matrix: Select an appropriate submatrix from the Cauchy matrix C as the generator matrix G.

[0059] b. Data block encoding:

[0060] Data block polynomial representation: Represent the combined spectrum X[4q+t] as a polynomial Where n represents the spectrum index after re-indexing, n∈(0,N-1), that is, the final data block is expressed as: D=[D(α 0 ),D(α 1),…,D(α n-1 )].

[0061] Use the generator matrix G to encode the data block and get the final encoded block E: E = G·D

[0062] 2. Coding block storage module

[0063] Receive the coding blocks from the coding module, check the coding blocks at this stage to verify data consistency, and then determine the storage node for each coding block through the storage allocator.

[0064] Regularly perform database data backup operations, generate and store erasure coded data blocks, store one coded block on each node to ensure distributed data storage, and transfer the coded blocks to the corresponding storage nodes for storage

[0065] 3. Data recovery module:

[0066] (1) Detecting data loss: When a database node detects that some data blocks are lost or damaged, it requests recovery from the recovery module.

[0067] (2) Obtaining available coding blocks: The recovery module obtains the remaining available coding blocks from the storage node.

[0068] (3) Decoding calculation D = G -1 ·E

[0069] (4) Recovered data sending: The recovery module sends the recovered data blocks back to the database node.

[0070] 4. Monitoring and management module

[0071] Monitor storage node and encoding block status in real time, record logs of backup and recovery processes, trigger alarms when data is lost or damaged, and allow administrators to view system status and manage data backup and recovery through a user interface.

[0072] Through the anomaly detection algorithm, when a storage node exceeds the preset threshold or exhibits abnormal behavior, an alarm is triggered to notify the administrator.

[0073] The administrator initiates corresponding data protection measures (such as temporarily disabling abnormal nodes, starting data migration processes, and backing up important data) based on the type and severity of the exception.

[0074] Example 2

[0075] Assume that there is a database data block D, which consists of 4 data blocks D0, D1, D2, and D3. Next, the improved erasure coding technology is used for backup and recovery, where the selected parameters k=4 and m=2, which means there are 4 data blocks and 2 check blocks, for a total of 6 coding blocks.

[0076] 1. Data verification and slicing: D = [D0, D1, D2, D3], assuming that the data of each data block is as follows:

[0077] D1:[1,2,3,, 4D2:[5,6,7,8], D3:[9,10,11,12], D4:[13,14,15,16]

[0078] 2. Calculate the rotation factor: Select the finite field GF(7), select a primitive root of the finite field α=3, and calculate

[0079]

[0080] 3. Use the Cauchy matrix to construct the generator matrix G: Select x i =[1,2,3,4] and y i =[1,2,3,4,5,6], calculate the Cauchy matrix C: Construct the generator matrix G.

[0081] 4. Encode data: Represent the data block as a polynomial and perform Fourier transform encoding. Assume that the encoded data block is as follows:

[0082] C1=[c11,c12,c13,c14,c15,c16]

[0083] C2=[c21,c22,c23,c24,c25,c26]

[0084] C3=[c31,c32,c33,c34,c35,c36]

[0085] C4=[c41,c42,c43,c44,c45,c46]

[0086] 5. Data recovery:

[0087] Assume that data blocks D2 and D3 are lost and data recovery is required: the recovery module obtains the remaining available coding blocks from the storage node, uses the generator matrix G and the coding blocks to perform decoding calculations, and recovers the original data: D = G -1 E. The recovery module sends the recovered data blocks back to the database node.

[0088] Compared with the existing technology, although it can provide basic data redundancy and recovery functions, it has insufficient recovery capabilities when multiple data blocks are lost or damaged at the same time, and lacks an effective pre-calculation mechanism for the frequently calculated parts, resulting in the need for repeated calculations in actual operation, which increases the calculation burden and response time of the system. This implementation method reflects that after the data is lost, the system can quickly obtain the available coding blocks from the storage node, and use the generation matrix G and the coding blocks for decoding to restore the original data, and use the pre-calculation of Fourier transform and rotation factors to support parallel computing during the encoding and decoding process. It has significant technical advantages in data reliability, error resistance, flexibility, data recovery success rate, and parallel computing support, and can better meet the high requirements of modern database systems for data backup and recovery.

[0089] The units, devices or modules described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. For the convenience of description, the above devices are described separately by functions divided into various modules. Of course, when implementing this application, the functions of each module can be implemented in the same or more software and / or hardware, or the modules that implement the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0090] Those skilled in the art also know that, in addition to implementing the controller in a purely computer-readable program code, the controller can be made to implement the same function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered as a hardware component, and the devices for implementing various functions included therein can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules for implementing the method and structures within the hardware component.

[0091] The present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc. that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0092] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a mobile terminal, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0093] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.

[0094] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present application. It should be understood that the above description is only a specific embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A data backup and recovery method based on high-performance erasure code optimization, which slices data and adds redundant information through data verification and slicing, erasure code technology, and Fourier transform coding, characterized in that: The method comprises the following steps: (1) Data slicing and encoding: Slice the original data and generate check blocks, and optimize the efficiency of erasure codes through improved Fourier transform and Cauchy matrix; (2) Coded block storage: verify and store the coded blocks, and regularly back up and store the erasure coded data blocks; (3) Data recovery: detect lost data, decode available blocks and recover the original data; (4) Monitoring and management: Real-time monitoring of encoding blocks and storage status, and providing a management interface.

2. A data backup and recovery method based on high-performance erasure code optimization according to claim 1, characterized in that: The specific steps of step (1) are as follows: 1.1 Data verification and slicing: Generate a verification code before data slicing. At the same time, the database node divides the data D in the database into pieces with a length of N = 4 w data blocks, each data block is denoted as D i , where i = 1, 2, ..., k, k is the number of data blocks, and w is an integer; 1.2 Determine the encoding parameters: Determine the parameters k and m, where m is the number of check blocks, and the total number of encoding blocks l = k + m. 1.3 Precompute the rotation factors: Select a finite field GF(p), where p is a prime number or a power of a prime number, α i is a primitive root of the finite field GF(p), and the rotation factor is Where q is the parameter of the rotation factor corresponding to the output frequency point of the spectrum, s is the index range of the sub-problem, and h is the imaginary unit. 1.4 Decomposing data blocks and calculating the spectrum 1.4.1 Data block D i Indicated as D i [4s+r], r is the subsequence index, r = 0, 1, 2, 3; 1.4.2 Calculating the subproblem spectrum in is the specific representation of the frequency rotation factor; 1.4.3 Combine the results of the sub-problems to get the final spectrum Where t is the index of the subproblem. 1.5 Data Encoding 1.5.

1. Select x i and i :Select k different elements from the finite field GF(p) as x i , and m different elements as y j ; Calculate the elements of the Cauchy matrix C Select an appropriate submatrix from the Cauchy matrix C as the generator matrix G; 1.5.

2. Express the combined spectrum X[4q+t] as Where n is the spectrum index after re-indexing, n∈(0,N-1), and the final data block D=[D(α 0 ),D(α 1 ),…,D(α n-1 )]; use the generator matrix G to encode the data block to obtain the final encoded block E = G·D.

3. A data backup and recovery system based on high-performance erasure code optimization, which slices data and adds redundant information through data verification and slicing, erasure code technology, and Fourier transform coding, characterized in that: The system includes the following modules: (1) Data slicing and encoding module: This module slices the original data and generates check blocks, and optimizes the efficiency of erasure codes through improved Fourier transform and Cauchy matrix; (2) Coded block storage module: This module verifies and stores the coded blocks, and regularly backs up and stores the erasure-coded data blocks; (3) Data recovery module: This module detects lost data, decodes available blocks and recovers the original data; (4) Monitoring and management module: This module monitors the encoding blocks and storage status in real time and provides a management interface.

4. A data backup and recovery system based on high-performance erasure code optimization as claimed in claim 3, characterized in that: The specific steps of data slicing and encoding module are as follows: 1.1 Data verification and slicing: Generate a verification code before data slicing. At the same time, the database node divides the data D in the database into pieces with a length of N = 4 w data blocks, each data block is denoted as D i , where i = 1, 2, ..., k, k is the number of data blocks, and w is an integer; 1.2 Determine the encoding parameters: Determine the parameters k and m, where m is the number of check blocks, and the total number of encoding blocks l = k + m. 1.3 Precompute the rotation factors: Select a finite field GF(p), where p is a prime number or a power of a prime number, α i is a primitive root of the finite field GF(p), and the rotation factor is Where q is the parameter of the rotation factor corresponding to the output frequency point of the spectrum, s is the index range of the sub-problem, and h is the imaginary unit. 1.4 Decomposing data blocks and calculating the spectrum 1.4.1 Data block D i Indicated as D i [4s+r], r is the subsequence index, r = 0, 1, 2, 3; 1.4.2 Calculating the subproblem spectrum in is the specific representation of the frequency rotation factor; 1.4.3 Combine the results of the sub-problems to get the final spectrum Where t is the index of the subproblem. 1.5 Data Encoding 1.5.

1. Select x i and i :Select k different elements from the finite field GF(p) as x i , and m different elements as y j ; Calculate the elements of the Cauchy matrix C Select an appropriate submatrix from the Cauchy matrix C as the generator matrix G; 1.5.

2. Express the combined spectrum X[4q+t] as Where n is the spectrum index after re-indexing, n∈(0,N-1), and the final data block D=[D(α 0 ),D(α 1 ),…,D(α n-1 )]; use the generator matrix G to encode the data block to obtain the final encoded block E = G·D.

Images