A data backup method, device, equipment, medium and product

Abstract

The present application discloses a data backup method, device, equipment, medium and product. The method encodes the target data to obtain a first target code and a second target code, concatenates the first target code and the second target code to obtain a target cascade code, and saves the target cascade code so that in the event that the target data is lost, the target data can be restored through the target cascade code. Since the cascade code constructs a long code by serially cascading short codes, the length of the cascade code is less than the length of the long code constructed by a single structure and a single form of code, and therefore has the advantages of low decoding complexity, strong error correction capability and high coding efficiency. This technology can be used to solve the cache recovery problem when different nodes fall behind, and can maximize the decoding success rate under certain storage redundancy to achieve effective recovery after data loss.

Description

Technical Field

[0001] The present application relates to the field of edge computing, and in particular to a data backup method, device, equipment, medium and product. Background Art

[0002] As the demand for processing massive amounts of data increases, distributed caching provides applications with fast data access services and improves data processing speed. However, large-scale distributed systems have the problem of node lag, resulting in cached data loss. Coding cache technology integrates storage and forward error correction coding technology to store redundant data on cache nodes, further improving the utilization efficiency of cache resources and achieving more efficient storage-communication resource replacement. It uses the local cache and distributed coding gain brought by coding cache technology to achieve low latency, high energy efficiency and other service requirements for cached data.

[0003] Currently, Maximum Distance Separable Code (MDS) is usually used to encode data for backup, but MDS has high computational complexity and low backup efficiency. Summary of the invention

[0004] The embodiments of the present application provide a data backup method, apparatus, device, medium and product, which can reduce the computational complexity of data backup and improve backup efficiency.

[0005] On the one hand, an embodiment of the present application provides a data backup method, including:

[0006] Obtain target data to be backed up;

[0007] Based on a preset concatenated code inner code type, encoding the target data to obtain a first target code;

[0008] Based on a preset cascade code outer code type, encoding the target data to obtain a second target code;

[0009] Concatenating the first target code and the second target code to obtain a target cascade code;

[0010] The target concatenated code is saved so that when the target data is lost, the target data can be restored through the target concatenated code.

[0011] On the other hand, the inner code type of the target concatenated code is a low density parity check code (LDPC); the target data includes a plurality of original partitions;

[0012] The step of encoding the target data based on a preset concatenated code inner code type to obtain a first target code comprises:

[0013] Based on LDPC, the original partition is encoded to obtain a first target code partition.

[0014] On the other hand, encoding the original partition based on LDPC to obtain a first target code partition includes:

[0015] Based on the historical data of nodes that have fallen behind, predict the number of nodes that have fallen behind at the same time;

[0016] Based on LDPC, the original partition is encoded according to the number of the fallen-behind nodes to obtain the first target code partition.

[0017] On the other hand, the outer code type of the target concatenated code is Luby Transform Code (LT);

[0018] The step of encoding the target data to obtain a second target code based on a preset cascade code outer code type comprises:

[0019] Based on LT, the original partition and / or the first target code partition are encoded to obtain a second target code partition.

[0020] On the other hand, encoding the original partition and / or the first target code partition based on LT to obtain a second target code partition includes:

[0021] Get the preset number of partitions for encoding;

[0022] Selecting the preset number of target code partitions from the original partitions and / or the first target code partitions;

[0023] Based on LT, the target coding partition is encoded to obtain a second target code partition.

[0024] On the other hand, obtaining the preset number of partitions for encoding includes:

[0025] Based on the historical dropped nodes data, when different numbers of nodes dropped at the same time, determine multiple first durations required for recalculating the way to recover data;

[0026] In the case of different numbers of nodes falling behind, determining the maximum number of target coding partitions with a second duration not greater than the corresponding first duration respectively for the target number of nodes falling behind; the second duration is the duration required for recovering the target data through the target concatenated code when the target number of nodes falls behind;

[0027] The maximum number is determined as the preset partition number.

[0028] On the other hand, selecting the preset number of target code partitions from the original partitions and / or the first target code partitions includes:

[0029] Obtaining the importance of each partition in the original partition and / or the first target code partition and the falling-behind probability of the corresponding node;

[0030] Based on the importance of the partitions and the falling-behind probability of the corresponding nodes, the target coding partitions of the preset number of partitions are selected.

[0031] On the other hand, after the target concatenated code is saved, the method further includes:

[0032] In the case where the target data is detected to be lost, recovering the target data based on the first target code;

[0033] In the event that the first target code fails to recover data, the target data is recovered based on the second target code.

[0034] On the other hand, an embodiment of the present application provides a data backup device, the device comprising:

[0035] An acquisition module, used to acquire target data to be backed up;

[0036] An encoding module, configured to encode the target data based on a preset concatenated code inner code type to obtain a first target code;

[0037] The encoding module is further used to encode the target data to obtain a second target code based on a preset cascade code outer code type;

[0038] A series connection module, used for connecting the first target code and the second target code in series to obtain a target cascade code;

[0039] The saving module is used to save the target concatenated code so as to restore the target data through the target concatenated code when the target data is lost.

[0040] In another aspect, an embodiment of the present application provides a data backup device, the device comprising: a processor and a memory storing computer program instructions;

[0041] When the processor executes the computer program instructions, the data backup method as described above is implemented.

[0042] On the other hand, an embodiment of the present application provides a computer-readable storage medium, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the data backup method as described above is implemented.

[0043] On the other hand, an embodiment of the present application provides a computer program product. When instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the data backup method as described above.

[0044] The data backup method of the embodiment of the present application first obtains the target data to be backed up; and based on the preset cascade code inner code type, encodes the target data to obtain the first target code; and then based on the preset cascade code outer code type, encodes the target data to obtain the second target code. Then the first target code and the second target code are connected in series to obtain the target cascade code, and the target cascade code is saved, so that in the case of target data loss, the target data can be restored by the target cascade code. Since the cascade code constructs the long code by serial cascade of short codes, the length of the cascade code is less than the length of the long code constructed by a single structure and a single form of code, so it has the advantages of low decoding complexity, strong error correction ability and high coding efficiency, requires less computing resources, is more practical, and the cascade code is suitable for the dynamic change of the number of nodes falling behind in large-scale computing. This technology can be used to solve the cache recovery problem when different nodes fall behind, and can improve the success rate of decoding as much as possible under certain storage redundancy, so as to achieve effective recovery after data loss. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0046] Figure 1 It is a flowchart of a data backup method provided by an embodiment of the present application;

[0047] Figure 2 A coding partition diagram of the LT code provided by one embodiment of the present application;

[0048] Figure 3 An encoded RDD partition backup graph provided by an embodiment of the present application;

[0049] Figure 4 is a structural schematic diagram of a data backup device provided by another embodiment of the present application;

[0050] Figure 5 It is a structural diagram of a data backup device provided in yet another embodiment of the present application. DETAILED DESCRIPTION

[0051] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0052] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0053] As the demand for processing massive amounts of data increases, distributed caches provide applications with fast data access services and improve data processing speed. However, large-scale distributed systems have the problem of node lag, which leads to cached data loss. Current distributed cache systems usually use forward error correction codes (such as MDS) to encode cache content and store redundant encoded content. When data is lost due to node failure, the required cached data is restored through the encoded data. However, the computational complexity of MDS codes is high, and it is difficult to dynamically adapt to the number of lag nodes. In addition, the time overhead of MDS decoding and disk reading will reduce the speed of task recovery, which is not conducive to the rapid recovery of tasks.

[0054] In order to solve the problems existing in the current scheme, the embodiments of the present application provide a data backup method, device, equipment, medium and product. In order to solve the important problems existing in the current distributed cache system, such as low recovery efficiency of the data of the fallen nodes, difficulty in adaptive coding based on the fallen nodes, and high compilation complexity in the existing coding scheme, a distributed cache solution with lightweight calculation, node adaptation and efficient coding is provided, which can significantly improve the stability and reliability of the distributed cache system.

[0055] The following first introduces the data backup method provided in the embodiment of the present application. Figure 1 is a flowchart of a data backup method provided by an embodiment of the present application; Figure 1 As shown, the method comprises the following steps:

[0056] S101: Acquire target data to be backed up.

[0057] This application mainly designs a cascade code to generate redundant Resilient Distributed Datasets (RDD) partition data to solve the problem of RDD partition loss of lagging nodes. Because there is a problem of node lagging during task processing, which will lead to cached data loss, it is necessary to back up the target data. The content of the target data to be backed up here can be set according to the actual situation. It can be the data of all nodes, that is, all data is encoded and stored to ensure data reliability; but encoding all data is labor-intensive and occupies more storage resources. Therefore, it is also possible to encode and save only the data on the nodes with a higher probability of lagging, and try to ensure data reliability while saving resources.

[0058] S102: Based on a preset concatenated code inner code type, the target data is encoded to obtain a first target code.

[0059] The present application does not limit the specific type of the inner code of the cascade code, which can be selected according to actual needs. Here, the inner code of the cascade code can use LDPC. LDPC has high coding efficiency, wide application, simple description and implementation, easy theoretical analysis and research, simple decoding and can be implemented in parallel, and is suitable for hardware implementation. The above-mentioned target data includes multiple original partitions, and the inner code type of the cascade code is specifically LDPC, that is, based on LDPC, the original partition is encoded to obtain the first target code partition. During the encoding process, the number of nodes that may fall behind at the same time can be determined based on the historical falling-behind data of the nodes (for example, the falling-behind probability is 0.002, then the number of nodes that fall behind at the same time for 1000 nodes is 2); then the LDPC check matrix is ​​determined based on this number of nodes, that is, the LDPC code of the system is generated.

[0060] S103: Based on a preset cascade code outer code type, the target data is encoded to obtain a second target code.

[0061] The present application does not limit the specific type of the outer code of the cascade code, which can be selected according to actual needs. Here, the outer code of the cascade code can use LT, and LT uses a simple and basic exclusive OR (XOR) to encode and decode. Specifically, based on LT, the original partition and / or the first target code partition are encoded to obtain the second target code partition.

[0062] The LT code is used as an outer code because it can adapt to the number of nodes that have fallen behind, thereby performing unequal difference protection. The LT with unequal difference protection can increase the probability that the RDD partitions of nodes with high importance and high probability of falling behind participate in the generation of coding partitions; at the same time, the maximum degree of the LT code is determined, taking into account the transmission delay of cache placement and recovery.

[0063] The degree distribution of the LT system code with unequal protection and the corresponding number of generated coding partitions can be determined based on the number of nodes that fall behind at the same time in the worst case. The present application provides a method for solving the maximum degree of the LT code (i.e., the maximum number of preset partitions), which can mainly solve the delay problem of data transmission for RDD partitions that need to participate in encoding during cache placement and recovery. By calculating the time required for encoding placement and decoding recovery and the time to recalculate the RDD partitions, the maximum degree is determined, which can theoretically ensure that the second duration used for encoding placement and decoding recovery is shorter than the first duration for recalculating the RDD partitions.

[0064] In addition, the present application may also adopt LT with unequal differential protection, that is, giving priority to protecting the RDD partitions of nodes with higher importance and higher probability of falling behind. First, according to the data importance of the RDD partitions on each node, the number of RDD partitions that the unequal differential protection system LT code expects to generate is determined; then the probability that the RDD partitions on each node participate in the generation of the RDD encoding partition is determined. This method is used to solve the recovery problem corresponding to different falling-behind probabilities of nodes, aiming to provide more redundancy for nodes with a high probability of falling behind, and to achieve effective recovery of RDD partitions in falling-behind nodes.

[0065] S104: Concatenate the first target code and the second target code to obtain a target cascade code.

[0066] S105: The target concatenated code is saved so that the target data can be restored through the target concatenated code when the target data is lost.

[0067] After the encoding is completed, the first target code and the second target code are connected in series to obtain the final target cascade code, and then the target cascade code is saved. In the event that the target data is lost, the target data can be restored through the target cascade code.

[0068] Here, a specific implementation scheme is provided. This embodiment proposes a distributed cache system based on cascade code for Spark, which includes an RDD partition weight calculation module, an encoding module, a decoding module, a cache placement module and a cache extraction module, wherein the RDD partition weight calculation module is used to calculate the importance of RDD partitions and the probability of node falling behind; the encoding module is used to take RDD partitions as input and output the encoding partitions of RDD partitions to achieve redundant storage; the decoding module is used to recover lost RDD partitions from RDD encoding partitions; the cache placement module is used to redundantly store the RDD generated by encoding to the working node, and the cache extraction module is used to extract the required RDD partitions and RDD encoding partitions from the neighboring working nodes, and then call the decoding module to recover the specified RDD partitions. This scheme can solve the problem of slow task recalculation caused by the falling of working nodes in Spark distributed computing.

[0069] First, the RDD partition weight calculation module is explained. Suppose a calculation task is generated during the calculation process. n RDD, use express.

[0070] Assume that the computing resources required for recalculating each RDD are , the priority of RDD cache is proportional to the computing resource requirements for recalculation, so let the importance of RDD be .

[0071] set up express m The probability of each node being left behind is obtained by counting the historical dropped status of each node. , since the higher the probability of falling behind, the greater the proportion of participating in backup should be, so let the backup importance of the node be .

[0072] In order to improve cache utilization, the probability of falling behind is lower than T The RDD partitions stored by the nodes do not participate in the backup. Therefore, the partitions corresponding to each RDD involved in the encoding can be set to .

[0073] Based on the importance of RDD and the probability of node falling behind, the importance of the partitions involved in encoding is calculated to obtain the importance of each RDD participating in the encoding partition. ,in .

[0074] Then there are the encoding module and the decoding module. For a given RDD, the RDD partition weight calculation module can be used to obtain the corresponding partitions involved in the encoding. , assume that the sizes of these partitions are consistent, (because the encoded RDD is generated by XOR between two RDDs, the sizes need to be consistent. If they are inconsistent, zero padding is performed according to the longest partition size).

[0075] Given a hyperparameter time T , within this time range, assuming that the average number of nodes that fall behind at the same time is N ,So k Partitions involved in encoding, divided into M Group, each group has k / M partitions, each group uses the system The code is encoded, where M The value of can be set empirically. N , the RDD partitions stored in the lagging nodes can be recovered with a high probability.

[0076] Through LDPC coding technology, we can get encoding partitions, but when the number of nodes that fall behind at the same time exceeds N , the RDD partitions of the fallen-behind nodes cannot be recovered.

[0077] Suppose there is a degree distribution The system LT code is used to generate the coding partition through unequal difference protection. Suppose the given length is The average redundancy of the system LT code is , that is, any The encoding partition can be recovered The cache system of the device can tolerate N' nodes falling behind at the same time, so the LT code needs to be generated When the number of nodes that are dropped at the same time does not exceed N', the RDD partitions stored by the dropped nodes can be recovered with a high probability.

[0078] The LT code encoding process of unequal differential protection is as follows:

[0079] Step 1: Select a degree through degree distribution d ,Right now d The partitions are XORed to generate a coded partition.

[0080] Step 2: Partition importance according to code Select one by one d Different partitions participate in the encoding, ensuring that the partition with greater importance has a greater probability of being selected to participate in the encoding.

[0081] The degree distribution in the above process can adopt the robust soliton distribution, which is set as , where the maximum degree distribution is determined by the following steps:

[0082] Step 1: Set Indicates the time required to recalculate the RDD. Indicates that the degree is d The time required to restore the specified RDD from the encoded RDD, including transmission time and decoding time. It can be expressed as: ;

[0083] in Indicates the time required to transfer a partition of RDD. Indicates the time required for two partitions to perform an XOR operation. In order for the encoding cache to be meaningful, , thus obtaining a maximum degree .

[0084] Step 2: Set express n The probability of a node failure. represents the time required for partition execution to complete, which can be expressed as: ;

[0085] in Indicates the time required for normal execution. Indicates the time required for recalculation. If caching is used, the time required for the partition to complete is: ;

[0086] in Indicates the time required to generate the code, Indicates the time required to resolve node failures through caching. Due to different failure probabilities, it is required to encode whether to generate partitions. , then by: ;

[0087] The maximum degree that can be calculated .

[0088] Step 3: Let be the maximum degree of generating coding partitions, then: ;

[0089] It can be seen that the maximum degree in this method is not , but .

[0090] Figure 2 The coding partition diagram of the LT code provided by one embodiment of the present application; the coding partition generated by the above steps is as follows Figure 2 As shown, the upper and lower parts are the generated coding partitions, the coding partition (1, 3, 5, 7) is the coding partition generated by the XOR of partition 1, partition 3, partition 5 and partition 7; the coding partition (4, 5) is the coding partition generated by partition 4 and partition 5.

[0091] After the encoding is completed, the data is saved and retrieved through the cache placement module and the extraction module.

[0092] Encoded backup writing is to asynchronously generate the encoded cache content. The specific steps are as follows:

[0093] Step 1: Each time the master node receives a notification from a worker node that an RDD calculation has been completed, it calculates the importance of the RDD.

[0094] Step 2: The Master node regularly obtains the health assessment data of each node from the node health detection module, calculates the probability of the node falling behind based on historical and real-time data, and thus calculates the importance of the node backup. The importance calculation is shown in the weight calculation module.

[0095] Step 3: The Master determines the RDD partitions that need to be backed up and their importance based on the above two importance calculation methods.

[0096] Step 4: The Master designates a node to generate the encoding partitions and requires other RDDs involved in the encoding to send the required partitions to the node. After the node that calculates the encoding partitions generates the encoding partitions, it distributes the encoding partitions to the designated working nodes for backup according to the backup strategy set by the Master.

[0097] Figure 3 The encoded RDD partition backup diagram provided by one embodiment of the present application; Figure 3 As shown, the generated coded partitions are stored in different nodes. For example, the coded partition (1, 2, 5) of RDD-1 is stored in node 1, and the coded partition (3, 4, 6) of RDD-2 is stored in node 2. If node 2 falls behind and partition 2 of RDD-1 is lost, partition 2 of RDD-1 can be restored by XORing partition -1 of RDD-1 of node 1 and coded partition (1, 2, 5) with partition -5 of RDD-1 of node 5. It should be noted that Figure 3 The scheme is only one example of the present application. Generally, the code partitions generated by LDPC and LT are evenly stored in different nodes.

[0098] The specific steps of encoding recovery reading are as follows:

[0099] Step 1: After the Master detects that a node has fallen behind, it checks the local index table to find and restore the segment where the node's computing task is located.

[0100] Step 2: The Master uses LDPC decoding to check the participating working nodes and their corresponding RDD partitions required to restore the RDD partition. If the LDPC code cannot be decoded, it tries to decode it with the LT code. If both fail, the cache is invalid.

[0101] Step 3: If it can be decoded using LDPC code or LT code, the Master designates a new node to replace the fallen node, and then notifies other working nodes involved in decoding to send the RDD partition to the new node.

[0102] Step 4: The new node uses LDPC decoding or LT code to recover the required RDD partition.

[0103] Step 5: The new node continues the calculation starting from the RDD.

[0104] This application proposes a cascade code design for Spark's RDD distributed cache system, which can quickly recover the RDD partitions lost by the fallen-behind nodes. The design of the unequal difference protection strategy based on the probability of node falling-behind can increase the probability of important RDD partitions and RDD partitions of nodes with high probability of falling-behind participating in encoding, thereby increasing the probability of RDD partitions being recovered after being lost. In addition, the maximum degree design of the LT code considering the communication overhead between nodes can solve the problem of increased communication delay caused by the need for too many other RDD partitions to participate in decoding when recovering the lost RDD partition by setting the maximum degree.

[0105] This application is based on the Spark platform, and aims at the distributed data processing needs in the production environment, and develops a coded cache program for parallel execution of distributed clusters. The advantage is that there is no need to pay too much attention to network communication and program fault tolerance on the Spark platform. In addition, the Spark platform also provides enough interfaces to quickly implement common tasks. The coded distributed cache model in this application realizes calculation and processing by means of the Spark platform, and various processing flows can be integrated together more simply and with lower consumption on the Spark platform. The memory data between nodes is backed up to each other by coding, which improves the memory utilization of the nodes. At the same time, the communication overhead between nodes is taken into account by setting the maximum degree. Compared with the existing distributed cache scheme, no additional distributed cache service system is required.

[0106] The data backup method of the embodiment of the present application first obtains the target data to be backed up; and based on the preset cascade code inner code type, encodes the target data to obtain the first target code; and then based on the preset cascade code outer code type, encodes the target data to obtain the second target code. Then the first target code and the second target code are connected in series to obtain the target cascade code, and the target cascade code is saved, so that in the case of target data loss, the target data can be restored by the target cascade code. Since the cascade code constructs the long code by serial cascade of short codes, the length of the cascade code is less than the length of the long code constructed by a single structure and a single form of code, so it has the advantages of low decoding complexity, strong error correction ability and high coding efficiency, requires less computing resources, is more practical, and the cascade code is suitable for the dynamic change of the number of nodes falling behind in large-scale computing. This technology can be used to solve the cache recovery problem when different nodes fall behind, and can improve the success rate of decoding as much as possible under certain storage redundancy, so as to achieve effective recovery after data loss.

[0107] The inner code type and outer code type of the target cascade code are not limited in the above embodiments. The embodiments of the present application provide a specific implementation scheme, and the inner code type of the target cascade code is specifically selected as LDPC. The present application mainly solves the problem of RDD partition loss of lagging nodes. Usually, the target data includes multiple original partitions. Based on the pre-set inner code type of the cascade code, the target data is encoded to obtain the first target code, including: based on LDPC, the original partition is encoded to obtain the first target code partition. Specifically, based on the historical lagging data of the node, the number of lagging nodes at the same time can be predicted; based on LDPC, the original partition is encoded according to the number of lagging nodes to obtain the first target code partition.

[0108] The solution provided in the embodiment of the present application adopts LDPC as the inner code of the target concatenated code, and provides a specific coding scheme. Compared with MDS, the coding efficiency is higher and the application is wider through LDPC coding.

[0109] Furthermore, in an embodiment of the present application, the outer code type of the target cascade code is LT; then based on the preset cascade code outer code type, the target data is encoded to obtain a second target code, including: based on LT, encoding the original partition and / or the first target code partition to obtain a second target code partition.

[0110] When encoding based on LT, one of the original partitions or the first target code can be used, or both can be used. Here, the encoding of both the original partition and the first target code partition is used as an example. In the encoding stage, the first target code partition is first generated using the LDPC code, and then the second target code partition is generated using the LT code. Assume that the original partitions include: partition 1, partition 2, and partition 3. A coding partition 4 is generated through the LDPC code. The coding partition 4 and the original three partitions form 4 partitions; the LT code generates a coding partition 5 through these 4 partitions.

[0111] This solution adopts serial cascading. For example, if there are 10 partitions, the LDPC code generates 2 coded partitions, so there are 12 partitions. Then the LT code is used to generate 2 more coded partitions, so there are 14 partitions in the end.

[0112] The embodiment of the present application adopts LT code as the outer code of the cascade code, which can adapt to the number of fallen-behind nodes and perform unequal differential protection.

[0113] When encoding based on LT, all partitions can be encoded, but more resources are occupied. Therefore, the preset number of partitions for encoding can be obtained first, and the target encoding partitions of the preset number of partitions can be selected from the original partitions and / or the first target code partitions; finally, based on LT, the target encoding partitions are encoded to obtain the second target code partitions.

[0114] The specific value of the preset number of partitions is set according to the actual situation. This embodiment provides a specific determination scheme for the preset number of partitions, including: based on the historical data of the nodes falling behind, when different numbers of nodes fall behind at the same time, determine the multiple first durations required to recover the data by recalculating. Then, when different numbers of nodes fall behind, determine the maximum number of target coding partitions whose second duration is not greater than the corresponding first duration for the target number of nodes falling behind, and finally determine the above-determined maximum number as the preset number of partitions. Among them, the second duration is the duration required to recover the target data through the target cascade code when the node falls behind the target number.

[0115] The embodiment of the present application provides a method for solving the maximum degree of LT code (the maximum value of the preset number of partitions). The maximum degree is obtained based on the second duration required for calculating the encoding placement and decoding recovery and the first duration for recalculating the RDD partitions, which can theoretically ensure that the encoding placement and decoding recovery are smaller than recalculating the RDD. The degree of LT code distribution determined by the maximum degree solution method of this scheme, relative to the traditional degree determination method, not only greatly ensures that the encoding technology in the distributed cache system can be superior to the multi-copy technology, but also reduces the computational overhead when encoding and decoding the LT code, thereby solving the problem of data transmission delay for the RDD partitions that need to participate in the encoding during cache placement and recovery.

[0116] The above embodiment also mentioned that it is necessary to give priority to backing up the RDD partitions of nodes with high importance and high probability of falling behind. Therefore, this embodiment provides an implementation plan, which selects a preset number of target coding partitions from the original partitions and / or the first target code partitions, including: obtaining the importance of each partition in the original partition and / or the first target code partition and the corresponding node falling behind probability; based on the partition's importance and the corresponding node falling behind probability, selecting a preset number of target coding partitions.

[0117] This application proposal provides an LT system code with unequal differential protection, which can determine the number of partitioned RDDs that the unequal differential protection system LT code expects to generate according to the backup importance of the RDD partitions on each node, and determine the probability that the RDD partitions on each node participate in the generation of RDD encoding partitions. Through unequal differential protection technology, more redundant check and retransmission mechanisms can be adopted for the data of important partitions to ensure the accuracy and integrity of the data; and for the data of unimportant partitions, the redundant check and retransmission mechanisms can be appropriately reduced to save bandwidth and time resources. Therefore, based on this technology, more redundancy can be provided for nodes with a high probability of falling behind, so as to achieve efficient recovery of RDD partitions in the falling-behind nodes.

[0118] In practical applications, after the target concatenated code is saved, when the target data is detected to be lost, the data needs to be restored based on the saved target concatenated code. Because the first target code and the second target code are generated, when restoring, the target data can be restored based on the first target code first, and when the first target code fails to restore the data, the target data can be restored based on the second target code.

[0119] During decoding, since LDPC codes are designed based on the probability of falling behind, which is generally based on the average value, when the number of falling behind nodes is less than the average value, the partitions lost by the falling behind nodes can be recovered with a high probability through the two coded partitions of LDPC. When the number of falling behind nodes is greater than the average value, it is likely that the partitions cannot be decoded, and the coded partitions generated by the LT code are continued to be used for decoding.

[0120] The solution provided by the embodiment of the present application can efficiently complete the recovery of lost data after data loss. Data recovery can be achieved through both inner and outer codes, and the success rate of decoding can be increased as much as possible under certain storage redundancy.

[0121] In order to solve the above technical problems, the present application provides a data backup device. Figure 4 is a structural diagram of a data backup device provided by another embodiment of the present application; Figure 4 As shown, the device comprises:

[0122] The acquisition module 401 is used to acquire the target data to be backed up;

[0123] The encoding module 402 is used to encode the target data based on the preset concatenated code inner code type to obtain a first target code;

[0124] The encoding module 402 is further used to encode the target data based on a preset cascade code outer code type to obtain a second target code;

[0125] A series connection module 403, configured to connect the first target code and the second target code in series to obtain a target cascade code;

[0126] The saving module 404 is used to save the target concatenated code so as to restore the target data through the target concatenated code when the target data is lost.

[0127] On the other hand, the encoding module is specifically used to encode the original partition based on the low-density parity-check code to obtain the first target code partition.

[0128] On the other hand, the encoding module is specifically used to predict the number of stragglers at the same time based on the historical stragglers data of the nodes;

[0129] Based on the low-density parity-check code, the original partition is encoded according to the number of stragglers to obtain the first target code partition.

[0130] On the other hand, the encoding module is specifically used to encode the original partition and / or the first target code partition based on the Lubricant transformation code to obtain the second target code partition.

[0131] On the other hand, the encoding module is specifically used to obtain the preset number of partitions for encoding;

[0132] Selecting a preset number of target code partitions from the original partitions and / or the first target code partitions;

[0133] Based on the Lubi transform code, the target code partition is encoded to obtain a second target code partition.

[0134] On the other hand, the acquisition module is specifically used to determine multiple first durations required for recalculating the way to recover data when different numbers of nodes are dropped at the same time based on the historical dropped data of the nodes;

[0135] In the case of different numbers of nodes falling behind, the maximum number of target coding partitions with a second duration not greater than the corresponding first duration is determined for the target number of nodes falling behind respectively; the second duration is the duration required to recover the target data through the target concatenated code in the case of the target number of nodes falling behind;

[0136] The maximum number is determined as the number of preset partitions.

[0137] On the other hand, the encoding module is specifically used to obtain the importance of each partition in the original partition and / or the first target code partition and the falling-behind probability of the corresponding node;

[0138] Based on the importance of the partitions and the probability of falling behind of the corresponding nodes, a target encoding partition with a preset number of partitions is selected.

[0139] On the other hand, the data backup device further includes: a recovery module for recovering the target data based on the first target code after the target concatenated code is saved and in the case where the target data is detected to be lost;

[0140] The recovery module is also used to recover the target data based on the second target code when the first target code fails to recover the data.

[0141] Figure 5 is a structural diagram of a data backup device provided by another embodiment of the present application; Figure 5 As shown, the data backup device may include a processor 501 and a memory 502 storing computer program instructions.

[0142] Specifically, the processor 501 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0143] The memory 502 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 502 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 502 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 502 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 502 is a non-volatile solid-state memory.

[0144] The memory 502 may include read-only memory (ROM), random access memory (RAM), magnetic disk storage media devices, optical storage media devices, flash memory devices, electrical, optical or other physical / tangible memory storage devices. Thus, in general, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to an aspect of the present disclosure.

[0145] The processor 501 implements any one of the data backup methods in the above embodiments by reading and executing computer program instructions stored in the memory 502 .

[0146] In one example, the data backup device may further include a communication interface 503 and a bus 504. The processor 501, the memory 502, and the communication interface 503 are connected via the bus 504 and communicate with each other.

[0147] The communication interface 503 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0148] Bus 504 includes hardware, software or both, and couples the components of the data backup device to each other. For example, but not limitation, the bus may include an accelerated graphics port (AGP) or other graphics bus, an enhanced industrial standard architecture (EISA) bus, a front-side bus (FSB), a hypertransport (HT) interconnection, an industrial standard architecture (ISA) bus, an infinite bandwidth interconnection, a low pin count (LPC) bus, a memory bus, a microchannel architecture (MCA) bus, a peripheral component interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a serial advanced technology attachment (SATA) bus, a video electronics standard association local (VLB) bus or other suitable buses or a combination of two or more of these. Where appropriate, bus 504 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the present application considers any suitable bus or interconnection.

[0149] In addition, in combination with the data backup method in the above embodiment, the embodiment of the present application can provide a computer storage medium to implement. The computer storage medium stores computer program instructions; when the computer program instructions are executed by the processor, any one of the data backup methods in the above embodiment is implemented.

[0150] An embodiment of the present application also provides a computer program product, including a computer program, which implements any one of the data backup methods in the above embodiments when the computer program is processed and executed.

[0151] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0152] The functional blocks shown in the structural block diagram described above can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application-specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier. "Machine-readable medium" may include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROMs, flash memories, erasable ROMs (EROMs), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0153] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0154] The above reference is to a data backup method, device, equipment, medium and product flowchart and / or block diagram according to an embodiment of the present disclosure. Various aspects of the present disclosure are described. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0155] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. A data backup method, characterized in that: include: Obtain target data to be backed up; The target data includes a plurality of original partitions; Based on a low-density parity-check code, encoding the original partition to obtain a first target code partition; Based on the Lubi transformation code, the original partition and / or the first target code partition are encoded to obtain a second target code partition; Connecting the first target code partition and the second target code partition in series to obtain a target cascade code; The first target code partition is an inner code of the target concatenated code, and the second target code partition is an outer code of the target concatenated code; The target concatenated code is saved so that when the target data is lost, the target data can be restored through the target concatenated code.

2. The data backup method according to claim 1, characterized in that: The step of encoding the original partition based on a low-density parity check code to obtain a first target code partition comprises: Based on the historical data of nodes that have fallen behind, predict the number of nodes that have fallen behind at the same time; Based on a low-density parity-check code, the original partition is encoded according to the number of the stragglers to obtain the first target code partition.

3. The data backup method according to claim 1, characterized in that: The step of encoding the original partition and / or the first target code partition based on the Lube transformation code to obtain a second target code partition comprises: Get the preset number of partitions for encoding; Selecting the preset number of target code partitions from the original partitions and / or the first target code partitions; Based on the Lubi transform code, the target code partition is encoded to obtain a second target code partition.

4. The data backup method according to claim 3, characterized in that: The obtaining of the preset number of partitions for encoding includes: Based on the historical dropped nodes data, when different numbers of nodes dropped at the same time, determine multiple first durations required for recalculating the way to recover data; In the case of different numbers of nodes falling behind, determining the maximum number of target coding partitions with a second duration not greater than the corresponding first duration respectively for the target number of nodes falling behind; the second duration is the duration required for recovering the target data through the target concatenated code when the target number of nodes falls behind; The maximum number is determined as the preset partition number.

5. The data backup method according to claim 3, characterized in that: The selecting the preset number of target code partitions from the original partitions and / or the first target code partitions comprises: Obtaining the importance of each partition in the original partition and / or the first target code partition and the falling-behind probability of the corresponding node; Based on the importance of the partitions and the falling-behind probability of the corresponding nodes, the target coding partitions of the preset number of partitions are selected.

6. The data backup method according to claim 1, characterized in that: After the target concatenated code is saved, the method further includes: In the case where the target data is detected to be lost, recovering the target data based on the first target code partition; In the event that data recovery by the first target code partition fails, the target data is recovered based on the second target code partition.

7. A data backup device, characterized in that: The device comprises: An acquisition module, used for acquiring target data to be backed up; the target data includes a plurality of original partitions; An encoding module, used for encoding the original partition based on a low-density parity-check code to obtain a first target code partition; The encoding module is further used to encode the original partition and / or the first target code partition based on the Lubi transformation code to obtain a second target code partition; A concatenation module, used for concatenating the first target code partition and the second target code partition to obtain a target cascade code; the first target code partition is an inner code of the target cascade code, and the second target code partition is an outer code of the target cascade code; The saving module is used to save the target concatenated code so as to restore the target data through the target concatenated code when the target data is lost.

8. A data backup device, characterized in that: The device comprises: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the data backup method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the data backup method according to any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the data backup method according to any one of claims 1 to 6.

Images