Routing method and device for distributed erasure code deduplication system and readable medium

Abstract

The invention discloses a routing method and device for a distributed erasure code deduplication system and a readable medium, and is used for solving the problems that in the erasure code deduplication system, part of data blocks are insufficient in reliability and the system performance is reduced, and the method comprises the following steps: adjusting the sequence of deduplication and erasure code coding, it is guaranteed that each data block of the file can participate in erasure code coding, and the same reliability guarantee is provided for each data block of the file; in combination with a cuckoo hash algorithm, a local deduplication technology is used for replacing global deduplication, so that the read-write performance of the system is improved; and a decentralized structure is designed, so that the problem of cluster failure caused by failure of a single node is avoided, and the performance of the system under a high-load condition is further improved. According to the invention, the read-write performance of the system is improved while high-reliability guarantee is provided for the system, and the performance improvement is more obvious under the condition of high load.

Description

Technical Field

[0001] The present invention relates to the field of storage technology, and in particular to a routing method, device and readable medium for a distributed erasure code deduplication system. Background Art

[0002] In order to cope with the challenges brought by data expansion, improve the storage utilization of data centers and reduce storage costs, deduplication technology has been widely used in large-scale storage systems with high duplication rates to accurately identify and eliminate duplicate data in the system, and significantly improve storage efficiency. Compared with traditional compression technology, deduplication technology can eliminate data redundancy between multiple files, greatly improve the storage efficiency of the system, and effectively reduce the overall cost of data transmission and storage. The application of deduplication technology enables files to be stored as a reference set of shared data objects or data blocks, so that files with the same data blocks have dependencies, that is, data blocks are shared by multiple files. This feature fundamentally changes the reliability of files. The failure of a data block may cause multiple files to be inaccessible, so a fault-tolerant mechanism is needed to help the system maintain high availability. Redundant replication technology and erasure coding technology are often used to provide effective means for data fault tolerance for the system.

[0003] Redundant replication technology refers to providing reliability for the system by strategically increasing data redundancy. This means that not all data blocks within the system have the same degree of reliability guarantee. However, for such systems, the failure of any file will have serious consequences, so reliability guarantees need to be provided for all files.

[0004] Erasure coding technology adds new checksum data to the original data to associate the data blocks of the file. When failures such as data block loss or errors occur, as long as the number of failed data blocks does not exceed the fault tolerance limit set by the system, the erasure coding can use the remaining valid data blocks for decoding operations to restore the failed data. Erasure coding technology uses less redundant data and provides the same fault tolerance as the multi-backup mechanism for all data blocks in the storage system.

[0005] The erasure code deduplication system combines erasure code technology with deduplication technology. Erasure code can provide reliability guarantee for all data blocks in the deduplication system, not just some data blocks, ensuring the integrity and availability of all data. This fusion solution inherits the advantages of both technologies, and further improves the storage efficiency of the system on the basis of ensuring system reliability. Through this combination, the system can find a balance between data redundancy and data deduplication, thereby providing a more efficient and reliable data storage solution. Bo Han et al. split the file into multiple data blocks and performed deduplication operations. After that, they applied erasure code technology to encode these data blocks; R-ADMAD spliced ​​multiple files into larger files and then performed the above operations; EEC-Dedup further reduced zero-byte padding; CodePlugin divided the file into sub-files to obtain higher storage efficiency. The existing work has the following shortcomings:

[0006] (1) Insufficient system reliability. When building an erasure code deduplication system, existing research often fails to fully consider the impact of file deletion operations on the reliability of erasure codes. When a file is deleted in the system, blocks indexed by multiple files may have other data blocks in the encoding stripe deleted, destroying the integrity of its erasure code stripe. As a result, blocks indexed by multiple files cannot be restored by the erasure code, and reliability is lost.

[0007] (2) System throughput decreases. In erasure code deduplication systems, existing research often uses global deduplication technology. Although this technology can effectively eliminate duplicate data, it also brings great computing pressure to the system. Global deduplication requires scanning and comparing data in the entire system, which consumes a lot of computing resources. In addition, existing systems mostly use a centralized layout. This layout can maintain high performance when the data processing volume is small, but when facing high load, the processing capacity of the central node may become a bottleneck, resulting in a decrease in system performance. Summary of the invention

[0008] The purpose of this application is to propose a routing method, device and readable medium for a distributed erasure code deduplication system in response to the above-mentioned technical problems.

[0009] In a first aspect, the present invention provides a routing method for a distributed erasure code deduplication system, comprising the following steps:

[0010] Get the file to be written, divide the file to be written into blocks of equal length to obtain k data blocks, perform erasure coding on the k data blocks, generate m check blocks, the k data blocks and the m check blocks constitute k+m file blocks, and calculate the fingerprint value of each file block in the file to be written;

[0011] Construct a local fingerprint table and a global file index table. The local fingerprint table records the fingerprint value, reference count and storage address of each file block in the written file stored in each node in the storage cluster. The global file index table records the unique identification code of the written file and the fingerprint value and node information of all file blocks in the written file. The node information includes the unique identifier of the node in the storage cluster.

[0012] The fingerprint value of one of the file blocks in the file to be written is queried in the local fingerprint table. In response to determining that the fingerprint value that is the same as the fingerprint value of one of the file blocks in the file to be written is queried in the local fingerprint table, it is determined that one of the file blocks in the file to be written is a duplicate block, and the reference count of the duplicate block in the local fingerprint table is accumulated; in response to determining that the fingerprint value that is the same as the fingerprint value of one of the file blocks in the file to be written is not queried in the local fingerprint table, it is determined that one of the file blocks in the file to be written is not a duplicate block, and the fingerprint value of one of the file blocks in the file to be written is stored on one of the nodes in the storage cluster, and the fingerprint value, reference count and storage address of one of the file blocks in the file to be written are recorded in the local fingerprint table;

[0013] Repeat the previous step until all file blocks in the file to be written are queried in the local fingerprint table. The file to be written becomes a written file and generates a corresponding unique identification code. It is combined with the fingerprint values ​​of all corresponding file blocks and the node information corresponding to all file blocks and updated in the global file index table.

[0014] Preferably, the fingerprint value of the file block is calculated using the cuckoo hash algorithm, and the fingerprint value of the file block is allocated and stored on the nodes of the storage cluster, wherein file blocks with different fingerprint values ​​in the same file are mapped to different nodes in the storage cluster, and file blocks with the same fingerprint values ​​in different files are mapped to the same node in the storage cluster.

[0015] Preferably, an access counter is used to record the number of accesses to each node in the storage cluster. In response to determining that the number of accesses to the allocated node corresponding to the file block exceeds a threshold, a cuckoo hash algorithm is used to calculate all nodes that can store the fingerprint value of the file block, and the node with the least stored fingerprint value or the least number of accesses among all the nodes that can store is selected as the node to be transferred. The fingerprint value of the file block is copied from the allocated node to the transferred node. If the fingerprint value of the file block cannot be found in the local fingerprint table of the allocated node, the fingerprint value of the file block is queried in the local fingerprint table of the transferred node. If the fingerprint value of the file block cannot be found in the local fingerprint table of the transferred node, the fingerprint value of the file block is queried in the local fingerprint table of the allocated node. Repeat the above query process until the fingerprint value of the file block is queried and the corresponding storage address is obtained to read the file block.

[0016] As a preference, it also includes:

[0017] Get the unique identification code of the file to be accessed;

[0018] In response to determining that a unique identification code identical to the unique identification code of the file to be accessed is queried in the global file index table, fingerprint values ​​and node information of all file blocks of the file to be accessed are acquired;

[0019] Traverse each node information, query the storage address corresponding to the fingerprint value that is the same as the fingerprint value of the file block in the file to be accessed in the local fingerprint table of the corresponding node according to the node information, and read the corresponding file block in the storage address;

[0020] Repeat the above steps until all node information is traversed.

[0021] As a preference, it also includes:

[0022] Get the unique identification code of the file to be deleted;

[0023] In response to determining that a unique identification code identical to the unique identification code of the file to be deleted is queried in the global file index table, fingerprint values ​​and node information of all file blocks of the file to be deleted are obtained;

[0024] Traverse each node information, query the local fingerprint table of the corresponding node according to the node information to find the number of references corresponding to the fingerprint value that is the same as the fingerprint value of the file block in the file to be deleted, and reduce the number of references by one;

[0025] Repeat the above steps until all node information is traversed, and the file blocks with a reference count of 0 are regarded as invalid blocks;

[0026] In response to the number of invalid blocks in one of the nodes in the storage cluster reaching a threshold, the invalid blocks in one of the nodes in the storage cluster are deleted.

[0027] As a preferred method, the method further includes: dividing the global file index table by the unique identification code to obtain a plurality of sub-tables having the same data item format as the global file index table, and storing each sub-table on a corresponding node in the storage cluster, and storing a backup table of the sub-table on a neighboring node of each node, wherein N characters of the unique identification code of the file in each sub-table are the same, and the sub-table is the same as the backup table;

[0028] The process of querying in the global file index table is equivalent to querying in the sub-table or backup table. If the sub-table cannot be accessed, the backup table is accessed.

[0029] In a second aspect, the present invention provides a routing device for a distributed erasure code deduplication system, comprising:

[0030] The erasure code encoding module is configured to obtain a file to be written, divide the file to be written into blocks of equal length to obtain k data blocks, perform erasure code encoding on the k data blocks, generate m check blocks, the k data blocks and the m check blocks constitute k+m file blocks, and calculate the fingerprint value of each file block in the file to be written;

[0031] An index relationship establishment module is configured to construct a local fingerprint table and a global file index table, wherein the local fingerprint table records the fingerprint value, reference count and storage address of each file block in the written file stored in each node in the storage cluster, and the global file index table records the unique identification code of the written file and the fingerprint values ​​and node information of all file blocks in the written file, wherein the node information includes the unique identifier of the node in the storage cluster;

[0032] A deduplication module is configured to query a fingerprint value of one of the file blocks in the file to be written in a local fingerprint table, and in response to determining that a fingerprint value identical to the fingerprint value of one of the file blocks in the file to be written is queried in the local fingerprint table, determine that one of the file blocks in the file to be written is a duplicate block, and accumulate reference counts of the duplicate blocks in the local fingerprint table; in response to determining that a fingerprint value identical to the fingerprint value of one of the file blocks in the file to be written is not queried in the local fingerprint table, determine that one of the file blocks in the file to be written is not a duplicate block, store the fingerprint value of one of the file blocks in the file to be written on one of the nodes in the storage cluster, and record the fingerprint value, reference count and storage address of one of the file blocks in the file to be written in the local fingerprint table;

[0033] The repeat module is configured to repeatedly execute the previous module until all file blocks in the file to be written are queried in the local fingerprint table, the file to be written becomes a written file and generates a corresponding unique identification code, and is updated in the global file index table in combination with the fingerprint values ​​of all corresponding file blocks and the node information corresponding to all file blocks.

[0034] In a third aspect, the present invention provides an electronic device comprising one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0035] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0036] In a fifth aspect, the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] (1) The routing method for a distributed erasure code deduplication system proposed in the present invention enables all data blocks to participate in erasure code encoding, thereby providing the same reliability protection for all data blocks and solving the problem of insufficient reliability of some data blocks in the system.

[0039] (2) The routing method for the distributed erasure code deduplication system proposed in the present invention adopts a local deduplication scheme instead of a global deduplication scheme, thereby avoiding the extra computing pressure brought by the use of a global deduplication scheme and improving the read and write performance of the system.

[0040] (3) The decentralized routing scheme for the distributed erasure code deduplication system proposed in the present invention adopts a decentralized load balancing scheme, which solves the drawbacks of the centralized structure, improves the system read and write performance under high load conditions, solves the consistency and synchronization problems of metadata management in the distributed erasure code deduplication system, and realizes lock-free processing of read and write conflicts. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 A schematic flow chart of a routing method for a distributed erasure code deduplication system according to an embodiment of the present application;

[0043] Figure 2 A schematic diagram of erasure code encoding for a routing method for a distributed erasure code deduplication system according to an embodiment of the present application;

[0044] Figure 3 A schematic diagram of file block allocation for a routing method of a distributed erasure code deduplication system according to an embodiment of the present application;

[0045] Figure 4 A schematic diagram of the structure of a sub-table of a global file index table of a routing method for a distributed erasure code deduplication system according to an embodiment of the present application;

[0046] Figure 5 A schematic diagram of a process of file block load balancing in a routing method for a distributed erasure code deduplication system according to an embodiment of the present application;

[0047] Figure 6 This is a graph showing the experimental results of the read throughput of the routing method for a distributed erasure code deduplication system according to an embodiment of the present application;

[0048] Figure 7 This is a graph showing experimental results of write throughput of a routing method for a distributed erasure code deduplication system according to an embodiment of the present application;

[0049] Figure 8 This is a graph showing experimental results of read throughput with different erasure code parameters for a routing method for a distributed erasure code deduplication system according to an embodiment of the present application;

[0050] Fig. 9 This is a graph showing experimental results of write throughput with different erasure code parameters for a routing method for a distributed erasure code deduplication system according to an embodiment of the present application;

[0051] Fig.10 This is a graph showing the experimental results of a read latency of a routing method for a distributed erasure code deduplication system according to an embodiment of the present application;

[0052] Fig.11 This is a graph showing the experimental results of a write latency of a routing method for a distributed erasure code deduplication system according to an embodiment of the present application;

[0053] Fig.12 This is a diagram showing the storage efficiency experimental results of a routing method for a distributed erasure code deduplication system according to an embodiment of the present application;

[0054] Fig.13 A reliability experiment result diagram of a routing method for a distributed erasure code deduplication system according to an embodiment of the present application;

[0055] Fig.14 A schematic diagram of a routing device for a distributed erasure code deduplication system according to an embodiment of the present application;

[0056] Fig.15 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] Figure 1 A routing method for a distributed erasure code deduplication system provided by an embodiment of the present application is shown, comprising the following steps:

[0059] S1, obtain the file to be written, divide the file to be written into blocks of equal length to obtain k data blocks, encode the k data blocks with erasure codes, generate m check blocks, the k data blocks and the m check blocks constitute k+m file blocks, and calculate the fingerprint value of each file block in the file to be written.

[0060] Specifically, refer to Figure 2 , divide the file into equal-length blocks to obtain data blocks and then perform erasure coding. Each data block in the file participates in the calculation of the check block. In other words, the file is divided into k equal-length data blocks using an equal-length file block scheme; the k divided data blocks are encoded using erasure codes, and all data blocks participate in data block encoding to generate m check blocks. The erasure coding process is to obtain k+m file blocks by performing matrix multiplication of a matrix of size (k+m)×k and a vector of size k, where the upper half of the matrix of size (k+m)×k is a unit matrix of size k×k, and the lower half is a generator matrix of size m×k. The k divided data blocks and the generated m check blocks form a file erasure code stripe. For the file erasure code stripe composed of k data blocks and m check blocks generated by erasure coding, calculate the fingerprint value of each file block. Reference Figure 3, use the cuckoo hash algorithm to calculate the fingerprint values ​​of all k+m file blocks in the file erasure code stripe and map them to the cluster nodes. The cuckoo hash algorithm can ensure that any file block in the file is distributed on different nodes, regardless of whether it is a duplicate block. For the same file blocks of different files, it can ensure that they are concentrated on the same node as much as possible; send the fingerprint value of the file block to the corresponding node for storage, and the data of the file block is not sent yet. Figure 3 In the figure, different letters represent different file blocks, file blocks with the same letters are the same file block, different colors represent file blocks belonging to different files, the dotted box represents the check block, and the brackets behind it represent the reference count.

[0061] S2, build a local fingerprint table and a global file index table. The local fingerprint table records the fingerprint value, reference count and storage address of each file block in the written file stored in each node in the storage cluster. The global file index table records the unique identification code of the written file and the fingerprint value and node information of all file blocks in the written file. The node information includes the unique identifier of the node in the storage cluster.

[0062] In a specific embodiment, the fingerprint value of the file block is calculated using the cuckoo hash algorithm, and the fingerprint value of the file block is allocated and stored on the nodes of the storage cluster, wherein file blocks with different fingerprint values ​​in the same file are mapped to different nodes in the storage cluster, and file blocks with the same fingerprint values ​​in different files are mapped to the same node in the storage cluster.

[0063] Specifically, the local fingerprint table is used to manage data stored in the local node to facilitate local deduplication and file block reading. The main data items of the local fingerprint table include information such as the fingerprint value of the file block, the reference count of the file block, and the storage address of the file block. Each node corresponds to a local fingerprint table, which is used to record information such as the fingerprint value of the file block mapped on the node, the reference count of the file block, and the storage address of the file block. The fingerprint value of a file block is a hash value that uniquely identifies the content of the file block, and can quickly identify and merge duplicate file blocks. The reference count records the number of times each file block is referenced. When the reference count is reduced to zero, it means that the file block is no longer needed and can be deleted to free up storage space. The storage address of the file block provides a convenient way to access and manage data blocks. The storage address of the file block can be a logical address or a physical address, and can be set according to needs. The global file index table is used to provide read and write services for files. The structure of the global file index table is as follows: Figure 4As shown. The data items in the global file index table mainly include the unique identification code (FID) of the file and the fingerprint values ​​of all file blocks of the file and the node information corresponding to the fingerprint values. The node information includes the unique identifier of the node in the storage cluster. For example, the node corresponding to file block 1 is node 1, and node 1 is the node information corresponding to file block 1. For each file in the RS (k, m) system, its metadata will record the unique identification code (FID) of 1 file and the fingerprint values ​​of k+m file blocks and the node information corresponding to the fingerprint values. When a user needs to access a file, he needs to provide the unique identification code (FID) of the file and then query the information of the file in the global file index table. If the query result shows that the file does not exist in the global file index table, the access will fail; if the query result shows that the file exists in the global file index table, the fingerprint value of the read file block and the node information corresponding to the fingerprint value will be used to go to the corresponding node in turn to perform the file block reading operation. The fingerprint value of at least one file block can be stored on one of the nodes in the storage cluster.

[0064] S3, querying the fingerprint value of one of the file blocks in the file to be written in the local fingerprint table, in response to determining that the fingerprint value that is the same as the fingerprint value of one of the file blocks in the file to be written is queried in the local fingerprint table, determining that one of the file blocks in the file to be written is a duplicate block, and accumulating the reference count of the duplicate block in the local fingerprint table; in response to determining that the fingerprint value that is the same as the fingerprint value of one of the file blocks in the file to be written is not queried in the local fingerprint table, determining that one of the file blocks in the file to be written is not a duplicate block, and storing the fingerprint value of one of the file blocks in the file to be written on one of the nodes in the storage cluster, and recording the fingerprint value, reference count and storage address of one of the file blocks in the file to be written in the local fingerprint table.

[0065] Specifically, local deduplication is used in each node to reduce the additional storage of duplicate data blocks. In the process of data deduplication, the fingerprint value of the received file block is queried in the local fingerprint table corresponding to the node, and whether the file block is a duplicate block is determined based on the query result. If the fingerprint value of the received file block is the same as the fingerprint value in the local fingerprint table, it is determined that the file block is a duplicate block, otherwise it is not a duplicate block. If it is the same file block, its corresponding fingerprint value is also the same. Therefore, it can be confirmed whether the file block is a duplicate block by querying the local fingerprint table based on the fingerprint value of one of the file blocks in the file to be written. For duplicate blocks, the reference count of the file block in the local fingerprint index table is increased by 1. If the file block is not a duplicate block, the node will send a request to obtain the fingerprint value of the file block and store it in the node, and update the local fingerprint table.

[0066] In a specific embodiment, an access counter is used to record the number of accesses to each node in the storage cluster. In response to determining that the number of accesses to the allocated node corresponding to the file block exceeds a threshold, a cuckoo hash algorithm is used to calculate all nodes that can store the fingerprint value of the file block, and the node with the least stored fingerprint value or the least number of accesses among all the nodes that can store is selected as the transferred node. The fingerprint value of the file block is copied from the allocated node to the transferred node. If the fingerprint value of the file block cannot be found in the local fingerprint table of the allocated node, the fingerprint value of the file block is queried in the local fingerprint table of the transferred node. If the fingerprint value of the file block cannot be found in the local fingerprint table of the transferred node, the fingerprint value of the file block is queried in the local fingerprint table of the allocated node. Repeat the above query process until the fingerprint value of the file block is queried and the corresponding storage address is obtained to read the file block.

[0067] Specifically, for the check blocks generated by the erasure code, due to their low repetition rate, they are mostly stored in the distributed erasure code deduplication system in a separate form. When allocating nodes to them, the cuckoo hash algorithm is used to calculate all storable nodes and select the node with the least storage pressure. Since the purpose of the embodiment of the present application is to deduplicate data, the node with the least storage content is used, and the node with the least access can also be used, depending on the specific needs. Figure 5 , node 3 sends a read request for file block A to node 1, which is route ①. However, due to load problems, node 1 fails to return data to node 3 or the data return times out, which is route ②. Therefore, when node 3 resends, it uses the cuckoo hash algorithm to calculate the fingerprint value of file block A and obtains another storage location of file block A, which is node 2. File block A is hot data. After node 1 detects that it is hot data, it has copied file block A to node 2. After that, node 3 will actively try to send a read request for file block A to another storage node (node ​​2) of file block A, which is route ③. Since file block A exists on node 2, node 2 returns file block A to node 3, which is route ④. If file block A does not exist on node 2, node 3 will resend the request to node 1, and then repeat this cycle until file block A is obtained.

[0068] Each node will record the hot and cold conditions of each corresponding file block. At the beginning, the file block is mapped on the assigned node. As the number of accesses increases, when the number of accesses to the assigned node corresponding to the file block exceeds the threshold, the file block is determined to be hot data. After the hot data is found, the file block is sent to the transferred node calculated by the cuckoo hash algorithm. When the data access to the assigned node corresponding to the file block fails, the transferred node obtains the fingerprint value of the file block. For each file block, all its storable nodes are calculated, and the node with the least storage pressure is selected. And an access counter is used for each node. If the number of accesses to a file block in the node exceeds the threshold, it is considered to be hot data and needs to be transferred. In order to reduce the storage pressure of the access counter, the embodiment of the present application can adopt an access counter based on Count-Min Sketch. The embodiment of the present application selects nodes with less storage pressure to store file blocks, and uses the solution of automatic transfer of hot data to solve the problem of local access blocking.

[0069] S4, repeat the previous step until all file blocks in the file to be written are queried in the local fingerprint table, the file to be written becomes a written file and generates a corresponding unique identification code, and is updated in the global file index table in combination with the fingerprint values ​​of all corresponding file blocks and the node information corresponding to all file blocks.

[0070] Specifically, when all file blocks in the file to be written are queried in the local fingerprint table, the deduplication process of all data in the file to be written is completed, and the fingerprint values ​​of all file blocks that are not duplicate blocks are stored in the nodes of the storage cluster and updated in the local fingerprint table. After the file block transmission is completed, the local fingerprint table and global file index table of the node are updated to ensure the final consistency of data and solve the consistency and conflict problems of data access.

[0071] In a specific embodiment, it also includes:

[0072] Get the unique identification code of the file to be accessed;

[0073] In response to determining that a unique identification code identical to the unique identification code of the file to be accessed is queried in the global file index table, fingerprint values ​​and node information of all file blocks of the file to be accessed are acquired;

[0074] Traverse each node information, query the storage address corresponding to the fingerprint value that is the same as the fingerprint value of the file block in the file to be accessed in the local fingerprint table of the corresponding node according to the node information, and read the corresponding file block in the storage address;

[0075] Repeat the above steps until all node information is traversed.

[0076] Specifically, during the data access process, the unique identification code (FID) of the file to be accessed is first obtained, and the global file index table is queried based on the unique identification code (FID) of the file to be accessed. If a unique identification code that is the same as the unique identification code (FID) of the file to be accessed is queried, the fingerprint values ​​and node information of all file blocks corresponding to the unique identification code in the global file index table are obtained, and then each fingerprint value is queried in the local fingerprint table of the node corresponding to each node information to obtain the storage address of the corresponding file block, and the corresponding file block is read.

[0077] In a specific embodiment, it also includes:

[0078] Get the unique identification code of the file to be deleted;

[0079] In response to determining that a unique identification code identical to the unique identification code of the file to be deleted is queried in the global file index table, fingerprint values ​​and node information of all file blocks of the file to be deleted are obtained;

[0080] Traverse each node information, query the local fingerprint table of the corresponding node according to the node information to find the number of references corresponding to the fingerprint value that is the same as the fingerprint value of the file block in the file to be deleted, and reduce the number of references by one;

[0081] Repeat the above steps until all node information is traversed, and the file blocks with a reference count of 0 are regarded as invalid blocks;

[0082] In response to the number of invalid blocks in one of the nodes in the storage cluster reaching a threshold, the invalid blocks in one of the nodes in the storage cluster are deleted.

[0083] Specifically, during the file deletion process, it is necessary to first obtain the unique identification code of the file to be deleted. Similarly, the global file index table is queried based on the unique identification code (FID) of the file to be deleted. If a unique identification code that is the same as the unique identification code (FID) of the file to be deleted is queried, the fingerprint values ​​and node information of all file blocks corresponding to the unique identification code in the global file index table are obtained, and then each fingerprint value is queried in the local fingerprint table of the node corresponding to each node information to reduce the reference count of the corresponding file block by one.

[0084] When performing a file deletion operation, you only need to determine the nodes where all the file blocks of the file are located, and send notifications to these nodes to reduce the reference count of the corresponding file blocks by 1. When the reference count of a file block is reduced to 0, the file block is determined to be an invalid block, and the file block will not be deleted immediately. The system will determine whether the conditions for garbage collection are met. Only when the number of invalid blocks in the node reaches the threshold will garbage collection be performed to free up storage space.

[0085] The file update operation is to treat the file update as a file write and file delete operation. Specifically, when a file needs to be updated, the old file is deleted first, and then the new file is written. The advantage of this processing method is that it avoids the complex operation of file block migration caused by the change of fingerprint value after the metadata is updated. The file block does not need to be migrated, and the metadata does not need to be frequently modified. This not only simplifies the system's operation process, but also reduces the complexity of the system.

[0086] In a specific embodiment, the method further includes: dividing the global file index table by the unique identification code to obtain a plurality of sub-tables having the same data item format as the global file index table, and storing each sub-table on a corresponding node in the storage cluster, and storing a backup table of the sub-table on a neighboring node of each node, wherein N characters of the unique identification code of the file in each sub-table are the same, and the sub-table is the same as the backup table;

[0087] The process of querying in the global file index table is equivalent to querying in the sub-table or backup table. If the sub-table cannot be accessed, the backup table is accessed.

[0088] Specifically, the global file index table is divided into multiple sub-tables and stored on the nodes in the storage cluster, and a backup table of the sub-table exists on the neighboring nodes of each node, with redundant data, to prevent a node from being suddenly inaccessible. Each item in the global file index table includes a unique identification code (FID) of the file and the fingerprint values ​​of all corresponding file blocks and node information, wherein FID is the unique index of the file. In one embodiment, the first few bits of the unique identification code (FID) of the file can be used as the sub-table ID, so as to divide the global file index table into multiple sub-tables, each sub-table stores a part of the data items in the global file index table, and the first few bits (sub-table ID) of the unique identification code (FID) of the file of all data items stored in each sub-table are the same. It should be noted that the division of sub-tables and the number of backup tables can be flexibly set, and any bit of the unique identification code (FID) of the file can be used as the sub-table ID. Multiple sub-tables can also be stored on the node, which can be flexibly adjusted according to user needs to meet different scenarios and needs. Under normal circumstances, only the sub-table is used, and the backup table is accessed only when the access speed is slow or the node is damaged. The contents of the two tables are the same, so decentralized data indexes can be used as much as possible to achieve load balancing and avoid a certain node being accessed all the time.

[0089] The following is a performance test of the routing method for the distributed erasure code deduplication system proposed in the present invention.

[0090] The read and write performance of the routing method for the distributed erasure code deduplication system proposed in the present invention is evaluated through large-scale real machine experiments, and the reliability of the present invention is evaluated through large-scale simulation experiments. The present invention (represented by DraEC) is compared with another most advanced redundant conversion method CodePlugin, which divides a file into multiple sub-files to obtain the highest data storage efficiency; and the present invention is compared with the RS erasure code storage system (represented by NaiveRS).

[0091] A. Large-scale real-machine experimental environment

[0092] Two Dell-EMC PowerEdge R750 servers were used, equipped with 256GB of DDR4 memory and two Intel 16-core processors with a frequency of 2.4GHz. The operating system installed on the server is Ubuntu 20.04, the kernel version is Linux 5.4.0, and the compiler used is GCC 9.4.0. Three real data sets are used as test data sets. These data sets have different repetition rates and sizes, which can effectively evaluate the performance and effects in different scenarios. Among them, the Linux Kernel data set contains one hundred different versions of Linux kernel files, the Wikipedia data set is composed of two versions of Wikipedia English data, and the Tensorflow data set contains one hundred different versions of Tensorflow installation packages. Each server runs 8 service processes, each process has 4 threads, and the CodePlugin and RS erasure code systems also deploy an additional metadata service process with 16 threads. In order to demonstrate the performance improvement of the present invention under congestion, additional read and write requests are injected by increasing the number of processes in the read and write throughput experiment.

[0093] A.1 Read throughput experiment:

[0094] Figure 6 The experimental results of read throughput show that the present invention improves the read throughput by 17.6% compared with CodePlugin under congestion conditions.

[0095] A.2 Write throughput experiment:

[0096] Figure 7 The results of the read throughput experiment show that when no requests are added, the write throughput of the present invention has an average write performance improvement of 9.2% compared with CodePlugin, and an average write performance improvement of 18.2% compared with the RS code storage system. When the system is congested, the write performance advantage of the present invention is further expanded to an average of about 90.7%.

[0097] A.3 Read throughput experiment with different erasure code parameters:

[0098] Figure 8 The performance change results of reading files by changing the encoding scheme are shown. The results show that under different encoding schemes, the present invention only produces a small degradation in reading performance, and under any encoding scheme, the present invention is better than CodePlugin and RS erasure code systems.

[0099] A.4 Experiments on write throughput with different erasure code parameters:

[0100] Fig. 9 The performance change results of writing files when changing the encoding scheme are shown. The results show that under different encoding schemes, the writing performance of the present invention decreases the least, and under any encoding scheme, the present invention is better than CodePlugin and RS erasure code systems.

[0101] A.5 Read latency experiment:

[0102] Fig.10 As shown in the experimental results of read latency, the present invention does not produce performance degradation on different data sets and different encoding schemes.

[0103] A.6 Write delay experiment:

[0104] Fig.11 As for the writing delay experimental results, the writing delay of the present invention is better than CodePlugin on different data sets and different encoding schemes, and the file writing delay is reduced by 36.4% on average.

[0105] A.7 Storage efficiency:

[0106] Fig.12 The experimental results of storage efficiency show that compared with the RS erasure code system, the present invention brings an average storage space saving rate of 11.6%. Compared with CodePlugin, the present invention does not introduce too much additional storage overhead, which is 0.6% on average.

[0107] B. Reliability simulation experiment

[0108] The embodiment of this application uses a mathematical model for evaluating the data reliability of the erasure code deduplication system to perform reliability analysis, and reasonably infers the reliability formula of CodePlugin through the reliability formula of the erasure code. When building the model, some necessary assumptions and simplified operations are made to make the model more practical and easy to understand:

[0109] First, the failures between nodes are independent of each other. In a real storage cluster, there may be a certain fault-cooperation relationship between different nodes, but the nodes selected by the present invention are often dispersed, and the total number of data blocks and check blocks of each stripe is not large, so this assumption is reasonable.

[0110] Second, it is assumed that the probability of node failure remains constant during the time slice and that all data blocks in the system are available at the beginning of the simulation.

[0111] Third, the appearance of duplicate blocks is random, without considering the spatial and temporal locality of duplicate blocks. It is assumed that duplicate blocks appear randomly at any position in the data set, that is, the probability of each data block being a duplicate block is the same.

[0112] B.1 Reliability test:

[0113] Fig.13 The reliability experiment results are shown in Figure 2, where the horizontal axis is the number of check blocks and the vertical axis is the ratio of the node failure rate of DraEC and CodePlugin. The results show that the reliability of the present invention is better than that of CodePlugin in different coding schemes. In the common erasure code parameters RS(4,2), RS(6,3), RS(10,4), and RS(12,5), the reliability of the present invention is improved by 337%, 486%, 641%, and 702%, respectively.

[0114] Further references Fig.14 As an implementation of the methods shown in the above figures, the present application provides an embodiment of a routing device for a distributed erasure code deduplication system. Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0115] The embodiment of the present application provides a routing device for a distributed erasure code deduplication system, including:

[0116] The erasure code encoding module 1 is configured to obtain a file to be written, divide the file to be written into equal-length blocks to obtain k data blocks, perform erasure code encoding on the k data blocks, generate m check blocks, the k data blocks and the m check blocks constitute k+m file blocks, and calculate the fingerprint value of each file block in the file to be written;

[0117] An index relationship establishment module 2 is configured to construct a local fingerprint table and a global file index table, wherein the local fingerprint table records the fingerprint value, reference count and storage address of each file block in the written file stored in each node in the storage cluster, and the global file index table records the unique identification code of the written file and the fingerprint values ​​and node information of all file blocks in the written file, wherein the node information includes the unique identifier of the node in the storage cluster;

[0118] The deduplication module 3 is configured to query the fingerprint value of one of the file blocks in the file to be written in the local fingerprint table, and in response to determining that the fingerprint value that is the same as the fingerprint value of one of the file blocks in the file to be written is queried in the local fingerprint table, determine that one of the file blocks in the file to be written is a duplicate block, and accumulate the reference count of the duplicate block in the local fingerprint table; in response to determining that the fingerprint value that is the same as the fingerprint value of one of the file blocks in the file to be written is not queried in the local fingerprint table, determine that one of the file blocks in the file to be written is not a duplicate block, and store the fingerprint value of one of the file blocks in the file to be written on one of the nodes in the storage cluster, and record the fingerprint value, reference count and storage address of one of the file blocks in the file to be written in the local fingerprint table;

[0119] Repeating module 4 is configured to repeatedly execute the above two modules until all file blocks in the file to be written are queried in the local fingerprint table, the file to be written becomes a written file and generates a corresponding unique identification code, and is updated in the global file index table in combination with the fingerprint values ​​of all corresponding file blocks and the node information corresponding to all file blocks.

[0120] Fig.15 Schematic diagram of the hardware structure of the electronic device provided by the embodiment of the present invention. Fig.15 As shown, the electronic device of this embodiment includes: a processor 1501 and a memory 1502; wherein the memory 1502 is used to store computer-executable instructions; the processor 1501 is used to execute the computer-executable instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant description in the above method embodiment.

[0121] Optionally, the memory 1502 may be independent or integrated with the processor 1501 .

[0122] When the memory 1502 is independently provided, the electronic device further includes a bus 1503 for connecting the memory 1502 and the processor 1501 .

[0123] The embodiment of the present invention further provides a computer storage medium, in which computer execution instructions are stored. When the processor 1501 executes the computer execution instructions, the above method is implemented.

[0124] The embodiment of the present invention further provides a computer program product, including a computer program. When the computer program is executed by the processor 1501, the above method is implemented.

[0125] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0126] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to implement the solution of this embodiment.

[0127] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, or each module may exist physically separately, or two or more modules may be integrated into one unit. The above-mentioned module-composed unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0128] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor 1501 to perform some steps of the methods of various embodiments of the present application.

[0129] It should be understood that the processor 1501 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or the processor 1501 may be any conventional processor 1501, etc. The steps of the method disclosed in the invention may be directly embodied in the hardware processor 1501 for execution, or may be executed by a combination of hardware and software modules in the processor 1501.

[0130] The memory 1502 may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk, or an optical disk.

[0131] The bus 1503 may be an Industry Standard Architecture (ISA), a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus 1503 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus 1503 in the drawings of the present application is not limited to only one bus 1503 or one type of bus 1503.

[0132] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0133] An exemplary storage medium is coupled to the processor 1501, so that the processor 1501 can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor 1501. The processor 1501 and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor 1501 and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0134] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0135] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A routing method for a distributed erasure code deduplication system, characterized in that: The following steps are involved: Obtain a file to be written, divide the file to be written into blocks of equal length to obtain k data blocks, perform erasure coding on the k data blocks to generate m check blocks, the k data blocks and the m check blocks constitute k+m file blocks, and calculate the fingerprint value of each file block in the file to be written; Constructing a local fingerprint table and a global file index table, wherein the local fingerprint table records the fingerprint value, reference count and storage address of each file block in the written file stored in each node in the storage cluster, and the global file index table records the unique identification code of the written file and the fingerprint values ​​and node information of all file blocks in the written file, wherein the node information includes a unique identifier of the node in the storage cluster; The fingerprint value of one of the file blocks in the file to be written is queried in the local fingerprint table, and in response to determining that a fingerprint value identical to the fingerprint value of one of the file blocks in the file to be written is queried in the local fingerprint table, one of the file blocks in the file to be written is determined to be a duplicate block, and a reference count of the duplicate block in the local fingerprint table is accumulated; In response to determining that a fingerprint value identical to the fingerprint value of one of the file blocks in the file to be written cannot be found in the local fingerprint table, determining that one of the file blocks in the file to be written is not a duplicate block, storing the fingerprint value of one of the file blocks in the file to be written on one of the nodes in the storage cluster, and recording the fingerprint value, reference count, and storage address of one of the file blocks in the file to be written in the local fingerprint table; Repeat the previous step until all file blocks in the file to be written are queried in the local fingerprint table, the file to be written becomes a written file and generates a corresponding unique identification code, and is updated in the global file index table in combination with the fingerprint values ​​of all corresponding file blocks and the node information corresponding to all file blocks.

2. The routing method for a distributed erasure code deduplication system according to claim 1, characterized in that: The fingerprint value of the file block is calculated using a cuckoo hash algorithm, and the fingerprint value of the file block is allocated and stored on the nodes of the storage cluster, wherein file blocks with different fingerprint values ​​in the same file are mapped to different nodes in the storage cluster, and file blocks with the same fingerprint values ​​in different files are mapped to the same node in the storage cluster.

3. The routing method for a distributed erasure code deduplication system according to claim 1, characterized in that: An access counter is used to record the number of accesses to each node in the storage cluster. In response to determining that the number of accesses to the allocated node corresponding to the file block exceeds a threshold, a cuckoo hash algorithm is used to calculate all nodes that can store the fingerprint value of the file block, and a node with the least stored fingerprint value or the least number of accesses among all the nodes that can store the file block is selected as a node to be transferred, and the fingerprint value of the file block is copied from the allocated node to the transferred node. If the fingerprint value of the file block cannot be found in the local fingerprint table of the allocated node, the fingerprint value of the file block is queried in the local fingerprint table of the transferred node. If the fingerprint value of the file block cannot be found in the local fingerprint table of the transferred node, the fingerprint value of the file block is queried in the local fingerprint table of the allocated node. The above query process is repeated until the fingerprint value corresponding to the file block is queried and the corresponding storage address is obtained to read the file block.

4. The routing method for a distributed erasure code deduplication system according to claim 1, characterized in that: Also includes: Get the unique identification code of the file to be accessed; In response to determining that a unique identification code identical to the unique identification code of the file to be accessed is found in the global file index table, fingerprint values ​​and node information of all file blocks of the file to be accessed are acquired; Traversing each node information, searching the local fingerprint table of the corresponding node according to the node information for a storage address corresponding to a fingerprint value that is the same as the fingerprint value of the file block in the file to be accessed, and reading the corresponding file block from the storage address; Repeat the above steps until all node information is traversed.

5. The routing method for a distributed erasure code deduplication system according to claim 1, characterized in that: Also includes: Get the unique identification code of the file to be deleted; In response to determining that a unique identification code identical to the unique identification code of the to-be-deleted file is found in the global file index table, fingerprint values ​​and node information of all file blocks of the to-be-deleted file are acquired; Traversing each node information, searching the local fingerprint table of the corresponding node according to the node information for the number of references corresponding to the fingerprint value that is the same as the fingerprint value of the file block in the file to be deleted, and reducing the number of references by one; Repeat the above steps until all node information is traversed, and the file blocks with a reference count of 0 are regarded as invalid blocks; In response to the number of invalid blocks in one of the nodes in the storage cluster reaching a threshold, the invalid blocks in one of the nodes in the storage cluster are deleted.

6. The routing method for a distributed erasure code deduplication system according to claim 4 or 5, characterized in that: Also includes: The global file index table is divided into several sub-tables having the same data item format as the global file index table by using the unique identification code, and each sub-table is stored on a corresponding node in the storage cluster, and a backup table of the sub-table is stored on a neighboring node of each node, wherein N characters of the unique identification code of the file in each sub-table are the same, and the sub-table is the same as the backup table; The process of querying in the global file index table is equivalent to querying in the sub-table or backup table. If the sub-table cannot be accessed, the backup table is accessed.

7. A routing device for a distributed erasure code deduplication system, characterized in that: include: an erasure code encoding module, configured to obtain a file to be written, divide the file to be written into blocks of equal length to obtain k data blocks, perform erasure code encoding on the k data blocks, generate m check blocks, the k data blocks and the m check blocks constitute k+m file blocks, and calculate a fingerprint value of each file block in the file to be written; An index relationship establishment module is configured to construct a local fingerprint table and a global file index table, wherein the local fingerprint table records the fingerprint value, reference count and storage address of each file block in the written file stored in each node in the storage cluster, and the global file index table records the unique identification code of the written file and the fingerprint values ​​and node information of all file blocks in the written file, wherein the node information includes a unique identifier of the node in the storage cluster; a deduplication module configured to query the local fingerprint table for a fingerprint value of one of the file blocks in the file to be written, and in response to determining that a fingerprint value identical to the fingerprint value of one of the file blocks in the file to be written is queried in the local fingerprint table, determine that one of the file blocks in the file to be written is a duplicate block, and accumulate a reference count of the duplicate block in the local fingerprint table; In response to determining that a fingerprint value identical to the fingerprint value of one of the file blocks in the file to be written cannot be found in the local fingerprint table, determining that one of the file blocks in the file to be written is not a duplicate block, storing the fingerprint value of one of the file blocks in the file to be written on one of the nodes in the storage cluster, and recording the fingerprint value, reference count, and storage address of one of the file blocks in the file to be written in the local fingerprint table; The repetition module is configured to repeatedly execute the previous module until all the file blocks in the file to be written are queried in the local fingerprint table, the file to be written becomes a written file and generates a corresponding unique identification code, and is updated in the global file index table in combination with the fingerprint values ​​of all the corresponding file blocks and the node information corresponding to all the file blocks.

8. An electronic device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images