Deduplication-oriented high-reliability low-redundancy erasure code coding repair system and method

Abstract

The invention belongs to the technical field of computer storage, and particularly discloses a deduplication-oriented high-reliability low-redundancy erasure code coding repair system and method. The method comprises the following steps; blocking original data to obtain a plurality of data blocks, deleting duplicated data blocks in the plurality of data blocks and recording the number of quoted times of remaining data blocks, dividing the remaining data blocks into high-quoted blocks and low-quoted blocks based on the number of quoted times of the remaining data blocks and a first preset threshold, and dividing the high-quoted blocks into a first number of data sub-block codes Generating a second number of check sub-blocks, dividing a low reference block into a third number of data sub-blocks, and encoding to generate a fourth number of check sub-blocks, wherein the first number is less than the third number and the second number is greater than the fourth number; when a data block is detected to be lost, directly repairing the remainder of the strip where the low reference block is located in the strip where the high reference block is located; when the number of the available blocks is reduced to the fifth number, the strip where the high reference block is located is repaired, and multiple times of repairing are integrated into one-time batch repairing during repairing.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer storage, and more particularly relates to a high-reliability low-redundancy erasure code encoding and repair system and method for deduplication. BACKGROUND

[0002] With the advent of the big data era, the global data volume is growing explosively, and the storage system is facing the challenge of managing massive data. Data deduplication technology effectively reduces storage overhead by deleting duplicate data blocks and retaining only unique copies, and is widely used in cold data scenarios such as backup and archiving. However, deduplicated data blocks are shared by multiple files, and once a highly referenced block (referenced by multiple files) is lost, a large number of files will become unavailable, significantly increasing the reliability risk.

[0003] To ensure reliability, existing technologies mostly use copy or erasure code redundancy strategies. The copy strategy achieves fault tolerance by storing multiple copies of data, but the storage overhead is high; traditional erasure codes use the same parameters for all data blocks, such as (4, 2) code. However, data blocks with high reference counts require higher fault tolerance, while data blocks with low reference counts account for a high proportion, usually over 85%, and have large block sizes. Using high-redundancy parameters will result in waste of storage costs. In addition, erasure code repair requires reading k blocks for decoding, and frequent repair will result in a large amount of I / O and computing overhead, affecting system performance. Therefore, existing technologies cannot balance reliability, storage overhead, and repair performance. SUMMARY

[0004] In view of the defects of the prior art, the purpose of the present application is to provide a high-reliability low-redundancy erasure code encoding and repair system and method for deduplication, aiming to solve the problem that the prior art cannot balance reliability, storage overhead, and repair performance.

[0005] To achieve the above-mentioned purpose, in a first aspect, the present application provides a high-reliability low-redundancy erasure code encoding and repair method for deduplication, comprising: blocking the original data to obtain a plurality of data blocks; deleting duplicate data blocks in the plurality of data blocks and recording the reference counts of the remaining data blocks; dividing the remaining data blocks into high-reference blocks and low-reference blocks based on the reference counts of the remaining data blocks and a first preset threshold, dividing the high-reference blocks into a first number of data subblocks, encoding to generate a second number of check subblocks, dividing the low-reference blocks into a third number of data subblocks, and encoding to generate a fourth number of check subblocks, wherein the first number is less than the third number, and the second number is greater than the fourth number; When the data block loss is detected, the low reference block located stripe is directly repaired, when the remaining available block quantity of the high reference block located stripe is reduced to the fifth quantity, the high reference block located stripe is repaired, and multiple repairs are integrated into one batch repair during the repair.

[0006] The application divides data into blocks, distinguishes high reference blocks and low reference blocks according to the reference times, adopts high fault-tolerant erasure code for the high reference blocks, reduces the risk of multiple file unavailability caused by loss, adopts low redundancy parameters for the low reference blocks, effectively reduces the storage space overhead compared with the traditional erasure code and multiple copy strategy, integrates multiple repairs into one when repairing the high reference block located stripe, effectively reduces the repair time and I / O overhead, and can simultaneously consider reliability, storage overhead and repair performance.

[0007] According to the high-reliability low-redundancy erasure code encoding and repairing method for deduplication provided by the application, the method further comprises: Calculate the fingerprint information of the plurality of data blocks.

[0008] According to the high-reliability low-redundancy erasure code encoding and repairing method for deduplication provided by the application, the deleting the duplicate data blocks in the plurality of data blocks comprises: The data blocks with the same fingerprint information are determined as duplicate data blocks, and the duplicate data blocks are deleted.

[0009] The application determines the duplicate data blocks based on the fingerprint information, improves the determination accuracy.

[0010] According to the high-reliability low-redundancy erasure code encoding and repairing method for deduplication provided by the application, the method further comprises: When the data block loss is detected, when the I / O request quantity of the storage system is lower than the second preset threshold, the high reference block located stripe is repaired, and multiple repairs are integrated into one batch repair during the repair.

[0011] The application triggers the high reference block located stripe repair when the storage system is in a low load state, avoiding affecting the performance of the storage system.

[0012] In the second aspect, the application provides a high-reliability low-redundancy erasure code encoding and repairing system for deduplication, comprising a block module, a deduplication module, a hybrid encoding module and a repairing module, wherein: The output end of the block module is connected with the input end of the deduplication module, and is used to divide the original data into blocks to obtain a plurality of data blocks; The output end of the deduplication module is connected with the input end of the hybrid encoding module, and is used to delete the duplicate data blocks in the plurality of data blocks and record the reference times of the remaining data blocks; An output end of the hybrid encoding module is connected with an input end of the repair module, and the repair module is configured to divide the remaining data blocks into high-reference blocks and low-reference blocks based on the reference times of the remaining data blocks and a first preset threshold, divide the high-reference blocks into a first number of data sub-blocks, encode to generate a second number of check sub-blocks, divide the low-reference blocks into a third number of data sub-blocks, and encode to generate a fourth number of check sub-blocks, wherein the first number is less than the third number, and the second number is greater than the fourth number. The repair module is configured to directly repair the low-reference blocks when the data blocks are detected to be lost, repair the high-reference blocks when the number of remaining available blocks in the stripe where the high-reference blocks are located decreases to a fifth number, and integrate multiple repairs into one batch repair when repairing.

[0013] According to the high-reliability and low-redundancy erasure code encoding and repair system for deduplication provided in the application, the block dividing module is further configured to: Calculate the fingerprint information of the plurality of data blocks.

[0014] According to the high-reliability and low-redundancy erasure code encoding and repair system for deduplication provided in the application, the deduplication module is specifically configured to: Determine data blocks with the same fingerprint information as repeated data blocks, and delete the repeated data blocks.

[0015] According to the high-reliability and low-redundancy erasure code encoding and repair system for deduplication provided in the application, the repair module is further configured to: When the data blocks are detected to be lost, repair the high-reference blocks when the I / O request amount of the storage system is lower than a second preset threshold, and integrate multiple repairs into one batch repair when repairing.

[0016] In a third aspect, the application provides an electronic device, comprising: at least one memory configured to store a program; and at least one processor configured to execute the program stored in the memory, and when the program stored in the memory is executed, the processor is configured to execute the deduplication-oriented high-reliability and low-redundancy erasure code encoding and repair method described in the first aspect or any possible implementation manner of the first aspect.

[0017] In a fourth aspect, the application provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program, and when the computer program runs on a processor, the processor executes the deduplication-oriented high-reliability and low-redundancy erasure code encoding and repair method described in the first aspect or any possible implementation manner of the first aspect.

[0018] In a fifth aspect, the present application provides a computer program product, which, when executed on a processor, causes the processor to perform the deduplication-oriented high-reliability low-redundancy erasure code encoding and repairing method described in the first aspect or any possible implementation manner of the first aspect.

[0019] It can be understood that the beneficial effects of the above-mentioned second aspect to the fifth aspect can be referred to the related description in the first aspect, which will not be repeated here.

[0020] In general, compared with the prior art, the above technical solutions conceived by the present application have the following beneficial effects: The present application performs block deduplication on data, and distinguishes high-reference blocks and low-reference blocks according to the number of references, adopts high-fault-tolerant erasure code for the high-reference blocks to reduce the risk of multiple file unavailability caused by loss, and adopts low-redundancy parameters for the low-reference blocks, which effectively reduces the storage space overhead compared with the traditional erasure code and the multiple copy strategy, integrates multiple repairs into one when repairing the strip where the high-reference blocks are located, effectively reduces the repair time and I / O overhead, and can simultaneously consider reliability, storage overhead and repair performance. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0022] Figure 1 is a flowchart of the deduplication-oriented high-reliability low-redundancy erasure code encoding and repairing method provided by the embodiment of the present application; Figure 2 is a flowchart of the hybrid encoding scheme provided by the embodiment of the present application; Figure 3 is a flowchart of the integrated repair strategy provided by the embodiment of the present application; Figure 4 is a structural diagram of the deduplication-oriented high-reliability low-redundancy erasure code encoding and repairing system provided by the embodiment of the present application; Figure 5 is a structural diagram of the electronic device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solutions and advantages of the present application more clear, the following will further describe the present application in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0024] The term "and / or", used herein, is a description of an association relationship of associated objects, which means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. The symbol " / " used herein means an or relationship of associated objects, for example, A / B means A or B.

[0025] In the embodiments of the present application, the word "exemplary" or "for example" is used to mean serving as an example, instance, or illustration. Any embodiment or design presented as "exemplary" or "for example" in the embodiments of the present application should not be construed as being superior or more advantageous than other embodiments or design solutions. In fact, the word "exemplary" or "for example" is used only to present related concepts in a specific manner.

[0026] In the description of the embodiments of the present application, unless otherwise specified, "a plurality of" means two or more, for example, a plurality of processing units means two or more processing units, and the like; a plurality of elements means two or more elements, and the like.

[0027] Next, the embodiments of the present application will be described in combination with Figures 1-3 The de-duplication-oriented high-reliability low-redundancy erasure code encoding and repair method provided in the embodiments of the present application is introduced.

[0028] Figure 1 The flowchart of the de-duplication-oriented high-reliability low-redundancy erasure code encoding and repair method provided in the embodiments of the present application is shown in FIG. 1, which comprises the following steps: Figure 1 Step 100, the original data is divided into blocks to obtain a plurality of data blocks; Optionally, the de-duplication-oriented high-reliability low-redundancy erasure code encoding and repair method provided in the embodiments of the present application can be applied to a client-server (C / S) architecture, in which the client is responsible for data preprocessing, and the server is responsible for core de-duplication, encoding and repair management.

[0029] Optionally, after receiving a user file, the client first performs a data block operation to cut the original data into data blocks of variable sizes, and then transmits the data blocks to the de-duplication module of the server.

[0030] Optionally, the client can use a content-defined chunking algorithm to calculate the hash value of the data through a sliding window, and determine the block boundary when the hash value meets the preset condition, so as to cut the original data into data blocks of variable sizes, typically ranging from 4KB to 64KB.

[0031] Step 110, the duplicate data blocks in the plurality of data blocks are deleted, and the reference times of the remaining data blocks are recorded; ​Optionally, the server performs deduplication on the received data blocks, deletes duplicate data blocks, and only updates the reference times of the blocks, i.e., increases the reference times by 1.

[0032] Optionally, the non-duplicate data blocks can be temporarily stored in a temporary storage area.

[0033] In step 120, the remaining data blocks are divided into high-reference blocks and low-reference blocks based on the reference times of the remaining data blocks and a first preset threshold, the high-reference blocks are divided into a first number of data sub-blocks, a second number of check sub-blocks are generated by encoding, the low-reference blocks are divided into a third number of data sub-blocks, and a fourth number of check sub-blocks are generated by encoding, wherein the first number is less than the third number, and the second number is greater than the fourth number. Optionally, the remaining data blocks are divided into high-reference blocks and low-reference blocks according to whether the reference times of the remaining data blocks are greater than or equal to a first preset threshold, and the high-reference blocks are greater than or equal to the first preset threshold, and the low-reference blocks are less than the first preset threshold.

[0034] Optionally, the first preset threshold can be set according to requirements, such as 2, or 3, or 4, and generally does not exceed 4.

[0035] Optionally, the values of the first number, the second number, the third number, and the fourth number can be set according to requirements, but it is necessary to ensure that the first number is less than the third number, and the second number is greater than the fourth number.

[0036] Figure 2 is a flowchart of a hybrid encoding scheme provided by the embodiments of the present application, as shown in Figure 2 As shown, first, the high-reference blocks and the low-reference blocks are distinguished according to the reference times, and the classification is realized through an adaptive threshold: the initial threshold is set to 2, and as the number of data blocks in the system increases, the threshold is dynamically adjusted to ensure that the proportion of the low-reference blocks in all blocks exceeds 90%; for the high-reference blocks, a high-fault-tolerant erasure code parameter (k1, m1) is used, and a typical selection is (4, 4) RS code, i.e., the original data block is divided into 4 data sub-blocks, 4 check sub-blocks are generated by encoding, and a stripe containing 8 blocks is formed, which can tolerate the loss of 4 blocks. For the low-reference blocks, a low-redundancy erasure code parameter (k2, m2) is used, and a typical selection is (8, 2) RS code, i.e., the original data block is divided into 8 data sub-blocks, 2 check sub-blocks are generated by encoding, and a stripe containing 10 blocks is formed, with a redundancy of 1.25, which is lower than that of (4, 2) code under the same fault-tolerant capability (tolerating the loss of 2 blocks).

[0037] Optionally, the encoding process is implemented by means of the Jerasure library, which supports efficient encoding and decoding of RS codes and uses the Intel SSE instruction set to improve processing speed. After encoding, the data sub-blocks and the check sub-blocks are stored in different storage nodes (such as the OSD devices of a Ceph cluster) in a distributed manner, ensuring that the blocks in a stripe are not concentrated in the same node.

[0038] Step 130, when detecting the loss of a data block, directly repairing the stripe where the low-reference block is located, repairing the stripe where the high-reference block is located when the number of remaining available blocks in the stripe decreases to the fifth number, and integrating multiple repairs into one batch repair when repairing.

[0039] When detecting the loss of a data block, the integrated repair strategy is adopted to reduce repair overhead. When detecting the loss of a block caused by a storage node failure (determined by the node heartbeat mechanism or read request failure), the block is first marked as unavailable, but repair is not immediately performed.

[0040] Figure 3 is a flowchart of the integrated repair strategy provided by the embodiments of the present application, as shown in Figure 3 When detecting the loss of a data block, directly and immediately repairing the stripe where the low-reference block is located, and for the stripe where the high-reference block is located, if the number of remaining available blocks in the stripe decreases to the fifth number, triggering passive repair. When repairing, reading the first number of available blocks (such as 4 blocks in a (4, 4) code) from the stripe, performing decoding calculation through the Jerasure library, recovering all lost blocks at once, and writing the recovered blocks to a new available node.

[0041] Optionally, after repair is completed, the metadata of the stripe is updated, all blocks are marked as available, and the stripe is removed from the repair queue.

[0042] Optionally, the value of the fifth number can be set according to requirements, and in an embodiment of the present application, the value of the fifth number is set to the value of the first number + 1, such as 5 in a (4, 4) code, to reduce the number of repairs and avoid excessive frequency of repair.

[0043] Optionally, for the encoded stripe, the storage location, erasure code parameters, and reference relationship of each block are recorded, and the metadata is stored in a distributed metadata server. At the same time, a garbage collection operation is periodically performed to clean up blocks with a reference count of 0 (i.e., blocks that are no longer referenced by any file), and the storage space is released.

[0044] The de-duplication-oriented high-reliability low-redundancy erasure code encoding and repair method provided in the application blocks and de-duplicates data, and distinguishes high-reference blocks and low-reference blocks according to the number of references, adopts high-fault-tolerant erasure code for the high-reference blocks to reduce the risk of multiple file unavailability caused by loss, and adopts low-redundancy parameters for the low-reference blocks, which effectively reduces the storage space overhead compared with the traditional erasure code and the multiple copy strategy, and when repairing the strip where the high-reference blocks are located, multiple repairs are integrated into one, effectively reducing the repair time and I / O overhead, and the reliability, storage overhead and repair performance can be considered at the same time.

[0045] In some embodiments, the method further comprises: calculating fingerprint information of the plurality of data blocks.

[0046] After the client completes the blocking of the original data, the fingerprint of each data block is calculated, and a unique identifier is generated by using a SHA-256 hash function as the fingerprint information of the data block.

[0047] Optionally, the client generates metadata containing block fingerprint information, block length, offset and file information, and transmits the metadata to the server side together with the data block.

[0048] In some embodiments, step 410 specifically comprises: determining that data blocks with the same fingerprint information are duplicate data blocks, and deleting the duplicate data blocks.

[0049] The server queries whether the fingerprint information of the current data block already exists in the system through a fingerprint index, the index adopts a key-value storage structure, the key is the block fingerprint information, and the value is the physical storage location and the number of references of the block. If the fingerprint information already exists, it is determined to be a duplicate block, and only the number of references of the block is updated (i.e. increased by 1 reference); if the fingerprint information does not exist, it is marked as a non-duplicate block, the number of references is recorded as 1, and the block data is temporarily stored in a temporary storage area.

[0050] In some embodiments, the method further comprises: When detecting the loss of data blocks, repairing the strip where the high-reference blocks are located when the I / O request amount of the storage system is lower than a second preset threshold, and integrating multiple repairs into one batch repair when repairing.

[0051] Optionally, the second preset threshold can be set according to experience and the total processing capacity of the cluster, for example, it can be set to 5%~10% of the total processing capacity of the cluster, for example, the number of requests per second can be 1000~1500 times per second.

[0052] When the I / O request amount of the storage system is lower than a second preset threshold, it can be determined that the system is in a low load state, at which time the active repair of the stripe where the high reference block is located can be triggered, and multiple repairs can be integrated into a batch repair when repairing.

[0053] Figure 4 is a structural schematic diagram of a high-reliability low-redundancy erasure code encoding and repair system provided by the embodiment of the present application, as shown in Figure 4 The system 400 includes a blocking module 410, a deduplication module 420, a hybrid encoding module 430, and a repair module 440, wherein: The output end of the blocking module 410 is connected to the input end of the deduplication module 420, for blocking the original data to obtain a plurality of data blocks; Optionally, the high-reliability low-redundancy erasure code encoding and repair system provided by the embodiment of the present application adopts a client-server (C / S) architecture, the client is responsible for data preprocessing, and the server is responsible for core deduplication, encoding, and repair management.

[0054] Optionally, after the client receives a user file, the data blocking operation is first performed by the blocking module 410 to cut the original data into data blocks of variable sizes, and then the data blocks are transmitted to the deduplication module 420 of the server.

[0055] Optionally, the blocking module 410 can perform data blocking by a content-defined blocking algorithm, calculate the hash value of the data through a sliding window, and determine the blocking boundary when the hash value meets a preset condition, cut the original data into data blocks of variable sizes, and the typical range is 4KB to 64KB.

[0056] The output end of the deduplication module 420 is connected to the input end of the hybrid encoding module 430, for deleting the duplicate data blocks in the plurality of data blocks and recording the reference times of the remaining data blocks; The deduplication module 420 performs deduplication processing on the received data blocks, deletes the duplicate data blocks, and only updates the reference times of the blocks, that is, increases the reference times by 1.

[0057] Optionally, the non-duplicate data blocks can be temporarily stored in a temporary storage area.

[0058] The output end of the hybrid encoding module 430 is connected to the input end of the repair module 440, for dividing the remaining data blocks into high reference blocks and low reference blocks based on the reference times of the remaining data blocks and a first preset threshold, dividing the high reference blocks into a first number of data subblocks, encoding to generate a second number of check subblocks, dividing the low reference blocks into a third number of data subblocks, and encoding to generate a fourth number of check subblocks, wherein the first number is less than the third number, and the second number is greater than the fourth number; Optionally, the hybrid encoding module 430 divides the remaining data blocks into high-reference blocks and low-reference blocks according to whether the reference times of the remaining data blocks are greater than or equal to a first preset threshold value. The high-reference blocks are greater than or equal to the first preset threshold value, and the low-reference blocks are less than the first preset threshold value.

[0059] Optionally, the first preset threshold value can be set according to requirements, such as 2, or 3, or 4, and generally does not exceed 4.

[0060] Optionally, the values of the first quantity, the second quantity, the third quantity, and the fourth quantity can be set according to requirements, but it needs to be ensured that the first quantity is less than the third quantity, and the second quantity is greater than the fourth quantity.

[0061] The hybrid encoding module 430 first distinguishes the high-reference blocks from the low-reference blocks according to the reference times, and classifies them through an adaptive threshold value: the initial threshold value is set to 2, and the threshold value is dynamically adjusted as the number of data blocks in the system increases, to ensure that the proportion of the low-reference blocks in all blocks exceeds 90%; for the high-reference blocks, a high-fault-tolerant erasure code parameter (k1, m1) is used, and a typical selection is (4, 4) RS code, that is, the original data block is divided into 4 data sub-blocks, 4 check sub-blocks are generated by encoding, and a strip containing 8 blocks is formed, which can tolerate the loss of 4 blocks. For the low-reference blocks, a low-redundancy erasure code parameter (k2, m2) is used, and a typical selection is (8, 2) RS code, which divides the original data block into 8 data sub-blocks, encodes to generate 2 check sub-blocks, and forms a strip containing 10 blocks, with a redundancy of 1.25. Compared with (4, 2) code, the storage overhead is reduced under the same fault tolerance capability (tolerating 2 block losses).

[0062] Optionally, the encoding process is realized by means of the Jerasure library, which supports efficient encoding and decoding of RS code, and uses the Intel SSE instruction set to improve processing speed. After encoding is completed, the data sub-blocks and the check sub-blocks are distributed and stored in different storage nodes (such as OSD devices of a Ceph cluster), to ensure that the blocks in the strip are not concentrated in the same node.

[0063] The repair module 440 is used for directly repairing the strip where the low-reference blocks are located when detecting the loss of data blocks, repairing the strip where the high-reference blocks are located when the number of remaining available blocks in the strip decreases to a fifth quantity, and integrating multiple repairs into a batch repair when repairing. The repair module 440 is responsible for handling the case of data block loss, and adopts an integrated repair strategy to reduce repair overhead. When detecting that a storage node failure causes a block loss (determined through a node heartbeat mechanism or a read request failure), the block is first marked as unavailable, but repair is not immediately performed.

[0064] When a data block loss is detected, the stripe containing the low-referenced block is immediately repaired. For the stripe containing the high-referenced block, if the number of remaining available blocks in the stripe drops to the fifth number, passive repair is triggered. During repair, the first number of available blocks are read from the stripe (e.g., 4 blocks are read in (4,4) code), decoding calculations are performed using the Jerasure library to recover all lost blocks at once, and the recovered blocks are written to new available nodes.

[0065] Optionally, after the repair is complete, update the stripe's metadata, mark all blocks as available, and remove the stripe from the repair queue.

[0066] Optionally, the value of the fifth quantity can be set according to requirements. In one embodiment of this application, the value of the fifth quantity is set to the value of the first quantity + 1, such as the fifth quantity being 5 in the (4,4) code, in order to reduce the number of repairs and avoid repairs being too frequent.

[0067] Optionally, the system may also include a storage management module responsible for persistent data storage and metadata maintenance. For the encoded stripes, the storage location, erasure coding parameters, and reference relationships of each block are recorded, and the metadata is stored in a distributed metadata server. Simultaneously, garbage collection is performed periodically to clean up blocks with a reference count of 0 (i.e., blocks no longer referenced by any file), freeing up storage space.

[0068] In some embodiments, the segmentation module 410 is further configured to: Calculate fingerprint information for multiple data blocks.

[0069] After the block segmentation module 410 completes the block segmentation, it calculates a fingerprint for each data block and uses the SHA-256 hash function to generate a unique identifier, which serves as the fingerprint information of the data block.

[0070] Optionally, the client generates metadata containing block fingerprint information, block length, offset, and file information, and transmits the metadata along with the data block to the server.

[0071] In some embodiments, the deduplication module 420 is specifically used for: Data blocks with identical fingerprint information are identified as duplicate data blocks and deleted.

[0072] The deduplication module 420 queries the fingerprint index to check whether the fingerprint information of the current data block already exists in the system. This index adopts a key-value storage structure, where the key is the block fingerprint information and the value is the physical storage location and reference count of the block. If the fingerprint information already exists, it is determined to be a duplicate block, and only the reference count of the block is updated (i.e., one reference is added); if the fingerprint information does not exist, it is marked as a non-duplicate block, its reference count is recorded as 1, and the block data is temporarily stored in a temporary storage area.

[0073] In some embodiments, the repair module 440 is further configured to: When a data block is detected as missing, if the I / O request volume of the storage system is lower than the second preset threshold, the stripe containing the highly referenced block is repaired, and multiple repairs are integrated into a single batch repair during the repair process.

[0074] Optionally, the second preset threshold can be set based on experience and the total processing capacity of the cluster. For example, it can be set to 5% to 10% of the total processing capacity of the cluster, or the number of requests per second can be 1000 to 1500.

[0075] When the I / O request volume of the storage system is lower than the second preset threshold, it can be determined that the system is in a low-load state. At this time, active repair of the stripe containing the highly referenced block can be triggered, and multiple repairs can be integrated into a batch repair during the repair process.

[0076] Based on the methods in the above embodiments, Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5 As shown in the illustration, this application provides an electronic device that may include a processor 510, a communications interface 520, a memory 530, and a communication bus 540. The processor 510, communications interface 520, and memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions stored in the memory 530 to execute the high-reliability, low-redundancy erasure coding repair method for deduplication described in the above embodiment.

[0077] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the high-reliability, low-redundancy erasure coding repair method for deduplication in the various embodiments of this application.

[0078] Based on the methods in the above embodiments, this application provides a computer-readable storage medium storing a computer program. When the computer program runs on a processor, it causes the processor to execute the high-reliability, low-redundancy erasure coding repair method for deduplication in the above embodiments.

[0079] Based on the method in the above embodiment, the embodiment of the present application provides a computer program product, when the computer program product runs on a processor, makes the processor execute the deduplication-oriented high-reliability low-redundancy erasure code encoding and repairing method in the above embodiment.

[0080] It can be understood that the processor in the embodiment of the present application can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor can be a microprocessor or any conventional processor.

[0081] The method steps in the embodiment of the present application can be realized by hardware or by the processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in a random access memory (RAM), a flash memory, a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically EPROM (EEPROM), a register, a hard disk, a mobile hard disk, a CD-ROM or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor, so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC.

[0082] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions can be stored in or transmitted by a computer readable storage medium. The computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through a wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media sets. The available media can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0083] It can be understood that various numerical numbers involved in the embodiments of the present application are only distinguished for convenience of description, and are not used to limit the scope of the embodiments of the present application.

[0084] Those skilled in the art easily understand that the above only describes the preferred embodiments of the present application and is not used to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A highly reliable, low-redundancy erasure coding repair method for deduplication, characterized in that, include: The original data is divided into blocks to obtain multiple data blocks; Delete duplicate data blocks from the plurality of data blocks, and record the number of times the remaining data blocks are referenced; Based on the number of references of the remaining data blocks and a first preset threshold, the remaining data blocks are divided into high-reference blocks and low-reference blocks. The high-reference blocks are divided into a first number of data sub-blocks and encoded to generate a second number of verification sub-blocks. The low-reference blocks are divided into a third number of data sub-blocks and encoded to generate a fourth number of verification sub-blocks. The first number is less than the third number, and the second number is greater than the fourth number. When a data block loss is detected, the stripe containing the low-referenced block is repaired directly. When the number of remaining available blocks in the stripe containing the high-referenced block drops to the fifth number, the stripe containing the high-referenced block is repaired, and multiple repairs are integrated into a single batch repair during the repair process.

2. The highly reliable, low-redundancy erasure coding repair method for deduplication as described in claim 1, characterized in that, The method further includes: Calculate the fingerprint information of the multiple data blocks.

3. The highly reliable, low-redundancy erasure coding repair method for deduplication as described in claim 2, characterized in that, The step of deleting duplicate data blocks from the plurality of data blocks includes: Data blocks with identical fingerprint information are identified as duplicate data blocks and deleted.

4. The highly reliable, low-redundancy erasure coding repair method for deduplication as described in claim 1, characterized in that, The method further includes: When a data block is detected as missing, if the I / O request volume of the storage system is lower than the second preset threshold, the stripe containing the highly referenced block is repaired, and multiple repairs are integrated into a single batch repair during the repair process.

5. A highly reliable, low-redundancy erasure coding repair system for deduplication, characterized in that, It includes a block segmentation module, a deduplication module, a hybrid encoding module, and a repair module, among which: The output of the segmentation module is connected to the input of the deduplication module, and is used to segment the original data into multiple data blocks. The output of the deduplication module is connected to the input of the hybrid encoding module, and is used to delete duplicate data blocks in the multiple data blocks and record the number of times the remaining data blocks are referenced; The output of the hybrid encoding module is connected to the input of the repair module. It is used to divide the remaining data block into high-reference blocks and low-reference blocks based on the number of times the remaining data block is referenced and a first preset threshold. The high-reference blocks are divided into a first number of data sub-blocks and encoded to generate a second number of verification sub-blocks. The low-reference blocks are divided into a third number of data sub-blocks and encoded to generate a fourth number of verification sub-blocks. The first number is less than the third number, and the second number is greater than the fourth number. The repair module is used to directly repair the stripe containing the low-referenced block when a data block loss is detected. When the number of remaining available blocks in the stripe containing the high-referenced block drops to the fifth number, the stripe containing the high-referenced block is repaired. During the repair process, multiple repairs are integrated into a single batch repair.

6. The high-reliability, low-redundancy erasure coding repair system for deduplication as described in claim 5, characterized in that, The segmentation module is also used for: Calculate the fingerprint information of the multiple data blocks.

7. The high-reliability, low-redundancy erasure coding repair system for deduplication as described in claim 6, characterized in that, The deduplication module is specifically used for: Data blocks with identical fingerprint information are identified as duplicate data blocks and deleted.

8. The high-reliability, low-redundancy erasure coding repair system for deduplication as described in claim 5, characterized in that, The repair module is also used for: When a data block is detected as missing, if the I / O request volume of the storage system is lower than the second preset threshold, the stripe containing the highly referenced block is repaired, and multiple repairs are integrated into a single batch repair during the repair process.

9. An electronic device, characterized in that, include: At least one memory for storing computer programs; At least one processor is configured to execute a program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to execute the deduplication-oriented, highly reliable, low-redundancy erasure coding repair method as described in any one of claims 1-4.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is run on the processor, the processor performs the high-reliability, low-redundancy erasure coding repair method for deduplication as described in any one of claims 1-4.

Images