Method and apparatus for repairing binary large object storage based on hybrid encoding

Abstract

The application belongs to the technical field of computer storage, and specifically discloses a binary large object storage repair method and device based on hybrid coding, which comprises the following steps: acquiring a plurality of binary large objects (Blob), and performing non-systematic minimum storage regeneration (MSR) coding on small-sized Blob in the plurality of Blob and systematic MSR coding on large-sized Blob in the plurality of Blob; storing the coded plurality of Blob in a plurality of storage nodes in a balanced manner; determining a corresponding repair strategy based on the type of a Blob to be repaired, and repairing the Blob to be repaired by using a grouping and parallel repair technology based on the repair strategy and the coded plurality of Blob in the plurality of storage nodes. The method can improve the Blob repair efficiency.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer storage, and more particularly relates to a binary large object storage repair method and device based on hybrid encoding. BACKGROUND

[0002] With the rapid development of cloud computing, big data and distributed storage technology, distributed storage systems have been widely used in storing massive binary large objects (Blob). These Blobs need to use erasure coding technology in the storage process to improve the fault tolerance and data reliability of the system. However, the existing erasure coding methods still face some performance bottlenecks in the repair and reading process, especially when dealing with Blob data of different sizes, among which

[0003] 1) Small Blob repair performance bottleneck: In the existing erasure code repair, the traditional systematic minimum storage regeneration (MSR) coding has low repair efficiency for small Blobs, especially in the repair process, which needs to perform frequent non-continuous I / O access, resulting in a significant decrease in repair bandwidth and reading performance.

[0004] 2) Large Blob repair bottleneck: For large Blobs, although the systematic MSR coding can guarantee efficient regular reading performance, due to its high overhead of repair bandwidth and node I / O operation, the repair efficiency is low.

[0005] 3) Parallel repair challenge: The existing repair methods fail to fully utilize the parallel processing capability of the storage system, resulting in unbalanced disk I / O load in the repair process, and the parallelism and efficiency of the repair process cannot be fully improved.

[0006] Therefore, the existing technology has not effectively solved the problems of unbalanced repair bandwidth and parallel repair performance in high-density, distributed storage systems, especially in the storage scenario dominated by small Blobs. SUMMARY

[0007] In view of the defects of the prior art, the purpose of the present application is to provide a binary large object storage repair method and device based on hybrid encoding, which aims to solve the problems of insufficient repair bandwidth and low repair efficiency of the existing repair scheme for small Blobs.

[0008] To achieve the above purpose, in a first aspect, the present application provides a binary large object storage repair method based on hybrid encoding, comprising:

[0009] A plurality of binary large objects (BLOBs) are obtained, and small BLOBs in the plurality of BLOBs are non-systematically minimum storage regeneration (MSR) encoded, and large BLOBs in the plurality of BLOBs are systematically MSR encoded;

[0010] The encoded plurality of BLOBs are stored in a plurality of storage nodes in a balanced manner;

[0011] Based on a type of a BLOB to be repaired, a corresponding repair strategy is determined, and the BLOB to be repaired is repaired using a grouping and parallel repair technology based on the repair strategy and the encoded plurality of BLOBs in the plurality of storage nodes.

[0012] The application adopts a hybrid encoding strategy based on BLOB size, uses non-systematic MSR encoding on small BLOBs to optimize repair bandwidth and reduce I / O amplification effect, uses systematic MSR encoding on large BLOBs to ensure reading efficiency, and improves repair efficiency through a parallel repair technology.

[0013] According to a binary large object storage and repair method based on a hybrid encoding provided by the application, the small BLOBs in the plurality of BLOBs are non-systematically minimum storage regeneration (MSR) encoded, which includes:

[0014] Small blocks with inter-block locality in the small BLOBs are MSR encoded using a merge-split-encode scheme;

[0015] Small blocks with intra-block locality in the small BLOBs are MSR encoded using a split-merge-encode scheme.

[0016] The application splits and recombines BLOB data blocks based on internal locality for small BLOBs to form optimized encoding strips, so that reading any blob can decode the same number of bytes as the blob from the parity blocks, without reading amplification.

[0017] According to a binary large object storage and repair method based on a hybrid encoding provided by the application, the large BLOBs in the plurality of BLOBs are systematically MSR encoded, which includes:

[0018] The large BLOBs are divided into a plurality of fixed-size blocks;

[0019] The plurality of fixed-size blocks are systematically MSR encoded.

[0020] The application divides large BLOBs into fixed-size blocks and performs systematic MSR encoding, so that the original data blocks and the redundant data blocks in the encoding strips are uniformly distributed on a plurality of nodes.

[0021] According to the binary large object storage repair method based on hybrid coding provided in the application, the corresponding repair strategy is determined according to the type of the Blob to be repaired, and the repair strategy comprises the following steps:

[0022] If the Blob to be repaired is a small Blob, the repair strategy is determined as a rotating selection strategy, and data is selected from the least used sub-blocks for repair;

[0023] If the Blob to be repaired is a large Blob, the repair strategy is determined as a repair rule according to systematic MSR coding, and the minimum cost path is selected for repair.

[0024] In the repair process, the dynamic load balancing strategy is used, the rotating selection and grouping parallel repair method are used, the load between the storage nodes is balanced, and disk I / O overload is avoided.

[0025] According to the binary large object storage repair method based on hybrid coding provided in the application, after the encoded multiple Blobs are balanced and stored in the multiple storage nodes, the method further comprises the following steps:

[0026] When reading the small Blob, the reading path is optimized based on locality analysis;

[0027] When reading the large Blob, the data block is directly accessed through systematic MSR coding.

[0028] When reading the small Blob, the reading path is optimized based on locality analysis, and the related data block is directly accessed, so that the reading of redundant data is reduced; when reading the large Blob, the data block is directly accessed through systematic MSR coding, so that the efficient reading performance is ensured.

[0029] According to the binary large object storage repair method based on hybrid coding provided in the application, when reading the Blob, the sub-block fragmentation technology is used for parallel decoding.

[0030] When reading, the sub-block fragmentation technology is used for parallel decoding, so as to balance the node load and improve the reading performance.

[0031] In a second aspect, the application provides a binary large object storage repair device based on hybrid coding, comprising:

[0032] The encoding module is configured to acquire multiple binary large objects (Blobs) and perform non-systematic minimum storage regeneration (MSR) coding on small Blobs in the multiple Blobs and perform systematic MSR coding on large Blobs in the multiple Blobs;

[0033] The storage module is configured to balance and store the encoded multiple Blobs in multiple storage nodes.

[0034] The repair module is configured to determine a corresponding repair strategy based on a type of the Blob to be repaired, and to repair the Blob to be repaired based on the repair strategy and the encoded Blobs in the plurality of storage nodes using a grouping parallel repair technique.

[0035] In a third aspect, the present application provides an electronic device, comprising: at least one memory configured to store a program; and at least one processor configured to execute the program stored in the memory, and when the program stored in the memory is executed, the processor is configured to execute the method for repairing a Binary Large Object storage based on hybrid encoding described in the first aspect or any possible implementation manner of the first aspect.

[0036] In a fourth aspect, the present application provides a computer readable storage medium, and the computer readable storage medium stores a computer program, and when the computer program is run on a processor, the processor is caused to execute the method for repairing a Binary Large Object storage based on hybrid encoding described in the first aspect or any possible implementation manner of the first aspect.

[0037] In a fifth aspect, the present application provides a computer program product, and when the computer program product is run on a processor, the processor is caused to execute the method for repairing a Binary Large Object storage based on hybrid encoding described in the first aspect or any possible implementation manner of the first aspect.

[0038] It can be understood that the beneficial effects of the above-mentioned second aspect to sixth aspect can be referred to the related description in the first aspect, and will not be repeated here.

[0039] Overall, compared with the prior art, the above technical solutions conceived by the present application have the following beneficial effects:

[0040] (1) Hybrid encoding strategy is adopted based on the size of the Blob, non-systematic MSR encoding is used on small Blobs to optimize repair bandwidth and reduce I / O amplification effect, systematic MSR encoding is used on large Blobs to ensure reading efficiency, and parallel repair technology is used to improve repair efficiency.

[0041] (2) In the repair process, a dynamic load balancing strategy is used, and through the rotation selection and grouping parallel repair method, the load between the storage nodes is balanced, and disk I / O overload is avoided. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0043] Figure 1 is a flowchart of a binary large object storage repair method based on hybrid coding provided by an embodiment of the application;

[0044] Figure 2 is a schematic diagram of a merge-split-encoding scheme provided by an embodiment of the application;

[0045] Figure 3 is a schematic diagram of a split-merge-encoding scheme provided by an embodiment of the application;

[0046] Figure 4 is a schematic diagram of packet parallel repair provided by an embodiment of the application;

[0047] Figure 5 is a structural schematic diagram of a binary large object storage repair device based on hybrid coding provided by an embodiment of the application;

[0048] Figure 6 is a structural schematic diagram of an electronic device provided by an embodiment of the application. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of the application clearer, the following will further describe the application in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the application, and are not used to limit the application.

[0050] The term "and / or" in this paper is a description of the association relationship of the associated object, which means that there can be three relationships, for example, A and / or B, which can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. The symbol " / " in this paper represents the relationship of or, for example, A / B represents A or B.

[0051] In the embodiments of the application, the words such as "exemplary" or "for example" are used to represent as an example, illustration or description. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of the application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. On the contrary, the words such as "exemplary" or "for example" are intended to present the relevant concept in a specific way.

[0052] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "a plurality of" is two or more, for example, a plurality of processing units means two or more processing units, and the like; a plurality of elements means two or more elements, and the like.

[0053] Next, combined with Figures 1-5 The method for repairing storage of binary large objects based on hybrid encoding provided in the embodiments of the present application is introduced.

[0054] Figure 1 The flowchart of the method for repairing storage of binary large objects based on hybrid encoding provided in the embodiments of the present application is shown in Figure 1 As shown in the figure, the method comprises the following steps:

[0055] Step 100, a plurality of binary large objects (BLOBs) are acquired, and small BLOBs in the plurality of BLOBs are subjected to non-systematic minimum storage regeneration (MSR) encoding, and large BLOBs in the plurality of BLOBs are subjected to systematic MSR encoding;

[0056] The original data block is a data unit directly segmented from a file, and the check block is a redundant data block generated through an encoding algorithm. The existence of the check block enables the system to reconstruct the lost data in the case of partial data block loss.

[0057] In a high-density server based on erasure code A file is divided into original data blocks, and the original data blocks are encoded into total data blocks through an encoding matrix, wherein, , is the number of check blocks, The set of the total data blocks is called a strip. The data blocks are distributed on n disks, and any disk failure can be tolerated. When any one of the data blocks is invalid, in order to repair the invalid data block, data needs to be read from other surviving disks. Specifically, data blocks need to be read These data blocks can be original data blocks or check blocks. Then, through the decoding algorithm of the erasure code, the invalid data block is repaired by using the data blocks.

[0058] The present application first receives Blob data of different sizes, and judges whether the Blob is a small Blob or a large Blob according to the size of the Blob; then, for the small Blob, non-systematic MSR encoding is adopted based on the principle of access locality to optimize the repair bandwidth; and for the large Blob, systematic MSR encoding is adopted to maintain efficient regular reading performance.

[0059] MSR is one of the encoding methods in erasure code, mainly used in data storage systems to improve data reliability and fault tolerance. Erasure code divides data into multiple segments and adds some redundant segments, so that even if some data segments are lost or damaged, the original data can be recovered through the remaining data segments.

[0060] Optionally, the Blob can be determined as a small Blob or a large Blob by a preset threshold and the size of the Blob. If the size of the Blob is greater than the preset threshold, it is considered as a large Blob, and if the size of the Blob is less than or equal to the preset threshold, it is considered as a small Blob.

[0061] Optionally, the single-block repair time under Clay encoding can be analyzed, including network transmission, disk reading and seeking, and calculation time, to find that when the Blob size is small, the disk seeking time dominates, and the threshold is determined as the preset threshold.

[0062] Specifically, the single-block repair time of Clay encoding can be obtained by T = TN + TD + TS + TC, where TN is the data transmission time on the network, TD is the disk reading time, TS is the disk seeking time, and TC is the calculation time.

[0063] For data transmission time, each of the n-1 helper nodes in (n, k) Clay encoding transmits 1 / (n-k) blocks to the requester, the repair bandwidth is (n-1)B / (n-k)k, and thus the data transmission time is TN = (n-1)B / (n-k)kw; for disk reading and seeking time, assuming that the I / O throughput of sequentially reading data from the disk is θ and the average disk seeking time is s, then TD = (n-1)B / (n-k)kθ and TS = λs can be obtained, where λ is the number of I / Os; finally, the single-block repair time is T = (n-1)B / (n-k)kw + (n-1)B / (n-k)kθ + λs + TC.

[0064] The Blob size refers to the storage space occupied by the Blob, and T starts to be dominated by TS, which means that in the total repair time T, the disk seeking time TS becomes the largest component, i.e. it occupies a major proportion of the total repair time, so the Blob size can be adjusted from large to small until the corresponding blob size T starts to be dominated by TS, and then the size is determined as the threshold.

[0065] Optionally, the systematic MSR encoding can be Clay encoding or the like.

[0066] Step 110, store the encoded multiple Blobs in the multiple storage nodes evenly.

[0067] After encoding the Blob, the encoded data is distributed in the form of strips on multiple storage nodes to ensure balanced distribution of data and improve storage reliability and repair efficiency.

[0068] Step 120, based on the type of Blob to be repaired, determine the corresponding repair strategy, and based on the repair strategy and the encoded multiple Blobs in the multiple storage nodes, use the grouping parallel repair technology to repair the Blob to be repaired.

[0069] When the system has data damage, trigger the repair operation, adopt different repair strategies for different types of Blob, and use the grouping parallel repair technology to perform repair tasks in parallel among multiple storage nodes, improving repair efficiency.

[0070] Using the grouping parallel repair technology, i.e. decomposing the block to be repaired into multiple sub-blocks, using the parallel capability of the storage node to repair multiple sub-blocks in multiple nodes at the same time, and improving repair efficiency through dynamic task allocation algorithm.

[0071] The application provides a binary large object storage repair method based on hybrid encoding, which adopts a hybrid encoding strategy based on Blob size, uses non-systematic MSR encoding on small Blobs to optimize repair bandwidth and reduce I / O amplification effect, uses systematic MSR encoding on large Blobs to ensure reading efficiency, and improves repair efficiency through parallel repair technology.

[0072] In some embodiments, the non-systematic minimum storage regeneration MSR encoding of the small Blob in the multiple Blobs in step 100 specifically includes:

[0073] The small blocks with inter-block locality in the small Blob are subjected to a merge-split-encode scheme for MSR encoding;

[0074] The small blocks with intra-block locality in the small Blob are subjected to a split-merge-encode scheme for MSR encoding.

[0075] For small Blobs, non-systematic MSR encoding is adopted, the Blob data block is split and recombined based on internal locality to form an optimized encoding strip.

[0076] Specifically, different non-systematic MSR encoding methods are used to address intra-block locality and inter-block locality. Intra-block locality means that a small blob itself is typically accessed as a whole, like a photograph. In other words, if any byte of the blob is read, the rest should be prefetched. Inter-block locality means that multiple blobs are typically accessed together. Specifically, these blobs often share the same attributes in their metadata, such as user ID or application ID.

[0077] Figure 2 This is a schematic diagram of the merging-splitting-encoding scheme provided in the embodiments of this application, as shown below. Figure 2 As shown, for data groups with strong inter-blob locality, multiple blobs are merged and then encoded to reduce the performance overhead caused by access dispersion. The merge-split-encode scheme first merges multiple small blobs with inter-blob locality into a fixed-size group, for example, 4MB by default; then splits the group into k data blocks; finally, the k data blocks are encoded into n parity blocks using (n,k) non-systematic MSR encoding. Thus, when reading any blob in the group, the NCBlob needs to read k parity blocks from disk into Dynamic Random Access Memory (DRAM) to reconstruct all blobs in the group, but all remaining blobs except the currently read blob will also be read immediately due to inter-blob locality. This ensures that the remaining blobs can be read directly from DRAM instead of disk, thereby reducing read amplification.

[0078] Figure 3 This is a schematic diagram of the split-merge-encoding scheme provided in the embodiments of this application, such as... Figure 3 As shown, the split-merge-encode scheme first splits a small blob into k equal-sized sub-blobs; then, it merges the k sub-blobs of multiple blobs into k data blocks; finally, it encodes the k data blocks into n parity blocks using (n, k) non-systematic MSR encoding. In this way, reading any blob can decode only the same number of bytes as the blob from the parity blocks, without read amplification.

[0079] In particular, the three blobs (i.e. Blob1, Blob2 and Blob3) are first divided into k = 4 sub-blobs, and then the sub-blobs are merged into data blocks, for example, for the first data block 1, it is formed by merging the first sub-blob of all three blobs. Finally, the four data blocks are encoded into six parity blocks using a (6, 4) non-systematic MSR code. To read Blob1, only the first segment of bytes from each parity block is read to decode Blob1, where the segment size is equal to the size of the sub-blob of Blob1, so that reading Blob1 only incurs the size of Blob1 and does not cause read amplification.

[0080] Preferably, the merge-divide-encode scheme is first employed to encode those small blobs with inter-blob locality, and then the divide-merge-encode scheme is employed to encode the remaining small blobs which usually have intra-blob locality.

[0081] In some embodiments, the systematic MSR encoding of the large blobs in the plurality of blobs in step 100 specifically comprises:

[0082] dividing the large blob into a plurality of fixed-size blocks;

[0083] systematically MSR encoding the plurality of fixed-size blocks.

[0084] The large blob is divided into fixed-size blocks and systematically MSR encoded, so that the original data blocks and the redundant data blocks in the encoded stripe are evenly distributed across the plurality of nodes.

[0085] In some embodiments, the determining of the corresponding repair strategy based on the type of the blob to be repaired in step 120 specifically comprises:

[0086] if the blob to be repaired is a small blob, determining the repair strategy to be a rotating selection strategy, and selecting data from the least used sub-blocks for repair;

[0087] if the blob to be repaired is a large blob, determining the repair strategy to be a repair rule according to the systematic MSR encoding, and selecting a minimum cost path for repair.

[0088] For large blobs and small blobs, a rotating sub-block selection repair scheme can be generalized. If the blob to be repaired is a small blob, the repair strategy is determined to be a rotating selection strategy, and data is selected from the least used sub-blocks for repair. If the blob to be repaired is a large blob, the repair strategy is determined to be a repair rule according to the systematic MSR encoding, and a minimum cost path is selected for repair.

[0089] Each storage node takes turns to select different sub-blocks to ensure the sustainability of repair, and ensures that the data blocks after repair still satisfy the MDS and repair MDS properties through two-stage checking.

[0090] Optionally, the repair scheme can realize single-block repair parallelization. The NCBlob realizes a group-based parallel repair scheme based on the concept of partial parallel repair, which decomposes the repair operation of non-systematic MSR encoding into parallel partial sub-operations.

[0091] Figure 4 is a schematic diagram of the grouping parallel repair provided by the embodiments of the present application, as Figure 4 As shown in the left drawing, a (5, 3) NCBlob has five nodes {N1, N2, N3, N4, N5}, and each node stores a block composed of n-k = 2 sub-blocks. SPR first divides each sub-block into n = 5 slices, and then collects each k(n-k) = 6 slices from all n-k = 2 sub-blocks of k = 3 nodes in a cyclic manner to form groups Gi (1≤i≤n) (for example, group G1 contains 6 slices from 6 sub-blocks of 3 nodes {N1, N2, N3}). Decoding each group can obtain the original data of the same offset in k = 3 data blocks (for example, G1 can decode the first slice of each of the six sub-blocks), because the contents of the same offset in a strip will be encoded or decoded together at the byte level. In this way, all five nodes (each providing six slices from three groups) can be used evenly in parallel reading.

[0092] Single-block repair parallelization. The NCBlob realizes a group-based parallel repair scheme based on the concept of partial parallel repair, which decomposes the repair operation of non-systematic MSR encoding into parallel partial sub-operations.

[0093] Figure 4The right figure shows a (7,4) NCBlob using GPR. Repairing the lost chunk can be decomposed into repairing three subchunks p'1,1, p'1,2 and p'1,3. First, all six helpers are divided into two groups. For each group, a collector is selected to collect the selected subchunks for repair (e.g., N5 and N7 are the collectors). Then, the helpers responsible for decoding the lost subchunks to be functionally repaired are selected. For example, N5 collects {p2,1, p3,1, p4,1, p5,1}, which can provide a linear combination part of the repaired subchunk (i.e., ai,2 p2,1 + ai,3 p3,1 + ai,4 p4,1 + ai,5 p5,1), and N7 collects {p6,1, p7,1}, which can provide ai,6 p6,1 + ai,7 p7,1. Finally, the required subchunks p'1,1, p'1,2 and p'1,3 are sent to the requester R to repair the lost chunk, where p'1,i = ai,2 p2,1 + ai,3 p3,1 +... + ai,7 p7,1. In this way, we can use N5 and N7 to transmit a partial linear combination of each subchunk, so that we can finally repair three subchunks p'1,1, p'1,2 and p'1,3 in parallel.

[0094] In some embodiments, after step 110, the method further comprises:

[0095] Step 111, when reading a small Blob, the read path is optimized based on locality analysis;

[0096] Step 112, when reading a large Blob, the data chunk is directly accessed through systematic MSR encoding.

[0097] When reading a Blob, the read path of a small Blob is optimized based on locality analysis, directly accessing related data chunks, and reducing the reading of redundant data; a large Blob directly accesses data chunks through systematic encoding, ensuring efficient read performance.

[0098] In some embodiments, subchunk slicing technology is used for parallel decoding when reading a Blob.

[0099] Subchunk slicing technology is used for parallel decoding when reading a Blob to balance node load and improve read performance.

[0100] Figure 5 is a structural schematic diagram of a binary large object storage repair device based on hybrid encoding provided by the embodiments of the present application, as Figure 5 shown, the device comprises an encoding module 510, a storage module 520 and a repair module 530, wherein:

[0101] Encoding module 510 is used to obtain multiple binary large object blobs, and to perform non-systematic minimum storage regeneration (MSR) encoding on the small blobs among the multiple blobs, and to perform systematic MSR encoding on the large blobs among the multiple blobs.

[0102] Storage module 520 is used to evenly store multiple encoded blobs across multiple storage nodes;

[0103] Repair module 530 is used to determine the corresponding repair strategy based on the type of the Blob to be repaired, and to repair the Blob to be repaired using grouped parallel repair technology based on the repair strategy and the encoded Blobs in multiple storage nodes.

[0104] It should be understood that the above-described device is used to execute the methods in the above embodiments. The implementation principle and technical effect of the corresponding program modules in the device are similar to those described in the above methods. The working process of the device can be referred to the corresponding process in the above methods, and will not be repeated here.

[0105] Based on the methods in the above embodiments, Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown in the illustration, this application provides an electronic device that may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640. The processor 610, communications interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can invoke logical instructions stored in the memory 630 to execute the binary large object storage repair method based on hybrid encoding described in the above embodiment.

[0106] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the binary large object storage repair method based on hybrid encoding described in the various embodiments of this application.

[0107] Based on the method in the above embodiment, the embodiment of the application provides a computer readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the mixed coding based binary large object storage repair method in the above embodiment.

[0108] Based on the method in the above embodiment, the embodiment of the application provides a computer program product, which makes the processor execute the mixed coding based binary large object storage repair method in the above embodiment when the computer program product runs on the processor.

[0109] It can be understood that the processor in the embodiment of the application can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor can be a microprocessor or any conventional processor.

[0110] The method steps in the embodiment of the application can be realized in the form of hardware or by the processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in a random access memory (RAM), a flash memory, a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically EPROM (EEPROM), a register, a hard disk, a mobile hard disk, a CD-ROM or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor, so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC.

[0111] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions can be stored in or transmitted by a computer readable storage medium. The computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through a wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media sets. The available media can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0112] It can be understood that various numerical numbers involved in the embodiments of the present application are only distinguished for convenience of description, and are not used to limit the scope of the embodiments of the present application.

[0113] Those skilled in the art easily understand that the above only describes the preferred embodiments of the present application and is not used to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A hybrid encoding based binary large object storage repair method, characterized in that, The method comprises the following steps: acquire a plurality of binary large objects (BLOBs), and perform non-systematic minimum storage regeneration (MSR) encoding on small BLOBs in the plurality of BLOBs and systematic MSR encoding on large BLOBs in the plurality of BLOBs; store the encoded plurality of BLOBs in a plurality of storage nodes in a balanced manner; determine a corresponding repair strategy based on the type of a BLOB to be repaired, and perform repair on the BLOB to be repaired based on the repair strategy and the encoded plurality of BLOBs in the plurality of storage nodes by using a grouping and parallel repair technology; the non-systematic MSR encoding on the small BLOBs comprises the following steps: performing MSR encoding on small blocks with inter-block locality in the small BLOBs by using a merge-split-encoding scheme; performing MSR encoding on small blocks with intra-block locality in the small BLOBs by using a split-merge-encoding scheme; the systematic MSR encoding on the large BLOBs comprises the following steps: dividing the large BLOBs into a plurality of fixed-size blocks; performing systematic MSR encoding on the plurality of fixed-size blocks; the determination of the corresponding repair strategy based on the type of the BLOB to be repaired comprises the following steps: if the BLOB to be repaired is a small BLOB, determining that the repair strategy is a rotating selection strategy, and selecting data from the least used sub-blocks for repair; if the BLOB to be repaired is a large BLOB, determining that the repair strategy is a repair rule according to systematic MSR encoding, and selecting a minimum cost path for repair.

2. The mixed encoding based binary large object storage repair method of claim 1, wherein, After the encoded plurality of BLOBs are stored in the plurality of storage nodes in a balanced manner, the method further comprises the following steps: when reading the small BLOBs, optimizing a reading path based on locality analysis; when reading the large BLOBs, directly accessing the data blocks by using systematic MSR encoding.

3. The mixed encoding based binary large object storage repair method of claim 2, wherein, When reading the BLOBs, a sub-block slicing technology is used for parallel decoding.

4. A hybrid coding based binary large object storage repair apparatus, characterized by, The method comprises the following steps: an encoding module is configured to acquire a plurality of binary large objects (BLOBs), and perform non-systematic minimum storage regeneration (MSR) encoding on small BLOBs in the plurality of BLOBs and systematic MSR encoding on large BLOBs in the plurality of BLOBs; a storage module is configured to store the encoded plurality of BLOBs in a plurality of storage nodes in a balanced manner; a repair module is configured to determine a corresponding repair strategy based on the type of a BLOB to be repaired, and perform repair on the BLOB to be repaired based on the repair strategy and the encoded plurality of BLOBs in the plurality of storage nodes by using a grouping and parallel repair technology; the encoding module is specifically configured to: perform MSR encoding on small blocks with inter-block locality in the small BLOBs by using a merge-split-encoding scheme; perform MSR encoding on small blocks with intra-block locality in the small BLOBs by using a split-merge-encoding scheme; the encoding module is specifically configured to: divide the large BLOBs into a plurality of fixed-size blocks; perform systematic MSR encoding on the plurality of fixed-size blocks; the repair module is specifically configured to: If the Blob to be repaired is a small Blob, a repair strategy is determined as a rotating selection strategy, and data is selected from the least used sub-blocks for repair; If the Blob to be repaired is a large Blob, a repair strategy is determined as a repair rule according to systematic MSR coding, and a minimum cost path is selected for repair.

5. An electronic device, comprising: The method comprises the steps of: at least one memory for storing a computer program; at least one processor for executing the program stored in the memory, and when the program stored in the memory is executed, the processor is used to execute the binary large object storage repair method based on hybrid coding according to any one of claims 1-3.

6. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 5. When the computer program runs on the processor, the processor is caused to execute the binary large object storage repair method based on hybrid coding according to any one of claims 1-3.

7. A computer program product, characterised in that, When the computer program product runs on the processor, the processor is caused to execute the binary large object storage repair method based on hybrid coding according to any one of claims 1-3.

Images