Binary large object storage repairing method and device based on hybrid coding

Abstract

The invention belongs to the technical field of computer storage, and particularly discloses a binary large object storage repairing method and device based on hybrid coding, and the method comprises the steps: obtaining a plurality of binary large objects Blob, carrying out the non-systematic minimum storage regeneration MSR coding of a small Blob in the plurality of Blob, and carrying out the systematic MSR coding of a large Blob in the plurality of Blob; storing the plurality of coded Blobs in a plurality of storage nodes in a balanced manner; and determining a corresponding repair strategy based on the type of the to-be-repaired Blob, and repairing the to-be-repaired Blob by using a grouping parallel repair technology based on the repair strategy and the plurality of coded Blobs in the plurality of storage nodes. According to the method, the Blob repairing efficiency can be improved.

Description

Technical Field

[0001] The present application belongs to the field of computer storage technology, and more specifically, to a binary large object storage repair method and device based on hybrid coding. Background Art

[0002] With the rapid development of cloud computing, big data and distributed storage technology, distributed storage systems have been widely used to store massive binary large objects (Blobs). These blobs need to use erasure coding technology during storage to improve the system's fault tolerance and data reliability. However, existing erasure coding methods still face some performance bottlenecks during repair and reading, especially when processing Blob data of different sizes. 1) Performance bottleneck of small blob repair: In the existing erasure code repair, the traditional systematic minimum storage regeneration (MSR) coding has low repair efficiency for small blobs, especially when frequent non-continuous I / O access is required during the repair process, resulting in a significant decrease in repair bandwidth and read performance.

[0003] 2) Repair bottleneck of large blobs: For large blobs, although systematic MSR encoding can ensure efficient regular read performance, it suffers from low repair efficiency due to its high repair bandwidth and node I / O operation overhead.

[0004] 3) Parallel repair challenge: Existing repair methods fail to fully utilize the parallel processing capabilities of the storage system, resulting in an unbalanced disk I / O load during the repair process, and the parallelism and efficiency of the repair process cannot be fully improved.

[0005] Therefore, the existing technology fails to effectively solve the performance problems of uneven repair bandwidth and parallel repair in high-density, distributed storage systems, especially in scenarios where small blobs are mainly stored. Summary of the invention

[0006] In view of the defects of the prior art, the purpose of the present application is to provide a binary large object storage repair method and device based on hybrid coding, aiming to solve the problems of insufficient bandwidth and low repair efficiency of existing repair solutions in small Blob repair.

[0007] To achieve the above objectives, in a first aspect, the present application provides a binary large object storage repair method based on hybrid encoding, comprising: Obtain multiple binary large object blobs, perform non-systematic minimum storage regeneration MSR encoding on small blobs among the multiple blobs, and perform systematic MSR encoding on large blobs among the multiple blobs; The encoded multiple blobs are evenly stored in multiple storage nodes; Based on the type of the Blob to be repaired, a corresponding repair strategy is determined, and based on the repair strategy and the encoded multiple Blobs in the multiple storage nodes, the Blob to be repaired is repaired using a group parallel repair technology.

[0008] This application adopts a hybrid encoding strategy based on the Blob size. Non-systematic MSR encoding is used on small Blobs to optimize the repair bandwidth and reduce the I / O amplification effect. Systematic MSR encoding is used on large Blobs to ensure reading efficiency and improve the repair efficiency through parallel repair technology.

[0009] According to a binary large object storage repair method based on hybrid coding provided by the present application, the non-systematic minimum storage regeneration MSR coding of a small Blob among multiple Blobs includes: For small blocks with inter-block locality in small blobs, a merge-split-encode scheme is used for MSR encoding; A split-merge-encode scheme is used to perform MSR encoding on small blocks with intra-block locality in small blobs.

[0010] The present application splits and re-merges the Blob data blocks based on internal locality for small Blobs to form optimized encoding stripes, so that when reading any blob, the same number of bytes as the blob can be decoded only from the parity block without generating read amplification.

[0011] According to a binary large object storage repair method based on hybrid encoding provided by the present application, the method of performing systematic MSR encoding on a large Blob among multiple Blobs includes: Divide the large Blob into multiple fixed-size blocks; The plurality of fixed-size blocks are systematically MSR encoded.

[0012] This application divides large blobs into fixed-size blocks and performs systematic MSR encoding so that the original data blocks and redundant data blocks in the encoding stripe are evenly distributed on multiple nodes.

[0013] According to a binary large object storage repair method based on hybrid coding provided by the present application, the corresponding repair strategy is determined based on the type of the Blob to be repaired, including: If the blob to be repaired is a small blob, the repair strategy is determined to be a rotational selection strategy, and data is selected from the least used sub-block for repair; If the Blob to be repaired is a large Blob, the repair strategy is determined to select the minimum cost path for repair according to the repair rules of the systematic MSR encoding.

[0014] This application utilizes a dynamic load balancing strategy during the repair process, and balances the load between storage nodes through rotation selection and group parallel repair methods to avoid disk I / O overload.

[0015] According to a binary large object storage repair method based on hybrid encoding provided by the present application, after storing the encoded multiple blobs evenly in multiple storage nodes, the method further includes: When reading small blobs, optimize the read path based on locality analysis; When reading large blobs, data blocks are accessed directly through systematic MSR encoding.

[0016] When reading small blobs, this application optimizes the reading path based on locality analysis, directly accesses related data blocks, and reduces the reading of redundant data; when reading large blobs, it directly accesses data blocks through systematic MSR encoding to ensure efficient reading performance.

[0017] According to a binary large object storage repair method based on hybrid coding provided by the present application, sub-block sharding technology is used to perform parallel decoding when reading Blob.

[0018] This application uses sub-block sharding technology for parallel decoding when reading to balance node load and improve reading performance.

[0019] In a second aspect, the present application provides a binary large object storage and repair device based on hybrid coding, comprising: An encoding module, used for obtaining a plurality of binary large objects (Blobs), performing non-systematic minimum storage regeneration (MSR) encoding on small blobs among the plurality of Blobs, and performing systematic MSR encoding on large blobs among the plurality of Blobs; A storage module is used to store the encoded multiple blobs evenly in multiple storage nodes; The repair module is used to determine a corresponding repair strategy based on the type of the Blob to be repaired, and repair the Blob to be repaired using a group parallel repair technology based on the repair strategy and the encoded multiple Blobs in the multiple storage nodes.

[0020] In a third aspect, the present application provides an electronic device comprising: at least one memory for storing programs; and at least one processor for executing the programs stored in the memory. When the programs stored in the memory are executed, the processor is used to execute the binary large object storage repair method based on hybrid coding described in the first aspect or any possible implementation of the first aspect.

[0021] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the binary large object storage repair method based on hybrid coding described in the first aspect or any possible implementation of the first aspect.

[0022] In a fifth aspect, the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the binary large object storage repair method based on hybrid coding described in the first aspect or any possible implementation of the first aspect.

[0023] It can be understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0024] In general, the above technical solutions conceived by this application have the following beneficial effects compared with the prior art: (1) A hybrid encoding strategy is adopted based on the size of the Blob. Non-systematic MSR encoding is used on small Blobs to optimize the repair bandwidth and reduce the I / O amplification effect. Systematic MSR encoding is used on large Blobs to ensure reading efficiency, and the repair efficiency is improved through parallel repair technology.

[0025] (2) During the repair process, a dynamic load balancing strategy is used to balance the load between storage nodes through rotation selection and group parallel repair methods to avoid disk I / O overload. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0027] Figure 1 It is a flowchart of a binary large object storage repair method based on hybrid encoding provided in an embodiment of the present application; Figure 2is a schematic diagram of a merge-split-encode scheme provided in an embodiment of the present application; Figure 3 is a schematic diagram of a split-merge-encode solution provided in an embodiment of the present application; Figure 4 is a schematic diagram of group parallel repair provided by an embodiment of the present application; Figure 5 It is a structural diagram of a binary large object storage and repair device based on hybrid coding provided in an embodiment of the present application; Figure 6 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0029] The term "and / or" in this article is a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The symbol " / " in this article indicates that the associated objects are in an or relationship, for example, A / B means A or B.

[0030] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0031] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more than two. For example, multiple processing units refer to two or more processing units, etc.; multiple elements refer to two or more elements, etc.

[0032] Next, combine Figure 1-Figure 5 The binary large object storage repair method based on hybrid encoding provided in the embodiment of the present application is introduced.

[0033] Figure 1 is a flow chart of a binary large object storage repair method based on hybrid encoding provided in an embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps: Step 100, obtaining a plurality of binary large objects Blobs, performing non-systematic minimum storage regeneration MSR encoding on small Blobs among the plurality of Blobs, and performing systematic MSR encoding on large Blobs among the plurality of Blobs; The original data block is a data unit directly segmented from the file, and the check block is a redundant data block generated by the encoding algorithm. The existence of the check block enables the system to reconstruct the lost data when some data blocks are lost.

[0034] In an erasure code In a high-density server, a file is divided into The original data blocks are encoded into Total data blocks, of which, , is the number of check blocks, A collection of total data blocks is called a stripe. data blocks are distributed on n disks, and can tolerate any Disk failure. When any of the data blocks fails, in order to repair the failed data block, it is necessary to read data from other surviving disks. Specifically, it is necessary to read data blocks, which can be original data blocks or check blocks. Then, through the erasure code decoding algorithm, data blocks to repair the failed data blocks.

[0035] This application first receives Blob data of different sizes, and determines whether the Blob is a small Blob or a large Blob according to the size of the Blob; then for small Blobs, based on the principle of access locality, non-systematic MSR encoding is used to optimize the repair bandwidth; for large Blobs, systematic MSR encoding is used to maintain efficient conventional reading performance.

[0036] MSR is a coding method in erasure coding, which is mainly used in data storage systems to improve data reliability and fault tolerance. Erasure coding divides data into multiple fragments and adds some redundant fragments, so that even if some data fragments are lost or damaged, the original data can be restored through the remaining data fragments.

[0037] Optionally, a small Blob or a large Blob may be determined by a preset threshold and the size of the Blob. If the size of the Blob is greater than the preset threshold, it is considered a large Blob; if the size of the Blob is less than or equal to the preset threshold, it is considered a small Blob.

[0038] Optionally, by analyzing the single block repair time under Clay encoding, including network transmission, disk reading and seeking, and computing time, it is found that when the Blob size is small, the disk seek time dominates, and the threshold is determined to be a preset threshold.

[0039] Specifically, the single-block repair time of Clay encoding can be obtained by T = TN + TD + TS + TC, where TN is the data transmission time on the network, TD is the disk read time, TS is the disk seek time, and TC is the calculation time.

[0040] For data transmission time, the n-1 helper nodes in the (n, k) Clay code each transmit 1 / (n-k) blocks to the requester, and the repair bandwidth is (n-1)B / (n-k)k, so the data transmission time is TN = (n-1)B / (n-k)kw; for disk read and seek time, assuming that the I / O throughput of sequentially reading data from the disk is θ and the average disk seek time is s, then we can get TD = (n-1)B / (n-k)kθ and TS = λs, where λ is the number of I / O times; finally, the single block repair time is T = (n-1)B / (n-k)kw + (n-1)B / (n-k)kθ+λs+TC.

[0041] Blob size refers to the storage space occupied by the Blob. When T starts to be dominated by TS, it means that in the total repair time T, the disk seek time TS becomes the largest component, that is, it occupies the main proportion of the total repair time. Therefore, the Blob size can be adjusted from large to small until T of the corresponding blob size starts to be dominated by TS, and this size is determined as the threshold.

[0042] Optionally, the systematic MSR coding may be Clay coding or the like.

[0043] Step 110, storing the encoded multiple Blobs evenly in multiple storage nodes; After encoding the Blob, the encoded data is distributed in stripes across multiple storage nodes to ensure balanced data distribution and improve storage reliability and repair efficiency.

[0044] Step 120: determine a corresponding repair strategy based on the type of the Blob to be repaired, and repair the Blob to be repaired using a grouped parallel repair technology based on the repair strategy and the encoded multiple Blobs in the multiple storage nodes.

[0045] When data is damaged in the system, a repair operation is triggered. Different repair strategies are adopted for different types of blobs. The group parallel repair technology is used to execute the repair task in parallel among multiple storage nodes to improve the repair efficiency.

[0046] The group parallel repair technology is used, that is, the block to be repaired is decomposed into multiple sub-blocks, and the parallel capabilities of the storage nodes are utilized to repair multiple sub-blocks simultaneously between multiple nodes, and the repair efficiency is improved through a dynamic task allocation algorithm.

[0047] The present application provides a binary large object storage repair method based on hybrid coding, which adopts a hybrid coding strategy based on the Blob size, uses non-systematic MSR coding on small Blobs to optimize the repair bandwidth and reduce the I / O amplification effect, uses systematic MSR coding on large Blobs to ensure reading efficiency, and improves the repair efficiency through parallel repair technology.

[0048] In some embodiments, performing non-systematic minimum storage regeneration MSR encoding on a small Blob among the multiple Blobs in step 100 specifically includes: For small blocks with inter-block locality in small blobs, a merge-split-encode scheme is used for MSR encoding; A split-merge-encode scheme is used to perform MSR encoding on small blocks with intra-block locality in small blobs.

[0049] For small blobs, non-systematic MSR encoding is used to split and re-merge blob data blocks based on internal locality to form optimized coding stripes.

[0050] Specifically, different non-systematic MSR encoding methods are used for intra-block locality and inter-block locality, where intra-block locality is that a small blob itself is usually accessed as a whole, such as a photo. In other words, if any byte of the blob is read, the rest should be prefetched; inter-block locality is that multiple blobs are usually accessed together. Specifically, these blobs often share the same attributes in their metadata, such as user ID or application ID.

[0051] Figure 2 is a schematic diagram of the merge-split-encode scheme provided in the embodiment of the present application, such as Figure 2As shown in the figure, for data groups with strong cross-blob locality, multiple blobs are merged and encoded to reduce the performance overhead caused by access dispersion. The merge-split-encode scheme first merges multiple small blobs with inter-blob locality into a fixed-size group, for example, 4MB by default; then splits the group into k data blocks; finally, the k data blocks are encoded into n parity blocks through (n, k) non-systematic MSR encoding. In this way, when reading any blob in the group, NCBlob needs to read k parity blocks from the disk to the dynamic random access memory (DRAM) to reconstruct all blobs in the group, but except for the blob currently being read, all remaining blobs will also be read immediately due to inter-blob locality. This ensures that the remaining blobs can be read directly from DRAM instead of disk, thereby reducing read amplification.

[0052] Figure 3 is a schematic diagram of the split-merge-encode solution provided in the embodiment of the present application, such as Figure 3 As shown in Figure 1, the split-merge-encode scheme first splits a small blob into k sub-blobs of equal size; then merges the k sub-blobs of multiple blobs into k data blocks; finally, encodes the k data blocks into n parity blocks through (n, k) non-systematic MSR encoding. In this way, reading any blob can decode the same number of bytes as the blob only from the parity block without incurring read amplification.

[0053] Specifically, the three blocks (i.e., Blob1, Blob2, and Blob3) are first split into k=4 sub-blocks, and then these sub-blocks are merged into data blocks. For example, for the first data block 1, it is formed by merging the first sub-blocks of all three blocks. Finally, the four data blocks are encoded into six parity blocks using a (6, 4) non-systematic MSR code. Among them, reading Blob1 only requires reading the first segment of bytes from each parity block to decode Blob1, where the segment size is equal to the size of the sub-block of Blob1. In this way, reading Blob1 only transfers the size of Blob1 and does not cause read amplification.

[0054] Preferably, a merge-split-encoding scheme is firstly adopted to encode a plurality of small blocks with inter-block locality, and then a split-merge-encoding scheme is adopted for the remaining small blocks which generally have intra-block locality.

[0055] In some embodiments, performing systematic MSR encoding on a large Blob among the multiple Blobs in step 100 specifically includes: Divide the large Blob into multiple fixed-size blocks; The plurality of fixed-size blocks are systematically MSR encoded.

[0056] Large blobs are divided into fixed-size blocks and systematically encoded with MSR, so that the original data blocks and redundant data blocks in the encoding stripe are evenly distributed on multiple nodes.

[0057] In some embodiments, determining the corresponding repair strategy based on the type of the Blob to be repaired in step 120 specifically includes: If the blob to be repaired is a small blob, the repair strategy is determined to be a rotational selection strategy, and data is selected from the least used sub-block for repair; If the Blob to be repaired is a large Blob, the repair strategy is determined to select the minimum cost path for repair according to the repair rules of the systematic MSR encoding.

[0058] For large and small blobs, a generalizable method is to select a rotation-based sub-block selection repair scheme. If the blob to be repaired is a small blob, the repair strategy is determined to be a rotational selection strategy, and data is selected from the least used sub-block for repair. If the blob to be repaired is a large blob, the repair strategy is determined to be based on the repair rules of the systematic MSR encoding, and the minimum cost path is selected for repair.

[0059] Each storage node takes turns selecting different sub-blocks to ensure the sustainability of repair, and uses a two-stage check to ensure that the repaired data block still meets the MDS and repair MDS properties.

[0060] Optionally, the repair scheme can implement single-block repair parallelization. NCBlob implements a group-based parallel repair scheme based on the concept of partial parallel repair, which decomposes the repair operation of non-systematic MSR encoding into parallel partial sub-operations.

[0061] Figure 4 is a schematic diagram of group parallel repair provided by an embodiment of the present application, such as Figure 4As shown in the left figure, a (5, 3) NCBlob has five nodes {N1, N2, N3, N4, N5}, each of which stores a block consisting of n-k = 2 sub-blocks. SPR first divides each sub-block into n = 5 slices, and then collects each k(n-k) = 6 slices from all n-k = 2 sub-blocks of k = 3 nodes in a round-robin manner to form a group Gi (1≤i≤n) (for example, group G1 contains 6 slices of all 6 sub-blocks from 3 nodes {N1, N2, N3}). Decoding each group can obtain the original data of the same offset in k = 3 data blocks (for example, G1 can decode the first slice of each of the six sub-blocks), because the contents of the same offset in a stripe will be encoded or decoded together at the byte level. In this way, all five nodes (each providing six slices from three groups) can be used evenly in parallel reading.

[0062] Single-block repair parallelization. NCBlob implements a group-based parallel repair scheme based on the concept of partial parallel repair, which decomposes the repair operation of non-systematic MSR encoding into parallel partial sub-operations.

[0063] Figure 4 The figure on the right shows a (7, 4) NCBlob using GPR. Repairing the lost block can be decomposed into repairing three sub-blocks p′1,1, p′1,2 and p′1,3. First, all six helpers are divided into two groups. For each group, a collector is selected to collect the selected sub-blocks for repair (for example, N5 and N7 are collectors). Then the helpers responsible for decoding the lost sub-blocks to be functionally repaired are selected. For example, N5 collects {p2,1, p3,1, p4,1, p5,1}, which can provide the linear combination part of the repair sub-block (i.e., ai,2 p2,1 + ai,3 p3,1 + ai,4 p4,1 + ai,5 p5,1), and N7 collects {p6,1, p7,1}, which can provide ai,6 p6,1 + ai,7 p7,1. Finally, the required sub-blocks p′1,1, p′1,2, and p′1,3 are sent to the requester R to repair the lost block, where p′1,i = ai,2 p2,1 + ai,3 p3,1 + · · · + ai,7 p7,1. In this way, we can use N5 and N7 to transmit partial linear combinations of each sub-block, so that the three sub-blocks p′1,1, p′1,2, and p′1,3 can be repaired in parallel in the end.

[0064] In some embodiments, after step 110, the method further includes: Step 111, when reading a small Blob, optimize the reading path based on locality analysis; Step 112, when reading a large Blob, directly access the data block through the systematic MSR encoding.

[0065] When reading blobs, the read path optimization for small blobs is based on locality analysis, directly accessing related data blocks to reduce the reading of redundant data; large blobs directly access data blocks through systematic encoding to ensure efficient reading performance.

[0066] In some embodiments, sub-block sharding technology is used to perform parallel decoding when reading a Blob.

[0067] When reading blobs, sub-block sharding technology is used for parallel decoding to balance node load and improve reading performance.

[0068] Figure 5 is a schematic diagram of the structure of a binary large object storage and repair device based on hybrid coding provided in an embodiment of the present application, such as Figure 5 As shown, the device includes an encoding module 510, a storage module 520 and a repair module 530, wherein: The encoding module 510 is used to obtain a plurality of binary large objects Blobs, and perform non-systematic minimum storage regeneration MSR encoding on small Blobs among the plurality of Blobs, and perform systematic MSR encoding on large Blobs among the plurality of Blobs; The storage module 520 is used to evenly store the encoded multiple Blobs in multiple storage nodes; The repair module 530 is used to determine a corresponding repair strategy based on the type of the Blob to be repaired, and repair the Blob to be repaired using a group parallel repair technology based on the repair strategy and the encoded multiple Blobs in multiple storage nodes.

[0069] It should be understood that the above-mentioned device is used to execute the method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method, which will not be repeated here.

[0070] Based on the method in the above embodiment, Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6As shown, an embodiment of the present application provides an electronic device, which may include: a processor (processor) 610, a communication interface (Communications Interface) 620, a memory (memory) 630 and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call the logic instructions in the memory 630 to execute the binary large object storage repair method based on hybrid encoding in the above embodiment.

[0071] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the binary large object storage repair method based on hybrid encoding described in each embodiment of the present application.

[0072] Based on the method in the above embodiment, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the binary large object storage repair method based on hybrid encoding in the above embodiment.

[0073] Based on the method in the above embodiment, an embodiment of the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the binary large object storage repair method based on hybrid encoding in the above embodiment.

[0074] It is understandable that the processor in the embodiment of the present application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0075] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0076] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0077] It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.

[0078] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A binary large object storage repair method based on hybrid coding, characterized in that: include: Obtain multiple binary large object blobs, perform non-systematic minimum storage regeneration MSR encoding on small blobs among the multiple blobs, and perform systematic MSR encoding on large blobs among the multiple blobs; The encoded multiple blobs are evenly stored in multiple storage nodes; Based on the type of the Blob to be repaired, a corresponding repair strategy is determined, and based on the repair strategy and the encoded multiple Blobs in the multiple storage nodes, the Blob to be repaired is repaired using a group parallel repair technology.

2. The binary large object storage repair method based on hybrid coding according to claim 1 is characterized in that: The non-systematic minimum storage regeneration MSR encoding of a small Blob among the multiple Blobs includes: For small blocks with inter-block locality in small blobs, a merge-split-encode scheme is used for MSR encoding; A split-merge-encode scheme is used to perform MSR encoding on small blocks with intra-block locality in small blobs.

3. The binary large object storage repair method based on hybrid coding according to claim 1 is characterized in that: The method of performing systematic MSR encoding on a large Blob among multiple Blobs includes: Divide the large Blob into multiple fixed-size blocks; The plurality of fixed-size blocks are systematically MSR encoded.

4. The binary large object storage repair method based on hybrid coding according to claim 1 is characterized in that: The determining of a corresponding repair strategy based on the type of the Blob to be repaired includes: If the blob to be repaired is a small blob, the repair strategy is determined to be a rotational selection strategy, and data is selected from the least used sub-block for repair; If the Blob to be repaired is a large Blob, the repair strategy is determined to select the minimum cost path for repair according to the repair rules of the systematic MSR encoding.

5. The binary large object storage repair method based on hybrid coding according to claim 1 is characterized in that: After storing the encoded multiple Blobs evenly in multiple storage nodes, the method further includes: When reading small blobs, optimize the read path based on locality analysis; When reading large blobs, data blocks are accessed directly through systematic MSR encoding.

6. The binary large object storage repair method based on hybrid coding according to claim 5 is characterized in that: When reading a Blob, the sub-block sharding technology is used for parallel decoding.

7. A binary large object storage and repair device based on hybrid coding, characterized in that: include: An encoding module, used for obtaining a plurality of binary large objects (Blobs), performing non-systematic minimum storage regeneration (MSR) encoding on small blobs among the plurality of Blobs, and performing systematic MSR encoding on large blobs among the plurality of Blobs; A storage module is used to store the encoded multiple blobs evenly in multiple storage nodes; The repair module is used to determine a corresponding repair strategy based on the type of the Blob to be repaired, and repair the Blob to be repaired using a group parallel repair technology based on the repair strategy and the encoded multiple Blobs in the multiple storage nodes.

8. An electronic device, characterized in that: include: at least one memory for storing a computer program; At least one processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the binary large object storage repair method based on hybrid encoding as described in any one of claims 1-6.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program runs on a processor, the processor is enabled to execute the binary large object storage repair method based on hybrid encoding as described in any one of claims 1 to 6.

10. A computer program product, characterized in that When the computer program product runs on a processor, the processor is enabled to execute the binary large object storage repair method based on hybrid encoding as described in any one of claims 1 to 6.

Images