A dual-layer optimization-based erasure code encoding method and system

Abstract

This invention relates to an erasure coding method and system based on two-layer optimization. The method includes: obtaining storage parameters of the data to be stored; constructing an initial generator matrix structure and a reference table for characterizing the bit-level characteristics of elements; constructing a two-layer optimization model and using the two-layer optimization model to collaboratively optimize the initial generator matrix structure; the two-layer optimization model includes: an upper-layer optimization model configured to adjust the structural distribution of the generator matrix; and a lower-layer optimization model configured to schedule the bit-level operation paths after the generator matrix expansion based on the reference table and calculate the actual bit-level computational overhead; feeding back the bit-level computational overhead of the lower-layer optimization model to the upper-layer optimization model; and encoding the data to be stored based on the comprehensive evaluation result of the algebraic characteristics and the bit-level computational overhead. This invention achieves the collaborative unification of the generator matrix structure and XOR scheduling optimization through the two-layer optimization model, improving the coding rationality, practicality, and throughput.

Description

Technical Field

[0001] This invention belongs to the field of big data and storage coding technology, specifically relating to an erasure coding method and system based on two-layer optimization. Background Technology

[0002] Since 2010, with the rapid development of cloud computing, big data, and artificial intelligence technologies, the global data scale has grown exponentially, placing higher demands on the reliability, scalability, and performance of distributed storage systems. In ultra-large-scale storage systems, node failures, disk damage, and network anomalies occur frequently, necessitating effective redundancy mechanisms to ensure data security. While traditional multi-replica mechanisms (such as triple replicas) are simple to implement, their high storage redundancy and cost make them unsuitable for the economic demands of modern cloud storage. Therefore, erasure coding (EC), with its significantly lower storage overhead and stronger fault tolerance compared to multi-replica systems, has become the mainstream redundancy method for major cloud platforms and distributed systems. Systems such as Microsoft Azure Storage, Ceph, HDFS, and Facebook f4 all use RS series coding as their core redundancy scheme.

[0003] However, in the actual deployment of erasure coding, the computational overhead of encoding becomes the main bottleneck affecting system throughput and resource utilization. The encoding process of erasure coding involves algebraic layer computation over finite fields and bit-level XOR calculations. To achieve efficient execution on the CPU, existing systems generally adopt the method of expanding finite field multiplication into bit-wise XOR operations for encoding. However, in large-scale parameter (such as...) Under certain conditions, an encoding process may require tens of thousands of XOR operations, consuming CPU resources, increasing encoding latency, and consequently affecting I / O performance. Therefore, optimizing the encoding performance of erasure coding has become an important research direction in the field of distributed storage.

[0004] Current research on addressing the performance issues of erasure coding mainly focuses on two directions: one focuses on optimizing the generator matrix structure, typically aiming to reduce the Hamming weight (i.e., the number of non-zero elements); the other focuses on XOR scheduling optimization, attempting to reduce bit-level computation by merging duplicate intermediate results. Related studies include Plank et al.'s Jerasure (2009), Uber-CSHR (2012) algorithm, and Sathiamoorthy et al.'s Xorbas (VLDB2013). However, these studies often separate matrix design from XOR scheduling, neglecting the strong coupling between the two. More importantly, our team has discovered through extensive experiments that the generator matrix with the minimum Hamming weight is not unique, and the corresponding number of XOR operations varies significantly; there are even cases where a generator matrix with a larger Hamming weight has fewer XOR operations.

[0005] This phenomenon shows that relying solely on Hamming weight as an optimization metric cannot guarantee the optimality of bit-level coding computation, which is precisely the theoretical blind spot of existing coding optimization work.

[0006] Shortcomings of existing technology: (1) Separate optimization of the sparsity of the generated matrix and the scheduling complexity of XOR. Existing methods usually only minimize the Hamming weight without considering the bit-level scheduling results, which often leads to the situation where the structure is optimal but the computation is not optimal.

[0007] (2) The minimum Hamming weight is not equal to the minimum XOR order. Existing studies generally assume that the Hamming weight is positively correlated with the XOR order, which leads to a large amount of potential optimization space being ignored.

[0008] (3) Lack of cross-layer feedback mechanism. Existing methods cannot feed back XOR scheduling performance to the generator matrix design process, which makes it impossible to guide the structure selection from the perspective of bit-level performance. Summary of the Invention

[0009] To address the issues of fragmented optimization between the sparsity of the erasure coding generator matrix and the complexity of XOR scheduling, the neglect of a large amount of potential optimization space, and the lack of cross-layer feedback mechanisms, and to improve the coding performance of erasure coding, this invention provides a two-layer optimization-based erasure coding method in its first aspect. The method includes: obtaining storage parameters of the data to be stored; constructing an initial generator matrix structure and a reference table for characterizing the bit-level characteristics of elements; the reference table records the distribution or number of non-zero bits in the binary matrix corresponding to the finite field elements after the bit matrix expansion; constructing a two-layer optimization model; and using the two-layer optimization model to optimize the initial generator matrix. The generator matrix structure is collaboratively optimized; the two-layer optimization model includes: an upper-layer optimization model configured to adjust the structural distribution of the generator matrix to optimize its algebraic properties; and a lower-layer optimization model configured to schedule the bit-level operation paths after the generator matrix is ​​expanded based on the reference table, and calculate the actual bit-level computational overhead; the bit-level computational overhead of the lower-layer optimization model is fed back to the upper-layer optimization model, and the target generator matrix is ​​updated and output based on the comprehensive evaluation result of the algebraic properties and the bit-level computational overhead; the target generator matrix is ​​then used to encode the data to be stored.

[0010] In some embodiments of the present invention, the upper-level optimization model is configured to simultaneously estimate the change in bit-level computational overhead caused by the element change using the reference table when adjusting an element.

[0011] In some embodiments of the present invention, the upper-level optimization model employs a heuristic search algorithm to generate new candidate structures by performing neighborhood replacements in the candidate element set of the generating matrix; and during the search process, a tabu list is used to record visited structures to avoid local optima.

[0012] Furthermore, the two-layer optimization model also includes a penalty mechanism: if the bit-level computational overhead corresponding to the currently generated matrix structure exceeds a preset threshold, the probability of the matrix structure being selected as the target structure is reduced or the matrix structure is added to the tabu list.

[0013] In some embodiments of the present invention, the lower-level optimization model constructs intermediate vector reuse paths by identifying reusable substructures in the bit matrix, thereby minimizing the number of XOR operations required to perform encoding.

[0014] In some embodiments of the present invention, updating and outputting the target generation matrix includes: employing a diversity-preserving greedy algorithm to retain multiple different matrix structures as candidate matrix structures for iteration while screening low-level computational overhead structures.

[0015] A second aspect of the present invention provides an erasure coding system based on two-layer optimization, comprising: an acquisition module for acquiring storage parameters of data to be stored, constructing an initial generator matrix structure and a reference table for characterizing bit-level characteristics of elements; the reference table recording the distribution or number of non-zero bits in the binary matrix corresponding to the finite field elements after bit matrix expansion; a construction module for constructing a two-layer optimization model and using the two-layer optimization model to perform collaborative optimization on the initial generator matrix structure; the two-layer optimization model comprising: an upper-layer optimization model configured to adjust the structural distribution of the generator matrix to optimize the algebraic characteristics of the generator matrix; a lower-layer optimization model configured to schedule the bit-level operation paths after the generator matrix expansion based on the reference table and calculate the actual bit-level computational overhead; an update module for feeding back the bit-level computational overhead of the lower-layer optimization model to the upper-layer optimization model, updating and outputting a target generator matrix based on the comprehensive evaluation result of the algebraic characteristics and the bit-level computational overhead; and an encoding module for encoding the data to be stored using the target generator matrix.

[0016] A third aspect of the present invention provides an electronic device, comprising: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the erasure coding method based on two-layer optimization provided in the first aspect of the present invention.

[0017] In a fourth aspect, the present invention provides a computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the erasure coding method based on two-layer optimization provided in the first aspect of the present invention.

[0018] The beneficial effects of this invention are: (1) This invention proposes an erasure coding system based on a two-layer optimization model, which realizes the coordinated unification of generator matrix structure optimization and XOR scheduling optimization. This technology has a simple structure and clear parameters, and can effectively solve the problem of the separation between structure optimization and bit-level computation in the prior art, thereby improving the rationality and practicality of coding design.

[0019] (2) Under typical storage parameters, the present invention can reduce the number of XOR operations and increase the encoding throughput by 2 to 5 times. While maintaining the same redundancy overhead, the encoding system of the present invention exhibits significant performance advantages and is suitable for high-performance data encoding scenarios in cloud storage and distributed storage systems. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the basic process of the erasure coding method based on two-layer optimization in some embodiments of the present invention; Figure 2 This is a schematic diagram illustrating the specific process of the erasure coding method based on two-layer optimization in some embodiments of the present invention; Figure 3 This is a schematic diagram of the structure of a two-layer optimized erasure coding system in some embodiments of the present invention; Figure 4 This is a schematic diagram of the structure of an electronic device in some embodiments of the present invention. Detailed Implementation

[0021] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0022] Example 1 refer to Figure 1 and Figure 2In a first aspect of the present invention, an erasure coding method based on two-layer optimization is provided, comprising: S100. obtaining storage parameters of the data to be stored, constructing an initial generator matrix structure and a reference table for characterizing the bit-level characteristics of elements; the reference table records the distribution or number of non-zero bits in the binary matrix corresponding to the finite field elements after the bit matrix expansion; S200. constructing a two-layer optimization model and using the two-layer optimization model to perform collaborative optimization on the initial generator matrix structure; the two-layer optimization model includes: an upper-layer optimization model configured to adjust the structural distribution of the generator matrix to optimize the algebraic characteristics of the generator matrix; and a lower-layer optimization model configured to schedule the bit-level operation path after the generator matrix expansion based on the reference table and calculate the actual bit-level computational overhead; S300. feeding back the bit-level computational overhead of the lower-layer optimization model to the upper-layer optimization model, and updating and outputting a target generator matrix according to the comprehensive evaluation result of the algebraic characteristics and the bit-level computational overhead; and S400. encoding the data to be stored using the target generator matrix.

[0023] In step S100 of some embodiments of the present invention, the storage parameters of the data to be stored are obtained, an initial generation matrix structure and a reference table for characterizing the bit-level characteristics of the elements are constructed; the reference table records the distribution or number of non-zero bits in the binary matrix corresponding to the finite field elements after the bit matrix is ​​expanded. Specifically, S101. Construct the erasure coding generation matrix structure system; Based on the construction methods of systematic Reed-Solomon or Cauchy Reed-Solomon coding, the data block matrix and parity block matrix structures for coding are established. The goal of this step is to form an initial structural scheme that can be used for coding computation and to ensure that the generated matrix satisfies the MDS condition, thereby enabling the recovery of the original data content in the event of data loss. This step involves setting different finite field parameters (such as...) The initial structural form allows the system to form a structural space that can be used for iterative optimization.

[0024] S102. Construct a reference table for characterizing the bit-level properties of elements. Specifically, based on commonly used erasure coding configurations in engineering (in order of number of data blocks, number of parity blocks, and stripe unit size), such as (10,2,8), (6,2,8), and (12,3,8), typical parameter ranges for coding performance evaluation are collected. The evaluation system includes the structural compactness index of the generator matrix, the computational complexity index after bit-level expansion, and the overall throughput index of the coding system. These parameters constitute the system's input space, enabling this invention to stably and effectively optimize the coding structure under different conditions. The coding parameters and test set include, but are not limited to, RS codes, Cauchy RS codes, and XOR scheduling algorithms (such as Jerasure, GF-Complete, and Uber-CSHR); the coding test parameters used include... The construction method of field elements, matrix type (Cauchy / Vandermonde), bit-matrix expansion method, XOR scheduling strategy, etc.

[0025] Typical test sets in this field have the following properties: diverse sparsity of generated matrices, controllable complexity of bit matrix structure, and significant differences in XOR scheduling for different matrices. These properties provide a stable and reliable test platform for this invention, used to verify the comprehensive performance of the optimization algorithm at the structure and scheduling layers.

[0026] It's understandable that the erasure coding problem has two substructures: an algebraic layer (generating matrix structure) and a bit-level layer (XOR scheduling path). The algebraic layer is responsible for determining... The matrix construction method at the top level determines the actual number of XOR operations through the bit matrix expansion of the field elements at the bit level. Traditional works mostly perform sparsity optimization only at the algebraic level, neglecting the non-linear coupling between the generator matrix structure and bit-level scheduling complexity. This makes it difficult to obtain the optimal coding structure in terms of actual computational performance by relying solely on Hamming weights. Therefore, in this invention, constructing a two-layer optimization model that can simultaneously describe structural characteristics and bit-level complexity is a necessary foundational work.

[0027] In step S200 of some embodiments of the present invention, a two-layer optimization model is constructed, and the initial generator matrix structure is collaboratively optimized using the two-layer optimization model; the two-layer optimization model includes: an upper-layer optimization model configured to adjust the structural distribution of the generator matrix to optimize the algebraic properties of the generator matrix; and a lower-layer optimization model configured to schedule the bit-level operation paths after the generator matrix is ​​expanded based on the reference table and calculate the actual bit-level computational overhead; the upper-layer optimization model is configured to simultaneously estimate the change in bit-level computational overhead caused by the element change using the reference table when adjusting elements.

[0028] The construction of the two-layer optimization model includes three processes: system initialization, the effect of the two-layer optimization model on the coding structure, and the coding structure update. System initialization: In this stage, the invention constructs an initial matrix structure for coding based on predetermined coding parameters (such as the number of data blocks, the number of parity blocks, and the width of the finite field). This structure consists of a systematic part and a redundant part, ensuring the correctness and integrity of the coding process. Simultaneously, to accurately quantify the algebraic characteristics and execution burden of different structures in subsequent structure optimization, the invention establishes a reference table to characterize the bit-level characteristics of finite field elements, describing the behavior of different elements during bit-level expansion. This table helps the system quickly identify the characteristics of the coding structure at different levels, making subsequent model analysis more efficient.

[0029] In addition, this stage generates multiple sets of alternative initial structures by applying different degrees of perturbation to the initial matrix structure, so that the system has structural diversity in the early stage of optimization, providing sufficient candidate structures for the two-layer model and ensuring that the optimization direction is global.

[0030] Specifically, firstly in a finite field Select the set of elements that satisfy the disjoint constraint. and Construct the initial Cauchy generation matrix Simultaneously, during the initialization phase, this invention also requires the construction of an ONES table (ONES matrix), which records the results of element-wise multiplication of finite fields after bit matrix expansion. The number of "1"s in a binary matrix. Since the changes in the generator matrix structure are mainly due to... and The neighborhood substitution caused by this allows us to use the ONES table without re-expanding the bit matrix, in order to... The time complexity is reduced to quickly calculate Hamming weight changes, thus significantly reducing the computational overhead of structural evaluation.

[0031] This invention also generates multiple sets of initial candidate structures during the initialization phase, through... and Several matrices with different sparse patterns are constructed by applying slight perturbations to the elements, ensuring good coverage of the structure space from the initial stage. These initial structures are then combined with the ONES table, enabling the subsequent two-layer optimization process to efficiently evaluate a large number of candidate structures with less computation.

[0032] Through the above methods, this invention constructs the initial structure set, bit matrix expansion mechanism, and structure evaluation tool required for the two-layer optimization solution, enabling the system to provide a stable and efficient foundation for subsequent searches. In the two-layer optimization model stage, the candidate generator matrices are systematically solved. The upper-layer optimization generates new candidate matrix structures through tabu search, aiming to reduce the sparsity of the generator matrix; the lower-layer optimization performs XOR path optimization on the bit matrix after the generator matrix expansion using the Uber-CSHR scheduling algorithm, obtaining the actual number of XOR operations.

[0033] Optionally, the upper-level optimization model employs a heuristic search algorithm to generate new candidate structures by performing neighborhood replacements in the candidate element set of the generating matrix; and during the search process, a tabu list is used to record visited structures to avoid local optima.

[0034] Specifically, the upper-level structure is generated by replacing sets. and The neighborhood structure is constructed from individual elements in the ONES table, which can quickly calculate the corresponding neighborhood structure after element replacement. The change in the bit matrix "1" allows for a rapid assessment of the Hamming weight without having to regenerate the entire bit matrix.

[0035] Furthermore, the two-layer optimization model also includes a penalty mechanism: if the bit-level computational overhead corresponding to the currently generated matrix structure exceeds a preset threshold, the probability of the matrix structure being selected as the target structure is reduced or the matrix structure is added to the tabu list.

[0036] Specifically, the tabu search mechanism avoids the search process from getting stuck in localized oscillations by recording recently visited inferior structures; the expectation criterion allows breaking tabus to accept structures that may improve XOR costs, making the upper-level search direction more flexible and global.

[0037] Optionally, the lower-level optimization model constructs intermediate vector reuse paths by identifying reusable substructures in the bit matrix, thereby minimizing the number of XOR operations required to perform encoding.

[0038] Specifically, the lower-level XOR scheduling evaluation uses the Uber-CSHR scheduling algorithm to detect reusable structures in the bit matrix, constructing intermediate vector reuse paths to minimize the number of XOR operations. The obtained XOR cost serves as a lower-level feedback metric to guide upper-level structure optimization and triggers a penalty mechanism: if the XOR cost of a certain structure fails to improve upon the historical best value, it is penalized and added to the tabu list, causing the upper layer to actively avoid that structure in subsequent iterations.

[0039] It is understandable that the system measures the processing load after matrix expansion from a bit-level perspective, including the characteristics of the binary matrix, the degree of reuse between different parts, and the computational burden generated during execution. The task of this layer is to ensure that the retained matrix is ​​not only structurally sound but also has a lower processing load during actual execution, which is more conducive to improving encoding throughput.

[0040] The system compares the evaluation results at two levels, retaining the advantageous parts of the matrix and replacing or adjusting the weaker parts, thus gradually evolving the structure towards a better performance at both the algebraic and execution levels. The key to this two-layer mechanism is that the structural and execution levels work collaboratively within the same system, ensuring that the optimization of the coding structure no longer relies solely on a single criterion, but rather gradually converges to a final solution more suitable for execution through multi-dimensional analysis.

[0041] The coding structure update phase: After the two-layer model completes the analysis of the current structure, this invention updates and organizes the candidate structures. The system selects matrices with better overall performance as the basic structure for the next stage, while discarding matrices with weak algebraic properties and high execution burden. To avoid the system focusing on a single structure type in a short period of time, this invention retains a certain number of differentiated candidate structures, keeping the optimization process open and diverse in search direction.

[0042] In step S300 of some embodiments of the present invention, updating and outputting the target generation matrix includes: employing a diversity-preserving greedy algorithm to retain multiple different matrix structures as candidate matrix structures for iteration while screening low-level computational overhead structures.

[0043] Specifically, the new structure generated through the above solution process is compared with the current structure set, and the matrix with better XOR cost and reasonable sparse structure is selected to enter the next generation of candidate sets. This invention uses a greedy strategy to screen the candidate structure set, making it not only more effective for XOR performance optimization, but also maintaining good diversity and uniformity of distribution in the structure space.

[0044] Understandably, the updated structure set will be re-input into the upper-level optimization module, allowing the system's solution process to continue. This optimization loop terminates when the set maximum number of iterations is reached or the XOR cost stops improving, thereby obtaining the optimal generator matrix that meets encoding requirements, has low bit-level computational cost, and is structurally stable.

[0045] The updated structure will then undergo another round of two-layer analysis, allowing the system optimization process to continue. As the structure improves, the processing power required for matrix encoding gradually decreases, and the usability and stability of the structure continuously improve. Ultimately, when the overall performance of the system no longer shows significant improvement or reaches the set conditions, this invention outputs a coding matrix obtained through stepwise optimization based on two-layer analysis. This matrix has a clear and reasonable algebraic structure, low bit-level execution burden, and can significantly improve the overall performance of the erasure coding system.

[0046] In step S400 of some embodiments of the present invention, the target generation matrix is ​​used to encode the data to be stored.

[0047] By running tests on multiple typical parameter combinations (such as (10,2,8), (6,4,8), (8,3,8), etc.), the optimal generator matrix and its corresponding best XOR scheduling path under each parameter setting can be obtained. The essence of the test is to compare the approximation between the solution set found by the algorithm of this invention and the theoretically optimal XOR count, and to evaluate the ability of the optimization result to achieve balance at both the structural and bit-level levels. The main evaluation indicators used include: actual XOR count, coding throughput, intermediate result reuse, and scheduling path stability. Experimental results demonstrate that this invention can effectively find a coding structure that approximates the theoretically optimal XOR, and its performance on the test function set is significantly better than existing methods.

[0048] In a specific embodiment of the present invention, the following steps are included: Step 1, determining the algebraic structure and bit-level computational characteristics of erasure coding; Step 2, collecting typical erasure coding parameters and coding test sets and determining the optimization-related parameters to be used; Step 3, designing a coding system based on a two-layer optimization model; Step 4, testing the performance of the system under multiple typical erasure coding parameter combinations.

[0049] In a specific embodiment of the present invention, the invention was tested using more than 10 sets of typical erasure coding parameters. The results show that the present invention can effectively find solutions close to the optimal XOR cost in coding structures of different scales, and exhibits overall performance superior to traditional methods (such as Jerasure). Both coverage and the inversion distance of XOR cost demonstrate that the system has significant advantages in convergence, diversity, and stability.

[0050] Example 2 refer to Figure 3In a second aspect, the present invention provides an erasure coding system 1 based on two-layer optimization, comprising: an acquisition module 11, configured to acquire storage parameters of data to be stored, construct an initial generator matrix structure and a reference table for characterizing the bit-level characteristics of elements; the reference table records the distribution or number of non-zero bits in the binary matrix corresponding to the finite field elements after the bit matrix expansion; a construction module 12, configured to construct a two-layer optimization model and use the two-layer optimization model to perform collaborative optimization on the initial generator matrix structure; the two-layer optimization model includes: an upper-layer optimization model configured to adjust the structural distribution of the generator matrix to optimize the algebraic characteristics of the generator matrix; a lower-layer optimization model configured to schedule the bit-level operation path after the generator matrix expansion based on the reference table and calculate the actual bit-level computational overhead; an update module 13, configured to feed back the bit-level computational overhead of the lower-layer optimization model to the upper-layer optimization model, and update and output the target generator matrix according to the comprehensive evaluation result of the algebraic characteristics and the bit-level computational overhead; and an encoding module 14, configured to encode the data to be stored using the target generator matrix.

[0051] Furthermore, the update module 13 includes: a filtering unit for using a greedy algorithm to filter low-level computational overhead structures; and a retention unit for retaining multiple different matrix structures as candidate matrix structures for iteration.

[0052] Example 3 refer to Figure 4 In a third aspect, the present invention provides an electronic device comprising: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the erasure coding method based on two-layer optimization of the first aspect of the present invention.

[0053] Electronic device 500 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 502 or a program loaded from storage device 508 into random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of electronic device 500. The processing unit 501, ROM 502, and RAM 503 are interconnected via bus 504. An input / output (I / O) interface 505 is also connected to bus 504.

[0054] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 508 including, for example, hard disks; and communication devices 509. Communication device 509 allows electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 4 An electronic device 500 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 4 Each box shown can represent a device or multiple devices as needed.

[0055] Specifically, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by a processing device 501, it performs the functions defined in the methods of embodiments of this disclosure. It should be noted that the computer-readable medium described in embodiments of this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0056] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more computer programs, which, when executed by the electronic device, cause the electronic device to: Computer program code for performing the operations of embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, C++, and Python—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0057] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0058] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A two-layer optimized erasure coding method, characterized in that, include: Obtain the storage parameters of the data to be stored, construct the initial generation matrix structure and the reference table used to characterize the bit-level characteristics of the elements; The reference table records the distribution or number of non-zero bits in the binary matrix corresponding to the finite field elements after the bit matrix expansion; A two-layer optimization model is constructed, and the initial generation matrix structure is collaboratively optimized using the two-layer optimization model; The two-layer optimization model includes: an upper-layer optimization model configured to adjust the structural distribution of the generator matrix to optimize its algebraic properties; and a lower-layer optimization model configured to schedule the bit-level operation paths of the expanded generator matrix based on the reference table and calculate the actual bit-level computation overhead. The bit-level computational overhead of the lower-level optimization model is fed back to the upper-level optimization model. Based on the comprehensive evaluation result of the algebraic characteristics and the bit-level computational overhead, the target generation matrix is ​​updated and output. The target generation matrix is ​​used to encode the data to be stored.

2. The erasure coding method based on two-layer optimization according to claim 1, characterized in that, The upper-level optimization model is configured to simultaneously estimate the change in bit-level computational overhead caused by the element change using the reference table when adjusting an element.

3. The erasure coding method based on two-layer optimization according to claim 1, characterized in that, The upper-level optimization model employs a heuristic search algorithm to generate new candidate structures by performing neighborhood replacements in the candidate element set of the generation matrix; and during the search process, a tabu list is used to record visited structures to avoid local optima.

4. The erasure coding method based on two-layer optimization according to claim 1, characterized in that, The lower-level optimization model constructs intermediate vector reuse paths by identifying reusable substructures in the bit matrix, thereby minimizing the number of XOR operations required to perform encoding.

5. The erasure coding method based on two-layer optimization according to claim 3, characterized in that, The two-layer optimization model also includes a penalty mechanism: if the bit-level computational overhead corresponding to the currently generated matrix structure exceeds a preset threshold, the probability of the matrix structure being selected as the target structure is reduced or the matrix structure is added to the tabu list.

6. The erasure coding method based on two-layer optimization according to claim 1, characterized in that, The process of updating and outputting the target generated matrix includes: employing a greedy algorithm to select low-level computational overhead structures while retaining multiple different matrix structures as candidate matrix structures for iteration.

7. A two-layer optimized erasure coding system, characterized in that, include: The acquisition module is used to acquire the storage parameters of the data to be stored, construct the initial generation matrix structure and the reference table used to characterize the bit-level characteristics of the elements; the reference table records the distribution or number of non-zero bits in the binary matrix corresponding to the finite field elements after the bit matrix expansion. A construction module is used to construct a two-layer optimization model and to perform collaborative optimization of the initial generation matrix structure using the two-layer optimization model; The two-layer optimization model includes: an upper-layer optimization model configured to adjust the structural distribution of the generator matrix to optimize its algebraic properties; and a lower-layer optimization model configured to schedule the bit-level operation paths of the expanded generator matrix based on the reference table and calculate the actual bit-level computation overhead. The update module is used to feed back the bit-level computational overhead of the lower-level optimization model to the upper-level optimization model, and update and output the target generation matrix based on the comprehensive evaluation result of the algebraic characteristics and the bit-level computational overhead. The encoding module is used to encode the data to be stored using the target generation matrix.

8. The erasure coding system based on two-layer optimization according to claim 7, characterized in that, The update module includes: The filtering unit is used to filter low-level computational overhead structures using a diversity-preserving greedy algorithm. Reservation units are used to retain multiple different matrix structures as candidate matrix structures for iteration.

9. An electronic device, comprising: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the erasure coding method based on two-layer optimization as described in any one of claims 1 to 6.

10. A computer-readable medium having a computer program stored thereon, wherein, When the computer program is executed by the processor, it implements the erasure coding method based on two-layer optimization as described in any one of claims 1 to 6.

Images