Erasure coding data layout methods, devices, equipment and products
By using time-window archive packages as input in a cold archiving system, utilizing erasure coding segmentation and recovery chain-aware layout constraints, and combining cold media topology graphs to optimize erasure coding layout, the problem of high recall cost and unpredictable timing during recovery in the cold archiving system is solved, achieving a low-cost and highly predictable recovery scheme.
Patent Information
- Application Number
- CN202611300759.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-26
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2046-08-26
AI Technical Summary
In existing cold archiving systems, erasure coding layout schemes do not use the recoverable time window of the relational database as layout input, resulting in high recall costs during recovery, unpredictable recovery time, and a lack of lifecycle pre-check mechanism.
By obtaining the time window archive as the erasure coding stripe input, the erasure coding algorithm is used to divide it into data shards and verification shards, generating recovery chain-aware layout constraints, and combining the cold media topology graph to calculate the window-level recall cost. Candidate shard layouts that meet the constraints are selected as shard sets and written into the shard layout index.
It reduces the overall recall cost during recovery, improves the predictability of recovery time, and enables the persistence and reusability of deployment decisions.
Smart Images

Figure CN122816984B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data storage technology, and in particular to an erasure coding data layout method, apparatus, device and product. Background Technology
[0002] Point-in-time recovery of relational databases typically relies on the coordination of base backups, write-ahead log segments, timeline history, backup lists, and repository metadata. When the target recovery time falls within a recoverable time window, the recovery system needs to recall not a single isolated object, but a collection of repository objects that can form a complete recovery chain.
[0003] To reduce long-term storage costs, cold archiving systems typically migrate backup objects to tape libraries, deep archive object storage, or other cold media, and improve data durability through erasure coding technology. Existing erasure coding layout schemes often use ordinary objects, ordinary file blocks, or ordinary prefixes as stripe inputs, and place shards according to single metrics such as capacity balancing, fault domain isolation, connection latency, single object read performance, or media health. However, existing techniques do not use the recoverable time window of the relational database as layout input, nor do they consider the recovery chain dependencies between basic backups, write-ahead log segments, timeline history, backup lists, and metadata objects as core constraints for shard layout. In environments with multiple tape libraries, multiple controllers, and multiple drive pools, data shards belonging to the same target recovery time window may be scattered across multiple libraries, controllers, and drive paths, resulting in high recall costs during recovery, unpredictable recovery times, and difficulty in guaranteeing recovery service level agreements. Summary of the Invention
[0004] The main purpose of this application is to provide an erasure coding data layout method, apparatus, device and product, which aims to solve the technical problems of high recall cost and unpredictable recovery time during recovery caused by random dispersion or simple concentration of fragments in existing cold archiving systems.
[0005] To achieve the above objectives, this application proposes an erasure coding data layout method, the method comprising: Obtain the time window archive package and use the time window archive package as an erasure coding stripe; The erasure code stripe is segmented using an erasure code encoding algorithm to obtain data fragments and check fragments. Based on the recovery dependencies between data objects in the time window archive, a recovery chain-aware layout constraint is generated. Based on the recovery chain-aware layout constraints and the pre-constructed cold media topology, candidate shard layouts are generated for the data shards and check shards. The window-level recall cost for each candidate shard layout when the target recovery time falls within the recoverable time window is calculated. Specifically, this includes: determining the candidate storage location set corresponding to each data shard and check shard based on the available storage nodes and available drivers in the cold media topology; selecting storage location combinations that meet the recovery chain integrity requirements from the candidate storage location set, based on the recovery chain-aware layout constraints, to generate multiple candidate shard layouts; and substituting each candidate shard layout into a preset window-level recall cost function to calculate the window-level recall cost for each candidate shard layout when the target recovery time falls within the recoverable time window. From the candidate sharding layouts, the candidate sharding layouts that meet the preset constraints and have the lowest window-level recall cost are selected as the sharding set and written into the sharding layout index.
[0006] In one embodiment, the step of dividing the erasure code stripe into erasure code segments using an erasure code encoding algorithm to obtain data segments and parity segments includes: The erasure coding algorithm uses preset erasure coding parameters to segment the erasure code stripe, generating data fragments and check fragments.
[0007] In one embodiment, the step of dividing the erasure code stripe into erasure code segments using an erasure code encoding algorithm to obtain data segments and parity segments includes: The erasure coding algorithm uses preset erasure coding parameters to segment the erasure code stripe, generating data fragments and check fragments.
[0008] In one embodiment, before generating multiple candidate shard layouts for the data shards and verification shards based on the recovery chain-aware layout constraints and a pre-constructed cold media topology, and calculating the window-level recall cost of each candidate shard layout when the target recovery time falls within the recoverable time window, the method further includes: Construct a cold media topology based on at least one of multiple tape libraries, cold archive libraries, controllers, drive pools, drives, slots, robotic arm paths, media health, and bandwidth windows; In one embodiment, the step of selecting candidate shard layouts from the candidate shard layouts that satisfy preset constraints and have the lowest window-level recall cost as a shard set, and writing them into the shard layout index, includes: Select a candidate fragmentation layout from the candidate fragmentation layouts that satisfies one or more of the following constraints: durability, reconfigurability, cold media fault domain isolation, recovery chain constraint, and future point-in-time recovery cost constraint. Based on the window-level recall cost corresponding to the fragmentation layout, select fragmentation layouts with window-level recall costs lower than a first preset threshold as fragmentation sets; Write the shard set into the shard layout index.
[0009] In one embodiment, after writing the shard set to the shard layout index, the method further includes: Obtain the recoverable time window affected by the lifecycle operation to be executed and the corresponding fragment layout in the fragment layout index; Calculate the pre-operation cost based on the fragmented layout, and after simulating the execution of the lifecycle operation, recalculate the post-operation cost, cost difference ratio, and reconfigurable state of the affected recoverable time window. Based on the pre-operation cost, post-operation cost, cost difference ratio, and reconfigurable state, the pre-inspection results are judged according to a preset priority. If the cost after the operation exceeds the second preset threshold, then a pre-detection decision for splitting is output. If the cost difference ratio exceeds a preset ratio threshold, a delayed pre-detection decision is output. If the recovery chain-aware layout constraint is broken, a pre-detection decision for re-layout is output. If the recoverable time window is in an unreconfigurable state, a pre-detection decision to refuse execution is output.
[0010] Furthermore, to achieve the above objectives, this application also proposes an erasure coding data layout apparatus, the apparatus comprising: The input acquisition module is used to acquire the time window archive package and input the time window archive package as the erasure code stripe; The erasure coding segmentation module is used to segment the erasure coding stripe using an erasure coding algorithm to obtain data fragments and check fragments. The constraint generation module is used to generate recovery chain-aware layout constraints based on the recovery dependencies between data objects in the time window archive. The cost calculation module is used to generate multiple candidate shard layouts for the data shards and check shards based on the recovery chain-aware layout constraints and a pre-constructed cold media topology map, and to calculate the window-level recall cost of each candidate shard layout when the target recovery time falls within the recoverable time window. Specifically, this includes: determining the candidate storage location set corresponding to each of the data shards and check shards based on the available storage nodes and available drivers in the cold media topology map; selecting storage location combinations that meet the recovery chain integrity requirements from the candidate storage location set in conjunction with the recovery chain-aware layout constraints to generate multiple candidate shard layouts; and substituting each candidate shard layout into a preset window-level recall cost function to calculate the window-level recall cost of each candidate shard layout when the target recovery time falls within the recoverable time window. The layout decision module is used to select the candidate shard layouts that meet the preset constraints and have the lowest window-level recall cost from the candidate shard layouts as a shard set, and write them into the shard layout index.
[0011] Furthermore, to achieve the above objectives, this application also proposes an erasure coding data layout apparatus, the apparatus comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the erasure coding data layout method as described above.
[0012] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the erasure coding data layout method as described above.
[0013] The one or more technical solutions proposed in this application have at least the following technical effects: by obtaining the time window archive package as the erasure coding stripe input and segmenting it through the erasure coding algorithm, data fragments and verification fragments are obtained; based on the recovery dependency relationship between each data object in the time window archive package, recovery chain-aware layout constraints are generated; based on the recovery chain-aware layout constraints and the pre-constructed cold media topology map, candidate fragment layouts are generated, and the window-level recall cost of each candidate fragment layout when the target recovery time falls within the recoverable time window is calculated; candidate fragment layouts that meet the constraints are selected from the candidate fragment layouts, and the candidate fragment layout with the lowest window-level recall cost is selected as the fragment set and written into the fragment layout index. Therefore, by using time-window archives as stripe input, we avoid reading irrelevant window packages or missing necessary pre-written log segments during recovery; by using recovery chain-aware layout constraints, we ensure that the shard set meets the requirements of recovery chain integrity and reconfigurability; by using cold media topology graphs and window-level recall costs, we reduce the overall recall cost and improve the predictability of recovery time; and by writing the optimal shard set into the index, we achieve the persistence and reusability of layout decisions. Attached Figure Description
[0014] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0016] Figure 1 This is a flowchart illustrating the first embodiment of the erasure coding data layout method of this application. Figure 2 This is a schematic diagram of the overall process of an embodiment of the erasure coding data layout method of this application; Figure 3 This diagram illustrates the relationship between the recoverable time window, the time window archive, and erasure coding fragments in this application. Figure 4 This is a schematic diagram of the cold media topology of the multi-cold archive library in this application; Figure 5 Flowchart for selecting the set of fragments based on the target recovery time during the recovery of this application; Figure 6 A flowchart of the pre-screening process for window-level recall costs before the lifecycle migration or rebalancing of this application; Figure 7 This is a schematic diagram of the module structure of the erasure coding data layout device according to an embodiment of this application; Figure 8 This is a schematic diagram of the device structure of the hardware operating environment involved in the erasure coding data layout method in the embodiments of this application.
[0017] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0018] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.
[0019] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.
[0020] The main solution of this application embodiment is as follows: A time-window archive is used as the input for erasure coding stripes, and the archive is segmented using an erasure coding algorithm to obtain data shards and verification shards. Recovery chain-aware layout constraints are generated based on the recovery dependencies between data objects in the time-window archive. Candidate shard layouts are generated based on these constraints and a pre-built cold media topology graph, and the window-level recall cost for each candidate shard layout when the target recovery time falls within the recoverable time window is calculated. Candidate shard layouts that meet the constraints are selected from the candidate shard layouts, and the candidate shard layout with the lowest window-level recall cost is chosen as the shard set and written to the shard layout index. Thus, by using the time-window archive as the stripe input, reading irrelevant window packets or missing necessary pre-written log segments during recovery is avoided; recovery chain-aware layout constraints ensure that the shard set meets the requirements of recovery chain integrity and reconfigurability; the cold media topology graph and window-level recall cost reduce the overall recall cost and improve the predictability of recovery time; and writing the optimal shard set to the index achieves persistence and reusability of layout decisions.
[0021] This application's embodiments take into account that existing erasure coding layout schemes in cold archiving systems mostly use ordinary objects, ordinary file blocks, or ordinary prefixes as stripe inputs, and place shards according to single indicators such as capacity balancing, fault domain isolation, connection latency, single object read performance, or media health. The above methods do not use the recoverable time window of the relational database as a layout input, nor do they consider the recovery chain dependencies between basic backups, write-ahead log segments, timeline history, backup lists, and metadata objects as core constraints for sharding layout. Specifically, this is reflected in the following: Recovery chain semantics are missing: The recovery time window of the relational database is not used as the layout input for recovery at a point in time. The basic backup and write-ahead log segments may be scattered to different storage locations, and the integrity of the recovery chain cannot be guaranteed during recovery.
[0022] Uncontrollable recall costs: Data fragments belonging to the same target recovery time window may be scattered across multiple libraries, controllers, and driver paths, resulting in increased tape loading times, increased robotic arm movement distances, intensified driver contention, and increased cross-controller traffic during recovery, leading to high recall costs.
[0023] Recovery time is unpredictable: Due to the random dispersion or simple concentration of shards, recovery time cannot be accurately estimated based on the physical state of the storage, and recovery service level agreements are difficult to guarantee.
[0024] Lack of constraint screening mechanism: Existing solutions do not take erasure code reconfigurability, cold medium fault domain isolation, recovery chain constraints and future point-in-time recovery cost constraints as joint optimization objectives for layout decisions, and cannot guarantee that the selected layout meets the system requirements in terms of durability, fault tolerance and recovery chain integrity.
[0025] Lack of lifecycle pre-inspection: Before lifecycle migration, tape rebalancing, fragment rewriting, or cold archive migration, the window-level recall cost and reconfigurable state after the operation are not simulated, and the impact of operation and maintenance on the existing layout cannot be predicted.
[0026] This application provides a solution that addresses the technical problems in existing cold archiving systems, such as high recall costs and unpredictable recovery times due to random or simple centralization of fragments.
[0027] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device, erasure coding data layout device, or system capable of implementing the above functions. The following description uses an erasure coding data layout system as an example to illustrate this embodiment and the subsequent embodiments.
[0028] Based on this, embodiments of this application provide an erasure coding data layout method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the erasure coding data layout method of this application.
[0029] In this embodiment, the erasure coding data layout method includes steps S10 to S50. The following provides a detailed explanation of each step.
[0030] like Figure 1 As shown, the first embodiment of this application proposes an erasure coding data layout method, the method comprising: Step S10: Obtain the time window archive package and input the time window archive package as an erasure code stripe; Specifically, the time window archive package corresponding to the recoverable time window generated by the upstream window index module is obtained, and the time window archive package is used as the erasure coding stripe input, instead of any object, any prefix, or ordinary backup block as the stripe input. The time window archive package corresponds to the recoverable time window defined by the database system identifier, backup namespace, basic backup identifier, timeline, log sequence number range, and recovery time range, and contains the basic backup required for the target recovery time within the recoverable time window, necessary write-ahead log segments, timeline history, backup list, and necessary repository metadata objects.
[0031] The time window archive package is the original data source for subsequent erasure coding segmentation and layout decisions. Therefore, by obtaining the archive package at the time window granularity, a data foundation is provided for the integrity of the recovery chain when restoring at a point in time.
[0032] Step S20: The erasure code stripe is segmented using an erasure code encoding algorithm to obtain data fragments and check fragments; Specifically, the erasure coding algorithm divides the time window archive, which serves as the input of the erasure coding stripe, into data fragments and verification fragments according to the preset erasure coding parameters. The erasure coding parameters can be set to k and m, where k is the number of data fragments and m is the number of verification fragments. That is, each time window archive is divided into k data fragments and m verification fragments.
[0033] Therefore, by dividing the time window archive into erasure coding fragments, a data foundation is provided for reliable storage and efficient recovery of data in the cold media topology.
[0034] Step S30: Generate recovery chain-aware layout constraints based on the recovery dependencies between data objects in the time window archive package; Specifically, recovery chain-aware layout constraints are generated based on the recovery dependencies of the base backup, necessary write-ahead log segments, timeline history, backup list, and metadata objects in the time window archive. These constraints ensure that when any target recovery time falls within the time window, the selected shard set can restore the recovery chain objects required for that target recovery time.
[0035] Therefore, by constraining the sharding layout of the base backup, write-ahead log segments, timeline history, backup list, and metadata objects through recovery chain-aware layout constraints, the sharding set selected during recovery can meet the requirements of database recovery chain integrity and reconfigurability.
[0036] Step S40: Based on the recovery chain-aware layout constraints and the pre-built cold media topology, generate candidate shard layouts for the data shards and verification shards, and calculate the window-level recall cost of each candidate shard layout when the target recovery time falls within the recoverable time window.
[0037] Specifically, this includes: determining the candidate storage location set corresponding to each of the data shard and the verification shard based on the available storage nodes and available drivers in the cold media topology diagram; selecting storage location combinations that meet the recovery chain integrity requirements from the candidate storage location set in conjunction with the recovery chain-aware layout constraints to generate multiple candidate shard layouts; and substituting each candidate shard layout into a preset window-level recall cost function to calculate the window-level recall cost of each candidate shard layout when the target recovery time falls within the recoverable time window. Specifically, candidate storage locations for each shard are determined based on available storage nodes and drivers in a pre-constructed cold media topology diagram. Combined with recovery chain-aware layout constraints, storage location combinations that meet recovery chain integrity requirements are selected from the candidate storage location set to generate multiple candidate shard layouts. Each candidate shard layout is then substituted into a preset window-level recall cost function to calculate the window-level recall cost for each candidate shard layout when the target recovery time falls within a recoverable time window. The window-level recall cost measures the overall cost of recalling the required set of shards when the target recovery time falls within the window.
[0038] Therefore, by combining recovery chain constraints and cold media physical conditions, the recall cost of each candidate sharding layout is evaluated, and the physical overhead of the storage media is transformed into a comparable quantitative indicator. This provides a basis for selecting a sharding layout that balances constraint compliance and low recall overhead, and improves the predictability of recovery time.
[0039] Step S50: From the candidate sharding layouts, select the candidate sharding layouts that meet the preset constraints and have the lowest window-level recall cost as a sharding set, and write it into the sharding layout index.
[0040] Specifically, candidate sharding layouts that satisfy one or more of the following constraints are selected from the candidate sharding layouts: erasure coding reconfigurability, cold media fault domain isolation, recovery chain-aware layout constraint, and time-point recovery cost constraint. Among the candidate sharding layouts that meet the constraints, the candidate sharding layout with the lowest window-level recall cost is selected as the sharding set, and the sharding set is written into the sharding layout index.
[0041] Thus, by writing the optimal set of shards into the index, the layout decision results are made persistent and reusable, reducing the overall recall cost of future target time recovery.
[0042] This embodiment's technical solution involves: acquiring a time window archive package and using it as an erasure coding stripe input; segmenting the erasure coding stripe using an erasure coding algorithm to obtain data shards and check shards; generating recovery chain-aware layout constraints based on the recovery dependencies between data objects in the time window archive package; acquiring the physical topology information and dynamic operating status information of the storage system to construct a cold media topology map; generating candidate shard layouts for the data shards and check shards based on the recovery chain-aware layout constraints and the cold media topology map, and calculating the window-level recall cost of each candidate shard layout when the target recovery time falls within the recoverable time window; selecting candidate shard layouts that meet preset constraints from the candidate shard layouts, and choosing the candidate shard layout with the lowest window-level recall cost from among the candidate shard layouts that meet the constraints as the shard set, and writing it into the shard layout index. Therefore, by obtaining the time window archive package as the erasure code stripe input, reading irrelevant window packages is avoided; by generating a recovery chain to perceive layout constraints, the integrity of the recovery chain is guaranteed; by constructing a cold media topology graph and calculating the window-level recall cost, the predictability of recovery time is improved; by selecting the set of shards that meet the constraints and have the lowest cost and writing them to the index, the persistence and reusability of the layout results are achieved, reducing the overall recall cost of recovery at the future target time.
[0043] This invention provides a multi-cold archive repository recall cost-aware erasure coding layout method based on a recoverable time window for point-in-time recovery. This method optimizes the sharding storage strategy of cold archived data across multiple archive repositories to reduce window-level data recall costs and meet recovery chain integrity constraints. The recoverable time window is defined as a recoverable interval jointly limited by the following metadata dimensions: database system identifier, backup namespace, base backup identifier, dependent backup chain, timeline, log sequence number range, recovery time interval, necessary write-ahead log segment range, timeline history, backup list, and necessary repository metadata. Based on this recoverable time window, all archived data objects required at a specific target recovery time can be determined. Correspondingly, the time window archive package refers to an archive package capable of reconstructing the set of all repository objects required for any target recovery time within the recoverable time window; this archive package is generated and provided by the upstream window index module.
[0044] In one feasible implementation, step S20, which involves dividing the erasure code stripe into erasure code segments using an erasure code encoding algorithm to obtain data fragments and check fragments, includes step S201: Step S201: The erasure coding algorithm is used to segment the erasure code stripe based on preset erasure code parameters to generate data fragments and check fragments.
[0045] Specifically, the erasure coding algorithm divides the time window archive, which serves as the input of the erasure coding stripe, into data fragments and verification fragments according to the preset erasure coding parameters. The erasure coding parameters can be set to k and m, where k is the number of data fragments and m is the number of verification fragments. That is, each time window archive is divided into k data fragments and m verification fragments.
[0046] The erasure coding parameters can be set to k=6 and m=3, meaning each time window archive is divided into 6 data fragments and 3 parity fragments. The time window archive can be reconstructed when any 6 available fragments satisfy the recovery chain constraints. For tenants with high recovery time targets, k=4 and m=2 can be set to reduce the number of fragments that need to be recalled simultaneously during recovery; for tenants with long-term storage and low recovery frequency, k=10 and m=4 can be set to improve storage efficiency.
[0047] The stripe size can be determined based on the logical size of the time window archive and the granularity of cold media read / write. For example, the target size of a single stripe can be configured to 768 GiB, where the archive size of each data shard is 128 GiB. When the time window archive is smaller than 768 GiB, the actual size of the time window archive is used as the stripe input and the fill length is recorded. When the time window archive is larger than 768 GiB, it is divided into multiple time window stripes according to the object dependency graph within the window, but each stripe is still bound to the same recoverable time window identifier and recovery chain constraint summary.
[0048] The fragment naming convention can be: {Backup Namespace} / {Time Window Identifier} / {Archive Package Identifier} / Eradication Code-k{k}-m{m} / Strip-{Strip Sequence Number} / Fragment-{Fragment Sequence Number}-{Role}.pkg, where the role is a data fragment or a verification fragment. The time window identifier is generated from the database system identifier, backup namespace, base backup identifier, timeline, log sequence number range, and recovery time range summary. The archive package identifier is bound to the backup manifest summary of the time window archive package. This naming convention enables the recovery scheduling module to first locate the recoverable time window based on the target recovery time, and then locate the fragment layout within that time window, avoiding reading irrelevant window packages.
[0049] Therefore, by using the archived time window corresponding to the recoverable time window as the erasure coding stripe input, the erasure coding layout directly serves the target time recovery chain of the relational database, rather than serving ordinary object reading, thereby reducing the risk of reading irrelevant window packets or missing necessary write-ahead log segments during recovery.
[0050] This embodiment, through the above-described scheme, segments the time-window archive package based on preset erasure coding parameters to generate data shards and verification shards, constructing an erasure coding sharding scheme for recoverable time windows. It not only uses the time window as the stripe input boundary to avoid the risk of recovery chain breaks, but also adapts to the recovery time targets and storage efficiency requirements of different tenants through erasure coding parameters. Furthermore, it binds time window identifiers to shard names, enabling precise location of the corresponding shard layout during recovery and avoiding reading irrelevant window packets. This ensures that the generated data shards and verification shards provide a reliable data foundation for subsequent candidate shard layout generation and recall cost assessment.
[0051] In one feasible implementation, step S30, generating recovery chain-aware layout constraints based on the recovery dependencies between data objects in the time window archive, includes step S301: Step S301: Generate recovery chain-aware layout constraints based on the recovery dependencies of at least one of the basic backup, necessary write-ahead log segments, timeline history, backup list, and metadata objects in the time window archive.
[0052] Specifically, based on the recovery dependencies among the base backup, write-ahead log segments, timeline history, backup list, and metadata objects in the time window archive, a recovery chain-aware layout constraint is generated. This recovery chain-aware layout constraint represents the necessary recovery relationships between objects within the window containing the target recovery time, ensuring that when any target recovery time falls within the time window, the selected shard set can restore the recovery chain objects required for that target recovery time.
[0053] Among them, the recovery chain-aware layout constraint can be recorded as a constraint expression, bitmap, object dependency graph, or summary of the object collection within the window; the constraint is read when generating candidate shard layouts to exclude layouts that would cause the necessary write-ahead log segments to be removed from the basic backup and thus lose the window recall capability.
[0054] This embodiment generates recovery chain-aware layout constraints through the above scheme. These constraints are used to ensure that when the target recovery time falls within the window, the selected shard set can restore the complete recovery chain object, including the window recallability, timeline continuity, and recovery visibility of the basic backup and necessary write-ahead log segments. Layouts that are not within the window recall capability are excluded when candidate shard layouts are generated. This provides a constraint basis for the integrity of the recovery chain for the subsequent generation and selection of candidate shard layouts, so that the shard set selected during recovery can meet the requirements of database recovery chain integrity and reconfigurability.
[0055] In one feasible implementation, before step S40, which involves generating multiple candidate shard layouts for the data shards and verification shards based on the recovery chain-aware layout constraints and the pre-constructed cold media topology, and calculating the window-level recall cost of each candidate shard layout when the target recovery time falls within the recoverable time window, the method further includes step S040: Step S040: Construct a cold media topology based on at least one of multiple tape libraries, cold archive libraries, controllers, drive pools, drives, slots, robotic arm paths, media health, and bandwidth windows.
[0056] Specifically, a cold media topology map is generated based on multiple tape libraries, cold archive libraries, controllers, drive pools, drives, slots, robotic arm paths, media health, and bandwidth windows. The nodes of the cold media topology map represent tape libraries, cold archive libraries, controllers, drive pools, drives, media, or archive areas, while the edges represent recall relationships between loading, moving, cross-controller reading, drive occupancy, or archive areas.
[0057] In addition, the fields of the cold media topology map can be collected from multiple types of real system states: tape library, slot, robotic arm path, drive, and media barcode fields can be exported from the tape library management interface, SCSI Medium Changer query, LTFS index, media inventory table, or tape library control software; controller, drive pool, port bandwidth, cross-controller link, and current load can be collected from the cold archive controller status table, storage network monitoring table, scheduling system status table, or bandwidth meter; object archive region, storage class, recovery class, expected recall delay, and requested quota can be collected from object archive library metadata, archive bucket lifecycle configuration, object recovery task status, and cloud archive interface response.
[0058] The topology acquisition module periodically generates topology snapshots, recording all the aforementioned identifiers and status fields. This module only adopts valid snapshots with timestamps and source identifiers. If a snapshot expires or key fields are missing, the corresponding candidate shard layout score is lowered, or the decision is delayed.
[0059] Therefore, by collecting topology fields from multiple real system states and periodically generating topology snapshots, a comprehensive, real-time, and reliable cold media topology map was constructed, providing an accurate data foundation for determining subsequent candidate storage locations and effectively avoiding layout decision errors caused by expired snapshots or missing key fields.
[0060] In one feasible implementation, step S40, which involves generating multiple candidate shard layouts for the data shards and check shards based on the recovery chain-aware layout constraints and a pre-constructed cold media topology map, and calculating the window-level recall cost of each candidate shard layout when the target recovery time falls within the recoverable time window, specifically includes: determining the candidate storage location set corresponding to each of the data shards and check shards based on the available storage nodes and available drivers in the cold media topology map; selecting storage location combinations that meet the recovery chain integrity requirements from the candidate storage location set in conjunction with the recovery chain-aware layout constraints to generate multiple candidate shard layouts; and substituting each candidate shard layout into a preset window-level recall cost function to calculate the window-level recall cost of each candidate shard layout when the target recovery time falls within the recoverable time window, including steps S401-S403: Step S401: Based on the available storage nodes and available drives in the cold media topology diagram, determine the candidate storage location set corresponding to each of the data shard and the parity shard. Specifically, the available storage nodes and drivers are read from the cold media topology diagram. Based on the erasure coding role and storage requirements of each shard, a set of candidate storage locations for each data shard and parity shard is determined. These candidate storage locations include at least one or more of the following: storage node identifier, driver identifier, slot identifier, and media identifier. Furthermore, based on the dynamic operating status information in the cold media topology diagram, storage nodes and drivers currently in a fault state, with loads exceeding a preset load threshold, or health levels below a preset health threshold are excluded from the candidate locations.
[0061] Therefore, by filtering available storage nodes and drives based on the cold media topology map, faulty, overloaded or unhealthy storage resources can be effectively eliminated, ensuring the actual availability and reliability of the candidate storage location set and avoiding recovery failures caused by sharding to unavailable or high-risk storage nodes.
[0062] Step S402: In conjunction with the recovery chain-aware layout constraints, select storage location combinations that meet the recovery chain integrity requirements from the candidate storage location set to generate multiple candidate sharding layouts; Specifically, from the candidate storage location set, various allocation combinations are made between each shard and each candidate storage location, and each combination is checked to see if it meets the recovery chain-aware layout constraints. If a combination causes the basic backup and necessary write-ahead log segments to be out of the same window recall capability, or if the timeline history and write-ahead log segments do not meet the timeline continuity requirement, or if the backup list and metadata objects are not visible in the recovery view, then the combination is excluded. Through the above filtering, all storage location combinations that meet the recovery chain integrity requirements are retained, generating multiple candidate shard layouts. Each candidate shard layout contains the correspondence between each shard and the target storage location in the storage node.
[0063] Therefore, by strictly screening storage location combinations through recovery chain-aware layout constraints, the integrity of each candidate shard layout in terms of recovery chain semantics is ensured, avoiding the risk of the basic backup and log segment becoming out of the same window recall capability or the recovery chain breaking due to improper layout scheme.
[0064] Step S403: Substitute each candidate fragment layout into the preset window-level recall cost function to calculate the window-level recall cost of each candidate fragment layout when the target recovery time falls into the recoverable time window.
[0065] Specifically, the window-level recall cost function can be expressed as: Cost (W,L) = 0.22 × Loading (W,L) + 0.12 × Robotic Arm (W,L) + 0.14 × Cross-Controller (W,L) + 0.24 × Latency (W,L) + 0.14 × Driver Busy (W,L) + 0.08 × Target Time Probability (W) - 0.06 × Parallel Gain (W,L). Where, W represents the recoverable time window restored at a given time point, and L represents the candidate shard layout. The normalization factors involved in this function include: "Load" is the normalized value of the number of tape loads or object archive recovery tasks, "Robot Arm" is the normalized value of the robot arm movement distance or media replacement path, "Cross Controller" is the normalized value of cross-controller read traffic, "Latency" is the normalized value of the expected recall latency, "Driver Busy" is the normalized value of driver contention, "Target Time Probability" is the target recovery time probability weight, and "Parallel Benefit" is the normalized value of the parallel recall benefit. The sum of the above default weights is normalized separately for the positive cost item and the deduction item. In actual deployment, it can be covered by tenant or backup namespace. The candidate shard layouts obtained in step S402 and their corresponding recoverable time windows are substituted into the window-level recall cost function to obtain the window-level recall cost for each candidate shard layout. The window-level recall cost is calculated based on the recoverable time window where the target recovery time is located and the necessary set of shards required to reconstruct the target recovery time. It includes at least normalized cost terms corresponding to tape loading, robotic arm movement, cross-controller reading, expected recall delay, driver contention, and parallel gains. When comparing candidate shard layouts, candidate shard layouts are first filtered according to the hard constraints, and then selected in ascending order according to the normalized window-level recall cost. When the minimum window-level recall cost is still not lower than the first preset threshold, it is not written into the shard layout index and an unsatisfiable state is output.
[0066] Furthermore, the calculated window-level recall cost will serve as the basis for candidate shard layout selection, while the differentiated artifacts referenced for selection are provided by the tenant-level recall strategy. The tenant-level configuration can be recorded as a recall strategy, which includes tenant identifier, backup namespace, service level, recovery time target, maximum window-level recall cost, maximum cost increment ratio, minimum number of reconfigurable shard sets, priority for controller affinity, priority for cross-drive pool parallelism, and weight configuration. Specifically, the recovery time target for the Gold service level can be 240 minutes, the maximum window-level recall cost is 0.45, the maximum cost increment ratio is 12%, and at least 3 reconfigurable shard sets are required; the recovery time target for the Silver service level can be 720 minutes, the maximum window-level recall cost is 0.65, the maximum cost increment ratio is 20%, and at least 2 reconfigurable shard sets are required; the recovery time target for the Archive service level can be 1440 minutes, the maximum window-level recall cost is 0.80, the maximum cost increment ratio is 30%, and at least 1 reconfigurable shard set is required.
[0067] Therefore, by substituting each candidate segment layout into the preset window-level recall cost function, the overall recall cost of each candidate segment layout when the target recovery time falls within the recoverable time window is quantitatively calculated, providing an objective and comparable quantitative evaluation basis for subsequent constraint screening and cost optimization.
[0068] This embodiment combines the construction of a cold media topology map with window-level recall cost calculation to build a physical awareness evaluation system oriented towards recoverable time windows. It not only collects topology fields from various real system states and periodically generates topology snapshots, providing a comprehensive, real-time, and reliable data foundation for determining candidate storage locations and effectively eliminating faulty, overloaded, or unhealthy storage resources, but also determines a set of candidate storage locations based on the cold media topology map. Combined with recovery chain-aware layout constraints, it filters storage location combinations that meet the recovery chain integrity requirements, generating multiple candidate shard layouts. Furthermore, it quantifies the overall recall cost of each candidate shard layout through a preset window-level recall cost function, thus providing an objective and comparable quantitative basis for subsequent constraint screening and cost optimization. This ensures that the final selected shard set has a controllable recall cost and predictable recovery time during recovery, reducing the overall recall cost for future target time recovery and improving recovery time predictability.
[0069] In one feasible implementation, step S50, which involves selecting candidate shard layouts from the candidate shard layouts that satisfy preset constraints and have the lowest window-level recall cost as a shard set, and writing them into the shard layout index, includes steps S501-S503: Step S501: Select a candidate fragmentation layout from the candidate fragmentation layouts that satisfies one or more of the following: durability, reconfigurability, cold media fault domain isolation, recovery chain constraint, and future point-in-time recovery cost constraint. Specifically, multiple candidate sharding layouts generated by S40 are filtered according to preset constraints. These constraints include: persistence constraints to ensure data is not lost during long-term storage; reconfigurability constraints to ensure the number of available shards is not less than the erasure coding parameter k, enabling the restoration of time-window archive packages; cold media fault domain isolation constraints to prevent shards in the same time window from clustering in the same library, controller, or driver pool, thus avoiding single points of failure; recovery chain constraints to ensure that basic backups and necessary write-ahead log segments are recallable within the same window, have a continuous timeline, and remain visible in the recovery view; and future point-in-time recovery cost constraints to ensure that the recall cost is lower than a preset window recall cost threshold when any target recovery time falls within this window. Based on satisfying all hard constraints, soft constraints such as point-in-time recovery cost constraints are combined to filter candidate sharding layouts that do not meet the requirements, retaining those that satisfy the constraints as sharding layouts for the next step of optimization.
[0070] Therefore, through multiple constraint screening, invalid candidate sharding layouts that do not meet the requirements of durability, fault tolerance and fault isolation are effectively eliminated, ensuring that the candidate sharding layouts meet the system requirements in terms of recovery chain integrity. At the same time, through the strategic selection of co-location layout and split layout, candidate solutions that take into account low recall cost and high parallel recall capability are provided for subsequent cost optimization, avoiding the risk of data unavailability or recovery chain breakage during recovery due to improper layout scheme.
[0071] Step S502: Based on the window-level recall cost corresponding to the fragmentation layout, select fragmentation layouts with window-level recall costs lower than a first preset threshold as fragmentation sets. Specifically, the window-level recall cost corresponding to each candidate shard layout that meets the constraints after screening in step S501 is obtained. Each window-level recall cost is compared with a first preset threshold, and candidate shard layouts with window-level recall costs lower than the first preset threshold are selected as shard sets. The window-level recall cost is used to characterize the overall cost required to recall the shard set that can recover the time point when the target recovery time falls into the recoverable time window. The first preset threshold is the maximum window-level recall cost configured in the tenant recall strategy. It can be configured according to the tenant service level agreement. Different service levels correspond to different thresholds. For example, the gold service level corresponds to a lower threshold (such as 0.45), and the archive service level corresponds to a higher threshold (such as 0.80). If there is no candidate shard layout with a window-level recall cost lower than the maximum window-level recall cost, an unsatisfiable state is output, and at least one of the following processes is triggered: regenerating candidate layouts, recalculating after reducing the load, re-layouting, or refusing to write to the shard layout index.
[0072] Therefore, by comparing the window-level recall cost with the maximum window-level recall cost, candidate sharding layouts that meet the recovery cost requirements can be effectively screened, ensuring that the overall recall cost of the selected sharding set during recovery is controllable, avoiding the recovery time exceeding the service level agreement requirements due to excessively high recall costs of the layout scheme, and improving the predictability of recovery time.
[0073] Step S503: Write the shard set into the shard layout index.
[0074] Specifically, the set of shards selected in step S502 is used as the target layout corresponding to the recoverable time window and written into the shard layout index. The shard layout index is stored in persistent storage medium, with the time window identifier as the key and the shard layout record as the value. The shard layout record includes at least each shard identifier, storage location, storage node identifier and corresponding window-level recall cost. The shard layout index is also used to locate the set of shards that meet the recovery chain constraints and cost threshold and their window-level recall cost based on the time window identifier when the target recovery time is received.
[0075] Therefore, by writing the shard set into the shard layout index, the layout decision results are made persistent and reusable, avoiding repeated layout optimization calculations during each recovery, thereby improving recovery efficiency and providing an efficient query basis for quickly locating and selecting the optimal shard set during subsequent actual recovery.
[0076] This embodiment combines multiple constraint filtering with cost threshold filtering, supplemented by persistent storage of the sharded layout index, to construct a layout decision scheme oriented towards recoverable time windows. It performs multiple filtering on candidate sharded layouts based on persistence, reconfigurability, fault domain isolation, recovery chain constraints, and future point-in-time recovery cost constraints, eliminating invalid candidate sharded layouts to ensure the layout meets requirements for recovery chain integrity. By comparing the window-level recall cost with the maximum window-level recall cost, it selects sharded layouts with controllable recovery costs. Furthermore, by writing the shard set into the sharded layout index, it achieves persistence and reusability of the layout decision results, avoiding recalculation during each recovery, thereby improving recovery efficiency and providing an efficient query basis for quickly locating and selecting the optimal shard set during subsequent actual recovery.
[0077] In one feasible implementation, after writing the shard set to the shard layout index, the method further includes steps S5031-S5033: Before lifecycle migration, tape rebalancing, fragment rewriting, or cold archive migration, simulate the window-level recall cost and reconfigurable state after the operation; if the simulation results cause the cost to exceed the threshold, the recovery chain constraint to be broken, or the window to be unreconfigurable, output a split, delay, rearrange, or rejection decision.
[0078] Step S5031: Obtain the recoverable time window affected by the lifecycle operation to be executed and the corresponding fragment layout in the fragment layout index; Specifically, the lifecycle pre-inspection module parses the description of the lifecycle operation to be executed, determines the range of storage nodes or media affected by the operation, and then determines a list of affected point-in-time recoverable time windows. Using the time window identifier as a query condition, the corresponding shard layout record is read from the shard layout index to obtain the shard layout information of each affected time window and the corresponding window-level recall cost as the baseline data before the operation. If there is no corresponding time window record in the shard layout index, the pre-inspection of that time window is skipped or an invalid window prompt is output.
[0079] Therefore, by parsing the lifecycle operation description to determine the affected time window range, and accurately reading the corresponding shard layout and pre-operation cost baseline from the shard layout index, an accurate data foundation is provided for subsequent simulation calculations, avoiding invalid pre-detection of irrelevant windows.
[0080] Step S5032: Calculate the pre-operation cost based on the sharding layout, simulate the execution of the lifecycle operation, and recalculate the post-operation cost, cost difference ratio, and reconfigurable state of the affected recoverable time window. Specifically, based on the acquired sharding layout, the pre-operation window-level recall cost for each affected time window is calculated as a baseline value, i.e., the pre-operation cost. Subsequently, the migration, rebalancing, or rewrite operations to be performed are mapped onto the cold media topology map. The storage node state, sharding distribution, and physical path changes after the operation are simulated. Based on the simulated topology state, the post-operation window-level recall cost for each affected time window is recalculated, i.e., the post-operation cost. The cost difference ratio is calculated based on the pre-operation cost and the post-operation cost: Cost difference ratio = (post-operation cost - pre-operation cost) / pre-operation cost. Simultaneously, the reconfigurable state after the simulated operation is checked to determine whether the number of available shards still meets the requirements of the erasure coding parameter k, and whether the recovery chain object can still be completely reconstructed. The simulation execution does not change the actual sharding layout; instead, it recalculates the post-operation cost and reconfigurable state based on the topology state, sharding distribution, and recovery chain constraints after the lifecycle operation.
[0081] Step S5033: Based on the pre-operation cost, post-operation cost, cost difference ratio, and reconfigurable state, perform a pre-detection result judgment according to a preset priority. If the cost after the operation exceeds the second preset threshold, then a pre-detection decision for splitting is output. If the cost difference ratio exceeds a preset ratio threshold, a delayed pre-detection decision is output. If the recovery chain-aware layout constraint is broken, a pre-detection decision for re-layout is output. If the recoverable time window is in an unreconfigurable state, a pre-detection decision to refuse execution is output.
[0082] Specifically, the lifecycle pre-inspection module judges four conditions sequentially according to preset priorities. When multiple conditions are met simultaneously, the judgment is made according to the priority of non-reconstructability, recovery chain constraint violation, cost difference ratio exceeding limit, and post-operation cost exceeding limit. First, it is judged whether the recoverable time window is in a non-reconstructable state. The non-reconstructable state is defined as one of the following four situations: the number of available shards is less than the erasure coding parameter k; the necessary write-ahead log segments, timeline history, backup list, or metadata objects cannot be reconstructed by any candidate shard set; all candidate shard sets fall into the same failure controller or the same unavailable driver pool; or the topology snapshot shows that the health of the involved media is lower than the preset health threshold and there are no alternative shards.
[0083] When a non-reconfigurable state is determined, a pre-check decision to refuse execution is output, preventing the lifecycle operation from proceeding. Next, it checks if the recovery chain-aware layout constraints have been violated. If, after the simulation operation, the basic backup and necessary write-ahead log segments are no longer within the same window's recall capability, or if timeline continuity and recovery visibility requirements cannot be met, a pre-check decision to re-layout is output, and the layout optimization module generates a new alternative layout that satisfies the constraints. Next, it checks if the cost difference ratio exceeds a preset threshold. If it does, a delayed pre-check decision is output, waiting for the load to decrease or conditions to improve before re-executing the pre-check. Finally, it checks if the post-operation cost exceeds a second preset threshold. If it does, a split pre-check decision is output, dividing the same time window into multiple sub-strips based on object dependencies for separate processing to distribute the cost increase.
[0084] If none of the above four checks trigger an anomaly, i.e. the window is still in a reconfigurable state after the operation, the recovery chain-aware layout constraints are not broken, the cost difference ratio does not exceed the preset ratio threshold, and the cost after the operation does not exceed the second preset threshold, then the lifecycle operation is determined to have passed the pre-check and is allowed to be executed, and the fragmented layout after the simulated operation is updated to the fragmented layout index.
[0085] Furthermore, both the aforementioned second preset threshold and preset ratio threshold can be configured according to the tenant service level agreement (SLA). The tenant SLA configuration can be recorded as a recall strategy and includes at least the tenant identifier, backup namespace, service level, target recovery time in minutes, maximum cost, maximum cost difference ratio, minimum number of reconfigurable sets, priority controller, priority parallel driver pool, and weight configuration. For example, the target recovery time in minutes for the Gold service level can be 240 minutes, the second preset threshold is 0.45, and the preset ratio threshold is 0.12; the target recovery time in minutes for the Silver service level can be 720 minutes, the second preset threshold is 0.65, and the preset ratio threshold is 0.20; and the target recovery time in minutes for the Archive service level can be 1440 minutes, the second preset threshold is 0.80, and the preset ratio threshold is 0.30.
[0086] Therefore, by judging four abnormal situations—non-reconfigurable state, restoration chain constraint violation, cost difference ratio exceeding limit, and post-operation cost exceeding limit—according to preset priorities, and outputting differentiated pre-inspection decisions such as refuse to execute, re-layout, delayed execution, and splitting, the system effectively avoids lifecycle operations from damaging the reconfigurability of the optimized layout and the integrity of the restoration chain, and ensures that the window-level recall cost is stable and controllable in the long term.
[0087] This embodiment uses lifecycle and rebalancing pre-checks to simulate window-level impacts before migration, rebalancing, or fragment rewriting, avoiding lifecycle operations from disrupting the reconfigurable state of the recoverable time window or significantly increasing recall costs.
[0088] The complete process of this embodiment can be referred to Figure 2 As shown. Figure 2 This is a schematic diagram of the overall process of an embodiment of the erasure coding data layout method of this application.
[0089] like Figure 2 As shown, the process includes steps S1 to S7 and has three core constraints: the input is the recovery time window packet as the erasure code stripe, the target is window-level recall cost rather than single-object latency, and the constraint is recovery chain integrity and reconfigurability. First, S1 obtains the recoverable time window and the time window archive packet and uses it as the erasure code stripe input. S2 generates recovery chain-aware layout constraints based on the recovery dependencies of various objects in the archive packet. S3 constructs a cold media topology map by combining multiple types of storage hardware resources. The cold media topology map includes libraries and controllers. S4 calculates the window-level recall cost and obtains candidate shard layouts. S5 filters shard layouts that meet the constraints and writes them to the index. S6 selects the set of shards with better cost based on the target time during actual recovery. S7 performs window-level pre-checks before lifecycle operations such as storage rebalancing and migration, and outputs a decision to allow, delay, or reject the reconfiguration based on the simulation results, forming a closed-loop processing flow.
[0090] like Figure 3 As shown, Figure 3 This diagram illustrates the relationship between the point-in-time recovery time window, the time window archive, and erasure coding fragments in this application. The recoverable time window for point-in-time recovery includes a recovery time interval and consists of a basic backup, necessary write-ahead log segments, timeline history, backup list, and metadata. The overall time window archive corresponding to this time window is used as the erasure coding stripe input. In the example, the stripe parameters k=6, m=3, and the stripe size is 768 GiB. After erasure coding processing, six data fragments (data 0-data 5) and three check fragments (check 0-check 2) are output. The window package can be reconstructed by selecting any six usable fragments that satisfy the recovery chain constraints. At the same time, the fragments are named and bound, and the binding content includes the namespace, time window identifier, archive identifier, stripe sequence number, and fragment sequence number to avoid misreading irrelevant window packages as ordinary objects or ordinary prefixes.
[0091] like Figure 4 As shown, Figure 4This is a schematic diagram of the cold media topology of the multiple cold archive libraries in this application. The left side shows tape library 1, which contains controller 1 and controller 2, corresponding to drive pool 1 and drive pool 2 respectively, and then connected to drive 1 and drive 2 respectively. Below are slots / media / barcodes and robotic arm paths. The right side shows cold archive library 2, which includes archive area 1, recovery task status and expected recall delay, lifecycle configuration and request quota. Topology snapshots collect timestamped library, controller, drive, media and load information, and generate edge weights based on this. Edge weights represent loading, moving, cross-controller reading and drive occupancy relationships. Layout optimization only uses valid snapshots. When a field is missing, a split, delay or rejection decision is triggered.
[0092] like Figure 5 As shown, Figure 5 The flowchart for selecting the shard set based on the target recovery time during the recovery process of this application is as follows: After inputting the target recovery time, the corresponding recoverable time window is first located. This process does not traverse irrelevant time window archives, but only selects within the current time window. Then, it is determined whether the necessary write-ahead logs and timelines are complete. If they are incomplete, the recovery fails, and it is determined that the necessary write-ahead logs in the window package are missing. If they are complete, the shard layout is obtained, the reconstructable shard set is enumerated, and then it is determined whether the recovery chain, topology, and cost thresholds are met. If they are met, the shard set with the lowest window-level recall cost is selected. If they are not met, a rollback selection is performed, replacing the fault controller shard or queuing the occupied driver, and the reconstructable shard set is re-enumerated.
[0093] like Figure 6 As shown, Figure 6 This is a flowchart for the window-level recall cost pre-check before migration or rebalancing in the application lifecycle. After inputting the proposed operation of migration, rebalancing, or rewriting, the affected recovery time window is located, and the topology and sharding status after the operation are simulated. Referring to the threshold examples of 12% for gold level, 20% for silver level, 30% for archiving, and a maximum of 8 candidate re-layouts, the cost before and after the operation and the difference are calculated. It is determined whether there are no less than N sets of reconfigurable states, and then the recovery chain constraints corresponding to the pre-write log and backup list are checked. Then, it is determined whether the service level threshold and recovery chain are met. Based on the judgment result, the processing results are output respectively: allow execution, delay and wait for load to decrease, re-layout and output alternative layout, and reject and record the rejection reason.
[0094] This embodiment addresses the challenges of high recall costs for tape / cold archive libraries during point-in-time recovery in relational database cold archiving scenarios. These challenges include susceptibility to storage layout disruptions in the recovery chain and the potential for window non-reconstruction during media rebalancing. The solution uses the recoverable time window for point-in-time database recovery as the basic control unit. The entire archive package within the time window is used as an erasure coding stripe input. This constructs a recovery chain-aware layout constraint and a cold media topology map containing various hardware resources. Erasure coding fragment layout decisions are made with window-level recall cost as the optimization objective. During the recovery phase, only the cost-optimized fragment set is selected within the target time window, avoiding the reading of irrelevant archive packages. Window-level pre-checks are performed before lifecycle operations such as migration, rebalancing, and rewriting to simulate and evaluate the fragment reconstructability and recall cost after the operation, outputting a decision to allow, delay, re-layout, or reject. In a hybrid storage environment with multiple cold archive libraries and tape libraries, the solution balances recovery chain integrity, fault domain isolation, and cold media physical recall overhead, ensuring the reliability and efficiency of point-in-time recovery services.
[0095] Compared with the prior art, the technical solution of this embodiment has the following advantages: By using the archived time window corresponding to the recoverable time window as the erasure coding stripe input, the erasure coding layout directly serves the relational database target time recovery chain, rather than ordinary object reads, thereby reducing the risk of reading irrelevant window packets or missing necessary write-ahead log segments during recovery.
[0096] By using recovery chain-aware layout constraints, the sharding layout of the base backup, write-ahead log segments, timeline history, backup list, and metadata objects is constrained, ensuring that the set of shards selected during recovery meets the requirements for database recovery chain integrity and reconfigurability.
[0097] By incorporating tape loading, robotic arm movement, controller affinity, driver contention, and parallel recall capabilities into layout calculations using cold media topology maps and window-level recall costs, the overall recall cost for future target time recovery can be reduced and recovery time predictability can be improved.
[0098] By performing lifecycle and rebalancing pre-checks, window-level impacts are simulated before migration, rebalancing, or fragmented rewriting, avoiding lifecycle operations from disrupting the reconfigurable state of recoverable time windows or significantly increasing recall costs.
[0099] Furthermore, to achieve the above objectives, this application also proposes an erasure coding data layout device, such as... Figure 7 As shown, the device includes: Input acquisition module 10 is used to acquire time window archive packages and input the time window archive packages as erasure code stripes; The erasure coding segmentation module 20 is used to segment the erasure coding stripe using an erasure coding algorithm to obtain data fragments and check fragments. The constraint generation module 30 is used to generate recovery chain-aware layout constraints based on the recovery dependencies between data objects in the time window archive package. The cost calculation module 40 is used to generate multiple candidate shard layouts for the data shards and check shards based on the recovery chain-aware layout constraints and a pre-constructed cold media topology map, and to calculate the window-level recall cost of each candidate shard layout when the target recovery time falls within the recoverable time window. Specifically, this includes: determining the candidate storage location set corresponding to each of the data shards and check shards based on the available storage nodes and available drivers in the cold media topology map; selecting storage location combinations that meet the recovery chain integrity requirements from the candidate storage location set in conjunction with the recovery chain-aware layout constraints to generate multiple candidate shard layouts; and substituting each candidate shard layout into a preset window-level recall cost function to calculate the window-level recall cost of each candidate shard layout when the target recovery time falls within the recoverable time window. The layout decision module 50 is used to select the candidate shard layouts that meet the preset constraints and have the lowest window-level recall cost from the candidate shard layouts as a shard set, and write them into the shard layout index.
[0100] The erasure coding data layout apparatus provided in this application, which adopts the erasure coding data layout method in the above embodiments, can reduce recall costs and improve the predictability of recovery time, thus meeting the requirements of database recovery chain integrity and reconfigurability.
[0101] Compared with the prior art, the erasure coding data layout device provided in this application has the same beneficial effects as the erasure coding data layout method provided in the above embodiments, and other technical features in the erasure coding data layout device are the same as the features disclosed in the above embodiments, and will not be repeated here.
[0102] Furthermore, to achieve the above objectives, this application also proposes an erasure coding data layout apparatus, the apparatus comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the erasure coding data layout method as described above.
[0103] The following is for reference. Figure 8The diagram illustrates a structural schematic suitable for implementing erasure coding data layout devices according to embodiments of this application. The erasure coding data layout devices in embodiments of this application may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 8 The erasure coding data layout device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.
[0104] like Figure 8 As shown, the erasure coding data layout device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the erasure coding data layout device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows erasure coding data layout equipment to communicate wirelessly or wiredly with other devices to exchange data. Although erasure coding data layout equipment with various systems is shown in the figures, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.
[0105] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.
[0106] The erasure coding data layout device provided in this application, employing the erasure coding data layout method in the above embodiments, can reduce recall costs and improve the predictability of recovery time, meeting the requirements of database recovery chain integrity and reconfigurability. Compared with the prior art, the beneficial effects of the erasure coding data layout device provided in this application are the same as those of the erasure coding data layout method provided in the above embodiments, and other technical features in this erasure coding data layout device are the same as those disclosed in the previous embodiment method, and will not be repeated here.
[0107] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.
[0108] In addition, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the erasure coding data layout method as described above.
[0109] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.
[0110] The aforementioned computer-readable storage medium may be included in the erasure coding data layout device; or it may exist independently and not be assembled into the erasure coding data layout device.
[0111] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by the erasure coding data layout device, the erasure coding data layout device: obtains a time window archive as an erasure coding stripe input and segments it using an erasure coding algorithm to obtain data fragments and verification fragments; generates recovery chain-aware layout constraints based on the recovery dependencies between data objects in the time window archive; generates candidate fragment layouts based on the recovery chain-aware layout constraints and a pre-constructed cold media topology map, and calculates the window-level recall cost of each candidate fragment layout when the target recovery time falls within the recoverable time window; selects the candidate fragment layout that satisfies the preset constraints and has the lowest window-level recall cost from the candidate fragment layouts as a fragment set, and writes it into the fragment layout index. Therefore, using time window archives as erasure code stripe input avoids reading irrelevant window packages or missing necessary write-ahead log segments during recovery; generating recovery chain-aware layout constraints ensures that the shard set selected during recovery meets the requirements of database recovery chain integrity and reconfigurability; using cold media topology graphs and window-level recall costs reduces the overall recall cost during recovery and improves the predictability of recovery time; and writing the shard set that meets the constraints and has the lowest recall cost into the shard layout index achieves the persistence and reusability of layout decision results.
[0112] Computer program code for performing the operations of this application can be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, and C++, and conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0113] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0114] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.
[0115] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the erasure coding data layout method described above. This can reduce recall costs and improve the predictability of recovery time, meeting the requirements for database recovery chain integrity and reconfigurability. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the erasure coding data layout method provided in the above embodiments, and will not be repeated here.
[0116] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the erasure coding data layout method as described above.
[0117] The computer program product provided in this application can reduce recall costs and improve the predictability of recovery time, meeting the requirements for database recovery chain integrity and reconfigurability. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the erasure coding data layout method provided in the above embodiments, and will not be repeated here.
[0118] The one or more technical solutions proposed in this application have at least the following technical effects: by obtaining the time window archive package as the erasure coding stripe input and segmenting it through the erasure coding algorithm, data fragments and verification fragments are obtained; based on the recovery dependency relationship between each data object in the time window archive package, recovery chain-aware layout constraints are generated; based on the recovery chain-aware layout constraints and the pre-constructed cold media topology map, candidate fragment layouts are generated, and the window-level recall cost of each candidate fragment layout when the target recovery time falls within the recoverable time window is calculated; from the candidate fragment layouts, the candidate fragment layouts that meet the preset constraints and have the lowest window-level recall cost are selected as fragment sets and written into the fragment layout index. Therefore, using time window archives as erasure code stripe input avoids reading irrelevant window packages or missing necessary write-ahead log segments during recovery; generating recovery chain-aware layout constraints ensures that the shard set selected during recovery meets the requirements of database recovery chain integrity and reconfigurability; using cold media topology graphs and window-level recall costs reduces the overall recall cost during recovery and improves the predictability of recovery time; and writing the shard set that meets the constraints and has the lowest recall cost into the shard layout index achieves the persistence and reusability of layout decision results.
[0119] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.
Claims
1. A method for erasure coding data layout, characterized in that, The method includes: Obtain the time window archive package and use the time window archive package as an erasure coding stripe; The erasure code stripe is segmented using an erasure code encoding algorithm to obtain data fragments and check fragments. Based on the recovery dependencies between data objects in the time window archive, a recovery chain-aware layout constraint is generated. Based on the recovery chain-aware layout constraints and the pre-constructed cold media topology, candidate shard layouts are generated for the data shards and check shards, and the window-level recall cost of each candidate shard layout when the target recovery time falls within the recoverable time window is calculated. Specifically, this includes: determining the candidate storage location set corresponding to each of the data shards and check shards based on the available storage nodes and available drivers in the cold media topology; selecting storage location combinations that meet the recovery chain integrity requirements from the candidate storage location set in conjunction with the recovery chain-aware layout constraints to generate multiple candidate shard layouts; and substituting each candidate shard layout into a preset window-level recall cost function to calculate the window-level recall cost of each candidate shard layout when the target recovery time falls within the recoverable time window. From the candidate sharding layouts, select the candidate sharding layouts that meet the preset constraints and have the lowest window-level recall cost as a sharding set, and write them into the sharding layout index; Obtain the recoverable time window affected by the lifecycle operation to be executed and the corresponding fragment layout in the fragment layout index; Calculate the pre-operation cost based on the fragmented layout, and after simulating the execution of the lifecycle operation, recalculate the post-operation cost, cost difference ratio, and reconfigurable state of the affected recoverable time window. Based on the pre-operation cost, post-operation cost, cost difference ratio, and reconfigurable state, the pre-inspection results are judged according to a preset priority. If the cost after the operation exceeds the second preset threshold, then a pre-detection decision for splitting is output. If the cost difference ratio exceeds a preset ratio threshold, a delayed pre-detection decision is output. If the recovery chain-aware layout constraint is broken, a pre-detection decision for re-layout is output. If the recoverable time window is in an unreconfigurable state, a pre-detection decision to refuse execution is output.
2. The method as described in claim 1, characterized in that, The step of dividing the erasure code stripe into erasure code segments using an erasure code encoding algorithm to obtain data fragments and parity fragments includes: The erasure coding algorithm uses preset erasure coding parameters to segment the erasure code stripe, generating data fragments and check fragments.
3. The method as described in claim 1, characterized in that, The step of generating recovery chain-aware layout constraints based on the recovery dependencies between data objects in the time window archive includes: Based on the recovery dependencies of at least one of the following in the time window archive: basic backup, necessary write-ahead log segments, timeline history, backup list, and metadata object, generate recovery chain-aware layout constraints.
4. The method as described in claim 1, characterized in that, Before generating candidate shard layouts for the data shards and verification shards based on the recovery chain-aware layout constraints and the pre-constructed cold media topology, and calculating the window-level recall cost of each candidate shard layout when the target recovery time falls within the recoverable time window, the method further includes: Construct a cold media topology based on at least one of the following: multiple tape libraries, cold archive libraries, controllers, drive pools, drives, slots, robotic arm paths, media health, and bandwidth windows.
5. The method as described in claim 1, characterized in that, The step of selecting candidate shard layouts from the candidate shard layouts that meet preset constraints and have the lowest window-level recall cost as a shard set, and writing them into the shard layout index, includes: Select a candidate fragmentation layout from the candidate fragmentation layouts that satisfies one or more of the following constraints: durability, reconfigurability, cold media fault domain isolation, recovery chain constraint, and future point-in-time recovery cost constraint. Based on the window-level recall cost corresponding to the fragmentation layout, select fragmentation layouts with window-level recall costs lower than a first preset threshold as fragmentation sets; Write the shard set into the shard layout index.
6. An erasure coding data layout device, characterized in that, The device includes: The input acquisition module is used to acquire the time window archive package and input the time window archive package as the erasure code stripe; The erasure coding segmentation module is used to segment the erasure coding stripe using an erasure coding algorithm to obtain data fragments and check fragments. The constraint generation module is used to generate recovery chain-aware layout constraints based on the recovery dependencies between data objects in the time window archive. The cost calculation module is used to generate multiple candidate shard layouts for the data shards and check shards based on the recovery chain-aware layout constraints and a pre-constructed cold media topology map, and to calculate the window-level recall cost of each candidate shard layout when the target recovery time falls within the recoverable time window. Specifically, this includes: determining the candidate storage location set corresponding to each of the data shards and check shards based on the available storage nodes and available drivers in the cold media topology map; selecting storage location combinations that meet the recovery chain integrity requirements from the candidate storage location set in conjunction with the recovery chain-aware layout constraints to generate multiple candidate shard layouts; and substituting each candidate shard layout into a preset window-level recall cost function to calculate the window-level recall cost of each candidate shard layout when the target recovery time falls within the recoverable time window. The layout decision module is used to select the candidate shard layouts that meet preset constraints and have the lowest window-level recall cost from the candidate shard layouts as a shard set, and write them into the shard layout index; obtain the recoverable time window affected by the lifecycle operation to be executed and the corresponding shard layout in the shard layout index; calculate the pre-operation cost based on the shard layout, simulate the execution of the lifecycle operation, and recalculate the post-operation cost, cost difference ratio, and reconfigurable state of the affected recoverable time window; judge the pre-detection result according to the pre-operation cost, post-operation cost, cost difference ratio, and reconfigurable state according to preset priority; if the post-operation cost exceeds a second preset threshold, output a splitting pre-detection decision; if the cost difference ratio exceeds a preset ratio threshold, output a delayed pre-detection decision; if the recovery chain-aware layout constraint is violated, output a re-layout pre-detection decision; if the recoverable time window is in an unreconfigurable state, output a rejection pre-detection decision.
7. An erasure coding data layout device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the erasure coding data layout method as described in any one of claims 1 to 5.
8. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the erasure coding data layout method as described in any one of claims 1 to 5.
Citation Information
Patent Citations
Fault data recovery method and device, computer equipment and storage medium
CN120104054A
Erasure code-based fault-tolerant optimization method and system in distributed storage
CN120780534A
