A Disaster Recovery Method and System for Archival Data Based on Distributed Technology

Abstract

This application discloses a method and system for archival data disaster recovery based on distributed technology, relating to the field of data disaster recovery technology. The method includes: encoding the original archival data object to obtain K data fragments and M verification fragments, which are then distributed and stored in a cluster; real-time monitoring of the storage node status, and when a fault is detected, determining the set of fragments lost due to the fault, denoted as the set of fragments to be repaired; calculating the heat value of each fragment to be repaired, and predicting the availability of repair resources for multiple future time intervals, generating a repair task queue based on the heat value and the availability of repair resources; selecting a subset of surviving fragments for repair by calculating their network pressure index, and generating repair fragments through linear encoding operations; performing hash verification, and storing the verified fragments in healthy storage nodes. This application aims to solve the problems of large impact on business performance, high latency in critical data recovery, and severe network traffic redundancy during data repair, effectively suppressing repair storms.

Description

Technical Field

[0001] This invention relates to the field of data disaster recovery technology, and in particular to a method and system for archival data disaster recovery based on distributed technology. Background Technology

[0002] With the deepening of digital transformation, various organizations have generated massive amounts of archival data (such as electronic documents, audio and video materials, and database archives) that need to be preserved long-term. Disaster recovery systems based on distributed technology have become the preferred solution for storing this type of data due to their high scalability and high reliability. Such systems typically use erasure coding technology to encode archival data into blocks and then distribute them across multiple geographically distributed nodes.

[0003] However, when a large storage node or an entire rack fails in the system, a data reconstruction (repair) process must be initiated. Traditional methods treat all lost data blocks equally, immediately initiating a full read and reconstruction. This process instantly consumes a large amount of network bandwidth and computing resources, triggering a "repair storm," leading to a severe decline in front-end business performance or even service interruption, creating a risky chain reaction where "repairing one failure may induce a second failure." For archival data with low access frequency but extremely large data volume and relatively lenient recovery window requirements, this "stormy" repair approach is neither economical nor necessary.

[0004] Current technologies lack differentiated and intelligent repair scheduling mechanisms that address the hot / cold nature of archival data and business tolerance, and also fail to fully optimize the data throughput of the reconstruction process itself. Therefore, a new disaster recovery method that can smoothly handle repair traffic and ensure business continuity is urgently needed. Summary of the Invention

[0005] This invention provides a method for disaster recovery of archival data based on distributed technology, comprising:

[0006] The original archive data object is encoded using erasure coding to obtain K data fragments and M verification fragments. The K+M fragments are distributed and stored in storage nodes located in different fault domains in the cluster, and the metadata of the fragment location mapping relationship is recorded.

[0007] Real-time monitoring of storage node status; when a node or rack failure is detected, the set of shards lost due to the failure is determined based on the metadata and recorded as the set of shards to be repaired, and real-time load data of each node is obtained.

[0008] Based on the original archive metadata, calculate the heat value of each fragment to be repaired in the fragment set to be repaired, and predict the amount of repair resources available in multiple future time intervals based on real-time load data. Generate a repair task queue based on the heat value and the amount of repair resources available.

[0009] According to the repair task queue, a subset of surviving fragments for repair is selected from the surviving fragments that have the same erasure code as each fragment to be repaired by calculating their network pressure index, and the data blocks in the subset are read and repaired fragments are generated by linear coding operation corresponding to the erasure code. The size of the surviving fragment subset is equal to the minimum number of fragments required for the erasure code reconstruction.

[0010] The regenerated repair shards are hash-verified. Once the verification is successful, they are stored in healthy storage nodes, and the metadata is updated.

[0011] The aforementioned archive data disaster recovery method based on distributed technology calculates the heat value of each fragment to be repaired in the fragment set according to the original archive metadata, predicts the availability of repair resources for multiple future time intervals based on real-time load data, and generates a repair task queue based on the heat value and the availability of repair resources, including:

[0012] For each shard to be repaired, a heat value representing its urgency is calculated based on the access history and business attributes of its nodes.

[0013] Based on the current real-time load data of each node, the available repair resources for multiple future time intervals are predicted using time series forecasting methods.

[0014] Based on the heat value and the availability of repair resources, high-resolution fragments are preferentially allocated to the nearest repair window with sufficient resources, generating a repair task queue with smooth temporal distribution and controlled load.

[0015] The aforementioned method for disaster recovery of archive data based on distributed technology involves selecting a subset of surviving fragments for repair from among the surviving fragments that share the same erasure code as each fragment to be repaired, based on a repair task queue, by calculating their network pressure index, and then reading data blocks from these fragments and generating repair fragments through linear coding operations corresponding to the erasure codes. The method includes:

[0016] From the currently surviving fragments that belong to the same erasure coding group as the fragment to be repaired, calculate the network pressure index based on the network topology distance and node load, and select a subset of surviving fragments for repair;

[0017] Data blocks are read from each node of the surviving shard subset, and repair shards are generated by a dedicated repair computing node through linear coding operations corresponding to the erasure code on the data flow path to the target new node.

[0018] The aforementioned method for disaster recovery of archival data based on distributed technology selects a subset of surviving fragments for repair from other currently surviving fragments belonging to the same erasure coding group as the fragment to be repaired, based on network topology distance and node load to calculate the network pressure index, including:

[0019] Identify all currently surviving fragments belonging to the same (K, M) erasure coding group as the fragment to be repaired, and calculate all possible combinations of surviving fragments of size K;

[0020] Predict the network stress index that each combination will experience during the repair process;

[0021] Select the combination with the smallest total amount of data that needs to be transferred across racks as the selected live shard subset.

[0022] The aforementioned archival data disaster recovery method based on distributed technology reads data blocks from each node of the surviving shard subset, and generates repair shards by a dedicated repair computing node on the data flow path to the target new node through linear coding operations corresponding to the erasure coding, including:

[0023] Send a unique coding coefficient vector to each of the selected K surviving nodes;

[0024] Each surviving node uses the coefficient vector locally to perform an inner product calculation with the data fragments stored on the node to generate intermediate encoded blocks;

[0025] Each node streams intermediate coded blocks to the target node, which then adds the received K intermediate coded blocks element by element to obtain the regenerated lost fragments.

[0026] An archival data disaster recovery system based on distributed technology includes:

[0027] The data fragmentation encoding module is used to encode the original archive data object using erasure coding to obtain K data fragments and M verification fragments. The K+M fragments are distributed and stored in storage nodes located in different fault domains in the cluster, and the metadata of the fragment location mapping relationship is recorded.

[0028] The fault detection module is used to monitor the status of storage nodes in real time. When a node or rack fault is detected, the set of shards lost due to the fault is determined based on the metadata and recorded as the set of shards to be repaired. The module also obtains the real-time load data of each node.

[0029] The repair scheduling and planning module is used to calculate the heat value of each fragment to be repaired in the fragment set based on the original archive metadata, predict the availability of repair resources for multiple time intervals in the future based on real-time load data, and generate a repair task queue based on the heat value and the availability of repair resources.

[0030] The source selection parallel regeneration module is used to select a subset of surviving fragments for repair from the surviving fragments that have the same erasure code as each fragment to be repaired by calculating their network pressure index according to the repair task queue, and read the data blocks therein to generate repair fragments through linear coding operation corresponding to the erasure code. The size of the surviving fragment subset is equal to the minimum number of fragments required for the erasure code reconstruction.

[0031] The verification and update module is used to perform hash verification on the regenerated repair shards. After the verification is successful, the shards are stored in healthy storage nodes and the metadata is updated.

[0032] The beneficial effects achieved by this invention are as follows:

[0033] By using dynamic priority scheduling, the concentrated repair pressure is smoothly distributed across multiple time windows, preventing network and computing resources from being exhausted instantly and effectively suppressing repair storms. This significantly reduces the amount of data transfer required for individual repair tasks, improving repair speed and overall system efficiency. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0035] Figure 1 This is a flowchart of a disaster recovery method for archive data based on distributed technology, provided in Embodiment 1 of this application.

[0036] Figure 2 This is a schematic diagram of an archive data disaster recovery system based on distributed technology provided in Embodiment 2 of this application. Detailed Implementation

[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] Example 1

[0039] like Figure 1 As shown, Embodiment 1 of this application provides a method for disaster recovery of archival data based on distributed technology, including:

[0040] S1: Encode the original archive data object using erasure coding to obtain K data fragments and M verification fragments. Distribute the K+M fragments in storage nodes located in different fault domains in the cluster and record the metadata of the fragment location mapping relationship.

[0041] First, erasure coding technology transforms the original archive data object into a fault-tolerant distributed storage format. Specifically, after receiving the original archive data object, it is divided into data blocks according to a preset fixed block size (e.g., 1MB or 4MB). Then, based on the storage pool's reliability strategy, the erasure coding parameters (K, M) are determined. Here, K represents the number of data fragments, and M represents the number of parity fragments. This parameter determines the fault tolerance of the coding group, allowing the simultaneous loss of any M fragments without rendering the data unusable.

[0042] Next, an integrated erasure coding library (e.g., based on Reed-Solomon or LRC) is invoked, taking the preprocessed data block as input. Specific linear algebra operations (matrix multiplication over a finite field) are performed to generate K data slices and M parity slices. Each generated slice is assigned a unique identifier within the system.

[0043] Then, based on the cluster's physical topology, independent fault domain units are identified, with physical racks or data center availability zones as the granularity. A sharding placement algorithm is executed to complete the transmission and writing of shards. Specifically, the input includes the cluster topology containing the coding group to be stored (K+M shards), the fault domain identifiers of the tagged nodes, and the real-time storage load information of the nodes. Initialization involves creating an empty sharding placement plan and initializing the fault domain usage counters specific to the coding group. Subsequently, the K+M shards are randomly sorted to avoid distribution bias. Next, healthy nodes whose load does not exceed the safety threshold are selected one by one to form an initial candidate set. Then, the fault domain with the fewest shards placed in the current coding group is selected first using the fault domain usage counter, and candidate nodes within the domain are selected. The node with the lowest storage space utilization or the fewest existing shards within the domain is selected as the target node. Finally, the sharding placement plan, the corresponding fault domain counters, and the node load information are updated. After all sharding decisions are completed, the sharding placement plan is output, and the transmission and writing of shards are completed in parallel accordingly.

[0044] Finally, the mapping relationship for this storage operation is created and persisted in a separate metadata service cluster or distributed database. The generated metadata includes at least:

[0045] Object-level metadata records the globally unique ID, size, hash value, etc. of the original file object; shard-level mapping table records the precise correspondence between each shard (identified by shard ID) and its physical storage location (storage node address, fault domain ID, local path), and associates it with its corresponding object ID. It also records the integrity checksum of each shard (such as CRC32 or SHA-256 hash value).

[0046] For example, for a 4GB archive file using (16, 6) erasure coding, the system divides and encodes it into 16 data fragments and 6 parity fragments. These 22 fragments are then distributed and stored across at least 7 nodes (in practice, this is typically equal to or greater than M+1) different physical racks. The metadata server precisely records all 22 mapping relationships, such as "fragment D1 is stored on node A of rack Rack01". In this way, the loss of any subsequent 6 fragments can be tolerated and recovered.

[0047] S2: Monitor the status of storage nodes in real time. When a node or rack failure is detected, determine the set of shards lost due to the failure based on the metadata, record it as the set of shards to be repaired, and obtain the real-time load data of each node.

[0048] A lightweight agent is deployed on each storage node and network switch to report its health status to a centralized cluster monitoring service at fixed heartbeat intervals (e.g., every second). The reported data includes: node liveness signals (confirming node processes are running via heartbeat packets), hardware status indicators (CPU temperature, disk SMART status, memory ECC error count, etc.), and service reachability (simulating client requests to verify that the storage service ports on the nodes are read and written correctly). Simultaneously, the monitoring service detects rack-level faults (such as switch failures or rack-wide disconnections due to power outages) by inspecting the link status of the top-of-rack switches.

[0049] The following situations are considered valid fault events and trigger the repair process immediately:

[0050] Node-level failure: A storage node loses heartbeats more than a preset threshold number of times (e.g., 3 times) consecutively, or its service port remains unreachable.

[0051] Rack-level fault: All nodes in a rack are simultaneously disconnected, and the link status of its top-level switch is abnormal.

[0052] Once a fault is confirmed, a structured fault event notification is immediately generated, which includes at least: fault type (node / rack), fault entity identifier (node ​​ID / rack ID), and fault occurrence timestamp. The metadata service is then accessed to perform a reverse query: "Find all shard records in the storage location field whose node or fault domain matches the fault entity identifier." All shard identifiers in the query results are then aggregated to generate a set of shards to be repaired. This set precisely corresponds to all data shards and checksum shards that have become inaccessible due to this fault.

[0053] At the same time, at the moment the fault is triggered, a real-time load data snapshot of all currently surviving nodes is requested from the cluster monitoring service, including: node identifier, resource utilization, capacity information, etc.

[0054] For example, the monitoring system detected that node "Node_58" located in rack "Rack05" had consecutively lost heartbeats, indicating a node-level failure. A metadata query revealed that three different encoding groups each had one shard (shard IDs: `FID_A123`, `FID_B456`, and `FID_C789`) stored on this node. These three shards were then added to the set of shards to be repaired. Simultaneously, the average CPU utilization of the remaining 95 nodes in the cluster was observed to be 45%, and the average network bandwidth utilization was 60%.

[0055] S3: Based on the original archive metadata, calculate the heat value of each fragment to be repaired in the fragment set to be repaired, and predict the amount of repair resources available in multiple future time intervals based on real-time load data. Generate a repair task queue based on the heat value and the amount of repair resources available.

[0056] The process includes calculating the heat value of each fragment to be repaired in the fragment set based on the original archive metadata, predicting the availability of repair resources for multiple future time intervals based on real-time load data, and generating a repair task queue based on the heat value and the availability of repair resources. This process includes the following sub-steps:

[0057] S31: For each shard to be repaired, calculate the heat value representing its urgency based on the access history and business attributes of its nodes.

[0058] The heat value of each fragment in the set of fragments to be repaired is calculated to characterize its urgency in the current business context. This calculation is based on dynamic data access characteristics and static business attributes. Specifically, for each target fragment in the set of fragments to be repaired, data retrieval is performed. By querying metadata, based on the fragment's identifier, the globally unique identifier of the original archive data object to which it belongs is traced back and determined. Then, using the identifier as the key, the cluster's distributed access log database is queried to retrieve the total number of successfully responded read or write requests for that archive object within a configurable preset historical period.

[0059] Next, using the identifier as the key, the system queries the preset business rules database or configuration file to obtain the static attribute weights (archive security level, retention period, department or project priority, etc.) pre-configured for the archive object. Then, it obtains the highest number of accesses for all archive objects within the same preset historical period.

[0060] The historical access frequencies are compared with the highest access frequencies of all archive objects within the same statistical period. Through normalization, an access frequency factor between 0 and 1 is obtained. This factor reflects the relative activity level of the object in the recent period.

[0061] Based on the various business attributes of the retrieved archive objects, the corresponding weight scores are retrieved from a pre-defined attribute weight mapping table. A weighted average method or a maximum value method is used to combine the weight scores of multiple attributes into a business attribute factor. Finally, the final heat value is calculated according to the heat value calculation formula. The specific formula is as follows:

[0062]

[0063] Where R represents the final thermal value; Indicates the popularity factor of visits; Indicates business attribute factors; Indicates the weighting coefficient; Indicates the length of the statistical time window; This represents the time index, indicating the dth day (or dth time slice) within the statistics window. This represents the original number of visits on day d; This represents the time decay weight on day d, indicating that the closer the access is to the current time, the higher its value. This represents the total visit value after time weighting, highlighting recent visit trends. This represents the total number of global system calls on day d, which refers to the total number of file access requests processed by the entire storage cluster on day d. It represents the basic access behavior weight, which is a configurable coefficient used to distinguish the value differences of different access types (such as read and write) or access sources (such as internal access and external API calls); This indicates a time-specific adjustment factor; for example, a magnification factor set for such time periods, taking into account business cycles (such as more frequent access to specific files during month-end or quarter-end financial settlement periods). It represents a non-linear scaling up / down index, which is a non-linear adjustment to the basic access popularity, and more finely depicts the value curve of access behavior; It represents a non-linear scaling index, controlling the non-linear scaling of the internal value of business attributes;

[0064] Indicates the total number of business attributes (such as security level, time limit, etc.); Represents the i-th or j-th business attribute; This represents the original score of the i-th business attribute, which is the base score obtained by converting the specific value of the attribute (such as "top secret" or "permanent") according to the preset mapping table. This represents the weight of the i-th business attribute; Indicates the basic weighted business value; This represents the enhancement / weakening value of the association between attribute i and attribute j, indicating the additional value correction generated when both attributes occur simultaneously. For example, when "Security Classification = Top Secret" and "Retention Period = Permanent" occur simultaneously, their value may far exceed the simple sum of their individual values. A positive value indicates a positive reinforcement effect; conversely, certain attribute combinations may result in value cancellation. It is a negative number; This represents the total correlation adjustment value, which is the sum of the correlation values ​​between all possible pairwise combinations of attributes, reflecting the contribution of the interaction between attributes to the overall value; This represents the revised total business value; The normalization coefficient represents the correlation effect and is used to reconcile the dimensions of the correlation correction value with the basic weighted sum to prevent the correlation correction term from excessively affecting the result. This represents the time decay coefficient, used to dynamically adjust the strength of the correlation effect; The standardized baseline value is composed of the total attribute weights and the adjusted total correlation correction value. It is used to standardize the corrected total business value, stabilizing the result within a reasonable range (usually fluctuating around 1), and avoiding numerical explosion due to an excessive number of attributes or correlation items. This represents a nonlinear adjustment index, which plays a role in compressing the value range and smoothing differences, enabling the system to finely control the distribution of business value.

[0065] S32: Based on the current real-time load data of each node, use time series forecasting methods to predict the availability of repair resources for multiple future time intervals;

[0066] Based on real-time load data from nodes, real-time performance metrics of all surviving storage nodes are retrieved at fixed sampling intervals (e.g., every second or every 5 seconds) to form power time-series data. Key metrics include: computing resource metrics (e.g., average CPU utilization of each node), network resource metrics (e.g., ingress / egress bandwidth utilization of each node's service network interface card), and storage I / O metrics (e.g., I / O wait queue length or utilization of each node's data disk). After cleaning (e.g., removing abnormal spikes) the power time-series data, it is aggregated by time window (e.g., by taking the average value) to form a clean time-series dataset for predictive analysis.

[0067] Next, a resource availability model is established for each type of critical resource (CPU, network bandwidth, disk I / O) to dynamically generate an accurate repair resource budget. Specifically, at the node level, a time series forecasting algorithm (such as Holt-Winters exponential smoothing) is used to independently predict the future CPU, network, and disk I / O utilization of each node based on its historical performance data. At the resource level, the predicted values ​​of the same type of resource at the node level are aggregated by time interval to obtain a cluster-level resource load prediction, and a dynamic safety threshold mechanism is introduced—this threshold is dynamically adjusted according to the business cycle pattern and prediction confidence—by calculating available quantity = max(0, (safety threshold - predicted load) × total resources), the theoretical available quantity of each type of resource is obtained. At the cluster level, based on the preset repair task resource consumption template (indicating the standard consumption of CPU, network, and I / O per unit task), the barrel principle is used for multi-resource coordination: the available quantity of each type of resource is converted into the number of equivalent tasks that can be supported, and the minimum value is taken as the final schedulable repair resource budget for that time interval. It is updated periodically in a rolling manner, outputting a standardized sequence of available repair resources for a future series of time windows.

[0068] S33: Based on the heat value and the available repair resources, high-segment fragments are preferentially allocated to the nearest repair window with sufficient resources, generating a repair task queue with smooth temporal distribution and controlled load.

[0069] A multi-constraint greedy matching algorithm is executed to generate a repair task queue. Specifically, the input includes a list of shards to be repaired, sorted in descending order of heat value, and a sequence of available repair resources representing the available repair capacity for each future time window. For each high-level shard, future time windows are scanned sequentially, and the shard is assigned to the earliest window where the remaining available resources are sufficient to cover the estimated repair cost of that shard. The available resources for that window are deducted in real-time after assignment. If the available resources for all existing windows are insufficient, the shard is temporarily suspended, and a rolling update of resource predictions is triggered to find a more distant feasible window. After the initial queue is generated, load balancing is fine-tuned to make the task distribution on the timeline smoother. The final output is a detailed execution plan ordered by time windows, with each window containing a specific list of tasks. This process runs dynamically; when a new fault occurs or resource usage deviates from the prediction, the queue is rescheduled, ensuring that repair tasks are executed in an orderly and smooth manner according to business urgency under strictly controlled resource constraints.

[0070] S4: Based on the repair task queue, select a subset of surviving fragments for repair from the surviving fragments that have the same erasure code as each fragment to be repaired by calculating their network pressure index, and read the data blocks therein to generate repair fragments through linear coding operations corresponding to the erasure code. The size of the surviving fragment subset is equal to the minimum number of fragments required for the erasure code reconstruction.

[0071] The process involves selecting a subset of surviving fragments for repair from among the surviving fragments that share the same erasure code as each fragment to be repaired, based on the repair task queue and calculating their network pressure index. Data blocks from this subset are then read and processed using linear coding operations corresponding to the erasure code to generate repair fragments. The size of this subset of surviving fragments is equal to the minimum number of fragments required for erasure code reconstruction. This process includes the following sub-steps:

[0072] S41: From the currently surviving fragments that belong to the same erasure coding group as the fragment to be repaired, calculate the network pressure index based on the network topology distance and node load, and select a subset of surviving fragments for repair;

[0073] The process of selecting a subset of surviving fragments for repair from other currently surviving fragments belonging to the same erasure coding group as the fragment to be repaired, based on network topology distance and node load, and calculating the network pressure index, includes the following sub-steps:

[0074] S411: Determine all currently surviving fragments belonging to the same (K,M) erasure coding group as the fragment to be repaired, and calculate all possible combinations of surviving fragments of size K;

[0075] Based on the aforementioned repair task queue, the complete coding group information to which the target shard belongs is first determined by querying metadata. Then, combined with real-time fault status, all accessible live shards are filtered from this coding group, and their number is verified to meet the minimum value (K) required for erasure coding reconstruction. Simultaneously, each live shard is associated with the network topology identifier (e.g., rack ID) of its host node. The core operation is to perform combinatorial mathematical calculations, exhaustively enumerating all possible combinations of randomly selecting K shards from the live shard set, and generating a key feature vector for each combination containing its shard list, rack distribution, and optional node load snapshots. Finally, a structured list of candidate combinations is output.

[0076] S412: Predict the network stress index that each combination will experience during the repair process;

[0077] Based on the candidate combination list It contains M candidate combinations. Each combination contains K surviving shard identifiers and their associated rack IDs and node IDs. The regenerated shards are stored in the new node and its associated rack. The bandwidth utilization data of each node's network ports at the current moment is based on the real-time network load data of the nodes. For each candidate combination... Calculate its network stress index The calculation formula is as follows:

[0078]

[0079] This represents the weighting coefficient, reflecting the degree of importance attached to the repair completion time; It is a combination of all candidate repair sources; This represents the effective amount of data that needs to be transmitted from a single source node (considering the size of the data block required for erasure coding regeneration). From the source rack To the target rack The expected available bandwidth between; ,in, This refers to the theoretical bandwidth of the link. For this critical link ( arrive Historical average utilization of bottleneck links on the path (reflecting long-term load); The current business cycle factor (e.g., 0.7 for weekday peak hours and 1.0 for nighttime) is used to estimate future bandwidth availability. Indicates the source node Instantaneous load factor, It combines CPU and network queue length; among them, Source node The instantaneous CPU utilization percentage at the current sampling moment; Preset CPU utilization trigger threshold; when When the ratio is less than 1, it indicates that the node has sufficient computing power; when When the ratio is ≥1, it indicates that the node has entered an "overload" state; Source node The average length of the network interface sending queue is an indicator reflecting the network egress congestion status. It is the nonlinear amplification index of the load effect ( This indicates that the impact of high-load nodes on transmission performance deteriorates exponentially;

[0080] This represents the weighting coefficient, reflecting the degree of importance attached to "avoiding network congestion"; It is a combination The set of all shared critical links (such as the uplink of the core switch) that the data transmission path from all source nodes to the target nodes passes through. This means that the link with the largest potential increase in utilization rate among all critical links is taken as the representative of risk. The reaction calculation determines how much data will pass through this specific critical link if this combination is selected. ; This indicates that for a given source node and repair target node , Returning to the current network topology state, data from Transmit to An ordered set of physical links traversed; This represents the effective amount of data that needs to be transmitted from a single source node (considering the size of the data block required for erasure coding regeneration). This indicates the critical link The remaining safety capacity, ,in To prevent congestion, a safety threshold is required. This represents the current utilization rate of the link. Refers to critical network links The theoretical total bandwidth capacity.

[0081] The calculated As a network stress index for this combination, and the combination Associated storage, output includes network stress index. A list of candidate combinations.

[0082] S413: Select the combination with the smallest total amount of data that needs to be transferred across racks as the selected live fragment subset.

[0083] Based on the network pressure index calculated for each candidate combination, a deterministic minimization search algorithm is executed. All candidate combinations are traversed, their network pressure index values ​​are directly compared, and the combination with the smallest value is selected as the initial optimal solution. The initial selection result is then subjected to real-time resource verification. If the status of associated nodes or links is found to be severely deteriorated, a degradation selection mechanism is triggered to reselect a suboptimal combination from the remaining valid combinations. Finally, the verified optimal combination is determined as the surviving shard subset and encapsulated as a repair source descriptor containing information on all shards and nodes.

[0084] S42: Read data blocks from each node of the surviving fragment subset, and generate repair fragments by a dedicated repair computing node through linear coding operations corresponding to the erasure code on the data flow path to the target new node.

[0085] The process involves reading data blocks from each node in the surviving shard subset, and generating repair shards via a dedicated repair computing node on the data flow path to the target new node, using linear coding operations corresponding to the erasure code. This includes the following sub-steps:

[0086] S421: Send a unique coding coefficient vector to each of the selected K surviving nodes;

[0087] Based on the selected optimal subset of surviving fragments (containing K specific surviving fragments) and the target fragment information, the coding coordination module in the repair executor dynamically generates a set of dedicated coding coefficient vectors. This generation process is based on the generator matrix of erasure codes and solving specific linear equations (such as using Lagrange interpolation) to ensure that the coefficient vector calculated for each source node in the subset can correctly aggregate its local coding results at the target node to regenerate the lost fragments. Subsequently, the system encapsulates the coefficient vectors, the target node address, and the operation commands into a repair instruction package, which is then distributed to each source node in parallel and reliably through an efficient communication layer.

[0088] S422: Each surviving node uses the coefficient vector locally to perform an inner product calculation with the data fragments stored in the node to generate an intermediate coded block;

[0089] After parsing the repair instruction package locally, the specified raw fragment data is read from memory and divided into fixed-size data blocks. Then, the node uses the received dedicated encoding coefficient vector to independently perform linear encoding calculations (such as scalar multiplication or matrix multiplication) over a finite field on each data block, generating the corresponding encoded data block. To maximize efficiency, the node employs multi-core parallel technology to accelerate computation and sequentially combines all encoded data blocks into a complete intermediate encoded block for the node. Crucially, the node does not need to wait for all data to be computed; instead, it immediately initiates streaming network transmission to the target node after the first data block is encoded, achieving overlap between computation and network I / O and significantly reducing end-to-end latency. By offloading and distributing computationally intensive tasks to the data location, the transformation of data from its "raw form" to an "intermediate form suitable for regeneration and aggregation" is completed at the source, greatly reducing the computational burden on the target node and ensuring fundamental optimization of resource consumption and time efficiency in the repair process.

[0090] S423: Each node streams the intermediate coded blocks to the target node. The target node adds the received K intermediate coded blocks element by element to obtain the regenerated lost fragments.

[0091] The stream-based alignment and aggregation mechanism significantly reduces repair latency, and end-to-end integrity verification ensures the absolute reliability of the regenerated data. Specifically, the target node triggers the process after receiving the first data packet carrying the repair task ID, dynamically receiving intermediate encoded block data streams from K source nodes. By maintaining a sequential receive buffer for each source node, out-of-order data blocks are aligned. Once all data blocks from all K nodes corresponding to the same logical position arrive, an element-wise addition operation on a finite field is performed to regenerate the original data block in real time and write it to the main buffer (a main buffer matching the final size of the fragment to be regenerated is allocated in memory and initialized to all zeros (zero elements are 0 in the applicable finite field)). When all data blocks have been calculated and written, fragment regeneration is declared complete.

[0092] If a source node experiences a prolonged interruption in its flow, exceeding the amount of data it should transmit, that node is considered to have failed. If, before regeneration is complete, there are still enough surviving source nodes to complete the regeneration process, the target node requests the repair executor to select a replacement node from the remaining surviving nodes and re-receive the new version of the affected data blocks. If too many nodes fail, the entire repair task is rolled back and rescheduled.

[0093] S5: Perform hash verification on the regenerated repair shards. If the verification passes, store them in a healthy storage node and update the metadata.

[0094] Once all data blocks have been aggregated, the target node calculates the cryptographic hash value of the regenerated shard and compares it with the original hash value recorded in the metadata. If the verification passes, the target node asynchronously persists the complete data in the main cache to local storage, creating a new valid shard. If the verification fails, it indicates that an uncorrectable error may have occurred during the aggregation process (such as silent corruption of source data, incorrect coefficients, or calculation errors). The target node discards the data in the main cache and reports the failure to the repair executor, triggering a completely new repair task (e.g., selecting a different combination of source nodes).

[0095] Example 2

[0096] like Figure 2 As shown, Embodiment 2 of this application provides an archive data disaster recovery system based on distributed technology, including:

[0097] Data fragmentation encoding module 21: Encodes the original archive data object using erasure coding to obtain K data fragments and M verification fragments, and distributes the K+M fragments in storage nodes located in different fault domains in the cluster, and records the metadata of the fragment location mapping relationship.

[0098] Fault detection module 22: Real-time monitoring of storage node status. When a node or rack fault is detected, the set of shards lost due to the fault is determined based on the metadata and recorded as the set of shards to be repaired. Real-time load data of each node is also obtained.

[0099] Repair scheduling planning module 23: Based on the original archive metadata, calculate the heat value of each fragment to be repaired in the fragment set to be repaired, and predict the availability of repair resources in multiple future time intervals based on real-time load data, and generate a repair task queue based on the heat value and the availability of repair resources.

[0100] Source selection parallel regeneration module 24: According to the repair task queue, select a subset of surviving fragments for repair from the surviving fragments that have the same erasure code as each fragment to be repaired by calculating its network pressure index, and read the data blocks therein to generate repair fragments through linear coding operation corresponding to the erasure code. The size of the surviving fragment subset is equal to the minimum number of fragments required for the erasure code reconstruction.

[0101] Verification and update module 25: Used to perform hash verification on the regenerated repair shards. After the verification is successful, the shards are stored in healthy storage nodes and the metadata is updated.

[0102] Corresponding to the above embodiments, the present invention provides a computer storage medium, including: at least one memory and at least one processor;

[0103] The memory is used to store one or more program instructions;

[0104] A processor is used to run one or more program instructions to execute a distributed data disaster recovery method for archives.

[0105] Corresponding to the above embodiments, this embodiment of the invention provides a computer-readable storage medium containing one or more program instructions, which are executed by a processor to provide a method for archival data disaster recovery based on distributed technology.

[0106] The embodiments disclosed in this invention provide a computer-readable storage medium storing computer program instructions. When the computer program instructions are executed on a computer, the computer performs the aforementioned archival data disaster recovery method based on distributed technology.

[0107] In this embodiment of the invention, the processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0108] The various methods, steps, and logic diagrams disclosed in the embodiments of this invention can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The processor reads information from the storage medium and, in conjunction with its hardware, completes the steps of the above methods.

[0109] The storage medium can be memory, such as volatile memory or non-volatile memory, or may include both volatile and non-volatile memory.

[0110] Among them, non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory.

[0111] Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (Synchlink DRAM, SLDRAM), and direct memory bus RAM (DRRAM).

[0112] The storage media described in the embodiments of the present invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0113] Those skilled in the art will recognize that, in one or more of the examples above, the functions described in this invention can be implemented using a combination of hardware and software. When applied as software, the corresponding functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transmission of computer programs from one place to another. Storage media can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0114] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for archiving data disaster recovery based on distributed technology, characterized in that, include: The original archive data object is encoded using erasure coding to obtain K data fragments and M verification fragments. The K+M fragments are distributed and stored in storage nodes located in different fault domains in the cluster, and the metadata of the fragment location mapping relationship is recorded. Real-time monitoring of storage node status; when a node or rack failure is detected, the set of shards lost due to the failure is determined based on the metadata and recorded as the set of shards to be repaired, and real-time load data of each node is obtained. Based on the original archive metadata, the heat value of each fragment to be repaired in the fragment set is calculated, and the availability of repair resources for multiple future time intervals is predicted based on real-time load data. A repair task queue is generated based on the heat value and the availability of repair resources. Specifically, the list of fragments to be repaired, sorted in descending order of heat value, and the sequence of available repair resources representing the available repair capacity for each future time window are used as input. For each high-value fragment, future time windows are scanned sequentially, and it is assigned to the earliest window where the remaining available resources are sufficient to cover the estimated repair cost of that fragment. The available resources of that window are deducted in real time after allocation. If the available resources of all existing windows are insufficient, the fragment is suspended and the rolling update of resource prediction is triggered to find a more distant feasible window. After the initial queue is generated, load balancing is fine-tuned to make the task distribution on the time axis smoother. The final output is a detailed execution plan ordered by time window, with each window containing a specific task list. According to the repair task queue, a subset of surviving fragments for repair is selected from the surviving fragments that have the same erasure code as each fragment to be repaired by calculating their network pressure index, and the data blocks in the subset are read and repaired fragments are generated by linear coding operation corresponding to the erasure code. The size of the surviving fragment subset is equal to the minimum number of fragments required for the erasure code reconstruction. The regenerated repair shards are hash-verified. Once the verification is successful, they are stored in healthy storage nodes, and the metadata is updated.

2. The method of claim 1, wherein the method is characterized by, Based on the original archive metadata, calculate the heat value of each fragment to be repaired in the fragment set, and predict the availability of repair resources for multiple future time intervals based on real-time load data. Generate a repair task queue based on the heat value and the availability of repair resources, including: For each shard to be repaired, a heat value representing its urgency is calculated based on the access history and business attributes of its nodes. Based on the current real-time load data of each node, the available repair resources for multiple future time intervals are predicted using time series forecasting methods. Based on the heat value and the availability of repair resources, high-resolution fragments are preferentially allocated to the nearest repair window with sufficient resources, generating a repair task queue with smooth temporal distribution and controlled load.

3. The method of claim 1, wherein the method further comprises: Based on the repair task queue, a subset of surviving fragments with the same erasure codes as each fragment to be repaired is selected for repair by calculating their network pressure index. Data blocks from these fragments are then read and processed using linear coding operations corresponding to the erasure codes to generate repair fragments, including: From the currently surviving fragments that belong to the same erasure coding group as the fragment to be repaired, calculate the network pressure index based on the network topology distance and node load, and select a subset of surviving fragments for repair; Data blocks are read from each node of the surviving shard subset, and repair shards are generated by a dedicated repair computing node through linear coding operations corresponding to the erasure code on the data flow path to the target new node.

4. The method of claim 3, wherein the method further comprises: From the currently surviving fragments belonging to the same erasure coding group as the fragment to be repaired, a subset of surviving fragments for repair is selected based on the network pressure index calculated according to network topology distance and node load, including: Identify all currently surviving fragments belonging to the same (K, M) erasure coding group as the fragment to be repaired, and calculate all possible combinations of surviving fragments of size K; Predict the network stress index that each combination will experience during the repair process; Select the combination with the smallest total amount of data that needs to be transferred across racks as the selected live shard subset.

5. The method of claim 3, wherein the method further comprises: Data blocks are read from each node of the surviving shard subset, and along the data flow path to the target new node, a dedicated repair computing node generates repair shards through linear coding operations corresponding to the erasure code, including: Send a unique coding coefficient vector to each of the selected K surviving nodes; Each surviving node uses the coefficient vector locally to perform an inner product calculation with the data fragments stored on the node to generate intermediate encoded blocks; Each node streams intermediate coded blocks to the target node, which then adds the received K intermediate coded blocks element by element to obtain the regenerated lost fragments.

6. An archival data disaster recovery system based on distributed technology, characterized by, include: The data fragmentation encoding module is used to encode the original archive data object using erasure coding to obtain K data fragments and M verification fragments. The K+M fragments are distributed and stored in storage nodes located in different fault domains in the cluster, and the metadata of the fragment location mapping relationship is recorded. The fault detection module is used to monitor the status of storage nodes in real time. When a node or rack fault is detected, the set of shards lost due to the fault is determined based on the metadata and recorded as the set of shards to be repaired. The module also obtains the real-time load data of each node. The repair scheduling and planning module calculates the heat value of each fragment in the fragment set to be repaired based on the original archive metadata, and predicts the availability of repair resources for multiple future time intervals based on real-time load data. It then generates a repair task queue based on the heat value and the available repair resources. Specifically, it takes as input a list of fragments to be repaired in descending order of heat value and a sequence of available repair resources representing the repair capacity available for each future time window. For each high-value fragment, it sequentially scans future time windows, assigning it to the earliest window where the remaining available resources are sufficient to cover the estimated repair cost of that fragment, and deducting the available resources for that window in real time after assignment. If the available resources for all existing windows are insufficient, the fragment is temporarily suspended, and a rolling update of resource predictions is triggered to find a more distant feasible window. After the initial queue is generated, load balancing is fine-tuned to make the task distribution on the timeline smoother. Finally, it outputs a detailed execution plan ordered by time windows, with each window containing a specific task list. The source selection parallel regeneration module is used to select a subset of surviving fragments for repair from the surviving fragments that have the same erasure code as each fragment to be repaired by calculating their network pressure index according to the repair task queue, and read the data blocks therein to generate repair fragments through linear coding operation corresponding to the erasure code. The size of the surviving fragment subset is equal to the minimum number of fragments required for the erasure code reconstruction. The verification and update module is used to perform hash verification on the regenerated repair shards. After the verification is successful, the shards are stored in healthy storage nodes and the metadata is updated.

7. A computer readable storage medium characterized in that, It includes one or more program instructions, which are executed by a processor as described in any one of claims 1-5, to provide a method for archival data disaster recovery based on distributed technology.

Images