An intelligent disaster recovery backup and recovery system and method for multi-node data processing

Abstract

This invention belongs to the field of data disaster recovery technology. It discloses an intelligent disaster recovery backup and restoration system and method for multi-node data processing, including a dependency resolution module for obtaining processing dependencies between data processing nodes, constructing a node dependency topology graph, and determining the data objects of each data processing node and their corresponding sets of directly dependent nodes and sets of indirectly dependent nodes within a preset hop count range; a disaster recovery assessment module for calculating the business association depth score of data objects based on the sets of directly and indirectly dependent nodes, and calculating the non-renewability score in conjunction with the renewability attributes of the data objects, determining a comprehensive disaster recovery value score based on the business association depth score and the non-renewability score; and a backup orchestration module for mapping each data object to a corresponding backup level based on the comprehensive disaster recovery value score. This significantly improves the overall disaster recovery efficiency and business continuity assurance capabilities of the system.

Description

Technical Field

[0001] This invention relates to the field of data disaster recovery technology, and more specifically, to an intelligent disaster recovery backup and restoration system and method for multi-node data processing. Background Technology

[0002] With the widespread application of big data platforms, distributed computing frameworks, and multi-node collaborative processing systems, an increasing amount of business data needs to undergo processing such as collection, cleaning, computation, aggregation, and distribution across multiple data processing nodes according to predetermined dependencies. In such systems, a data object is often not only used by the current node but also serves as input data for multiple downstream nodes in subsequent processing. Therefore, if any critical data object is damaged, lost, or its host node malfunctions, the damage may propagate along the dependency chain, ultimately affecting the normal operation of the entire business processing chain.

[0003] Most existing data disaster recovery and backup technologies adopt a uniform configuration approach, such as performing full or incremental backups at fixed time intervals, or determining backup strategies based solely on single indicators such as storage type and access frequency. While this approach can achieve basic data protection, it typically does not fully consider the different business roles of data objects within the entire multi-node processing topology, nor does it comprehensively analyze the difficulty of rebuilding data after loss, making it difficult to implement truly targeted disaster recovery protection for different data objects.

[0004] Current technologies for assessing data importance typically consider only direct dependencies or simple call relationships between data objects, failing to reflect the true impact of multi-level dependency structures on business operations. For indirect dependencies, equal-weighted statistics or simple cumulative methods are often used, neglecting the objective law of gradual attenuation of impact as dependency path length increases, easily exaggerating the influence of distant dependency nodes. Furthermore, current technologies generally do not effectively limit the propagation range of indirect dependencies, which can introduce a large number of nodes with low relevance to the target data in complex topologies, increasing computational overhead and reducing the stability of assessment results. In addition, the lack of a unified normalization mechanism makes it difficult to directly compare importance assessment results obtained from systems of different sizes, thus affecting the consistency of disaster recovery strategies.

[0005] On the other hand, existing technologies typically treat business importance and recovery costs separately, or simply categorize data based on whether it is recoverable, without further distinguishing between original data and intermediate data that can be recalculated from upstream data, nor quantifying the differences in regeneration costs between different types of regenerable data. This leads to some data objects that, while reconstructable, have high reconstruction costs being assigned similar disaster recovery priorities to ordinary intermediate results, while truly non-regenerable critical original data is difficult to accurately identify, thus affecting the rationality of backup resource allocation and recovery sequence arrangement.

[0006] Furthermore, when a node fails or data is abnormal in the system, existing technologies typically perform recovery operations according to a preset order or human experience. They lack a dynamic recovery mechanism that combines node dependencies and data value scoring, which can easily lead to situations where non-critical data is recovered first while critical business links are recovered later, resulting in extended overall business interruption time.

[0007] In view of this, the present invention proposes an intelligent disaster recovery and backup system and method for multi-node data processing to solve the above problems. Summary of the Invention

[0008] To overcome the aforementioned deficiencies of the prior art and to achieve the above objectives, the present invention provides the following technical solution: an intelligent disaster recovery and backup system for multi-node data processing, comprising: The dependency resolution module is used to obtain the processing dependencies between data processing nodes, construct a node dependency topology graph, and determine the data objects of each data processing node and the corresponding set of directly dependent nodes and the set of indirectly dependent nodes within a preset hop count range. The disaster recovery assessment module is used to calculate the business association depth score of data objects based on the set of directly dependent nodes and the set of indirectly dependent nodes, and to calculate the non-renewability score in combination with the renewability attribute of the data objects. The comprehensive disaster recovery value score is determined based on the business association depth score and the non-renewability score. The backup orchestration module is used to map each data object to the corresponding backup level based on the comprehensive disaster recovery value score, and generate a backup plan that includes snapshot method, backup cycle and copy retention strategy according to the backup level. The anomaly monitoring module is used to monitor the running status of each data processing node. After detecting an anomaly in the running status, it identifies the affected data objects based on the node dependency topology graph and generates fault context information. The driver recovery module is used to determine the preconditions for recovery and the recovery execution order based on the fault context information, node dependency topology diagram and comprehensive disaster recovery value score, and call the corresponding backup plan to complete the node data recovery.

[0009] Preferably, the method for constructing a node-dependent topology graph includes: Collect the operation records generated during the operation of each data processing node. The operation records include transaction logs, task scheduling records, interface call records and data read and write records. Extract the node identifier, data object identifier and timestamp from the operation records. Based on the output and read correspondence of the same data object across different data processing nodes, and the task call relationship between different data processing nodes, determine the processing dependency relationship between upstream and downstream data processing nodes; Each data processing node is used as a graph node, and processing dependencies are used as directed edges, with the direction of the directed edges pointing from the upstream data processing node to the downstream data processing node. All graph nodes and directed edges are associated and organized to generate a node dependency topology graph.

[0010] Preferably, the method for determining the data objects of each data processing node includes: Based on the node dependency topology, read the output records of each data processing node, and determine the data files, data tables, cached results and message results generated and output by the corresponding data processing node as the data objects of the data processing node. Establish the association between each data object and the data processing node that generates the data object. For any target data object, obtain the upstream neighboring node corresponding to the incoming edge of the data processing node associated with the target data object in the node dependency topology graph, and determine the set of upstream neighboring nodes as the set of direct dependent nodes of the target data object. Starting with the data processing node associated with the target data object, the graph is traversed in the reverse direction of the directed edges in the node dependency topology graph. The number of edges traversed during the traversal is accumulated and used as the hop count. Within the preset hop count range, the remaining reachable upstream nodes other than the set of directly dependent nodes are obtained, and the set of the remaining reachable upstream nodes is determined as the set of indirect dependent nodes of the target data object.

[0011] Preferably, the method for calculating the business association depth score of the data object includes: For any target data object, obtain the set of directly dependent nodes and the set of indirectly dependent nodes within a preset hop count range corresponding to the target data object, and count the total number of nodes in the node dependency topology graph; calculate the number of nodes in the set of directly dependent nodes and the number of nodes in the set of indirectly dependent nodes respectively. The number of indirect dependent nodes is reduced by a preset indirect dependency discount factor. The number of direct dependent nodes is added to the number of indirect dependent nodes after the reduction. The sum is then normalized using the total number of nodes to obtain the business association depth score of the target data object.

[0012] Preferably, the method for determining the comprehensive disaster recovery value score includes: For any target data object, obtain the regenerable attribute of the target data object; when the target data object cannot be regenerated by the upstream data object, set the regenerable flag of the target data object to 1 and set the non-regenerability score to 1; when the target data object can be regenerated by the upstream data object, set the regenerable flag of the target data object to 0. Calculate the regeneration cost required to regenerate the target data object, obtain the maximum regeneration cost among all regenerable data objects in the system, and determine the non-regenerability score of the target data object using the ratio of the regeneration cost to the maximum regeneration cost; normalize the business association depth score and non-regenerability score of the target data object, and perform a weighted summation according to preset weight coefficients to obtain the comprehensive disaster recovery value score of the target data object.

[0013] Preferably, the method for mapping each data object to a corresponding backup level includes: Several backup levels are pre-set, and a corresponding scoring range is configured for each backup level. The scoring ranges are arranged in order of comprehensive disaster recovery value score from low to high. For any target data object, the comprehensive disaster recovery value score of the target data object is obtained, and the comprehensive disaster recovery value score is compared with each scoring range. When the comprehensive disaster recovery value score falls into a certain score range, the target data object is mapped to the backup level corresponding to that score range. The backup levels include the first backup level, the second backup level, and the third backup level, and the data protection strength corresponding to the first backup level, the second backup level, and the third backup level increases sequentially.

[0014] Preferably, the method for generating the backup scheme includes: A mapping table between backup levels and backup strategy parameters is pre-established, where each backup level corresponds to a different snapshot method type, backup execution cycle, and copy retention strategy. For any target data object, after determining the corresponding backup level, the snapshot method, backup cycle, and copy retention strategy corresponding to the backup level are obtained from the mapping table. When the target data object is at the first backup level, the snapshot method is configured as a full snapshot, the backup period is configured as the first period parameter, and the replica retention policy is configured as the first retention parameter; when the target data object is at the second backup level, the snapshot method is configured as an incremental snapshot, the backup period is configured as the second period parameter, and the replica retention policy is configured as the second retention parameter. When the target data object is at the third backup level, the snapshot method is configured as a high-frequency incremental snapshot method, the backup period is configured as the third period parameter, and the copy retention policy is configured as the third retention parameter; and the obtained snapshot method, backup period and copy retention policy are combined to generate the corresponding backup scheme.

[0015] Preferably, the method for generating fault context information includes: The system continuously acquires the running status information of each data processing node, including heartbeat status information, task execution feedback information, interface call response information, and data read / write log information. The running status of the node is determined based on the running status information. When the running status of any data processing node meets the preset abnormal judgment conditions, the running abnormality of that data processing node is determined to have occurred. After identifying the abnormal data processing node, the module propagates and traverses downstream nodes along the data flow direction based on the node dependency topology graph to identify all data processing nodes that are directly or indirectly dependent on the abnormal data processing node and the corresponding affected data objects. After completing the identification of the scope of impact, the anomaly monitoring module automatically generates fault context information.

[0016] Preferably, the method for invoking the corresponding backup scheme to complete node data recovery includes: The abnormal data processing node identifier, the set of affected data objects, and the dependency propagation path information in the fault context are parsed, and a recovery dependency subgraph is constructed based on the node dependency topology graph to represent the dependency constraint relationship between the affected nodes. The recovery prerequisites are determined based on the directed dependencies in the recovery dependency subgraph, wherein any affected data processing node can only enter the recovery execution phase after the data object corresponding to the upstream dependent node has been recovered. Based on meeting the prerequisites for recovery, each affected data object is associated with its corresponding comprehensive disaster recovery value score, and the recovery execution order is determined according to the comprehensive disaster recovery value score from high to low. Then, according to the backup level of each data object, the corresponding backup plan is called, the corresponding backup copy is obtained from the backup storage, and data rollback or version reconstruction operations are performed to complete the node data recovery.

[0017] A smart disaster recovery and backup method for multi-node data processing, characterized by comprising: S1. Obtain the processing dependencies between each data processing node, construct a node dependency topology graph, and determine the data objects of each data processing node and the corresponding set of directly dependent nodes and the set of indirectly dependent nodes within a preset hop count range. S2. Calculate the business association depth score of the data object based on the set of directly dependent nodes and the set of indirectly dependent nodes, and calculate the non-renewability score in combination with the renewability attribute of the data object. Determine the comprehensive disaster recovery value score based on the business association depth score and the non-renewability score. S3. Based on the comprehensive disaster recovery value score, map each data object to the corresponding backup level, and generate a backup plan that includes snapshot method, backup cycle and copy retention strategy according to the backup level. S4. Monitor the running status of each data processing node. After detecting abnormal running status, identify the affected data objects based on the node dependency topology graph and generate fault context information. S5. Based on the fault context information, node dependency topology diagram and comprehensive disaster recovery value score, determine the preconditions for recovery and the recovery execution order, and call the corresponding backup plan to complete the node data recovery.

[0018] Compared with the prior art, the present invention has the following beneficial effects: By introducing a combined modeling mechanism of directly dependent nodes and indirectly dependent nodes within a preset hop count range, the business association depth score can comprehensively reflect the actual impact range of data objects in multi-level topologies. By introducing a preset indirect dependency discount coefficient, the impact of indirect dependent nodes is differentially attenuated, ensuring a reasonable weight distribution of different levels of dependencies in business impact calculations. By constraining the indirect dependency propagation path with a preset hop count range, the introduction of invalid remote dependent nodes is effectively avoided, improving computational stability and reducing redundant computational overhead. Through normalization of the total number of nodes, the business association depth score under different scale topologies has a unified dimension, improving the comparability of data object importance assessment results between different systems. This quantitative result of business association depth can be directly used for subsequent comprehensive disaster recovery value scoring calculations, thereby improving the accuracy and rationality of backup strategy grading and recovery priority ranking.

[0019] By weighting and fusing business relevance depth scores and non-reproducibility scores, the disaster recovery value score simultaneously reflects the business importance of data and the difficulty of reconstruction, improving the completeness of the assessment results. Introducing a non-reproducibility score assigns higher weight to data objects that cannot be regenerated from upstream data, thereby accurately identifying truly irreplaceable critical data. This comprehensive scoring mechanism provides a continuously separable quantitative basis for different data objects in backup level classification and recovery priority ranking, avoiding decision distortions caused by traditional binary or coarse-grained classification methods. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the structure of an intelligent disaster recovery and backup system for multi-node data processing according to the present invention; Figure 2 This is a schematic diagram of the intelligent disaster recovery and backup method for multi-node data processing according to the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Example

[0022] Please see Figure 1 As shown, this embodiment provides an intelligent disaster recovery and backup system for multi-node data processing, specifically including the following steps: The dependency resolution module is used to obtain the processing dependencies between data processing nodes, construct a node dependency topology graph, and determine the data objects of each data processing node and the corresponding set of directly dependent nodes and the set of indirectly dependent nodes within a preset hop count range. The disaster recovery assessment module is used to calculate the business association depth score of data objects based on the set of directly dependent nodes and the set of indirectly dependent nodes, and to calculate the non-renewability score in combination with the renewability attribute of the data objects. The comprehensive disaster recovery value score is determined based on the business association depth score and the non-renewability score. The backup orchestration module is used to map each data object to the corresponding backup level based on the comprehensive disaster recovery value score, and generate a backup plan that includes snapshot method, backup cycle and copy retention strategy according to the backup level. The anomaly monitoring module is used to monitor the running status of each data processing node. After detecting an anomaly in the running status, it identifies the affected data objects based on the node dependency topology graph and generates fault context information. The driver recovery module determines the preconditions and execution order for recovery based on fault context information, node dependency topology, and comprehensive disaster recovery value score, and then invokes the corresponding backup scheme to complete node data recovery. All modules are connected via wired and / or wireless means to enable data transmission between them.

[0023] Methods for constructing a node dependency topology graph include: Collect the operation records generated during the operation of each data processing node. The operation records include transaction logs, task scheduling records, interface call records and data read and write records. Extract the node identifier, data object identifier and timestamp from the operation records. Based on the output and read correspondence of the same data object across different data processing nodes, and the task call relationship between different data processing nodes, determine the processing dependency relationship between upstream and downstream data processing nodes; In this embodiment, it should be noted that after obtaining the standardized running records, the dependency parsing module identifies the data transmission relationship between different data processing nodes. If the first data processing node outputs data object d at a certain moment, and the second data processing node processes the same data object d as an input object at a subsequent moment, it indicates that the data processing behavior of the second data processing node depends on the processing result generated by the first data processing node. Therefore, the first data processing node is determined to be the upstream data processing node, and the second data processing node is determined to be the downstream data processing node, and there is a processing dependency relationship between the two.

[0024] In addition to determining dependencies based on the output and read relationships of the same data object, the dependency resolution module also analyzes task scheduling records and interface call records. If the execution record shows that the first data processing node actively calls the second data processing node to execute a specified task, or the execution trigger condition of the second data processing node depends on the task completion status of the first data processing node, then it is determined that the execution of the second data processing node depends on the processing result of the first data processing node. Similarly, the first data processing node is identified as the upstream data processing node, and the second data processing node is identified as the downstream data processing node.

[0025] For example, the data acquisition node outputs the raw data file raw_01, the data cleaning node reads raw_01 and generates the cleaned result clean_01, and the feature extraction node further reads clean_01 and generates the feature result feature_01. Based on the above execution records, it can be determined that there is a processing dependency between the data acquisition node and the data cleaning node, and also between the data cleaning node and the feature extraction node. If the task scheduling records further indicate that the feature extraction node is automatically triggered upon completion of the data cleaning node, then this call relationship further verifies the processing dependency between the two.

[0026] Each data processing node is used as a graph node, and processing dependencies are used as directed edges, with the direction of the directed edges pointing from the upstream data processing node to the downstream data processing node. All graph nodes and directed edges are associated and organized to generate a node dependency topology graph.

[0027] Methods for determining the data objects of each data processing node include: Based on the node dependency topology, read the output records of each data processing node, and determine the data files, data tables, cached results and message results generated and output by the corresponding data processing node as the data objects of the data processing node. Establish the association between each data object and the data processing node that generates the data object. For any target data object, obtain the upstream neighboring node corresponding to the incoming edge of the data processing node associated with the target data object in the node dependency topology graph, and determine the set of upstream neighboring nodes as the set of direct dependent nodes of the target data object. Starting with the data processing node associated with the target data object, the graph is traversed in the reverse direction of the directed edges in the node dependency topology graph. The number of edges traversed during the traversal is accumulated and used as the hop count. Within the preset hop count range, the remaining reachable upstream nodes other than the set of directly dependent nodes are obtained, and the set of the remaining reachable upstream nodes is determined as the set of indirect dependent nodes of the target data object.

[0028] In this embodiment, it should be noted that when the cumulative hop count does not exceed the preset hop count range L, the corresponding reachable upstream node is included in the candidate set; when the cumulative hop count exceeds the preset hop count range, traversal along the path is stopped. Finally, the remaining reachable upstream nodes in the candidate set, excluding the directly dependent node set, are determined as the indirect dependent node set of the target data object.

[0029] To avoid duplicate counting of the same data processing node during the business association depth score calculation, the direct dependency node set and the indirect dependency node set satisfy a mutual exclusion relationship. That is, the intersection of the direct dependency node set of the data object and the indirect dependency node set of the data object within a preset hop count range is equal to an empty set. When determining the indirect dependency node set, nodes belonging to the direct dependency node set among all reachable upstream nodes traversed within the preset hop count range are removed, and only the remaining reachable upstream nodes are determined as the indirect dependency node set.

[0030] For example, if node A outputs the original data object d1, node B reads d1 and generates data object d2, node C reads d2 and generates data object d3, and node D reads d3 and generates the target data object d4, then the generating node for the target data object d4 is node D. The direct upstream node of node D is node C, therefore the set of directly dependent nodes is {C}. When the preset hop count is 3, traversing upstream from node D can reach nodes C, B, and A. Among these, nodes B and A (excluding node C) are determined as the set of indirectly dependent nodes, i.e., {B, A}.

[0031] Methods for calculating the business association depth score of data objects include: For any target data object, obtain the set of directly dependent nodes and the set of indirectly dependent nodes within a preset hop count range corresponding to the target data object, and count the total number of nodes in the node dependency topology graph; calculate the number of nodes in the set of directly dependent nodes and the number of nodes in the set of indirectly dependent nodes respectively. The number of indirect dependent nodes is reduced by a preset indirect dependency discount factor. The number of direct dependent nodes is added to the number of indirect dependent nodes after the reduction. The sum is then normalized using the total number of nodes to obtain the business association depth score of the target data object.

[0032] Business Relevance Depth Score: ;in, The business association depth score of the target data object represents the data object. The scope of business impact within the entire multi-node data processing system; Index identifiers representing data objects, including data files, data tables, cached results, and message results; Represents data objects The set of nodes that directly depend on it; This indicates the number of nodes in the set of directly dependent nodes, i.e., the number of directly dependent nodes; Represents data objects Within the preset number of jumps The set of indirect dependent nodes within; Indicates the number of nodes in the indirectly dependent node set; This indicates the preset hop count range, used to limit the maximum number of edges allowed when searching for indirectly dependent nodes upstream in the node dependency topology graph. It is obtained by statistically analyzing the path length distribution between each data processing node in the node dependency topology graph and its upstream source node, and calculating the average of all path lengths. This represents the set of all nodes in the topology graph that a node depends on; This represents the total number of nodes in the node-dependent topology graph; This represents the preset indirect dependency discount coefficient, which ranges from 0 to 1 and is determined by statistically analyzing the average contribution ratio of indirect dependency nodes to the scope of business impact during historical operation. For example, a target data object has a set of direct dependent nodes containing 3 nodes and a set of indirect dependent nodes containing 5 nodes, with an indirect dependency discount factor. If the score is 0.4 and the total number of nodes in the node dependency topology graph is 20, then the business association depth score is: The result indicates that the business impact of the target data object, after reduction, accounts for approximately 25% of the entire data processing system.

[0033] Methods for determining the comprehensive disaster recovery value score include: For any target data object, obtain the regenerable attribute of the target data object; when the target data object cannot be regenerated by the upstream data object, set the regenerable flag of the target data object to 1 and set the non-regenerability score to 1; when the target data object can be regenerated by the upstream data object, set the regenerable flag of the target data object to 0. In this embodiment, it should be noted that whether the target data object can be recalculated and generated from the upstream data object is determined by analyzing the generation path information in the node dependency topology graph and the processing rules of the data processing nodes.

[0034] Specifically, the dependency parsing module records the correspondence between the input data object, processing rules, and output data object for each data processing node. The processing rules can be data cleaning rules, aggregation calculation rules, transformation scripts, query statements, interface processing logic, or task execution programs. If the processing rules for a data processing node corresponding to a certain data object are completely saved, and all upstream data objects that the data processing node depends on are available, then it is considered that the data object can be recalculated and generated based on the upstream data objects and corresponding processing rules.

[0035] Conversely, if any of the following conditions exist, the data object is determined to be unrecoverable through recalculation of the upstream data object: 1. The processing rules of the data processing node that generated the data object are missing, corrupted, or unexecutable; 2. At least one required upstream data object has no available replica; 3. The data objects originate from external systems through one-time pushes, manual entry, real-time collection, or temporary session information, and the original generation process cannot be repeated; 4. The data objects contain random disturbances, manual confirmation results, or external environmental status information during the generation process, which makes it impossible to reproduce them stably under the same input conditions.

[0036] For example, raw measurement data acquired in real time by sensors, manual approval results, and one-time response data returned by third-party systems are considered non-renewable data objects because their generation process cannot be repeated. Conversely, if a data object is calculated from an upstream data table using fixed SQL query statements, ETL transformation rules, or statistical scripts, and both the upstream data table and processing rules are completely preserved, then that data object is considered a renewable data object. This accurately distinguishes between original, irreplaceable data and recalculated derived data, providing a basis for calculating non-renewability scores.

[0037] Calculate the regeneration cost required to regenerate the target data object, obtain the maximum regeneration cost among all regenerable data objects in the system, and determine the non-regenerability score of the target data object using the ratio of the regeneration cost to the maximum regeneration cost; Non-reproducibility score of the target data object: ;in, The score representing the non-reproducibility of a target data object characterizes the difficulty of reconstructing the data object after it is lost; the higher the score, the more difficult it is to regenerate. The regenerable flag of the target data object. When the value is 1, it means that the data object cannot be recalculated and generated from the upstream data object. When the value is 0, it means that the data object can be recalculated and generated from the upstream data object. This represents the regeneration cost of the target data object, used to quantify the resource consumption required to regenerate the data object. It is obtained by weighting the required time, CPU, or memory consumption. This represents the maximum regeneration cost of all regenerable data objects in the system. The business association depth score and non-renewability score of the target data object are normalized and then weighted and summed according to preset weight coefficients to obtain the comprehensive disaster recovery value score of the target data object.

[0038] Overall disaster recovery value score of the target data object: ;in, This represents the overall disaster recovery value score of the target data object; The preset weighting coefficients represent the business relevance depth score; The preset weighting coefficients represent the non-renewability score; and The following methods are used to obtain the data: the duration of business interruption and the duration of data reconstruction after each data object experienced an anomaly during the historical operating cycle; the correlation coefficient between the business association depth score and the duration of business interruption, and the correlation coefficient between the non-renewability score and the duration of data reconstruction are calculated using the Pearson correlation coefficient; after normalizing the two correlation coefficients, the preset weight coefficients corresponding to the business association depth score and the non-renewability score are obtained, and the sum of the two is made equal to 1.

[0039] For example, if a target data object has a business association depth score of 0.65 and a non-reproducibility score of 0.90, the preset weighting coefficient for the business association depth score is set as follows: The preset weighting coefficient for the non-renewability score is 0.6. If set to 0.4, the overall disaster recovery value score is: The higher the score, the greater the business impact of the target data object and the greater the difficulty of recovery; therefore, it should receive higher priority in subsequent backup resource allocation and disaster recovery prioritization.

[0040] Methods for mapping data objects to corresponding backup levels include: Several backup levels are pre-set, and a corresponding scoring range is configured for each backup level. The scoring ranges are arranged in order of comprehensive disaster recovery value score from low to high. For any target data object, the comprehensive disaster recovery value score of the target data object is obtained, and the comprehensive disaster recovery value score is compared with each scoring range. When the comprehensive disaster recovery value score falls into a certain score range, the target data object is mapped to the backup level corresponding to that score range. The backup levels include the first backup level, the second backup level, and the third backup level, and the data protection strength corresponding to the first backup level, the second backup level, and the third backup level increases sequentially.

[0041] For example, a first backup level, a second backup level, and a third backup level can be pre-set, where: the first backup level corresponds to a scoring range. The second backup level corresponds to the following scoring range. The second backup level corresponds to the following scoring range. ;in, and To preset the comprehensive disaster recovery value scoring threshold, and to meet the following requirements: ; When the comprehensive disaster recovery value score of the target data object is less than When the overall disaster recovery value score is greater than or equal to the first backup level, it is mapped to the first backup level; when the overall disaster recovery value score is greater than or equal to the first backup level, it is mapped to the first backup level. and less than When the overall disaster recovery value score is greater than or equal to the second backup level, it is mapped to the second backup level. If the value is less than 1, it is mapped to the third backup level; the backup level classification for different data objects is achieved in the above manner.

[0042] The methods for generating backup plans include: A mapping table between backup levels and backup strategy parameters is pre-established, where each backup level corresponds to a different snapshot method type, backup execution cycle, and copy retention strategy. For any target data object, after determining the corresponding backup level, the snapshot method, backup cycle, and copy retention strategy corresponding to the backup level are obtained from the mapping table. When the target data object is at the first backup level, the snapshot method is configured as a full snapshot, the backup period is configured as the first period parameter, and the replica retention policy is configured as the first retention parameter; when the target data object is at the second backup level, the snapshot method is configured as an incremental snapshot, the backup period is configured as the second period parameter, and the replica retention policy is configured as the second retention parameter. When the target data object is at the third backup level, the snapshot method is configured as a high-frequency incremental snapshot method, the backup period is configured as the third period parameter, and the copy retention policy is configured as the third retention parameter; and the obtained snapshot method, backup period and copy retention policy are combined to generate the corresponding backup scheme.

[0043] In this embodiment, it should be noted that the backup orchestration module sets corresponding backup cycle parameters and copy retention parameters according to the backup level. The backup cycle parameters are inversely hierarchical to the backup level. The first cycle parameter corresponding to the first backup level is greater than the second cycle parameter corresponding to the second backup level, and the second cycle parameter is greater than the third cycle parameter corresponding to the third backup level, so as to achieve a gradual increase in backup frequency for different backup levels. Meanwhile, the copy retention parameters and backup levels have a positive hierarchical relationship, that is, the first retention parameter corresponding to the first backup level is less than the second retention parameter corresponding to the second backup level, and the second retention parameter is less than the third retention parameter corresponding to the third backup level, so that data objects with higher backup levels have higher historical copy redundancy. For example, the system pre-establishes a mapping table between backup levels and backup policy parameters, where the third backup level corresponds to a high protection policy configuration. Assuming a target data object is classified as the third backup level after comprehensive disaster recovery value scoring, the backup orchestration module reads the corresponding parameters from the mapping table, where the third period parameter is set to 30 minutes, the third retention parameter is set to 10 historical copy versions, and the snapshot method is a high-frequency incremental snapshot method.

[0044] During the strategy generation process, the system configures the snapshot method of the target data object as a high-frequency incremental snapshot method, so that it only captures the difference data between adjacent versions each time; the backup cycle is configured to 30 minutes, that is, an incremental backup task is triggered every 30 minutes; at the same time, the copy retention policy is configured to 10, that is, the 10 most recent historical version copies are retained in the storage system.

[0045] Finally, the system combines the above snapshot method, backup cycle, and copy retention strategy to generate a complete backup plan for the target data object. For example: snapshot method: high-frequency incremental snapshot; backup cycle: 30 minutes; number of copies retained: 10 versions.

[0046] Methods for generating fault context information include: The system continuously acquires the running status information of each data processing node, including heartbeat status information, task execution feedback information, interface call response information, and data read / write log information. The running status of the node is determined based on the running status information. When the running status of any data processing node meets the preset abnormal judgment conditions, the running abnormality of that data processing node is determined to have occurred. In this embodiment, it should be noted that the determination of the node's operating status is achieved by fusing and judging multi-source operating status information. Specifically, the anomaly monitoring module classifies and analyzes the continuously acquired operating status information. Among them, the center jump status information is used to characterize the node's liveness, the task execution feedback information is used to characterize the task execution success rate and blocking status, the interface call response information is used to characterize service reachability and response latency, and the data read and write log information is used to characterize the integrity and correctness of data operations. Based on this, various types of status information are mapped to corresponding status indicator values, and each indicator value is comprehensively evaluated according to preset status judgment rules to obtain the overall operating status of the node.

[0047] During the operation status determination process, if the heartbeat status of any data processing node is not updated within a preset time window, or the failure rate in the task execution feedback information exceeds the preset failure rate threshold, or the interface call response information times out continuously, or a key error mark appears in the data read and write log, the operation status of the node is determined to meet the preset abnormal judgment conditions, thereby determining that the data processing node has experienced an operational abnormality.

[0048] After identifying the abnormal data processing node, the module propagates and traverses downstream nodes along the data flow direction based on the node dependency topology graph to identify all data processing nodes that are directly or indirectly dependent on the abnormal data processing node and the corresponding affected data objects. After completing the identification of the scope of impact, the anomaly monitoring module automatically generates fault context information.

[0049] The fault context information includes the identifier of the abnormal data processing node, the type of abnormality, the time of occurrence of the abnormality, the set of affected data objects, and the dependency propagation path information. The abnormality monitoring module encapsulates the abnormal node identifier, the type of abnormality, and the time of occurrence of the abnormality in a structured manner, and associates and binds the set of affected data objects identified based on the node dependency topology graph with the corresponding dependency propagation path information to form a unified fault context data structure, which is then written into the fault event storage unit.

[0050] For example, during system monitoring, node A was identified as an abnormal node because its heartbeat signal had not been updated for three consecutive cycles. Simultaneously, task execution feedback showed that its recent task failure rate exceeded 80%. The anomaly monitoring module traversed downstream based on the node dependency topology graph, identifying nodes B and C and their corresponding data object sets as affected, and further determining the dependency propagation path A→B→C. The final generated fault context information includes: an abnormal node A identifier, anomaly type (heartbeat interruption and task failure), anomaly timestamp, affected data object set, and dependency propagation path A→B→C, thus providing complete contextual information for subsequent disaster recovery.

[0051] The methods for using the corresponding backup scheme to complete node data recovery include: The abnormal data processing node identifier, the set of affected data objects, and the dependency propagation path information in the fault context are parsed, and a recovery dependency subgraph is constructed based on the node dependency topology graph to represent the dependency constraint relationship between the affected nodes. In this embodiment, it should be noted that the drive recovery module first extracts the set of affected data processing nodes and their dependency propagation path information from the fault context information, and uses the affected data processing nodes as the initial range of the node set; it then filters the upstream and downstream dependency edges associated with the affected data processing nodes from the node dependency topology graph, retaining only the nodes within the fault propagation path range and their directed dependencies, thereby forming a recovery dependency subgraph containing only fault-related paths. This limits the recovery range and characterizes the recovery dependency order relationship between nodes.

[0052] The recovery prerequisites are determined based on the directed dependencies in the recovery dependency subgraph, wherein any affected data processing node can only enter the recovery execution phase after the data object corresponding to the upstream dependent node has been recovered. Specifically, for any affected data processing node, trace its upstream node backward along the directed edge in the recovery dependency subgraph. If there is a dependency relationship from the upstream node to the current node, the recovery completion status of the upstream node is used as a prerequisite for the current node. When all upstream dependent nodes have completed data recovery and entered the available state, it is determined that the current node meets the recovery prerequisite, thus allowing it to enter the recovery execution phase.

[0053] For example, in the case of a dependency relationship A→B→C, node B can only enter the recovery process after node A fails and completes recovery, and node C can only perform recovery operations after node B has recovered, thus ensuring that the recovery process strictly follows the data dependency topology order.

[0054] Based on meeting the prerequisites for recovery, each affected data object is associated with its corresponding comprehensive disaster recovery value score, and the recovery execution order is determined according to the comprehensive disaster recovery value score from high to low. Then, according to the backup level of each data object, the corresponding backup plan is called, the corresponding backup copy is obtained from the backup storage, and data rollback or version reconstruction operations are performed to complete the node data recovery.

[0055] The preset failure rate threshold is set by staff based on historical data analysis results. This historical analysis process includes the system collecting multiple failure rates and calculating their average value as a reference to obtain the preset failure rate threshold. Similarly, the preset comprehensive disaster recovery value score threshold is also set by staff based on the system's historical operating data and specific application scenario requirements, and is adjusted by staff during system operation according to the actual situation.

[0056] This embodiment introduces a combined modeling mechanism of directly dependent nodes and indirectly dependent nodes within a preset hop count range, enabling the business association depth score to comprehensively reflect the actual impact range of data objects in a multi-level topology. By introducing a preset indirect dependency discount coefficient, the impact of indirect dependent nodes is differentially attenuated, ensuring a reasonable weight distribution of different levels of dependencies in business impact calculation. By introducing a preset hop count range to constrain the indirect dependency propagation path, the introduction of invalid remote dependent nodes is effectively avoided, improving computational stability and reducing redundant computational overhead. Through normalization of the total number of nodes, the business association depth score under different scale topologies has a unified dimension, improving the comparability of data object importance assessment results between different systems. This quantitative result of business association depth can be directly used for subsequent comprehensive disaster recovery value scoring calculation, thereby improving the accuracy and rationality of backup strategy grading and recovery priority ranking.

[0057] By weighting and fusing business relevance depth scores and non-reproducibility scores, the disaster recovery value score simultaneously reflects the business importance of data and the difficulty of reconstruction, improving the completeness of the assessment results. Introducing a non-reproducibility score assigns higher weight to data objects that cannot be regenerated from upstream data, thereby accurately identifying truly irreplaceable critical data. This comprehensive scoring mechanism provides a continuously separable quantitative basis for different data objects in backup level classification and recovery priority ranking, avoiding decision distortions caused by traditional binary or coarse-grained classification methods. Example

[0058] Please see Figure 2 As shown, for parts not described in detail in this embodiment, please refer to the description in Embodiment 1. This embodiment provides an intelligent disaster recovery and backup method for multi-node data processing, including: S1. Obtain the processing dependencies between each data processing node, construct a node dependency topology graph, and determine the data objects of each data processing node and the corresponding set of directly dependent nodes and the set of indirectly dependent nodes within a preset hop count range. S2. Calculate the business association depth score of the data object based on the set of directly dependent nodes and the set of indirectly dependent nodes, and calculate the non-renewability score in combination with the renewability attribute of the data object. Determine the comprehensive disaster recovery value score based on the business association depth score and the non-renewability score. S3. Based on the comprehensive disaster recovery value score, map each data object to the corresponding backup level, and generate a backup plan that includes snapshot method, backup cycle and copy retention strategy according to the backup level. S4. Monitor the running status of each data processing node. After detecting abnormal running status, identify the affected data objects based on the node dependency topology graph and generate fault context information. S5. Based on the fault context information, node dependency topology diagram and comprehensive disaster recovery value score, determine the preconditions for recovery and the recovery execution order, and call the corresponding backup plan to complete the node data recovery. Example

[0059] This embodiment discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the operation mode of the intelligent disaster recovery backup system for multi-node data processing described above.

[0060] Since the electronic device described in this embodiment is the electronic device used to implement the intelligent disaster recovery backup and recovery system and method for multi-node data processing in the embodiments of this application, those skilled in the art can understand the specific implementation methods and various variations of the electronic device in this embodiment based on the intelligent disaster recovery backup and recovery system and method for multi-node data processing described in the embodiments of this application. Therefore, how the electronic device implements the method in the embodiments of this application will not be described in detail here. Any electronic device used by those skilled in the art to implement the intelligent disaster recovery backup and recovery system and method for multi-node data processing in the embodiments of this application falls within the scope of protection of this application.

[0061] It should be noted that all formulas in this manual are calculated by removing dimensions and taking their numerical values. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters and thresholds in the formulas are set by those skilled in the art according to the actual situation.

[0062] The above description is merely a preferred embodiment of the present invention, and the scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for users of ordinary technical skills, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. An intelligent disaster recovery and backup system for multi-node data processing, characterized in that, include: The dependency resolution module is used to obtain the processing dependencies between data processing nodes, construct a node dependency topology graph, and determine the data objects of each data processing node and the corresponding set of directly dependent nodes and the set of indirectly dependent nodes within a preset hop count range. The disaster recovery assessment module is used to calculate the business association depth score of data objects based on the set of directly dependent nodes and the set of indirectly dependent nodes, and to calculate the non-renewability score in combination with the renewability attribute of the data objects. The comprehensive disaster recovery value score is determined based on the business association depth score and the non-renewability score. The backup orchestration module is used to map each data object to the corresponding backup level based on the comprehensive disaster recovery value score, and generate a backup plan that includes snapshot method, backup cycle and copy retention strategy according to the backup level. The anomaly monitoring module is used to monitor the running status of each data processing node. After detecting an anomaly in the running status, it identifies the affected data objects based on the node dependency topology graph and generates fault context information. The driver recovery module is used to determine the preconditions for recovery and the recovery execution order based on the fault context information, node dependency topology diagram and comprehensive disaster recovery value score, and call the corresponding backup plan to complete the node data recovery.

2. The intelligent disaster recovery and backup system for multi-node data processing according to claim 1, characterized in that, The method for constructing the node-dependent topology graph includes: Collect the operation records generated during the operation of each data processing node. The operation records include transaction logs, task scheduling records, interface call records and data read and write records. Extract the node identifier, data object identifier and timestamp from the operation records. Based on the output and read correspondence of the same data object across different data processing nodes, and the task call relationship between different data processing nodes, determine the processing dependency relationship between upstream and downstream data processing nodes; Each data processing node is used as a graph node, and processing dependencies are used as directed edges, with the direction of the directed edges pointing from the upstream data processing node to the downstream data processing node. All graph nodes and directed edges are associated and organized to generate a node dependency topology graph.

3. The intelligent disaster recovery and backup system for multi-node data processing according to claim 2, characterized in that, The method for determining the data objects of each data processing node includes: Based on the node dependency topology, read the output records of each data processing node, and determine the data files, data tables, cached results and message results generated and output by the corresponding data processing node as the data objects of the data processing node. Establish the association between each data object and the data processing node that generates the data object. For any target data object, obtain the upstream neighboring node corresponding to the incoming edge of the data processing node associated with the target data object in the node dependency topology graph, and determine the set of upstream neighboring nodes as the set of direct dependent nodes of the target data object. Starting with the data processing node associated with the target data object, the graph is traversed in the reverse direction of the directed edges in the node dependency topology graph. The number of edges traversed during the traversal is accumulated and used as the hop count. Within the preset hop count range, the remaining reachable upstream nodes other than the set of directly dependent nodes are obtained, and the set of the remaining reachable upstream nodes is determined as the set of indirect dependent nodes of the target data object.

4. The intelligent disaster recovery and backup system for multi-node data processing according to claim 3, characterized in that, The method for calculating the business association depth score of the data object includes: For any target data object, obtain the set of directly dependent nodes and the set of indirectly dependent nodes within a preset hop count range corresponding to the target data object, and count the total number of nodes in the node dependency topology graph; calculate the number of nodes in the set of directly dependent nodes and the number of nodes in the set of indirectly dependent nodes respectively. The number of indirect dependent nodes is reduced by a preset indirect dependency discount factor. The number of direct dependent nodes is added to the number of indirect dependent nodes after the reduction. The sum is then normalized using the total number of nodes to obtain the business association depth score of the target data object.

5. The intelligent disaster recovery and backup system for multi-node data processing according to claim 4, characterized in that, The method for determining the comprehensive disaster recovery value score includes: For any target data object, obtain the regenerable attribute of the target data object; when the target data object cannot be regenerated by the upstream data object, set the regenerable flag of the target data object to 1 and set the non-regenerability score to 1; when the target data object can be regenerated by the upstream data object, set the regenerable flag of the target data object to 0. Calculate the regeneration cost required to regenerate the target data object, obtain the maximum regeneration cost among all regenerable data objects in the system, and determine the non-regenerability score of the target data object using the ratio of the regeneration cost to the maximum regeneration cost; normalize the business association depth score and non-regenerability score of the target data object, and perform a weighted summation according to preset weight coefficients to obtain the comprehensive disaster recovery value score of the target data object.

6. The intelligent disaster recovery and backup system for multi-node data processing according to claim 5, characterized in that, The method for mapping each data object to a corresponding backup level includes: Several backup levels are pre-set, and a corresponding scoring range is configured for each backup level. The scoring ranges are arranged in order of comprehensive disaster recovery value score from low to high. For any target data object, the comprehensive disaster recovery value score of the target data object is obtained, and the comprehensive disaster recovery value score is compared with each scoring range. When the comprehensive disaster recovery value score falls into a certain score range, the target data object is mapped to the backup level corresponding to that score range. The backup levels include the first backup level, the second backup level, and the third backup level, and the data protection strength corresponding to the first backup level, the second backup level, and the third backup level increases sequentially.

7. The intelligent disaster recovery and backup system for multi-node data processing according to claim 6, characterized in that, The method for generating the backup scheme includes: A mapping table between backup levels and backup strategy parameters is pre-established, where each backup level corresponds to a different snapshot method type, backup execution cycle, and copy retention strategy. For any target data object, after determining the corresponding backup level, the snapshot method, backup cycle, and copy retention strategy corresponding to the backup level are obtained from the mapping table. When the target data object is at the first backup level, the snapshot method is configured as a full snapshot, the backup period is configured as the first period parameter, and the replica retention policy is configured as the first retention parameter; when the target data object is at the second backup level, the snapshot method is configured as an incremental snapshot, the backup period is configured as the second period parameter, and the replica retention policy is configured as the second retention parameter. When the target data object is at the third backup level, the snapshot method is configured as a high-frequency incremental snapshot method, the backup period is configured as the third period parameter, and the copy retention policy is configured as the third retention parameter; and the obtained snapshot method, backup period and copy retention policy are combined to generate the corresponding backup scheme.

8. The intelligent disaster recovery and backup system for multi-node data processing according to claim 7, characterized in that, The method for generating fault context information includes: The system continuously acquires the running status information of each data processing node, including heartbeat status information, task execution feedback information, interface call response information, and data read / write log information. The running status of the node is determined based on the running status information. When the running status of any data processing node meets the preset abnormal judgment conditions, the running abnormality of that data processing node is determined to have occurred. After identifying the abnormal data processing node, the module propagates and traverses downstream nodes along the data flow direction based on the node dependency topology graph to identify all data processing nodes that are directly or indirectly dependent on the abnormal data processing node and the corresponding affected data objects. After completing the identification of the scope of impact, the anomaly monitoring module automatically generates fault context information.

9. The intelligent disaster recovery and backup system for multi-node data processing according to claim 8, characterized in that, The method for invoking the corresponding backup scheme to complete node data recovery includes: The abnormal data processing node identifier, the set of affected data objects, and the dependency propagation path information in the fault context are parsed, and a recovery dependency subgraph is constructed based on the node dependency topology graph to represent the dependency constraint relationship between the affected nodes. The recovery prerequisites are determined based on the directed dependencies in the recovery dependency subgraph, wherein any affected data processing node can only enter the recovery execution phase after the data object corresponding to the upstream dependent node has been recovered. Based on meeting the prerequisites for recovery, each affected data object is associated with its corresponding comprehensive disaster recovery value score, and the recovery execution order is determined according to the comprehensive disaster recovery value score from high to low. Then, according to the backup level of each data object, the corresponding backup plan is called, the corresponding backup copy is obtained from the backup storage, and data rollback or version reconstruction operations are performed to complete the node data recovery.

10. A method for intelligent disaster recovery and backup of multi-node data processing, implemented by an intelligent disaster recovery and backup system for multi-node data processing as described in any one of claims 1 to 9, characterized in that, include: S1. Obtain the processing dependencies between each data processing node, construct a node dependency topology graph, and determine the data objects of each data processing node and the corresponding set of directly dependent nodes and the set of indirectly dependent nodes within a preset hop count range. S2. Calculate the business association depth score of the data object based on the set of directly dependent nodes and the set of indirectly dependent nodes, and calculate the non-renewability score in combination with the renewability attribute of the data object. Determine the comprehensive disaster recovery value score based on the business association depth score and the non-renewability score. S3. Based on the comprehensive disaster recovery value score, map each data object to the corresponding backup level, and generate a backup plan that includes snapshot method, backup cycle and copy retention strategy according to the backup level. S4. Monitor the running status of each data processing node. After detecting abnormal running status, identify the affected data objects based on the node dependency topology graph and generate fault context information. S5. Based on the fault context information, node dependency topology diagram and comprehensive disaster recovery value score, determine the preconditions for recovery and the recovery execution order, and call the corresponding backup plan to complete the node data recovery.

Images