A data disaster backup and recovery drill method and system

Abstract

A data disaster backup and recovery drill method and system, wherein the method comprises: acquiring service node business traffic data, a plurality of calling service node calling relationships and data flow transfer records; constructing a service dependency directed graph, generating a calling service node hierarchical table; completing environment initialization in a preset drill environment; establishing a traffic mirroring channel in a target business environment, copying read-only business traffic in the target business environment to the preset drill environment in real time through the traffic mirroring channel; acquiring a first data change snapshot of the target business environment and a second data change snapshot in the preset drill environment; generating a data consistency verification baseline table; analyzing the data consistency verification baseline table to determine a data inconsistency event, updating the service dependency directed graph based on the data inconsistency event, and generating a comprehensive drill report. The application can improve the accuracy of data disaster backup and recovery drills.

Description

Technical Field

[0001] This application relates to the field of data disaster drill technology, specifically to a data disaster backup and recovery drill method and system. Background Technology

[0002] As enterprise information systems continue to expand in scale and become increasingly complex in architecture, data disaster backup and recovery capabilities have become crucial for ensuring business continuity. This is especially true in industries with extremely high data security requirements, such as finance and e-commerce, where regular disaster recovery drills have become an industry standard and regulatory requirement.

[0003] In existing technologies, disaster recovery drills are conducted using full data replication and environmental isolation. Specifically, the data in the production environment is first fully backed up to the drill environment, and then a failure scenario is simulated and the recovery process is executed in the drill environment.

[0004] However, modern distributed systems have complex dependencies between services, especially when there are numerous asynchronous calls and distributed transactions. This can lead to microsecond-level inconsistencies in the data state between different service nodes. While these subtle inconsistencies can usually be automatically corrected during normal business operations, they can cause services to fail to start or business processes to malfunction in disaster recovery scenarios. This reduces the accuracy of data disaster backup and recovery drills. Summary of the Invention

[0005] This application provides a data disaster backup and recovery drill method and system to improve the accuracy of data disaster backup and recovery drills.

[0006] The first aspect of this application provides a data disaster backup and recovery drill method, which includes: acquiring business traffic data of each calling service node in a target business environment, the calling relationship between multiple calling service nodes, and data flow records; constructing a service dependency directed graph based on the business traffic data, the calling relationship, and the data flow records, and generating a calling service node hierarchical table based on the service dependency directed graph, wherein each calling service node in the calling service node hierarchical table corresponds to a hierarchical number; in a preset drill environment, sequentially starting each calling service node to complete environment initialization according to the hierarchical number in the calling service node hierarchical table in ascending order; establishing a traffic mirroring channel in the target business environment, and transmitting the traffic data... Read-only business traffic in the target business environment is replicated in real time to the preset training environment through the traffic mirroring channel. The preset training environment is isolated from the target business environment. A first data change snapshot of the target business environment and a second data change snapshot of the preset training environment are obtained. The first data change snapshot and the second data change snapshot are data state change results including synchronization identifiers triggered by the same business request in their respective environments during the traffic replication process. Based on the first data change snapshot and the second data change snapshot, a data consistency verification baseline table is generated. The data consistency verification baseline table is analyzed to determine data inconsistency events. Based on the data inconsistency events, the service dependency directed graph is updated, and a comprehensive training report is generated.

[0007] Optionally, generating a hierarchical table of calling service nodes based on the service dependency directed graph specifically includes: calculating the in-degree and out-degree values ​​of each calling service node based on the service dependency directed graph; determining the level number of each calling service node based on the in-degree and out-degree values; and generating a hierarchical table of calling service nodes based on the level number.

[0008] Optionally, based on the in-degree value and the out-degree value, the level number of each calling service node is determined, specifically including: repeatedly executing steps S1-S3: S1, determining the calling service node in the service dependency directed graph that has not been assigned a level number and has an in-degree value of zero as the current level node; S2, assigning the current count value of the level number counter to all the current level nodes, with the initial value of the level number counter being one; S3, for each current level node, traversing all outgoing call relationships of the current level node, decrementing the in-degree value of the called node pointed to by each outgoing call relationship by one, incrementing the count value of the level number counter by one, and re-executing step S1 until the level number of each calling service node is determined.

[0009] Optionally, the method further includes: when the current count value of the level number counter is greater than a preset iteration threshold, combining all target calling service nodes in the service dependency directed graph whose level numbers are not determined into a loop dependency node set; traversing all outgoing call relationships of each target calling service node in the loop dependency node set, and based on the business traffic data, determining the outgoing call relationship corresponding to the minimum business traffic value as the target disconnect relationship; decrementing the in-degree value of the called node pointed to by the target disconnect relationship by one, so as to logically disconnect the loop dependency structure.

[0010] Optionally, a data consistency verification baseline table is generated based on the first data change snapshot and the second data change snapshot, specifically including: determining the first data change snapshot and the second data change snapshot with the same synchronization identifier as a data change snapshot pair; calculating the first data checksum of the first data change snapshot and the second data checksum of the second data change snapshot in each data change snapshot pair; associating the synchronization identifier, the first data checksum, and the second data checksum of the data change snapshot pair to generate a data consistency verification baseline table.

[0011] Optionally, data inconsistency events include permanent data inconsistency events and instantaneous over-threshold inconsistency events. The step of analyzing the data consistency verification baseline table to determine data inconsistency events specifically includes: identifying data change snapshot pairs in the data consistency verification baseline table where the first data checksum differs from the second data checksum as initial inconsistency snapshot pairs; for each initial inconsistency snapshot pair, based on the synchronization identifier of the initial inconsistency snapshot pair, extracting all associated data change snapshot pairs triggered by the same business request from the data flow record, whose trigger time is after the initial inconsistency snapshot pair; combining the initial inconsistency snapshot pair with all associated data change snapshot pairs in chronological order to form an inconsistency event tracing chain; and determining the sequence of events in the inconsistency event tracing chain. The first data checksum and the second data checksum of the last data change snapshot pair are compared: if they are not the same, the calling service node corresponding to the first initial inconsistent snapshot pair in the inconsistency event tracing chain is identified as the root cause failure node, and the inconsistency event tracing chain is identified as a permanent data inconsistency event; if they are the same, the time span of the inconsistency event tracing chain from the first initial inconsistent snapshot pair to the last data change snapshot pair is calculated, and it is determined whether the time span is greater than a preset business tolerance latency threshold; if so, the calling service node corresponding to the first initial inconsistent snapshot pair in the inconsistency event tracing chain is identified as a potential risk node, and the inconsistency event tracing chain is identified as an instantaneous threshold-exceeding inconsistency event.

[0012] Optionally, the method further includes: obtaining an input parameter sequence for algorithm decision-making in the target business environment; dividing the input parameter sequence into multiple input parameter sub-sequences based on multiple consecutive preset time windows, and obtaining a first algorithm decision result within each preset time window in the target business environment; executing the algorithm decision-making based on the multiple input parameter sub-sequences in the preset exercise environment to obtain a second algorithm decision result within each preset time window; calculating the decision difference value between the first algorithm decision result and the second algorithm decision result within the same preset time window; backtracking the system resource status data of the target time window, wherein the target time window is the preset time window in which the decision difference value is greater than a preset drift threshold; determining decision divergence factors based on the system resource status data, and updating the comprehensive exercise report based on the decision divergence factors.

[0013] In a second aspect, embodiments of this application provide a data disaster backup and recovery drill system, which includes: one or more processors and a memory; the memory is coupled to the one or more processors, and the memory is used to store computer program code, which includes computer instructions, and the one or more processors call the computer instructions to cause the data disaster backup and recovery drill system to perform the method described in the first aspect and any possible implementation thereof.

[0014] Thirdly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on a data disaster backup and recovery drill system, cause the data disaster backup and recovery drill system to perform the method described in the first aspect and any possible implementation thereof.

[0015] Fourthly, embodiments of this application provide a computer program product containing instructions that, when the computer program product is run on a data disaster backup and recovery drill system, cause the data disaster backup and recovery drill system to execute the method described in the first aspect and any possible implementation thereof.

[0016] In summary, one or more technical solutions provided in this application have at least the following technical effects or advantages:

[0017] 1. By constructing a directed graph of service dependencies and generating a hierarchical table of service call nodes, the calling relationships and startup order between different service nodes can be clearly defined, avoiding service initialization failures due to unmet dependencies. A traffic mirroring mechanism is introduced in the preset drill environment to copy read-only business traffic from the production environment to the isolated environment in real time, ensuring the drill process has realistic business characteristics without affecting the online system. Combined with data change snapshots triggered by the same business request in both environments, a data consistency verification baseline table is further established to identify and track microsecond-level data state differences. This effectively detects data inconsistency events caused by asynchronous calls or distributed transactions, and dynamically updates the service dependency model accordingly, improving the accuracy of data disaster backup and recovery drills.

[0018] 2. Based on the constructed directed graph of service dependencies, the in-degree and out-degree values ​​of each calling service node are further calculated. This allows for accurate identification of the position and role of each service node within the overall dependency structure. Then, by repeatedly executing a layered processing procedure based on the in-degree value—that is, identifying nodes with an in-degree of zero and no assigned layer number as current-layer nodes and assigning them a current-layer number, while traversing all outgoing call relationships and dynamically updating the in-degree value of the called nodes—the layering process is progressively completed, ensuring strict topological order and dependency integrity. Since this layering process is built upon real dependency relationships, and the generation of layer numbers involves progressively stripping away nodes with an in-degree of zero while dynamically advancing the layer division using an in-degree value update mechanism, it effectively avoids startup failures or execution anomalies caused by unmet dependencies during service initialization. Furthermore, this mechanism does not rely on manual configuration or static rule judgments but is entirely based on automatic calculation of the actual call graph driven by business traffic data, resulting in higher accuracy and adaptability.

[0019] 3. By comparing the checksums of the two environments in the data consistency verification baseline table, the initial inconsistent snapshot pairs are identified, and an inconsistency event tracing chain is formed based on the synchronization identifier, thus achieving complete link tracing of data inconsistency issues. By determining whether the link eventually restores consistency and whether the recovery time exceeds the business tolerance threshold, data inconsistency events can be classified as permanent or transient threshold-exceeding anomalies, and related service nodes can be located as root cause nodes or potential risk nodes. This not only improves the accuracy and classification capability of data consistency issues but also enhances the pertinence of fault location and risk assessment, providing strong support for anomaly analysis and dependency optimization in disaster recovery drills. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating a data disaster backup and recovery drill method disclosed in an embodiment of this application;

[0021] Figure 2 This is another flowchart illustrating a data disaster backup and recovery drill method disclosed in an embodiment of this application;

[0022] Figure 3 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of this application.

[0023] Explanation of reference numerals in the attached drawings: 301, Central Processing Unit; 302, Read-Only Memory; 303, Random Access Memory; 304, Bus; 305, Input / Output Interface; 306, Input Section; 307, Output Section; 308, Storage Section; 309, Communication Section; 310, Driver; 311, Removable Media. Detailed Implementation

[0024] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0025] In the description of the embodiments of this application, words such as "for example" or "for instance" are used to indicate examples, illustrations, or explanations. In the description of the embodiments of this application, the term "multiple" means two or more. The terms "comprising," "including," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0026] Figure 1 This is a flowchart illustrating a data disaster backup and recovery drill method according to an embodiment of this application. Please refer to... Figure 1 This application provides a data disaster backup and recovery drill method, the method comprising:

[0027] S101. Obtain the business traffic data of each calling service node in the target business environment, the calling relationship between multiple calling service nodes, and the data flow record;

[0028] In this embodiment, the target business environment refers to an online production system that is currently in full operation and carries actual business requests. It typically includes multiple business service nodes deployed in a distributed architecture, a database system, middleware, and network communication components. This environment provides complete business capability support and serves as the object system for simulation and reproduction required for disaster backup and recovery drills. A service node refers to a service unit deployed in the target business environment that possesses independent service capabilities and participates in the business processing flow. These nodes typically exist in the form of microservices, with each node responsible for handling specific business function modules, such as user authentication services, order processing services, payment gateway services, and inventory management services. Service nodes interact with each other through remote procedure calls (RPC), message queues, etc., forming the call chain of the business system.

[0029] In practice, business traffic data can be collected through traffic collection modules deployed on service gateways, service proxies, or service registries. These modules can perform non-intrusive monitoring of inter-service communication based on a sidecar architecture or network-layer traffic mirroring technology (such as iptables forwarding rules), extracting business call logs containing information such as request entry points, target service identifiers, request parameters, response results, and timestamps. By performing structured parsing of these logs, business traffic data such as call frequency, call sequence, and path characteristics of each calling service node during the actual business flow can be obtained.

[0030] To accurately determine the call relationships between services, the call chain information recorded in the logs can be used to reconstruct the call chain, combined with a distributed tracing system (such as a consistent Trace ID mechanism based on OpenTelemetry). Trace IDs can be used to connect the call paths of a complete business request across multiple service nodes, thereby identifying the upstream and downstream dependencies of each service node in the business process.

[0031] Data flow records are used to represent the trajectory of data state changes caused by business requests during their transmission between service nodes. To achieve this, a lightweight data change monitoring component can be integrated into each calling service node. This component can record in real time the data primary key, operation type (such as insert), data summary before and after the change (such as hash value), and corresponding business request identifier involved in each write operation, based on database binlog parsing, cache operation hooks, or business logic middleware instrumentation.

[0032] S102. Based on the business traffic data, the call relationship and the data flow record, construct a service dependency directed graph, and generate a call service node hierarchical table based on the service dependency directed graph. Each call service node in the call service node hierarchical table corresponds to a hierarchical number.

[0033] To reconstruct the actual dependency structure between services in the target business environment, a directed service dependency graph is first constructed based on the aforementioned collected business traffic data, call relationships, and data flow records. Specifically, the business traffic data includes information such as the source service node identifier, target service node identifier, request path, timestamp, request identifier, and response status for each call request; the call relationship refers to the call behavior of a service node to other service nodes during a complete business process, forming a service call chain through request chain tracing technology (e.g., based on the Trace ID mechanism); and the data flow records reflect the data state change trajectory caused by each service node processing the request during the call process, including fields such as data primary key, change content summary, belonging service node, and request identifier. When constructing the directed service dependency graph, each calling service node is abstracted as a vertex in the graph, and directed edges are added according to the call relationships to represent the call direction between services. If service A calls service B, a directed edge from node A to node B is added to the graph. Meanwhile, in order to enhance the ability of the service dependency graph to express the direction of data flow, the time series characteristics of upstream requests triggering downstream data changes reflected in the data flow records can be combined to further confirm the strength and directionality of the dependency relationship, thereby improving the accuracy of the dependency graph and its ability to express business semantics.

[0034] After the service dependency directed graph is constructed, a topology analysis operation is performed based on the graph to generate a hierarchical table of calling service nodes. Specifically, this includes the following steps: calculating the in-degree and out-degree values ​​of each calling service node based on the service dependency directed graph; determining the level number of each calling service node based on the in-degree and out-degree values; and generating a hierarchical table of calling service nodes based on the level number.

[0035] To provide a structural basis for the hierarchical numbering of service nodes, it is necessary to perform structural analysis on the completed service dependency directed graph, and then calculate the in-degree and out-degree values ​​of each calling service node in the graph. The service dependency directed graph is a graph structure with calling service nodes as vertices and the calling relationships between services as directed edges, where the direction of the edge represents the calling direction of the service, that is, from the caller to the callee.

[0036] The in-degree value represents the number of directed edges in the graph pointing to a specific calling service node, reflecting the degree to which that node is depended on by other services in the business processing path. The out-degree value represents the number of directed edges originating from that node and pointing to other service nodes, reflecting the number of other services that node depends on. By calculating the in-degree and out-degree, the structural position of each service in the system can be clearly determined, whether it depends on the starting point, intermediate nodes, or terminal nodes.

[0037] In practice, the in-degree and out-degree values ​​can be calculated by traversing all edges in the directed graph of service dependencies. During the traversal, for each directed edge (A→B), it indicates that service node A has called service node B, so the out-degree value of service node A is incremented by one, and the in-degree value of service node B is incremented by one. After the traversal is complete, the complete in-degree and out-degree values ​​for each calling service node can be obtained.

[0038] To illustrate with a simplified example: Suppose there are service calling nodes A, B, C, and D, where A calls B and C, B calls D, and C calls D. The dependent directed graph contains edges (A→B), (A→C), (B→D), and (C→D). In this structure, node A has an in-degree of 0 and an out-degree of 2, B has an in-degree of 1 and an out-degree of 1, C has an in-degree of 1 and an out-degree of 1, and D has an in-degree of 2 and an out-degree of 0.

[0039] The calculation of in-degree and out-degree values ​​not only provides key conditions for subsequent service layering but also reveals potential critical path nodes, highly dependent nodes, and possible loop risks within the system. After calculating the in-degree and out-degree values ​​of each calling service node based on the service dependency directed graph, to further clarify the startup order of each service node in the overall dependency structure, all calling service nodes are layered and numbered. This can specifically include the following steps:

[0040] Repeat steps S1-S3 until the level number of each calling service node is determined: S1, determine the calling service node in the service dependency directed graph that has no assigned level number and an in-degree value of zero as the current level node; S2, assign the current count value of the level number counter to all current level nodes, and the initial value of the level number counter is one; S3, for each current level node, traverse all outgoing call relationships of the current level node, decrement the in-degree value of the called node pointed to by each outgoing call relationship by one, and increment the count value of the level number counter by one.

[0041] In the specific implementation process, to achieve ordered layering of calling service nodes, it is necessary to first identify calling service nodes with no assigned layer number and an in-degree value of zero in the directed graph of service dependencies as nodes of the current layer. Here, "in-degree value of zero" means that the node has no upstream node calling it in the dependency graph. It is usually located at the beginning of the service dependency path and is the first service node called in the system, possessing natural initialization priority. The core purpose of this step is to accurately identify the group of service nodes that do not depend on any other uninitialized nodes in each round of layering operations, as a batch of nodes that can be started at the moment. During implementation, this can be achieved by traversing the set of nodes in the graph that have not yet been labeled with layer numbers, filtering out nodes with an in-degree value of zero, and forming the set of nodes of the current layer. This set will be uniformly assigned the current layer number in subsequent steps and removed from the dependency graph to advance the next round of layering operations. For example, in a dependency graph containing service nodes A, B, and C, if A is not called by any service, its in-degree value is zero, and it can be identified as a node of the current layer in the first round.

[0042] After identifying the current level node set, the next step is to uniformly number these nodes, that is, to assign the current count value of the level number counter to all current level nodes. The level number counter is an ordered integer variable used to record the currently assigned level, and its initial value is set to one, representing the first level in the service initialization order. By batch assigning the current count value to the current level nodes identified in this round, the service layer numbering is automated and structured, so that each service node has a unique level number in the final layer table that corresponds to the dependency order. This operation can be performed by traversing the current level node set and uniformly updating its level number field to the current count value. After the numbering is completed, the level identifier of each node is determined, providing a data foundation for the subsequent service startup order. For example, service A, which was identified as a current level node in the previous round, will be assigned level number 1, marking it as the starting node of the startup chain.

[0043] After assigning node numbers to the current level nodes, to further peel away the dependency graph and identify nodes in the next level, it is necessary to traverse all outgoing call relationships of the current level nodes. Specifically, for each current level node, traverse the target nodes connected to its outgoing edges (i.e., the downstream service nodes it calls), and decrement the in-degree value of these called nodes by one. The principle behind this operation is that the current level node has already been assigned a level number in this round, meaning it has been "initialized," and therefore its dependencies on downstream nodes can be considered "satisfied," and the corresponding in-degree value can be reduced. By decrementing the in-degree value of the target node in each outgoing call relationship, the structural state of the remaining nodes in the dependency graph can be dynamically updated, providing a basis for identifying nodes with an in-degree of zero in the next round. After processing the outgoing call relationships, increment the level number counter by one to enter the next round of layering iteration. For example, if service A was assigned level number 1 in the previous round, and service A calls services B and C, then the in-degree values ​​of services B and C will each decrease by one. If their in-degree values ​​become zero at this time, they will be identified as new current level nodes in the next round and assigned level number 2.

[0044] Through iterative execution of the above steps, each calling service node in the service dependency graph will be identified as the current level node and assigned a corresponding level number in a certain round, ultimately constructing a complete calling service node hierarchy table. This hierarchy table not only accurately reflects the service startup order but can also be used as a process control basis during the initialization of the exercise environment to ensure that service startup conforms to dependency logic and avoid dependency failures or business anomalies caused by incorrect service initialization order.

[0045] In the actual process of constructing the service node hierarchy table, if a circular dependency structure exists in the directed graph of service dependencies, the in-degree value of all nodes may never be zero, thus preventing further progress in the hierarchy numbering iteration process and causing the hierarchy logic to fall into an infinite loop. To solve this problem, this method further sets up an iteration protection mechanism. When the current count value of the hierarchy number counter exceeds a preset iteration threshold, it is determined that there is an unsolvable circular dependency relationship in the current dependency graph, and all service nodes that have not yet completed hierarchy numbering are uniformly identified as a set of circular dependency nodes. To break this loop and allow the hierarchy process to continue, it is necessary to analyze the call intensity of the nodes within the loop based on business traffic data, select the outbound call relationship with the least business impact as the target disconnect relationship, and disconnect this relationship logically to reduce the in-degree value of the called node. Specifically, this may include the following steps: when the current count value of the level number counter is greater than a preset iteration threshold, combine all target calling service nodes in the service dependency directed graph whose level numbers are not determined into a loop dependency node set; traverse all outgoing call relationships of each target calling service node in the loop dependency node set, and based on the business traffic data, determine the outgoing call relationship corresponding to the minimum business traffic value as the target disconnect relationship; decrement the in-degree value of the called node pointed to by the target disconnect relationship by one, so as to logically disconnect the loop dependency structure.

[0046] In this embodiment, to prevent the layering operation from failing to terminate due to loop structures in the service dependency directed graph during the layering process, an iteration threshold is set as an anomaly detection mechanism. The layer number counter is an integer variable used to mark the current layering round; its initial value is one, and it increments by one after each layering round. The iteration threshold is primarily based on the number of calling service nodes in the service dependency directed graph. Typically, this threshold is set to a constant multiple (e.g., 1.5 or 2 times) of the number of nodes to ensure that the number of iterations is sufficient to complete the layering of all nodes during normal topological sorting of the acyclic graph. Once this threshold is exceeded, an abnormal circular dependency is considered to exist, thus triggering a loop detection and disconnection mechanism in a timely manner to prevent the algorithm from getting stuck in infinite iteration. If the current value of the counter exceeds the preset iteration threshold, the system determines that there is a circular call chain in the current dependency graph that cannot be normally topologically sorted. At this point, to continue the layering process, all calling service nodes in the dependency graph that have not yet been assigned a layer number are considered as a potentially circular dependency set and uniformly combined into a circular dependency node set. The service nodes in this node set have direct or indirect call loops with each other, causing their in-degree values ​​to never reach zero. Without intervention, this will lead to an infinite loop in the topology sorting. This operation can be performed by scanning the node list where the level number field is still empty and merging them into a set based on their connections in the dependency graph. For example, if node A calls B, B calls C, and C calls A, and none of the three are numbered and all have mutual call relationships, then this is determined to be a cyclic dependency node set.

[0047] After identifying the loop-dependent node set, to "disconnect" the loop structure, it is necessary to further analyze the strength of the call relationships between service nodes within the set to minimize the impact of the disconnection operation on business logic. Therefore, this embodiment uses business traffic data for quantitative judgment. Business traffic data is the service call behavior log collected in real-time from the target business environment in step S101, recording indicators such as call frequency, average response time, and call success rate for each call path. By traversing all outgoing call relationships of each service node in the loop-dependent node set, the call frequency of each call edge is extracted as the business traffic value, and then the call relationship with the lowest business call frequency is identified and determined as the target disconnection relationship. The principle of this selection strategy is that disconnecting the edge with the lowest business frequency causes the least business impact, maximizing the integrity of the business structure while breaking the loop. For example, in a loop composed of nodes A, B, and C, if the call frequency of A→B is 500 times / minute, B→C is 120 times / minute, and C→A is only 10 times / minute, then C→A is selected as the target disconnection relationship.

[0048] After selecting a target to disconnect, the call path needs to be logically disconnected. Specifically, the in-degree value of the called node pointed to by the relationship is decremented by one to simulate the effect of its upstream dependency being removed. Since the entire layering logic determines whether a node can be assigned a layer number based on its in-degree value, reducing the in-degree value allows service nodes that would otherwise be unable to become starting nodes due to circular dependencies to have the possibility of being numbered, thus restoring the continuity of the topology sorting. This logical disconnection operation does not affect the actual business system structure; it only takes effect during the dependency modeling phase of the initialization of the exercise environment, aiming to ensure the termination and robustness of the layering algorithm. Continuing with the example above, if C→A is disconnected, the in-degree value of A is reduced from 1 to 0, allowing it to be identified as a node of the current layer and assigned a layer number in the next round, thereby breaking the entire loop and propelling the layering process forward.

[0049] S103. In the preset training environment, each of the calling service nodes is started sequentially in ascending order of the hierarchical number in the calling service node hierarchy table to complete the environment initialization.

[0050] In this embodiment, the preset training environment refers to an isolated operating environment that is pre-built for data disaster backup and recovery drills and is highly consistent with the target business environment in terms of service architecture, deployment configuration, and data model. This environment does not share state or resources with the real production system, has independent service node deployment capabilities and data processing capabilities, and is mainly used to handle read-only business traffic copied from the target business environment. It also simulates the disaster recovery process without affecting actual business operations, and is used to verify the effectiveness of backups and system recovery capabilities.

[0051] To ensure that the pre-set training environment accurately replicates the real dependencies and operational states between services in the target business environment, it is necessary to start each service node sequentially according to the hierarchical numbers recorded in the service node hierarchical table during the environment initialization phase. The service node hierarchical table is a data structure generated after constructing the directed service dependency graph and executing the hierarchical algorithm. Each service node participating in the service call is assigned a unique hierarchical number; a smaller hierarchical number indicates that it is closer to the bottom of the service dependency chain and is called earlier by other service nodes. This hierarchical number is passed upwards from the starting node of the business call chain, reflecting the order of service calls and the direction of dependencies.

[0052] After deploying all service nodes in the pre-defined training environment, each service must be started layer by layer, starting with the service node with the smallest layer number, based on the hierarchical numbering provided by the hierarchical table. This startup process involves loading the corresponding service node into a running state and initializing its dependent basic resources, such as database connections, configuration file loading, cache preheating, and internal threads, enabling it to handle requests or respond to upstream dependent calls. Because there are explicit call dependencies between services, if the startup order does not match the dependency direction—for example, starting an intermediate layer or upstream service that depends on other services first—it will cause connection failures, service registration errors, or configuration fetching errors during the initialization phase due to the inability to access downstream services. This can lead to the failure of the entire training environment startup or interruption of the business chain.

[0053] This startup sequence control can be implemented through automated scripts. For example, after system deployment, the scheduling module can read the service node hierarchy table and submit each service node to a container orchestration system (such as Kubernetes) or service orchestration platform in batches according to the ascending hierarchy number. Only after each hierarchy has completed startup and passed health checks will the startup command for the next hierarchy be triggered. This method ensures that when any service starts, its dependent downstream services are already available, thus achieving a closed-loop construction and stable operation of the service dependency chain. Taking a simplified set of service nodes as an example, if node number 1 is "user authentication service," node number 2 is "order processing service," and node number 3 is "payment settlement service," then the startup order must strictly follow the sequence of authentication service → order service → payment service to ensure the integrity of the call chain logic between service nodes.

[0054] By employing the above methods, the pre-set training environment can accurately reproduce the service call structure and execution sequence of the target business environment during initialization, avoiding service anomalies caused by missing dependencies or incorrect startup order, and providing a stable operational foundation for subsequent traffic mirroring and data consistency verification. This layered startup mechanism not only improves the stability and controllability of the training environment, but also provides crucial support for the automated and structured implementation of disaster recovery solutions.

[0055] S104. Establish a traffic mirroring channel in the target business environment, and copy the read-only business traffic in the target business environment to the preset training environment in real time through the traffic mirroring channel. The preset training environment is isolated from the target business environment.

[0056] To achieve a high-fidelity reproduction of the actual business processing in the target business environment and to verify the consistency of data processing across service nodes under data disaster scenarios, it is necessary to establish a traffic mirroring channel in the target business environment. This channel replicates read-only business traffic generated during actual business operations to a pre-set training environment in real time. The read-only business traffic referred to here means that after replication to the pre-set training environment, any data change side effects (such as persistent writing to the database) generated during processing in that environment will be intercepted or redirected to non-production storage, thereby ensuring that the actual data state of the target business environment is not affected. A traffic mirroring channel is a logical replication path built based on network layer or service mesh technology. Its function is to bypass and replicate client-initiated request data flowing through multiple service nodes without interrupting or interfering with the original business process, and synchronously forward it to a training environment with the same structure but logical isolation. In this way, the training environment can receive input requests that are completely identical to those in the target business environment, thus ensuring the comparability of service behaviors, state changes, and data processing paths during the training process.

[0057] During implementation, a traffic replication component is first deployed at the network ingress of the target business environment. This component can be implemented based on a mirror proxy, service mesh (such as Istio), or load balancer extension module, and has the ability to perform shallow copies of business requests. As a business request enters the target business environment's processing flow, its copy is forwarded in real-time to the ingress node of the training environment, thus forming a bidirectional, independent but consistent input processing path. To ensure data security and business isolation, all service nodes in the training environment are configured in read-only mode, disallowing write operations to external systems or shared resources, and avoiding side effects on the real system caused by mirrored traffic processing.

[0058] Furthermore, the training environment and the target business environment do not share state resources such as databases, caches, and message queues. This ensures that even if service response logic differs during the training, it will not affect the accuracy and stability of actual business operations. This isolation principle is typically implemented through logical network isolation (such as virtual private network segments), resource access control, and service registry differentiation. For example, if the "order service" in the target business environment receives a user's order request, a copy of this request will be simultaneously sent to the "order service" copy node in the training environment via a traffic mirroring channel. Both will process the request independently, but the processing result in the training environment will not be written to the real database; it will only be used for subsequent data snapshot comparison and consistency verification analysis.

[0059] By establishing a traffic mirroring channel in the target business environment and forwarding read-only business traffic to the preset drill environment in real time, lossless collection and drill reproduction of real business behavior can be achieved without interfering with business operations. This provides a realistic, controllable, and comparable input data foundation for disaster recovery drills, greatly improving the accuracy of the drills and the credibility of the assessments. This mechanism not only enhances the practicality and verifiability of the drill plan but also provides complete contextual data support for subsequent data comparison and service dependency optimization.

[0060] S105. Obtain a first data change snapshot of the target business environment and a second data change snapshot of the preset training environment. The first data change snapshot and the second data change snapshot are data state change results including synchronization identifiers triggered by the same business request in their respective environments during the traffic replication process.

[0061] To systematically verify the processing capabilities, data consistency, and disaster recovery accuracy of service nodes in the pre-set training environment, it is necessary to collect data state change results triggered by the same business request from both the target business environment and the pre-set training environment, forming comparable data snapshots. This process is accomplished by obtaining a first data change snapshot and a second data change snapshot. The first data change snapshot refers to the record of data state changes caused by the actual business request during the processing flow in the target business environment; the second data change snapshot refers to the corresponding data state change record formed by the processing of the same business request by each service node after it is replicated to the pre-set training environment through a traffic mirroring channel. Both snapshots contain a synchronization identifier field to identify whether they were triggered by the same business request.

[0062] The core objective of this step is to compare the data output results generated in the two environments under the same business input conditions to determine whether the exercise environment can correctly reproduce the data change behavior in the target business environment, provided that the initialization of each service node, dependency construction, and data processing logic configuration remain consistent. To achieve effective snapshot acquisition, a data change acquisition module needs to be deployed in both environments. This module can be implemented based on database log monitoring, middleware change event recording, or application-level data write interception. It captures the primary key information, field changes before and after, operation timestamps, and source service node identifiers for each data write operation, and encapsulates them uniformly into structured snapshot data items.

[0063] Each data change snapshot also needs to carry a synchronization identifier to uniquely identify whether the change was triggered by the same business request. This synchronization identifier can be generated by the traffic mirroring module when replicating the request and attached to the request context to be passed to the service node of the exercise environment, enabling accurate tracking of the original request. In this way, it can be ensured that the snapshot data collected in the two environments have a one-to-one correspondence, thus providing a foundation for subsequent data consistency verification. During the operation, the system continuously monitors the data write behavior in both environments and periodically stores the collected change snapshots in the comparison cache, preparing for the next stage of baseline verification.

[0064] For example, if the "Order Service" in the target business environment receives a user's order request and updates the record status in the order table from "Pending Payment" to "Submitted," this change will be recorded as the first data change snapshot. Simultaneously, a copy of this request is transmitted to the exercise environment, and the "Order Service" copy node performs the same process. If it also updates the corresponding record status to "Submitted," the system will capture this operation as the second data change snapshot. By comparing the changed field values, target objects, operation paths, and timestamps of the two snapshots, it can be determined whether the exercise environment consistently reproduces the behavior of the target environment in terms of business processing logic.

[0065] S106. Generate a data consistency verification baseline table based on the first data change snapshot and the second data change snapshot;

[0066] After collecting data state change snapshots triggered by the same business request in both the target business environment and the preset exercise environment, it is necessary to construct a unified comparison benchmark based on these data change snapshots to verify the consistency of the processing results. This benchmark table serves as the core data structure for subsequent consistency analysis. By associating first and second data change snapshots with the same synchronization identifier, it records the differences and comparison criteria in the data processing results, thus providing structured support for identifying data inconsistency events. Specifically, the process of generating the data consistency verification baseline table includes combining each pair of data change snapshots with the same synchronization identifier into a data change snapshot pair and calculating their corresponding data checksums. This allows for quick determination of data content consistency through checksum comparison. The steps include: identifying first and second data change snapshots with the same synchronization identifier as data change snapshot pairs; calculating the first data checksum of the first data change snapshot and the second data checksum of the second data change snapshot in each data change snapshot pair; and associating the synchronization identifier, the first data checksum, and the second data checksum of the data change snapshot pair to generate the data consistency verification baseline table.

[0067] In this embodiment, to achieve accurate consistency verification of data change results generated when the target business environment and the preset training environment process the same business request, it is necessary to pair the first and second data change snapshots collected in the two environments based on the synchronization identifier, forming a one-to-one corresponding data change snapshot pair. The synchronization identifier is a unique identifier uniformly generated by the system and attached to the business request during the traffic mirroring process. Its function is to mark the data state changes triggered by the same business request in the two environments, thereby ensuring that the subsequent snapshot correspondence has semantic accuracy and business timing consistency. This identifier can be implemented in the form of a unique request ID, transaction number, or tracing link identifier generated by the service mesh.

[0068] Combining a first data change snapshot and a second data change snapshot with the same synchronization identifier into a data change snapshot pair is a prerequisite for constructing a data consistency verification baseline table. This step is implemented by establishing a snapshot pairing cache in the system, classifying and organizing snapshot data from the two environments, and using the synchronization identifier as the primary key for fast lookup and matching. When two snapshots are detected to originate from the target business environment and a preset training environment respectively, and the synchronization identifier field is the same, the system combines them into a structured data change snapshot pair and stores it in a temporary mapping table. In this way, a corresponding relationship can be established for the data changes triggered by each business request in the two environments, providing a one-to-one comparison basis for subsequent consistency comparisons. For example, in a user payment request process, if the system generates a synchronization identifier of "TX123456", then the change snapshots involving the order table and payment record table collected in both environments will carry this identifier, and the system can automatically identify and combine them into a snapshot pair.

[0069] After the snapshot pairs are constructed, to achieve fast and efficient data consistency comparison, it is necessary to further calculate the data checksum for each data change snapshot pair. A data checksum is a hash value used to compress and summarize data content, reflecting the uniqueness and integrity of the data content at a relatively low cost. In this embodiment, the first data checksum of the first data change snapshot and the second data checksum of the second data change snapshot are calculated by concatenating their key fields (such as the target primary key, the changed field, and the values ​​before and after the change) in a preset order and then inputting them into a hash function (such as MD5 or SHA-256). The principle behind this method is that if the content of two snapshots is completely identical, their checksums will necessarily be identical; if there are differences in fields, values, or order, the checksum results will show significant deviations, thus achieving efficient consistency judgment. This process can be completed in batches in the background asynchronous calculation module, ensuring good performance even in large-scale data change scenarios.

[0070] After completing the checksum calculation, to facilitate subsequent consistency verification and anomaly analysis, the synchronization identifier, first data checksum, and second data checksum of each data change snapshot pair need to be uniformly associated and written into the data consistency verification baseline table. This baseline table is a structured data comparison record table used to summarize and record whether the data change content of each snapshot pair is consistent in the two environments. Each record in the table contains three core fields: the synchronization identifier indicates the source of the business request corresponding to the record; the first data checksum reflects the processing result of the target business environment; and the second data checksum reflects the processing result of the preset drill environment. The system can perform consistency comparison analysis based on this table to determine whether the two checksums are the same. If they are inconsistent, the source and scope of the difference will be further traced. Taking the snapshot pair with "TX123456" as the synchronization identifier as an example, if both checksums are "9f1a2b3c…", it means that the corresponding business processing results are consistent; if one is "9f1a2b3c…" and the other is "7d4e6f8a…", it needs to be marked as a potential inconsistency event for subsequent analysis module processing.

[0071] S107. Analyze the data consistency verification baseline table, determine data inconsistency events, update the service dependency directed graph based on the data inconsistency events, and generate a comprehensive exercise report.

[0072] After generating the data consistency verification baseline table, in order to further evaluate the processing accuracy of the preset exercise environment in disaster backup and recovery scenarios, it is necessary to conduct a systematic analysis of the data change snapshot pairs in the baseline table to identify data inconsistency events.

[0073] Data inconsistency events refer to situations during disaster backup and recovery drills where, due to logical differences, system latency, or state asynchrony between the target business environment and the preset drill environment when processing the same business request, the data state changes resulting from the same business request are inconsistent in the two environments. To facilitate accurate identification, classification, and subsequent optimization, inconsistency events are further subdivided into two categories: permanent data inconsistency events and transient inconsistency events exceeding thresholds.

[0074] A permanent data inconsistency event refers to a situation where, after the entire processing flow of a business request has ended, the corresponding data states in the two environments fail to reach consistency. This typically indicates a logical flaw, configuration difference, or abnormal service behavior in the exercise environment that cannot be automatically converged. In implementation, this type of event is addressed by verifying and comparing the snapshot pairs at the tail of the inconsistency event tracing chain. If they still differ, it can be determined as a permanent inconsistency, and the service node corresponding to the first snapshot pair in the chain can be identified as the root cause failure node.

[0075] Transient inconsistency events exceeding the threshold are a type of transient data inconsistency, characterized by initial differences in processing results between two environments, which converge in subsequent stages of the business process. These events are common in systems employing eventual consistency models, such as those caused by message queues or asynchronous write mechanisms, resulting in brief data delays. To ensure system availability and user experience, these events require further evaluation to determine if their duration exceeds a preset business tolerance latency threshold. If it exceeds the threshold, it is identified as a transient "over-tolerance" inconsistency event, and the initial inconsistency node is marked as a potential risk node, indicating a performance bottleneck or potential processing delay.

[0076] This step not only objectively verifies the effectiveness of the exercise but also serves as a crucial basis for reproducing the root causes of real-world failures and identifying potential risk nodes. By comparing the first and second data checksums of each snapshot pair, inconsistent snapshot pairs with discrepancies can be initially screened out. Combined with data flow records, an inconsistency event tracing chain is constructed along the propagation path of business requests at each service node. Through time-series analysis and consistency judgment of the data state evolution process in the tracing chain, data inconsistency events can be further classified into permanent data inconsistency events and instantaneous over-threshold inconsistency events. Based on this, the root cause failure node or potential risk node can be located, thus providing a logical basis and decision support for the dynamic adjustment of the service dependency structure and the generation of a comprehensive exercise report. Analyzing the data consistency verification baseline table to determine data inconsistency events specifically includes the following steps: Data change snapshot pairs in the data consistency verification baseline table whose first data checksum differs from the second data checksum are identified as initial inconsistency snapshot pairs; for each initial inconsistency snapshot pair, based on the synchronization identifier of the initial inconsistency snapshot pair, all associated data change snapshot pairs triggered by the same business request and whose trigger time is after the initial inconsistency snapshot pair are extracted from the data flow record; the initial inconsistency snapshot pair and all associated data change snapshot pairs are combined into an inconsistency event tracing chain according to time order; and the last data change snapshot pair in the inconsistency event tracing chain is determined to have the following characteristics: If the first data checksum is different from the second data checksum, and they are different, then the calling service node corresponding to the first initial inconsistent snapshot pair in the inconsistent event tracing chain is identified as the root cause failure node, and the inconsistent event tracing chain is identified as a permanent data inconsistency event. If they are the same, then the time span of the inconsistent event tracing chain from the first initial inconsistent snapshot pair to the last data change snapshot pair is calculated, and it is determined whether the time span is greater than a preset business tolerance latency threshold. If it is, then the calling service node corresponding to the first initial inconsistent snapshot pair in the inconsistent event tracing chain is identified as a potential risk node, and the inconsistent event tracing chain is identified as an instantaneous threshold-exceeding inconsistent event.

[0077] In analyzing the data consistency verification baseline table to identify data inconsistency events, the primary task is to discover discrepancies in data change snapshot pairs. These snapshot pairs consist of a first data change snapshot and a second data change snapshot generated under the same business request in the target business environment and a preset training environment. They are uniquely paired using a synchronization identifier. Comparing the values ​​of the first and second data checksums is crucial for determining whether the data processing results are consistent. The data checksum is a digest value obtained by hashing key fields in the snapshot (such as changed fields, values ​​before and after the change, etc.). It can efficiently determine whether two sets of data are identical without viewing the complete data content. If the checksums differ, it indicates a difference in the processing results of the same business request between the two environments, and this data change snapshot pair is identified as an initially inconsistent snapshot pair.

[0078] To further trace and analyze the propagation path of data inconsistencies, it is necessary to start with each initial inconsistency snapshot pair and, based on its synchronization identifier, extract subsequent data change snapshot pairs related to the business request from the system's data flow records. Data flow records refer to snapshots of the service call chain and the processing results of each node, recorded chronologically as the business request flows from the entry service through each calling service node. Using the synchronization identifier of the initial inconsistency snapshot pair as the query condition, all subsequent related snapshot pairs are extracted from this record and combined with the initial inconsistency snapshot pair in chronological order to construct a complete inconsistency event tracing chain. This tracing chain reflects the propagation path of a business request through each node and the evolution of the corresponding data state, serving as the fundamental data structure for determining the nature of inconsistency events and locating their root causes.

[0079] After the traceability chain is built, it is necessary to determine whether the data processing of the tail of the chain (i.e., the last data change snapshot pair) has eventually become consistent in both environments. Specifically, this is done by comparing the first and second data checksums of the snapshot pair again. If they are still inconsistent, it indicates that the inconsistency was not automatically corrected or converged in subsequent processing, representing a fundamental deviation in the business processing logic. Therefore, the calling service node corresponding to the first initially inconsistent snapshot pair in the traceability chain can be marked as the root cause failure node, and the traceability chain can be identified as a permanent data inconsistency event. This event indicates a stability problem in the exercise environment or a difference in business logic between the exercise and production environments, requiring further intervention and investigation.

[0080] However, if the last snapshot pair in the traceback chain shows that the data state has returned to consistency (i.e., the checksums are the same), it indicates that the data inconsistency may be due to temporary deviations caused by asynchronous processing, caching latency, or eventual consistency mechanisms in the service call chain. In this case, it is necessary to assess whether the duration of the inconsistency exceeds the system's acceptable range. This involves calculating the time span from the first initially inconsistent snapshot pair to the last snapshot pair in the traceback chain and comparing it with a preset business tolerance latency threshold. This threshold is a tolerance standard set according to the real-time requirements of the business system, reflecting the duration of transient data inconsistency that the business can accept. If the time span exceeds this threshold, the event can be determined as an instantaneous over-threshold inconsistency event, and the service node corresponding to the first initially inconsistent snapshot pair will be marked as a potential risk node, indicating that the node has some kind of latency risk or processing bottleneck.

[0081] For example, in an order system, if a user triggers calls to multiple service nodes during the order placement process, including operations such as inventory locking, coupon verification, and order entry, the system records its synchronization identifier as "ORD-20251118-001". During data consistency verification, an inconsistency is found in the checksum of the "Order Entry" node snapshot. Subsequently, the snapshot results of the "Inventory Locking" and "Coupon Verification" nodes are traced from the workflow records, and finally, consistency is restored in the "Order Approval" node snapshot. If this process takes longer than the set business tolerance delay of 10 seconds, the entire event is judged as a momentary inconsistency event exceeding the threshold, and the "Order Entry" node is identified as a potential risk node. If the checksum is never restored to consistency, it can be judged as a permanent data inconsistency event, and the "Order Entry" node is identified as the root cause failure node.

[0082] By combining the above steps, we can achieve the discovery of data deviations, the tracing of propagation paths, the location of root causes, and the classification of event nature, thus comprehensively improving the analytical depth and positioning accuracy of disaster recovery drills and providing a solid data foundation and support capabilities for the stability assessment of business systems and the optimization of drill strategies.

[0083] After analyzing the data consistency verification baseline table and identifying all data inconsistency events, the system needs to further rely on these analysis results to dynamically update the original service dependency directed graph, and generate a comprehensive exercise report based on the updated dependency structure. This enables a comprehensive assessment of the disaster backup and recovery exercise quality and problem localization. This step is not only a process of summarizing the results of the preceding analysis, but also an important output link for forming visual feedback and subsequent optimization strategies.

[0084] In practice, the system first maps each data inconsistency event back to the corresponding calling service node in the directed graph of service dependencies, based on the node information identified in that event. For permanent data inconsistency events, since the root cause fault node has been clearly identified, the system marks the outgoing call relationship of that node in the graph as a high-risk dependency path and adds fault labels such as "root cause fault" or "inconsistency propagation source" to the graph structure to visually demonstrate the potential impact of this node on the stability of downstream services. For transient inconsistency events exceeding the threshold, the system identifies the corresponding potential risk nodes and records indicators such as the duration of inconsistency and the cumulative number of occurrences, serving as an important basis for assessing service node performance fluctuations. To avoid misjudgments, the system can also introduce event weights or confidence models to set dynamic thresholds for update operations, ensuring the accuracy and stability of graph structure adjustments.

[0085] Based on the directed graph update, the system automatically generates a comprehensive exercise report. This report summarizes all data consistency test results, data inconsistency event statistics, node-level risk distribution, service link health, and other key indicators from the exercise, and provides a visual representation in conjunction with the updated service dependency graph. The report typically includes the following aspects: exercise overview statistics (such as exercise duration, total number of requests, number of snapshot pairs, consistency rate, etc.), details of data inconsistency events (including synchronization identifier, involved nodes, anomaly type, duration, etc.), service dependency graph change records (addition of risky paths, suggestions to break loops, etc.), and system-generated optimization suggestions (such as fixing differences in exercise environment configuration, enhancing node fault tolerance, and suggestions for call chain reconstruction, etc.).

[0086] For example, in a drill, if the "Inventory Service Node" is identified as the root cause of multiple events, the system will mark its outgoing edge to the "Order Service Node" as a red high-risk path in the service dependency graph and list it as a first-level fault node in the drill report, suggesting that its configuration, interface compatibility, or data initialization status in the drill environment should be checked first. If another "Payment Service Node" experiences multiple instantaneous inconsistency events exceeding the threshold, the system will add a performance fluctuation label to that node in the graph and provide suggestions in the report, such as considering adding a cache preheating mechanism or optimizing asynchronous processing strategies.

[0087] By dynamically updating the directed graph of service dependencies and generating comprehensive reports, the system achieves closed-loop management from data consistency verification and anomaly tracing to structured feedback, ensuring that disaster recovery drills are not only controllable and measurable, but also interpretable and optimizable, providing data support and decision-making basis for the enterprise's business continuity assurance.

[0088] Figure 2 This is another flowchart illustrating a data disaster backup and recovery drill method as described in this application. Please refer to... Figure 2 The present application embodiment provides a data disaster backup and recovery drill method, which further includes steps S201 to S206.

[0089] Building upon the analysis of service call consistency and data consistency, an algorithm decision consistency comparison mechanism is further introduced. This mechanism effectively identifies deviations in business decision-making behavior caused by fluctuations in system resource status, model parameter drift, or differences in exercise environment configuration. In exercise evaluation, the consistency of algorithmic behavior is of paramount importance, especially when core decision-making modules such as risk control, intelligent recommendation, and pricing engines are involved. Any minor change in model input or difference in system status can lead to divergences in business processing paths. Therefore, a systematic approach is needed to capture, quantify, and backtrack on these deviations. The following are examples of steps S201 to S206, elaborated and explained according to execution logic:

[0090] S201. Obtain the sequence of input parameters for algorithm decision-making in the target business environment;

[0091] In step S201, "target business environment" refers to the production-level application environment that currently handles business traffic and runs real service logic and algorithm models. The algorithm decision service deployed in this environment typically undertakes intelligent decision-making tasks that are strongly related to business processing results, such as risk control assessment, intelligent recommendation, customer profile matching, and pricing reasoning. "Algorithm decision" refers to the process of performing feature processing and model reasoning on business inputs based on the trained model and outputting decision results. "Input parameter sequence" refers to the set of all input variables used by the model in the reasoning process after the algorithm module receives the business request. It typically includes structured fields, semi-structured labels, environmental context parameters, and some system state indicators that may affect the model's behavior.

[0092] Since algorithmic decision-making is typically a "black box" process, its output is highly sensitive to input parameters. Therefore, in disaster recovery drills, to compare the consistency of algorithm behavior between the target business environment and the preset drill environment, it is essential to ensure that both models receive completely identical input parameters. To this end, during implementation, the system needs to implement non-intrusive data logging at the algorithm call entry point in the target business environment or connect to a gateway-level request parameter collection module to extract the input parameters of each algorithm decision request in real time and record them chronologically into a dedicated queryable data structure, forming an input parameter sequence. To ensure the traceability and comparability of the collected data, the system assigns a unique identifier (such as a request ID or business tracking identifier) ​​to each algorithm request and attaches metadata fields such as a timestamp, calling service node identifier, and algorithm version number, facilitating subsequent reproducible replay in the drill environment.

[0093] Input parameters can be acquired using two technical approaches: one is to extract request data flowing through the algorithm module via a service gateway or API middleware, suitable for relatively standardized interface call scenarios; the other is to use the algorithm service's own log collection component (such as using log frameworks like Logback or Log4j) to output input parameters to a log stream before model inference, and then have log parsing and structuring tools (such as Fluentd or Logstash) extract and store them, suitable for self-developed model services or embedded model containers. To avoid privacy leaks or data redundancy, the system can perform field-level anonymization and selective collection of input parameters, retaining only key feature fields that have a substantial impact on model behavior, such as user level, transaction amount, historical behavior count, and context time window.

[0094] S202. Divide the input parameter sequence into multiple input parameter sub-sequences based on multiple consecutive preset time windows, and obtain the first algorithm decision result in each preset time window in the target business environment;

[0095] A "preset time window" refers to a unit of division for aggregating input parameters along the time dimension. It is used to group consecutively arriving algorithm decision requests into multiple relatively independent analysis units according to their chronological order. The time window can be set to a fixed length (e.g., 5 minutes, 10 minutes) or dynamically adjusted based on the number of requests (e.g., one window for every 1000 requests). Its main function is to transform large-scale, continuous data streams into manageable and comparable discrete data blocks, thereby improving the efficiency and accuracy of subsequent consistency comparisons of algorithm behavior.

[0096] A "subsequence of input parameters" refers to a set of local parameters extracted from the original sequence of input parameters according to the time window division rules. Each subsequence of input parameters includes all algorithm request inputs received within the corresponding time window. Each record contains elements such as a request identifier, timestamp, set of input fields, and information about the request source node, ensuring complete reproducibility and traceability. Because business traffic exhibits significant temporal distribution characteristics, such as e-commerce peak periods and financial system settlement periods, directly comparing the entire sequence of input parameters without time window division can easily lead to wasted computational resources, masking of outliers, and confusion of behavioral patterns. Therefore, a time window mechanism is needed to establish a local analytical perspective.

[0097] In the implementation process, the system first sorts the input parameter sequence according to the timestamp field, and then slides it at preset time intervals to form multiple time window intervals. The input parameters within each time window constitute a complete input parameter subsequence. The system assigns a unique time window identifier to each subsequence to facilitate subsequent mapping with results in the training environment. Subsequently, for each input parameter subsequence, the system backtracks to the "first algorithm decision result" generated in the target business environment within the same time window. This "first algorithm decision result" refers to the output result generated by the target business environment after executing the algorithm model on each request in the input parameter subsequence, such as risk score, recommended product set, user tag result, or strategy hit result. This decision result needs to be associated one-to-one with the input parameter record to ensure accurate comparison during subsequent replication in the training environment.

[0098] The extraction of decision results can be achieved through two technical approaches: one is to perform structured parsing of the algorithm service's response logs, extract each output result, and aggregate them by time window; the other is to directly extract the historical decision records within the target time period from the model result cache or decision database, and then associate and match them with the input parameter request ID. To improve data quality, the system can also introduce an abnormal data filtering mechanism to mark and remove outlier output values ​​(such as empty results, error codes, and timeout responses) to prevent them from interfering with subsequent differential analysis.

[0099] By implementing this step, the system achieves time-window-level aggregation of input parameters and decision results, providing a structured data foundation for subsequent refined algorithm behavior replay, decision consistency comparison, and difference index statistics in the training environment. For example, in an intelligent recommendation system, the system collected a total of 1200 product recommendation requests within a time window from 14:00 to 14:05 on a certain day, and extracted the recommendation list generated by each request in the target business environment as the first algorithm decision result. These 1200 records, along with their output results, form a complete subsequence of input parameters and a corresponding decision set, which the system marks as "window number W_1400_1405" for subsequent model inference consistency verification in the training environment. This method effectively improves the organization of data processing and the granular control capability of comparative analysis, and is a key preliminary step for realizing algorithm behavior drift identification.

[0100] S203. In the preset training environment, the algorithm decision is executed based on multiple input parameter subsequences to obtain the second algorithm decision result within each preset time window;

[0101] During implementation, the system first starts the algorithm inference service in a preset training environment and loads an algorithm model that is logically consistent with the target business environment. To ensure the comparability of inference results, the model parameters, feature processors, feature dictionaries, normalization configurations, and model versions used in the training environment must be consistent with those of the target business environment or precisely matched in the version control system. Subsequently, the system sequentially sends each algorithm request input in each subsequence to the inference service for processing according to the time window number of each input parameter subsequence, and records the output results of each inference, finally aggregating them into a second set of algorithm decision results within that time window.

[0102] To improve the accuracy and efficiency of the exercise, the system can introduce a batch processing execution mechanism to send each subsequence into the model inference engine (such as TensorFlow Serving, TorchServe, ONNX Runtime, etc.) in batch input mode, reducing the overhead of repeated model loading; at the same time, an input-output comparison hook is introduced to bind the results generated by inference in the exercise environment with the input parameter identifiers one by one, ensuring that each result can be accurately traced in subsequent comparison processes.

[0103] The system also needs to monitor key performance indicators of the algorithm execution process in the training environment, such as inference time, resource utilization, and anomaly rate, to evaluate the stability of the model's operation in the training environment. This process can not only verify the integrity of the model components in the recovery environment, but also discover model behavior drift issues caused by configuration differences, resource scheduling, and latency of dependent services.

[0104] For example, in an intelligent risk control system, suppose the input parameter subsequence W_1400_1405 contains 1000 loan application requests. The training system sequentially feeds this subsequence into the risk control scoring model in the training environment for inference, obtaining a score value (such as 0.85, 0.42, etc.) for each request. These scores constitute the decision result of the second algorithm. The system structurally binds this result set with the input parameter request identifier and marks it as "training decision result set_W_1400_1405" for use in subsequent difference comparison steps.

[0105] Through this implementation process, the system enables the input of real algorithms from the target business environment into a controlled and isolated environment for restoration calculation. This can not only be used to evaluate the availability and consistency of the model in the disaster recovery environment, but also to identify potential differences in algorithm behavior, drift of environmental parameters and abnormal model execution without affecting production services, significantly enhancing the integrity and credibility of disaster recovery drills.

[0106] S204. Calculate the decision difference value between the decision result of the first algorithm and the decision result of the second algorithm within the same preset time window;

[0107] "Decision Difference Value" refers to a numerical metric that measures the degree of difference between the decision results of the first algorithm and the second algorithm. The calculation method for this metric needs to be specifically designed based on the output type of the algorithm decisions. For example, in a classification model, where the decision result is a category label (such as "pass" or "reject"), the difference value can be measured using classification accuracy, F1 score, or label matching rate; in a scoring model, where the result is a continuous numerical value (such as a credit score), the mean squared error, mean absolute error, or Pearson correlation coefficient can be used to measure the degree of difference; in a ranking model, where the result is a recommendation list or priority order, ranking consistency metrics such as NDCG (Normalized Diminished Cumulative Gain) and MAP (Mean Mean Accuracy) can be used for evaluation. The system needs to dynamically select an appropriate difference measurement method based on the type of model being compared to ensure that the difference value has discriminative power and business interpretability.

[0108] During implementation, the system first aligns the decision results of the first and second algorithms one-to-one according to the request identifier, using the time window number as an index, forming a set of decision result pairs. The alignment process must ensure consistent input parameters, maintained request order, and compatible model versions to guarantee the validity of the comparison results. Subsequently, for each pair of results, the system selects the corresponding difference calculation logic based on the decision type, calculates the difference index at the request level, and aggregates it at the window level (e.g., average difference value, difference ratio, deviation rate). Finally, the aggregated result is used as the overall decision difference value for that time window.

[0109] To enhance the sensitivity and robustness of difference detection, the system can also incorporate distributional statistical indicators, such as comparing the mean, variance, skewness, and kurtosis of the decision results from the first and second algorithms, to determine whether there is a statistically significant shift in the model's inference behavior during the exercise. Furthermore, by setting warning thresholds, time windows where the difference value exceeds a preset business tolerance limit can be marked and incorporated as "target time windows" into the system's resource status backtracking and root cause analysis processes in subsequent steps.

[0110] For example, in an anti-fraud scoring model, assuming the first algorithm's decision result is a set of real-valued scores (e.g., [0.91, 0.75, 0.22, 0.63]), and the second algorithm's decision result is the inference return value for the same input in the training environment (e.g., [0.88, 0.71, 0.35, 0.58]), the system can calculate the score difference for each pair of results (e.g., 0.03, 0.04, -0.13, 0.05), and then calculate the average absolute difference (approximately 0.0625) as the decision difference value for that time window. If this value is higher than a set drift threshold (e.g., 0.05), the window is marked as a behavior offset window, and the subsequent resource status backtracking process begins.

[0111] Through the above implementation methods, the system can quantitatively evaluate the consistency of algorithm behavior in different environments, transforming the originally difficult-to-observe changes in model output into a monitorable and alarmable numerical indicator system. This not only strengthens the algorithm verification capability in disaster recovery drills, but also provides a general technical path for model stability monitoring, deployment regression testing, and multi-environment behavior alignment.

[0112] S205. Backtrack the system resource status data of the target time window, wherein the target time window is the preset time window in which the decision difference value is greater than the preset drift threshold;

[0113] The target time window refers to a preset time window in which the corresponding decision difference value exceeds a preset drift threshold. The "target time window" in this step is a time interval dynamically determined based on the algorithm behavior offset recognition mechanism. Within this time interval, there are significant decision differences between the algorithm behavior in the training environment and the target business environment, which cannot be explained simply by the consistency of input parameters. Therefore, it is necessary to further explore the impact of the system's operating status on the model behavior.

[0114] A preset drift threshold is used to measure the maximum tolerable difference between the decision results of the first algorithm and the decision results of the second algorithm within the same time window. This threshold can be set based on historical exercise experience, business risk tolerance, or statistical analysis results, and is usually configured using a fixed value or a dynamic percentile value. For example, for a scoring model, an average difference value exceeding 0.05 can be set to be considered behavioral drift; for a classification model, a drop in accuracy exceeding 3% can trigger target window marking.

[0115] System resource status data refers to a set of hardware and software environment indicators related to the operation of the algorithm model within a target time window. This includes basic resource indicators at the operating system level such as CPU utilization, memory usage, disk read / write speed, network communication latency, and thread blocking duration; and business resource operation indicators at the algorithm service level such as container load, JVM heap memory usage, GC (garbage collection) frequency, model loading time, feature service response time, and downstream dependent interface latency. This type of data reflects the resource usage and system health status during model inference execution in the training environment, and is an important basis for locating the system-level root causes behind abnormal algorithm behavior.

[0116] During implementation, the system first determines the corresponding timestamp range based on the identified target time window and constructs query conditions for resource status data. Subsequently, the system queries all resource indicator data related to the algorithm service nodes within that time period by connecting to a multi-source monitoring platform or indicator acquisition system (such as Prometheus, Telegraf, Grafana, Zabbix, etc.). To improve the readability of data analysis, the system standardizes the raw collected indicators, unifies the time granularity (e.g., aggregation by second or minute), and organizes them in a structured manner according to indicator categories (CPU, memory, IO, network, etc.).

[0117] After data collection, the system further performs trend analysis and anomaly detection on key resource indicators. For example, it uses statistical methods such as sliding window averaging, Z-score standard deviation detection, or IQR quartile analysis to identify any sudden changes, overloads, or bottlenecks in resource usage within the target time window. If phenomena such as memory spikes, CPU full load, or sudden increases in network latency are detected, it is preliminarily determined that the resource status may affect the algorithm model's behavior. The system can also combine model runtime logs to verify whether abnormal situations occur in the decision-making process, such as model loading failures, feature missing, interface timeouts, or cache invalidations, serving as input conditions for subsequent identification of divergence factors.

[0118] For example, during a drill of a recommendation service, the system identified a significant difference in the ranking of recommendation results within the time window W_1500_1505. Further investigation of resource status data within this time window revealed that container memory usage in the drill environment exceeded 95% during that period, along with a significant increase in GC frequency, multiple blocking events in the JVM thread pool, and warnings in the model logs indicating that some model parameters had not been successfully loaded. Based on this, the system initially determined that the algorithm behavior deviation within this time window might be due to an incomplete inference environment caused by memory resource constraints.

[0119] By implementing this step, the system can extend the consistency analysis of algorithm behavior from the input parameter layer to the system operation layer, realizing a deep analysis path from "input comparison" to "execution environment insight," further improving the interpretability and problem closure capability of the algorithm verification stage in disaster recovery drill scenarios. Simultaneously, this step can also provide data support for resource allocation optimization, adjustment of model service elastic scheduling strategies, and improvement of operation and maintenance disaster recovery capabilities.

[0120] S206. Determine decision-making disagreement factors based on the system resource status data, and update the comprehensive exercise report based on the decision-making disagreement factors.

[0121] In step S206, decision divergence factors refer to the reasons for deviations in algorithm decision-making behavior caused by differences in non-model input levels such as system resource status, service dependency response, and model execution path, under the premise that the algorithm input parameters are consistent. This fundamentally reveals the specific triggering conditions for inconsistent model inference behavior in different environments. These factors may include, but are not limited to, model loading failure, missing features, service call timeouts, container resource bottlenecks, cache misses, differences in preprocessing logic, inconsistent JVM parameter configurations, and differences in inference engine versions.

[0122] During implementation, the system first aggregates and analyzes the system resource status data collected in the previous step, compares it with the standard operating baseline, and identifies abnormal indicators at the resource level. Then, it performs multi-dimensional cross-validation by combining algorithm service logs, model execution trajectories, and call chain tracing data to construct a "decision execution path diagram." This path diagram starts with the request identifier and connects the feature processing stage, model inference stage, dependent service call stage, and post-processing stage to form a complete model execution chain.

[0123] The system performs node-level anomaly detection on the execution path graph. For example, at the model loading node, it checks for abnormal loading time, insufficient memory, or missing files; at the feature service node, it checks for empty fields, request timeouts, or data offsets; and at the dependent service node, it checks for interface return errors or excessive response latency. Detection methods can combine rule matching, threshold alarms, log keyword recognition, and link time statistics. When an anomaly pattern on a certain execution path highly overlaps with the time of decision differences within the target time window, and the anomaly indicators exhibit consistency, repetition, or significance, the system marks this anomaly as a potential decision divergence factor.

[0124] Subsequently, the system categorizes and organizes all marked potential decision-making disagreement factors, generating a structured set of disagreement factors. Each disagreement factor includes fields such as factor type, scope of influence, triggering conditions, corresponding indicators, associated service nodes, and decision offset direction. For example, if insufficient memory is detected during model loading in the exercise environment within the target time window, resulting in some model components not being activated, and this phenomenon closely matches the behavior of changing the decision result from "high risk" to "low risk," then the system classifies this problem as a decision-making disagreement factor of the "model loading anomaly" type, with the scope of influence being "risk control scoring service" and the offset direction being "risk level downgrade."

[0125] After generating the set of divergence factors, the system writes them into the "Behavioral Consistency Analysis" section of the comprehensive exercise report and automatically highlights the corresponding service nodes that experienced anomalies in the service dependency graph, generating a view displaying the exercise results with root cause path visualization. Simultaneously, the system appends optimization suggestions based on the divergence factors at the end of the report, such as adjusting the JVM memory configuration of the exercise environment, synchronizing feature engineering versions, optimizing model deployment methods, or introducing model integrity verification mechanisms. This ensures that the comprehensive exercise report not only has evaluation and analysis functions but also decision support and operational guidance capabilities.

[0126] Please see Figure 3 This is a schematic diagram of the data disaster backup and recovery drill in the embodiments of this application.

[0127] It should be noted that, Figure 3 The structure of the data disaster backup and recovery drill system shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0128] like Figure 3 As shown, a data disaster backup and recovery drill system includes a central processing unit 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory 302 or a program loaded from a storage section 308 into a random access memory 303, such as executing the methods described in the above embodiments. The random access memory 303 also stores various programs and data required for system operation. The central processing unit 301, the read-only memory 302, and the random access memory 303 are interconnected via a bus 304. An input / output interface 305 is also connected to the bus 304.

[0129] The following components are connected to the input / output interface 305: an input section 306 including audio input devices, push-button switches, etc.; an output section 307 including an LCD display, audio output devices, indicator lights, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the input / output interface 305 as needed. A removable medium 311, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 310 as needed so that computer programs read from it can be installed into the storage section 308 as needed.

[0130] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing computer programs for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 309, and / or installed from removable medium 311. When the computer program is executed by central processing unit 301, it performs the various functions defined in the present invention.

[0131] It should be noted that specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, flash memory, optical fiber, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0132] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions, and operations of systems, methods, and computer program products according to various embodiments of the present invention. Specifically, a data disaster backup and recovery drill system according to this embodiment includes a processor and a memory. The memory stores a computer program, and when the computer program is executed by the processor, it implements a data disaster backup and recovery drill method provided in the above embodiment.

[0133] In another aspect, the present invention also provides a computer-readable storage medium, which may be included in a data disaster backup and recovery drill system described in the above embodiments; or it may exist independently and not assembled into the data disaster backup and recovery drill system. The storage medium carries one or more computer programs, which, when executed by a processor of the data disaster backup and recovery drill system, cause the data disaster backup and recovery drill system to implement the data disaster backup and recovery drill method provided in the above embodiments.

Claims

1. A method for data disaster backup and recovery drills, characterized in that, The method includes: Acquire business traffic data for each calling service node in the target business environment, the calling relationships between multiple calling service nodes, and data flow records; Based on the business traffic data, the call relationship, and the data flow record, a service dependency directed graph is constructed, and a call service node hierarchical table is generated based on the service dependency directed graph. Each call service node in the call service node hierarchical table corresponds to a hierarchical number. In the preset training environment, each of the calling service nodes is started sequentially in ascending order of the hierarchical number in the calling service node hierarchy table to complete the environment initialization; A traffic mirroring channel is established in the target business environment, and read-only business traffic in the target business environment is copied to the preset training environment in real time through the traffic mirroring channel. The preset training environment is isolated from the target business environment. Obtain a first data change snapshot of the target business environment and a second data change snapshot of the preset training environment. The first data change snapshot and the second data change snapshot are data state change results including synchronization identifiers triggered by the same business request in their respective environments during the traffic replication process. The first data change snapshot and the second data change snapshot, which have the same synchronization identifier, are identified as a data change snapshot pair; Calculate the first data checksum of the first data change snapshot and the second data checksum of the second data change snapshot in each data change snapshot pair; Based on the first data change snapshot and the second data change snapshot, a data consistency verification baseline table is generated; Analyze the data consistency verification baseline table to identify data inconsistency events, update the service dependency directed graph based on the data inconsistency events, and generate a comprehensive exercise report; the data inconsistency events include permanent data inconsistency events and instantaneous over-threshold inconsistency events. The analysis of the data consistency verification baseline table to identify data inconsistency events specifically includes: The data change snapshot pairs in the data consistency verification baseline table whose first data checksum is different from the second data checksum are identified as initial inconsistent snapshot pairs. For each of the initial inconsistent snapshot pairs, based on the synchronization identifier of the initial inconsistent snapshot pair, extract all associated data change snapshot pairs triggered by the same business request and whose triggering time is after the initial inconsistent snapshot pair from the data flow record, and combine the initial inconsistent snapshot pair and all the associated data change snapshot pairs into an inconsistent event tracing chain in chronological order; Determine whether the first data checksum and the second data checksum of the last data change snapshot pair in the inconsistency event tracing chain are the same: If they are not the same, the calling service node corresponding to the first initial inconsistent snapshot pair in the inconsistent event tracing chain is identified as the root cause failure node, and the inconsistent event tracing chain is identified as the permanent data inconsistency event. If they are the same, calculate the time span of the inconsistency event tracing chain from the first initial inconsistent snapshot pair to the last data change snapshot pair, and determine whether the time span is greater than the preset business tolerance latency threshold. If so, the calling service node corresponding to the first initial inconsistent snapshot pair in the inconsistent event tracing chain is identified as a potential risk node, and the inconsistent event tracing chain is identified as the instantaneous over-threshold inconsistent event.

2. The method according to claim 1, characterized in that, The process of generating a hierarchical table of calling service nodes based on the service dependency directed graph specifically includes: Based on the service dependency directed graph, calculate the in-degree and out-degree values ​​of each of the calling service nodes; Based on the in-degree value and the out-degree value, the level number of each calling service node is determined, and a calling service node hierarchical table is generated according to the level number.

3. The method according to claim 2, characterized in that, The step of determining the level number of each calling service node based on the in-degree value and the out-degree value specifically includes: Repeat steps S1-S3: S1. Determine the calling service node with an unassigned level number and an in-degree value of zero in the service dependency directed graph as the current level node; S2. Assign the current count value of the level number counter to all the current level nodes, where the initial value of the level number counter is one; S3. For each current level node, traverse all outgoing call relationships of the current level node, decrement the in-degree value of the called node pointed to by each outgoing call relationship by one, increment the count value of the level number counter by one, and re-execute step S1 until the level number of each calling service node is determined.

4. The method according to claim 3, characterized in that, The method further includes: When the current count value of the level number counter is greater than the preset iteration threshold, all target call service nodes in the service dependency directed graph that have not determined the level number are combined into a loop dependency node set. Traverse all outgoing call relationships of each target call service node in the loop dependency node set, and based on the business traffic data, determine the outgoing call relationship corresponding to the minimum business traffic value as the target disconnect relationship; Decrement the in-degree value of the called node pointed to by the target disconnect relationship by one to logically disconnect the circular dependency structure.

5. The method according to claim 1, characterized in that, The step of generating a data consistency verification baseline table based on the first data change snapshot and the second data change snapshot specifically includes: The synchronization identifier, the first data checksum, and the second data checksum of the data change snapshot pair are associated to generate a data consistency verification baseline table.

6. The method according to claim 1, characterized in that, The method further includes: Obtain the sequence of input parameters for algorithm decision-making in the target business environment; The input parameter sequence is divided into multiple input parameter sub-sequences based on multiple consecutive preset time windows, and the first algorithm decision result within each preset time window in the target business environment is obtained; In the preset training environment, the algorithm decision is executed based on multiple input parameter subsequences to obtain the second algorithm decision result within each preset time window; Calculate the decision difference between the first algorithm decision result and the second algorithm decision result within the same preset time window; The system resource status data of the target time window is traced back, where the target time window is the preset time window in which the decision difference value is greater than the preset drift threshold. Based on the system resource status data, decision-making disagreement factors are determined, and the comprehensive exercise report is updated based on these decision-making disagreement factors.

7. A data disaster backup and recovery drill system, characterized in that, The data disaster backup and recovery drill system includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to cause the data disaster backup and recovery drill system to perform the method as described in any one of claims 1-6.

8. A computer-readable storage medium comprising instructions, characterized in that, When the instruction is run on the data disaster backup and recovery drill system, the data disaster backup and recovery drill system performs the method as described in any one of claims 1-6.

9. A computer program product, comprising instructions, characterized in that, When the instruction is run on the data disaster backup and recovery drill system, the data disaster backup and recovery drill system performs the method as described in any one of claims 1-6.

Images