System disaster recovery control method, device, equipment and medium

Abstract

The application relates to the technical field of cloud computing, and discloses a system disaster control method, device, equipment and medium, the method comprises the following steps: constructing a fault injection model according to the fault type and the fault occurrence time corresponding to each historical fault record of a business system to be practiced; generating a business service link graph according to the dependency relationship among a plurality of preset businesses in the business system; injecting a preset fault signal into a target node according to the fault injection model, and switching the business flow of the target node to a redundant backup node according to the flow switching mapping relationship between the target node and the redundant backup node determined by the business service link graph; and adjusting the resource allocation parameters of the redundant backup node according to the operation performance index of the redundant backup node until the service capacity of the business system recovers to the target level before the fault injection. The application can be applied to the business system platform of financial technology, medical health and the like, and can improve the precision of disaster control.

Description

Technical Field

[0001] This invention relates to the field of cloud computing technology, and in particular to a system disaster recovery control method, apparatus, equipment and medium. Background Technology

[0002] With the ongoing digital transformation, the scale of core data carried by business systems in critical sectors such as finance and healthcare is increasing daily, and the requirements for business continuity are also constantly rising. If the main production system encounters unforeseen circumstances such as hardware failure, network outages, human error, or even natural disasters, it will directly lead to business stagnation and data loss, causing incalculable economic losses and a crisis of trust for enterprises and society. Against this backdrop, disaster recovery system construction has become an indispensable core component of enterprise information security architecture, and disaster recovery control, as a key means of verifying the effectiveness of disaster recovery systems and ensuring rapid business recovery in disaster scenarios, is becoming increasingly important.

[0003] Current disaster recovery control methods are mainly based on simulation control. Simulation control relies on simulated data and non-production environments to carry out operations, which makes it difficult to reproduce the high concurrency, multi-dependency, and complex link characteristics of real business scenarios. In particular, most simulation control methods use manual random fault setting, and the fault scenarios have a low degree of matching with the high-frequency fault types that occur in the actual production environment. It is difficult to cover typical risk points in real business operations, resulting in limited reference value of control results. Consequently, it is impossible to dynamically adjust resource allocation based on real-time performance indicators, often resulting in resource overload or idle resources on disaster recovery nodes, making it difficult to ensure that business service capabilities can be quickly restored to the health level before the fault.

[0004] For example, in the healthcare scenario, core systems such as HIS and EMR in tertiary hospitals need to operate 24 / 7. Traditional disaster recovery control often fails because the fault scenarios deviate from actual historical downtime records, and the resource configuration of disaster recovery nodes after control is unreasonable. This leads to delays in business switching such as registration and emergency triage when a real fault occurs, directly threatening the safety of patients' medical treatment. For example, in the fintech scenario, core banking transaction and payment clearing systems have extremely high requirements for strong data consistency. Traditional simulation control cannot reproduce the complex dependencies of high-concurrency transaction links. Manually set faults have low matching degree with the high-frequency risks in actual production. After switching, disaster recovery nodes are prone to resource overload or idleness, causing compliance risks such as transaction delays and data corruption. Both scenarios urgently need precise disaster recovery control solutions adapted to business characteristics to ensure the continuity and stability of core businesses under extreme scenarios.

[0005] Therefore, how to implement a system disaster recovery control method to improve the accuracy of disaster recovery control is a technical problem that urgently needs to be solved. Summary of the Invention

[0006] This invention provides a system disaster recovery control method, device, equipment, and medium, the main purpose of which is to solve the problem of low accuracy in disaster recovery control.

[0007] Firstly, to achieve the above objectives, the system disaster recovery control method provided by the present invention includes: Obtain a set of historical fault records of the business system to be tested, and extract the fault type and fault occurrence time corresponding to each historical fault record in the set of historical fault records; A fault injection model is constructed based on the fault type and fault occurrence time corresponding to all the historical fault records. Obtain multiple preset services from the business system, identify the dependencies between each preset service, and generate a business service link graph of the preset services based on the dependencies; Based on the service link graph, the target node and redundant backup node of each preset service are determined, and a traffic switching mapping relationship is established between the target node and the redundant backup node. According to the fault injection model, corresponding fault signals are injected into the target node, and the response status data of the business system is monitored in real time after each fault signal injection. When the response status data meets the preset disaster recovery switching trigger conditions, the service traffic of the target node is switched to the corresponding redundant backup node according to the traffic switching mapping relationship; After the business traffic switching is completed, the operating performance indicators of the redundant backup nodes are collected, and the resource allocation parameters of the redundant backup nodes are adjusted according to the operating performance indicators until the service capability of the business system is restored to the target level before the fault injection.

[0008] Secondly, the present invention also provides a system disaster recovery control device, comprising: The fault information extraction module is used to obtain a set of historical fault records of the business system to be tested, and to extract the fault type and fault occurrence time corresponding to each historical fault record in the set of historical fault records. The fault model construction module is used to construct a fault injection model based on the fault type and fault occurrence time corresponding to all the historical fault records. The relationship graph generation module is used to obtain multiple preset services in the business system, identify the dependency relationships between each preset service, and generate a business service link graph of the preset services based on the dependency relationships. The mapping relationship establishment module is used to determine the target node and redundant backup node of each preset service based on the service link graph, and to establish a traffic switching mapping relationship between the target node and the redundant backup node. The fault signal injection module is used to inject corresponding fault signals into the target node according to the fault injection model, and to monitor the response status data of the business system in real time after each fault signal injection. The disaster recovery control module is used to switch the service traffic of the target node to the corresponding redundant backup node according to the traffic switching mapping relationship when the response status data meets the preset disaster recovery switching trigger conditions. The disaster recovery module is used to collect the operating performance indicators of the redundant backup nodes after the business traffic switching is completed, and adjust the resource allocation parameters of the redundant backup nodes according to the operating performance indicators until the service capability of the business system is restored to the target level before the fault injection.

[0009] Thirdly, the present invention also provides an electronic device, the electronic device comprising: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the system disaster recovery control method described above.

[0010] Fourthly, the present invention also provides a computer-readable storage medium storing at least one computer program, which is executed by a processor in an electronic device to implement the system disaster recovery control method described above.

[0011] In this embodiment of the invention, a history-driven fault injection model is constructed to ensure the realism and repeatability of the drill scenarios. By combining the business service link graph to clarify node dependencies and primary / backup mapping relationships, a precise routing foundation for traffic switching is provided. After injecting a fault and monitoring the response status in real time, the system can automatically determine the triggering conditions and execute a switch, effectively shortening fault response and recovery time. After the switch is completed, performance indicators of the backup nodes are further collected and resource allocation parameters are dynamically adjusted, forming a complete closed loop of "injection-monitoring-determination-switching-optimization," ensuring that service capabilities are quickly restored to the pre-fault target level. This invention comprehensively verifies the effectiveness of the disaster recovery strategy and the system's fault tolerance resilience, significantly reducing the uncertainty and operational risks caused by manual intervention, providing efficient, controllable, and quantifiable technical support for ensuring business continuity, and effectively improving the accuracy of disaster recovery control. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a schematic diagram of an application environment for a cross-device data transmission method according to an embodiment of the present invention. Figure 2 This is a flowchart illustrating a cross-device data transmission method according to an embodiment of the present invention; Figure 3 yes Figure 2 A flowchart illustrating a specific implementation method of step S1; Figure 4 yes Figure 2 A flowchart illustrating a specific implementation method of step S2; Figure 5 yes Figure 2 A flowchart illustrating a specific implementation method of step S3; Figure 6 yes Figure 2 A flowchart illustrating a specific implementation of step S4; Figure 7 yes Figure 2 A flowchart illustrating a specific implementation of step S5; Figure 8 yes Figure 2 A flowchart illustrating a specific implementation of step S9; Figure 9 This is a functional block diagram of a cross-device data transmission apparatus provided in an embodiment of the present invention; Figure 10 This is a schematic diagram of the structure of an electronic device that implements a cross-device data transmission method according to an embodiment of the present invention; Figure 11 This is another structural schematic diagram of an electronic device that implements a cross-device data transmission method according to an embodiment of the present invention.

[0014] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0015] It should be noted that in the technical solutions disclosed in this invention, the acquisition of user information (personal image data (e.g., facial videos or pictures, facial feature videos or pictures, etc.) and personal privacy information (e.g., name, ID number, occupation, address, etc.)) is all completed with the user's knowledge and consent, and the acquisition of the relevant user information is legal and compliant.

[0016] To enable those skilled in the art to better understand the technical solutions of this disclosure, and to fully understand and implement the process of how this disclosure applies technical means to solve technical problems and achieve corresponding technical effects, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, not all embodiments. The embodiments of this disclosure and the various features within them can be combined with each other without conflict, and the resulting technical solutions are all within the protection scope of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort should fall within the protection scope of this disclosure.

[0017] It should be noted that the terms "first," "second," etc., used in this disclosure and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0018] This application provides a system disaster recovery control method. The executing entity of the system disaster recovery control method includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the device provided in this application: a server, a terminal, or other electronic devices. In other words, the system disaster recovery control method can be executed by software or hardware installed on a terminal device or a server device. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0019] The disaster recovery control method of this invention can be applied to, for example... Figure 1In this application environment, the client communicates with the server via a network. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will now be described in detail through specific embodiments.

[0020] Reference Figure 2 The diagram shown is a schematic flowchart of a system disaster recovery control method provided in an embodiment of the present invention. In this embodiment, the system disaster recovery control method includes: S1. Obtain the set of historical fault records of the business system to be tested, and extract the fault type and fault occurrence time corresponding to each historical fault record in the set of historical fault records.

[0021] In this embodiment of the invention, the fault type refers to the classification of the root causes that lead to the interruption or performance degradation of the business system, including but not limited to hardware faults (such as disk damage), software faults (such as application deadlock), network faults (such as link interruptions), data logic errors (such as accidental deletion), natural disasters, or power outages.

[0022] like Figure 3 As shown in this embodiment of the invention, the step of extracting the fault type and fault occurrence time corresponding to each historical fault record in the historical fault record set includes: S31. Perform data format identification on each historical fault record in the historical fault record set to obtain the record format type of each historical fault record; S32. Match the corresponding fault information parsing template from the preset fault parsing strategy library according to the record format type. The fault information parsing template includes fault type field positioning rules and fault occurrence time field positioning rules. S33. Determine the location of the fault type field from the historical fault records according to the fault type field location rules, and extract the field value of the fault type field from the location of the fault type field to obtain the fault type corresponding to each historical fault record; S34. Determine the location of the fault occurrence time field from the historical fault records according to the fault occurrence time field location rules, and extract the field value of the fault occurrence time field according to the location of the fault occurrence time field to obtain the fault occurrence time corresponding to each historical fault record.

[0023] In this embodiment of the invention, when identifying the data format of each historical fault record in the historical fault record set, the format classification technology based on multi-dimensional feature matching is used. Specifically, feature clustering is performed on the layout features, semantic layout logic, and information organization paradigm of each historical fault record to automatically distinguish the format belonging type of different historical fault records, thereby obtaining the record format type of each historical fault record.

[0024] In detail, the information arrangement patterns of records in different formats are abstracted into a standardized set of parsing rules in advance. Through the strong correlation mapping between format type and template identifier, the exclusive parsing template that is suitable for the current historical fault record format type can be quickly located. The two types of field positioning rules encapsulated in the template can be used to achieve targeted adaptation for the information distribution characteristics of records in different formats, avoiding information misalignment problems caused by general parsing logic.

[0025] Specifically, by using predefined fault class semantic features as anchors and combining them with the contextual information boundary constraints defined in the rules, the actual distribution location of the fault type field is accurately located in unstructured or semi-structured fault records. Then, through semantic feature matching and extraction operations, irrelevant and redundant information is filtered out, and finally, the accurate fault type identifier corresponding to each historical fault record is obtained.

[0026] The system uses a semantic paradigm of general time expression as the basis for recognition, relies on the pre-set time information arrangement boundaries in the rules to accurately determine the field position, and then uses a standardized cleaning operation of time sequence information to remove non-time-related interference content mixed in the records, finally obtaining the standardized fault occurrence time information corresponding to each historical fault record.

[0027] For example, in a healthcare scenario, the hospital information operation and maintenance system automatically pulls a collection of historical fault records from core systems such as HIS, LIS, and electronic medical records over the past three years. It accurately extracts the corresponding fault type from each fault record, covering categories such as billing link blockage, abnormal barcode flow of test results, interruption of medical record data synchronization, and timeout of medical insurance settlement interface. At the same time, it fully marks the specific occurrence time of each fault, covering different business periods such as outpatient peak, end-of-month medical insurance settlement, and night ward duty, forming a fault label dataset adapted to the characteristics of medical business.

[0028] For example, in a fintech scenario, a bank's intelligent operation and maintenance system retrieves a collection of historical fault records from core payment and clearing, online transfer, and merchant reconciliation systems over the past five years. From each fault record, it extracts the corresponding fault type, covering categories such as transaction link congestion, database lock contention, interface response timeout, and batch clearing queue backlog. Simultaneously, it marks the specific occurrence time of each fault, covering key periods such as end-of-month clearing peaks, daytime transaction peaks, and holiday business peaks, forming a fault feature dataset adapted to the characteristics of financial business.

[0029] In this embodiment of the invention, by automatically acquiring historical fault records and accurately extracting the fault type and occurrence time of each historical fault record, a structured fault knowledge graph can be built for the system, providing a reliable data foundation for subsequent fault disaster recovery drills, thereby significantly improving the scientificity and effectiveness of disaster recovery drills and effectively reducing the risk of business interruption.

[0030] S2. Construct a fault injection model based on the fault type and fault occurrence time corresponding to all the historical fault records.

[0031] In this embodiment of the invention, the fault injection model is a mathematical or logical abstract model constructed based on historical fault data, used to describe the statistical regularities, correlations, and triggering conditions between fault types and occurrence times. The fault injection model typically includes fault probability distributions, time series characteristics, inter-fault dependencies, and injection parameter templates. Its purpose is to controllably reproduce or simulate real fault scenarios in a training environment according to predetermined rules, thereby verifying the system's fault tolerance and recovery capabilities.

[0032] like Figure 4 As shown in this embodiment of the invention, the step of constructing a fault injection model based on the fault type and fault occurrence time corresponding to all the historical fault records includes: S41. Obtain the fault type and fault occurrence time corresponding to all historical fault records, and group all historical fault records according to the fault type to obtain multiple fault type groups; S42. Based on the fault occurrence time in each fault type group, calculate the time interval sequence between adjacent fault occurrence times in the fault type group, and fit the time interval probability distribution function corresponding to the fault type based on the time interval sequence; S43. Generate a fault occurrence time prediction function corresponding to the fault type based on the time interval probability distribution function and the first fault occurrence time in the fault type group; S44. Construct the model parameter structure of the fault injection model based on the fault type identifiers of all fault type groups and the corresponding fault occurrence time prediction functions; S45. Instantiate the model parameter structure into an executable fault injection model.

[0033] In this embodiment of the invention, the semantic features of fault types are used as the clustering dimension. Non-overlapping attribute aggregation operations are performed on all historical fault records, and historical fault records with the same fault attribute features are grouped into the same group, forming multiple fault type groups with clear boundaries and unified attributes, providing a regular grouped data foundation for subsequent time series feature analysis.

[0034] In detail, the fault occurrence records within the group are sorted in chronological order along the preset time dimension to extract the time span between two adjacent fault events, thereby forming a complete interval sequence. Then, through the multi-distribution hypothesis testing and fit calibration process, the probability distribution form with the highest matching degree with the sequence features is selected to complete the construction of the time interval probability distribution function for the corresponding fault type.

[0035] The earliest fault occurrence time within a group is used as the starting anchor point for deduction. Combined with the inherent evolution law of the verified time interval probability distribution, multiple rounds of recursive deduction of fault occurrence time are completed in the forward direction of time, forming a dedicated prediction logic that can cover the evolution law of fault occurrence time throughout the entire cycle, and obtaining the fault occurrence time prediction function for the corresponding fault type.

[0036] Furthermore, the unique identifier attribute of each type of fault is strongly associated with the corresponding prediction function logic. The attributes and logic of all fault categories are structurally collected according to a unified architecture specification, forming a standardized parameter framework with clear hierarchical relationships that can be identified and called by subsequent execution units, thus completing the parameter structure construction of the fault injection model.

[0037] Furthermore, the static parameter structure framework is adapted and connected with the underlying execution logic of the fault injection system to complete the runtime transformation of parameter attributes and the connection of logical links. This allows the encapsulated fault timing evolution rules to be directly called and executed by the system, ultimately generating a complete executable fault injection model that can be directly used in test scenarios.

[0038] In this embodiment of the invention, by integrating the type labels of historical faults with time series features to construct a fault injection model, discrete fault records can be transformed into a simulation rule base with temporal patterns and correlation logic. This model supports automated and programmable fault injection in the exercise environment according to the real fault distribution, thereby breaking through the limitations of traditional single fault simulation, making the verification of disaster recovery system closer to actual operational risks, helping to accurately assess the system's recovery capability under complex fault scenarios, and further improving the accuracy of system disaster recovery exercises.

[0039] S3. Obtain multiple preset services in the business system, identify the dependencies between the preset services, and generate a business service link graph of the preset services based on the dependencies.

[0040] In this embodiment of the invention, dependency refers to the interrelationship and constraint between preset services in terms of data flow, call chain, or resource consumption. For example, the order service depends on the identity information provided by the user authentication service, and the payment service depends on the transaction details generated by the order service. The business service link graph is a directed topology graph structure generated based on preset services and their dependencies. This graph uses business nodes as vertices and dependencies as edges, showing the complete call path from the user request entry point to backend resource support. It is the basic view for conducting fault impact analysis and formulating refined recovery strategies.

[0041] like Figure 5 As shown in this embodiment of the invention, generating the service link graph of the preset service based on the dependency relationship includes: S51. Obtain all preset services in the business system and the dependencies between each preset service; S52. Each of the preset services is taken as a service node, and any two of the service nodes are selected as the source service node and the target service node, respectively. S53. Construct corresponding directed edges based on the dependency relationship between the source service node and the target service node; S54. Using the source service node as the starting node and the target service node as the ending node, construct a service link graph of all the preset services obtained in the service system based on all the starting nodes, all the ending nodes and all the directed edges.

[0042] In this embodiment of the invention, various preset business entities predefined within the business system are collected, and the hierarchical dependencies and relationships such as calls, support, and linkages between different preset businesses are clarified, so as to complete the full information capture of business entities and business-related logic.

[0043] In detail, each independent preset business in the business system is abstracted into a basic business node in the graph structure. A single business node is randomly selected from all the determined business nodes as the starting source business node for link deduction, and at the same time, any other business node is selected as the ending target business node of the link.

[0044] Based on the real business dependency logic between the two types of business nodes that have been sorted out, the directional association relationship between the nodes is defined, and the corresponding directed association edge that can reflect the business flow logic is constructed to accurately map the business driving and transmission relationship between different business nodes.

[0045] Furthermore, by taking all selected source business nodes as the starting point of the overall link graph and all matched target business nodes as the ending point of the link graph, all participating business starting nodes, business ending nodes, and directed edges representing business relationships are integrated to complete the interconnection and integration of the overall structure. Finally, a business service link graph that can fully cover all preset businesses of the business system and truly restore the business flow and dependency logic is generated.

[0046] For example, in a healthcare scenario, the smart healthcare operation and maintenance system can batch acquire all pre-set medical services such as outpatient registration, doctor prescriptions, laboratory tests, report delivery, and medical insurance settlement. It can identify the call triggers, data flow, and resource sharing dependencies between these services. For instance, the prescription service depends on registration identity verification, and the report delivery depends on the return of laboratory data. Based on the association logic, it can generate a medical service link map covering the entire process, clearly presenting the flow path and association logic of each service node.

[0047] For example, in a fintech scenario, a bank's operation and maintenance management system can batch acquire all preset financial business data, such as user identity verification, account balance inquiry, transaction initiation, clearing and reconciliation, and result notification. It can identify the dependency relationships between these businesses, such as permission verification, data transmission, and process triggering. For instance, transaction initiation depends on balance verification, and clearing and reconciliation depends on transaction flow collection. Based on the correlation logic, a financial business service link graph covering the entire chain is generated, which fully presents the flow order and correlation path of each business node.

[0048] In this embodiment of the invention, by acquiring preset services and identifying their dependencies, a service link graph is generated, which can transform the originally scattered service nodes into a visualized global topology structure. In disaster recovery drills, this graph can be used to guide recovery priority ranking and resource allocation strategies, effectively avoiding secondary failures caused by unclear dependencies, thereby significantly improving the accuracy of system disaster recovery drills and the precision of service recovery.

[0049] S4. Based on the service link graph, determine the target node and redundant backup node for each preset service, and establish a traffic switching mapping relationship between the target node and the redundant backup node.

[0050] In this embodiment of the invention, the target node refers to the primary service node or instance (such as the primary database or primary application server) in the business service link graph that undertakes business processing functions and is in normal operating condition. It is the destination of business traffic under normal circumstances, and its health status directly determines business availability. It is also the object that needs to be protected or replaced during disaster recovery switchover. The redundant backup node is a standby service node or instance (such as a backup database or application cluster in a disaster recovery data center) that is pre-deployed to ensure business continuity and has the same or equivalent functions as the target node. It can take over business traffic when the target node fails and is a key redundant resource in the disaster recovery system.

[0051] Specifically, the traffic switching mapping relationship is a set of configuration rules and conversion logic defined in the routing rules that guide business traffic from the target node to the redundant backup node. This mapping relationship usually includes parameters such as IP address redirection, DNS resolution change, load balancing policy adjustment, and connection session persistence mechanism to ensure that the switching process is executable and rollbackable. It is the core control logic for realizing automated disaster recovery switching.

[0052] like Figure 6 As shown in this embodiment of the invention, determining the target node and redundant backup node for each preset service based on the service link graph includes: S61. Obtain the business node corresponding to each preset business in the business service link graph, and extract the node attribute information of each business node; S62. Take each of the preset service nodes as the node to be analyzed, and calculate the topological importance feature value of the node to be analyzed based on the number of predecessor nodes and successor nodes associated with the node to be analyzed in the service link graph. S63. Based on the node attribute information of the node to be analyzed, calculate the current processing capability feature value of the node to be analyzed, and combine the topological importance feature value and the current processing capability feature value of the node to be analyzed to calculate the comprehensive feature value of the node to be analyzed. S64. Select the preset number of nodes to be analyzed with the highest comprehensive feature value as the target nodes of the preset service, and select one or more nodes with the highest comprehensive feature value among the neighboring nodes that are directly adjacent to the target node in the service service link graph as preliminary candidate backup nodes. S65. Determine whether the preliminary candidate backup node is in the same physical fault domain as the target node; S66. If the preliminary candidate backup node and the target node are in the same physical fault domain, then the preliminary candidate backup node is removed and a new preliminary candidate backup node is selected from the remaining neighbor nodes. S67. If the preliminary candidate backup node and the target node are not in the same physical fault domain, the preliminary candidate backup node is directly determined as the redundant backup node for the preset service.

[0053] In this embodiment of the invention, a business service link graph is used as the data source. All business nodes belonging to a specific preset business are traversed, and the attribute parameter set attached to each node is retrieved through the graph's metadata interface. This parameter set contains key information such as the node's load capacity, processing speed, and its topological connection relationship in the link graph.

[0054] Specifically, taking each node to be analyzed as the center, based on the directed edge relationships in the business service link graph, the number of predecessor nodes pointing to the node and the number of successor nodes pointing to the node are counted respectively. Based on these two topological metrics, a weighted fusion algorithm is used to obtain the feature value reflecting the coreness of the node in the graph structure. The higher the feature value, the more significant the central role played by the node in the business data flow process.

[0055] In detail, based on the load capacity and processing speed in the node attribute information, the remaining processing potential of the node at the current moment is calculated through a preset capability assessment model, that is, the current processing capability feature value. This feature value and the topology importance feature value are input into a preset comprehensive evaluation function. According to the normalization and weighted synthesis technology, the two different dimensions of the measurement value are merged into a comprehensive feature value that can be uniformly ranked, thereby realizing the comprehensive quantification of node importance and real-time performance.

[0056] In this process, all nodes to be analyzed are sorted in descending order according to their comprehensive feature values. A preset number of nodes with the highest ranking are selected as target nodes for the preset services. Based on this, starting from the target nodes, a set of neighboring nodes with direct connection relationships with them is retrieved in the service link graph. The nodes in this set are then sorted again according to their comprehensive feature values, and the node with the highest value is selected as the initial candidate backup node, thereby ensuring that the backup node and the primary node have a direct and efficient connection path.

[0057] Furthermore, the system obtains the physical fault domain identifier of the preliminary candidate backup node and compares it with the physical fault domain identifier of the target node to determine whether they belong to the same fault boundary. If the determination result is the same fault domain, it means that the candidate node cannot provide effective redundancy when the primary node encounters a physical failure. Therefore, the system removes it and returns to the neighborhood search step to select again. If the determination result is different fault domains, it confirms that the node has physical isolation fault tolerance capability. The system directly marks it as the redundant backup node for the preset service, thereby completing the final determination of the high availability deployment scheme.

[0058] For example, in a healthcare scenario, based on the generated medical business service chain map, target nodes for carrying the main process are determined for each core pre-set business such as outpatient registration, test ordering, and medical insurance settlement. At the same time, modules with the same function deployed on independent server clusters are selected as redundant backup nodes. A traffic switching mapping relationship is established between the target nodes and the corresponding redundant backup nodes to ensure that business traffic can be quickly migrated to the backup nodes when the main node fails, thus ensuring the continuous operation of medical services.

[0059] For example, in a fintech scenario, based on the generated financial business service chain map, target nodes are identified for each core pre-set business such as identity verification, transaction clearing, and reconciliation to carry out the main process operation. At the same time, modules with the same function deployed in remote multi-active data centers are selected as redundant backup nodes. A traffic switching mapping relationship is established between the target nodes and the corresponding redundant backup nodes to ensure that business traffic can be quickly migrated to the backup nodes when the main node fails, thus ensuring the stability and uninterrupted operation of financial transactions.

[0060] In this embodiment of the invention, by determining the target node and redundant backup node based on the business service link graph and establishing the traffic switching mapping relationship between them, the abstract dependency topology can be transformed into an executable set of routing rules. This helps to achieve rapid and orderly transfer of business traffic when a failure occurs, avoids switching failures caused by routing errors or missing configurations, and improves the operational standardization and success rate of disaster recovery drills.

[0061] S5. Inject corresponding fault signals into the target node according to the fault injection model, and monitor the response status data of the business system in real time after each fault signal injection.

[0062] In this embodiment of the invention, response status data refers to a set of multi-dimensional indicators collected from the business system and its operating environment after the fault signal is injected. These indicators are used to characterize changes in system behavior, including but not limited to system-level indicators (such as CPU utilization, memory usage, and disk read / write latency), application-level indicators (such as request response time, error rate, and throughput), network-level indicators (such as packet loss rate and number of connections), and business-level indicators (such as transaction success rate and order processing volume). These indicators are used to quantitatively assess the impact of the fault on the system.

[0063] like Figure 7 As shown in this embodiment of the invention, injecting a corresponding fault signal into the target node according to the fault injection model includes: S71. Obtain the node identifier of the target node, and determine the deployment location of the target node in the business system based on the node identifier; S72. Generate a fault injection time point according to the fault occurrence time prediction function in the fault injection model, and match the fault signal template corresponding to the fault type to be injected from the preset fault signal library according to the fault injection model. S73. Generate a fault signal that conforms to the communication protocol specification of the target node according to the fault signal format and fault signal strength parameters in the fault signal template, and establish a signal transmission channel between the target node and the preset fault injector. S74. Based on the deployment location and the fault injection time point, the fault signal is sent to the target node through the signal transmission channel.

[0064] In this embodiment of the invention, based on the pre-acquired node identifier, a search is performed in the asset configuration database or infrastructure topology map of the business system. Through the mapping relationship between the identifier and physical coordinates, the deployment information such as the server rack number, network subnet segment, and process running environment of the node is located.

[0065] Specifically, the system invokes a pre-defined occurrence time prediction module within the fault injection model. This module, based on historical fault occurrence patterns and the current system load, uses a time-series analysis algorithm to calculate the target time point at which the fault signal should be injected, ensuring that the simulated behavior matches the actual timing of the fault. Simultaneously, the business system uses the fault type specified in the fault injection model as the query condition and performs template matching in a pre-defined fault signal library. This retrieves fault signal templates that are perfectly matched to the fault type in terms of waveform features and semantic encoding, ensuring a high degree of scenario realism in the injected content.

[0066] The process involves parsing the signal format description and signal strength parameters defined in the fault signal template, and encapsulating and converting the original signal content according to the communication protocol specifications adopted by the target node to generate a fault signal payload that can be correctly identified and processed within the target protocol stack. Simultaneously, the business system establishes a dedicated signal transmission channel between the fault injector and the target node by calling the system's underlying communication interface or proxy program. This channel is typically implemented using a virtual network connection or inter-process communication pipe to carry subsequent data transmission operations.

[0067] Furthermore, by using deployment location information as the addressing basis, the generated fault signal is encapsulated according to the network address or process identifier of the target node, and the signal transmission channel strictly follows the fault injection time point. When the preset timing is reached, the signal sending command is triggered, so that the fault signal is accurately delivered to the communication interface of the target node through the transmission channel to complete the execution of the entire fault injection action.

[0068] In this embodiment of the invention, by injecting a preset fault signal into the target node according to the fault injection model and monitoring the response status data of the business system in real time, the impact of real faults on the system can be simulated in a controlled environment. This can effectively verify the effectiveness of the disaster recovery strategy, accurately assess the fault tolerance boundary of the system, and thus significantly improve the business continuity assurance capability.

[0069] S6. Determine whether the response status data meets the preset disaster recovery switching trigger conditions.

[0070] S7. When the response status data does not meet the preset disaster recovery switching trigger conditions, the system disaster recovery control is stopped.

[0071] S8. When the response status data meets the preset disaster recovery switching trigger conditions, the service traffic of the target node is switched to the corresponding redundant backup node according to the traffic switching mapping relationship.

[0072] In this embodiment of the invention, the disaster recovery switchover trigger condition refers to a set of pre-defined quantitative thresholds or logical rules used to determine whether a business traffic switchover needs to be initiated. It is evaluated based on real-time monitored response status data and typically includes system performance indicators (such as response timeouts and error rate exceeding limits), availability indicators (such as node heartbeat loss and service unavailability duration), or custom business rules (such as transaction backlog). It serves as the decision-making basis between fault monitoring and switchover execution.

[0073] In this embodiment of the invention, response status data of the target node is continuously collected, and the response status data is compared item by item with preset disaster recovery switching trigger conditions. The trigger conditions are usually in the form of composite logic rules based on multi-dimensional indicators, such as a number of consecutive heartbeat losses accompanied by a request error rate exceeding a certain preset ratio. When the real-time collected health indicator curve completely matches all the judgment logic defined in the trigger conditions, the system's state machine transitions from the "normal operation state" to the "switching ready state", thereby generating a formal switching start command, avoiding false switching caused by sporadic jitter at a single point.

[0074] Specifically, the system's disaster recovery strategy library maintains a traffic switching mapping table in advance. This table uses the node identifier of the critical node as the query key and the network address and access port of the corresponding redundant backup node as the mapping value. Once a switching command is issued, the target node identifier that triggers the switching condition is used as the index to perform a precise search in the mapping table and quickly locate the redundant backup node pre-specified for that target node.

[0075] Furthermore, after obtaining the target address of the redundant backup node, the routing update interface of the traffic scheduling middleware is called to perform a back-switch operation on the original inbound traffic path of the target node: at the load balancer level, the backend server list of the virtual service is modified, the original target node is removed from the service group, and the redundant backup node is added to the active service group and given full traffic handling rights.

[0076] For traffic switching of stateful sessions, the process of migrating or rebuilding session information will be executed simultaneously to ensure that the client's request is seamlessly redirected to the new backend node without the client being aware of the connection interruption. This completes the final switching action of business traffic, allowing the redundant backup node to officially take over all service responsibilities of the original target node.

[0077] For example, in a healthcare scenario, when the response status data of core business nodes such as outpatient billing and medical insurance settlement triggers preset disaster recovery switching conditions such as interface timeout thresholds and service circuit breaker thresholds, the real-time business traffic such as patient registration and payment settlement carried by the original target node will be automatically and seamlessly switched to the corresponding redundant backup node for operation according to the pre-established traffic switching mapping relationship, so as to avoid problems such as outpatient window queuing and medical insurance settlement failures that affect the medical process.

[0078] For example, in a fintech scenario, when the response status data of core business nodes such as payment clearing and account verification triggers preset disaster recovery switching conditions such as response delay thresholds and error rate thresholds, the real-time business traffic such as user transfers and transaction deductions carried by the original target node is automatically and seamlessly switched to the corresponding redundant backup node for operation according to the pre-established traffic switching mapping relationship, so as to avoid risks that affect the continuity of financial services such as transaction interruption and fund accrual.

[0079] In this embodiment of the invention, by monitoring response status data in real time and matching it with preset disaster recovery switching trigger conditions, when the service damage reaches a threshold, the service traffic of critical nodes can be automatically switched to redundant backup nodes according to the traffic switching mapping relationship. This effectively shortens the time delay of human judgment and operation, ensures the timeliness and accuracy of switching actions, and provides efficient and reliable technical support for automated disaster recovery switching.

[0080] S9. After completing the business traffic switching, collect the operating performance indicators of the redundant backup node, and adjust the resource allocation parameters of the redundant backup node according to the operating performance indicators until the service capability of the business system is restored to the target level before the fault injection.

[0081] In this embodiment of the invention, the operational performance indicators refer to a set of quantitative parameters used to measure the actual carrying capacity and health status of a redundant backup node after business traffic is switched to it. These typically include throughput, response latency, CPU / memory utilization, disk I / O rate, network bandwidth usage, and the number of active database connection pools, used to assess whether the backup node meets the business operation requirements.

[0082] Specifically, resource allocation parameters refer to the set of configurable values ​​that control the supply and scheduling of computing resources for redundant backup nodes. These include, but are not limited to, CPU core allocation, memory heap size, disk I / O priority, network bandwidth rate limiting, database connection pool size, thread pool capacity, and cache space quota. Adjusting these parameters aims to dynamically optimize node performance to adapt to the business load after the switchover. The target level refers to the baseline service capability achieved by the business system under normal operating conditions before the disaster recovery switchover. It is usually defined by specific indicator thresholds, such as response time ≤ 200ms, error rate ≤ 0.1%, and throughput ≥ 1000TPS. It serves as the benchmark for determining whether recovery is complete, ensuring that the business quality after the backup node takes over is not lower than the pre-switch level.

[0083] like Figure 8 As shown in this embodiment of the invention, adjusting the resource allocation parameters of the redundant backup node according to the operational performance indicators until the service capability of the business system is restored to the target level before the fault injection includes: S81. Obtain the target level of the service capability of the business system before fault injection, and calculate the performance deviation value between each of the operational performance indicators and each indicator in the target level. S82. Determine the resource adjustment direction of the redundant backup node based on the performance deviation value; S83. According to the resource adjustment direction, the resource allocation parameters of the redundant backup node are gradually adjusted with a preset adjustment step size, and the updated operating performance indicators of the redundant backup node are collected again after each adjustment. S84. Determine whether the updated runtime performance indicators after each adjustment have reached or exceeded the target level; S85. If all the updated performance indicators after each adjustment reach or exceed the target level, then stop adjusting the resource allocation parameters. S86. If any of the updated performance indicators after each adjustment fails to reach the target level, the resource allocation parameter adjustment continues until the service capability of the business system is restored to the target level before the fault injection.

[0084] In this embodiment of the invention, the service capability indicators of the business system collected during normal operation before fault injection are used as the target level benchmark. This benchmark covers stable thresholds in multiple dimensions such as response latency, throughput rate, and resource utilization. During the stage after fault injection when the system switches to the redundant backup node, the actual operating performance indicators of the node are collected in real time, and each indicator is aligned to the corresponding dimension in the target level. The performance deviation value in each dimension is calculated by difference calculation.

[0085] In detail, each performance deviation value is analyzed for its positive / negative nature and magnitude: if the deviation value is positive, it indicates that the current indicator has exceeded the target level, and no adjustment is needed in that direction; if the deviation value is negative, it indicates that the current indicator has not reached the target level, and the corresponding resource parameters need to be adjusted along the compensation direction. Based on the distribution characteristics of the deviation values, the optimization direction of resource allocation is inferred—for example, if the response latency indicator lags behind, processing resources need to be increased; if the throughput is insufficient, bandwidth resources need to be expanded—thereby matching a targeted resource compensation strategy for each type of operational performance indicator.

[0086] Specifically, based on the adjustment direction, the resource allocation parameters of the redundant backup nodes are progressively modified with a preset adjustment step size as the change range for each operation. After each adjustment is completed, the updated running performance indicators of the node are immediately re-collected to form a new set of state snapshots. This allows the system to carefully observe the performance feedback brought about by each step of the operation, avoiding system oscillations or service interruptions caused by excessive adjustments at one time.

[0087] Furthermore, after each adjustment, the updated performance metrics are comprehensively compared with the target level. Only when all metrics reach or exceed the target level does the system determine that the service capability of the business system has been restored to the level before the fault injection, and then issues an instruction to stop adjusting, maintaining the current resource allocation state to achieve steady-state operation. If any metric is found to be below the target during the comparison process, the system determines that the convergence condition has not yet been met, and returns to step three to continue executing the next round of resource adjustments. This process is repeated iteratively until all performance metrics meet the target level, ultimately completing the full restoration of service capability.

[0088] In this embodiment of the invention, by collecting the operating performance indicators of redundant backup nodes and dynamically adjusting their resource allocation parameters accordingly until the service capabilities are restored to the target level before the fault injection, adaptive elastic optimization after the backup nodes take over is achieved. This can effectively cope with the performance degradation or resource bottlenecks that may occur after the switch, avoid the secondary service quality degradation caused by insufficient backup node configuration, ensure the integrity and stability of service recovery, and further improve the realism and reliability of disaster recovery drills.

[0089] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0090] like Figure 9 The diagram shown is a functional block diagram of a system disaster recovery control device provided in an embodiment of the present invention.

[0091] In this embodiment of the disclosure, a system disaster recovery control device is provided, which corresponds one-to-one with the system disaster recovery control method described in the above embodiments. For example... Figure 9 As shown, this system disaster recovery control device 100 can be installed in electronic devices. According to its functions, the system disaster recovery control device 100 includes a fault information extraction module 101, a fault model construction module 102, a relationship graph generation module 103, a mapping relationship establishment module 104, a fault signal injection module 105, a disaster recovery control module 106, and a disaster recovery module 107. Detailed descriptions of each functional module are as follows: The fault information extraction module 101 is used to obtain a set of historical fault records of the business system to be tested, and to extract the fault type and fault occurrence time corresponding to each historical fault record in the set of historical fault records; The fault model construction module 102 is used to construct a fault injection model based on the fault type and fault occurrence time corresponding to all the historical fault records; The relationship graph generation module 103 is used to acquire multiple preset services in the business system, identify the dependency relationships between each preset service, and generate a business service link graph of the preset services based on the dependency relationships. The mapping relationship establishment module 104 is used to determine the target node and redundant backup node of each preset service based on the service link graph, and to establish a traffic switching mapping relationship between the target node and the redundant backup node. The fault signal injection module 105 is used to inject corresponding fault signals into the target node according to the fault injection model, and to monitor the response status data of the business system in real time after each fault signal injection. The disaster recovery control module 106 is used to switch the service traffic of the target node to the corresponding redundant backup node according to the traffic switching mapping relationship when the response status data meets the preset disaster recovery switching trigger conditions. The disaster recovery module 107 is used to collect the operating performance indicators of the redundant backup node after the business traffic switching is completed, and adjust the resource allocation parameters of the redundant backup node according to the operating performance indicators until the service capability of the business system is restored to the target level before the fault injection.

[0092] In one embodiment, when the fault information extraction module 101 extracts the fault type and fault occurrence time corresponding to each historical fault record in the historical fault record set, it is used to: Data format identification is performed on each historical fault record in the historical fault record set to obtain the record format type of each historical fault record; According to the record format type, the corresponding fault information parsing template is matched from the preset fault parsing strategy library. The fault information parsing template includes fault type field positioning rules and fault occurrence time field positioning rules. The location of the fault type field is determined from the historical fault records according to the fault type field location rules, and the field value of the fault type field is extracted from the location of the fault type field to obtain the fault type corresponding to each historical fault record. The location of the fault occurrence time field is determined from the historical fault records according to the fault occurrence time field location rules, and the field value of the fault occurrence time field is extracted according to the location of the fault occurrence time field to obtain the fault occurrence time corresponding to each historical fault record.

[0093] In one embodiment, when the fault model construction module 102 constructs a fault injection model based on the fault type and fault occurrence time corresponding to all the historical fault records, it is used to: Obtain the fault type and fault occurrence time corresponding to all historical fault records, and group all historical fault records according to the fault type to obtain multiple fault type groups; Based on the fault occurrence time in each fault type group, calculate the time interval sequence between adjacent fault occurrence times in the fault type group, and fit the time interval probability distribution function corresponding to the fault type based on the time interval sequence; Based on the time interval probability distribution function and the first fault occurrence time in the fault type group, generate a fault occurrence time prediction function corresponding to the fault type; The model parameter structure of the fault injection model is constructed based on the fault type identifier of all fault type groups and the corresponding fault occurrence time prediction function. The model parameter structure is instantiated into an executable fault injection model.

[0094] In one embodiment, when the relationship graph generation module 103 generates a service link graph of the preset service based on the dependency relationship, it is used to: Obtain all preset services in the business system and the dependencies between each preset service; Each of the preset services is taken as a service node, and any two of the service nodes are selected as the source service node and the target service node, respectively. Construct corresponding directed edges based on the dependency relationship between the source service node and the target service node; Using the source service node as the starting node and the target service node as the ending node, a service link graph of all the preset services obtained in the service system is constructed based on all the starting nodes, all the ending nodes, and all the directed edges.

[0095] In one embodiment, when the mapping relationship establishment module 104 determines the target node and redundant backup node for each preset service based on the service link graph, it is used to: Obtain the business node corresponding to each preset business in the business service link graph, and extract the node attribute information of each business node; Each of the preset service nodes is taken as the node to be analyzed. Based on the number of predecessor nodes and successor nodes associated with the node to be analyzed in the service link graph, the topological importance feature value of the node to be analyzed is calculated. Based on the node attribute information of the node to be analyzed, calculate the current processing capability feature value of the node to be analyzed, and combine the topological importance feature value and the current processing capability feature value of the node to be analyzed to calculate the comprehensive feature value of the node to be analyzed. Select a preset number of nodes with the highest comprehensive feature value as the target nodes of the preset service, and select one or more nodes with the highest comprehensive feature value among the neighboring nodes that are directly adjacent to the target node in the service service link graph as preliminary candidate backup nodes. Determine whether the preliminary candidate backup node is in the same physical fault domain as the target node; If the preliminary candidate backup node and the target node are in the same physical fault domain, the preliminary candidate backup node is removed and a new preliminary candidate backup node is selected from the remaining neighboring nodes. If the preliminary candidate backup node and the target node are not in the same physical fault domain, the preliminary candidate backup node will be directly determined as the redundant backup node for the preset service.

[0096] In one embodiment, when the fault signal injection module 105 injects a corresponding fault signal into the target node according to the fault injection model, it is used to: Obtain the node identifier of the target node, and determine the deployment location of the target node in the business system based on the node identifier; The fault injection time point is generated according to the fault occurrence time prediction function in the fault injection model, and the fault signal template corresponding to the fault type to be injected is matched from the preset fault signal library according to the fault injection model. Based on the fault signal format and fault signal strength parameters in the fault signal template, a fault signal conforming to the target node communication protocol specification is generated, and a signal transmission channel is established between the target node and the preset fault injector. The fault signal is sent to the target node via the signal transmission channel based on the deployment location and the fault injection time.

[0097] In one embodiment, when the disaster recovery module 107 adjusts the resource allocation parameters of the redundant backup nodes according to the operational performance indicators until the service capability of the business system is restored to the target level before the fault injection, it is used to: Obtain the target level of the service capability of the business system before fault injection, and calculate the performance deviation value between each of the operational performance indicators and each indicator in the target level. The resource adjustment direction of the redundant backup node is determined based on the performance deviation value; According to the resource adjustment direction, the resource allocation parameters of the redundant backup nodes are gradually adjusted with a preset adjustment step size, and the updated operating performance indicators of the redundant backup nodes are collected again after each adjustment. Determine whether the updated performance indicators after each adjustment meet or exceed the target level; If all the updated performance metrics after each adjustment reach or exceed the target level, then stop adjusting the resource allocation parameters; If any of the updated performance metrics after each adjustment fails to reach the target level, the resource allocation parameter adjustment will continue until the service capability of the business system is restored to the target level before the fault injection.

[0098] In this invention, the specific limitations of a system disaster recovery control device can be found in the above-described limitations of the system disaster recovery control method, and will not be repeated here. Each module in the aforementioned system disaster recovery control device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0099] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 10As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of the system disaster recovery control method on the server side.

[0100] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 11 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps of the system disaster recovery control method on the client side.

[0101] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Obtain a set of historical fault records of the business system to be tested, and extract the fault type and fault occurrence time corresponding to each historical fault record in the set of historical fault records; A fault injection model is constructed based on the fault type and fault occurrence time corresponding to all the historical fault records. Obtain multiple preset services from the business system, identify the dependencies between each preset service, and generate a business service link graph of the preset services based on the dependencies; Based on the service link graph, the target node and redundant backup node of each preset service are determined, and a traffic switching mapping relationship is established between the target node and the redundant backup node. According to the fault injection model, corresponding fault signals are injected into the target node, and the response status data of the business system is monitored in real time after each fault signal injection. When the response status data meets the preset disaster recovery switching trigger conditions, the service traffic of the target node is switched to the corresponding redundant backup node according to the traffic switching mapping relationship; After the business traffic switching is completed, the operating performance indicators of the redundant backup nodes are collected, and the resource allocation parameters of the redundant backup nodes are adjusted according to the operating performance indicators until the service capability of the business system is restored to the target level before the fault injection.

[0102] In the several embodiments provided by this invention, it should be understood that the disclosed devices and apparatuses can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0103] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0104] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0105] In some embodiments of this example, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the method described in the above embodiments.

[0106] The readable storage medium of the present invention stores a computer program, which, when executed by a processor of an electronic device, can perform the following: Obtain a set of historical fault records of the business system to be tested, and extract the fault type and fault occurrence time corresponding to each historical fault record in the set of historical fault records; A fault injection model is constructed based on the fault type and fault occurrence time corresponding to all the historical fault records. Obtain multiple preset services from the business system, identify the dependencies between each preset service, and generate a business service link graph of the preset services based on the dependencies; Based on the service link graph, the target node and redundant backup node of each preset service are determined, and a traffic switching mapping relationship is established between the target node and the redundant backup node. According to the fault injection model, corresponding fault signals are injected into the target node, and the response status data of the business system is monitored in real time after each fault signal injection. When the response status data meets the preset disaster recovery switching trigger conditions, the service traffic of the target node is switched to the corresponding redundant backup node according to the traffic switching mapping relationship; After the business traffic switching is completed, the operating performance indicators of the redundant backup nodes are collected, and the resource allocation parameters of the redundant backup nodes are adjusted according to the operating performance indicators until the service capability of the business system is restored to the target level before the fault injection.

[0107] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0108] Computer-readable storage media may also store at least one computer-executable program / instruction, such as computer-readable instructions. Computer-readable storage media include, but are not limited to, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Computer-readable storage media may include, for example, read-only memory (ROM), hard disk, flash memory, etc. For example, a non-transitory computer-readable storage medium may be connected to a computing device such as a computer, and then, when the computing device executes the computer-readable instructions stored on the computer-readable storage medium, the various methods described above can be performed.

[0109] In addition, the computer device may include (but is not limited to) a data bus, an input / output (I / O) bus, a display, and input / output devices (e.g., keyboard, mouse, speakers, etc.).

[0110] In one embodiment, the at least one computer-executable instruction may also be compiled into or comprise a software product / computer program product, wherein one or more computer-executable instructions are executed by a processor to perform the steps of the various functions and / or methods in the embodiments described herein.

[0111] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Furthermore, any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory.

[0112] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0113] In the embodiments provided in this disclosure, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0114] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

[0115] It should be noted that any AI models, software tools, or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been authorized (with the knowledge and consent) by the relevant parties or has been fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

Claims

1. A system disaster recovery control method, characterized in that, The method includes: Obtain a set of historical fault records of the business system to be tested, and extract the fault type and fault occurrence time corresponding to each historical fault record in the set of historical fault records; A fault injection model is constructed based on the fault type and fault occurrence time corresponding to all the historical fault records. Obtain multiple preset services from the business system, identify the dependencies between each preset service, and generate a business service link graph of the preset services based on the dependencies; Based on the service link graph, the target node and redundant backup node of each preset service are determined, and a traffic switching mapping relationship is established between the target node and the redundant backup node. According to the fault injection model, corresponding fault signals are injected into the target node, and the response status data of the business system is monitored in real time after each fault signal injection. When the response status data meets the preset disaster recovery switching trigger conditions, the service traffic of the target node is switched to the corresponding redundant backup node according to the traffic switching mapping relationship; After the business traffic switching is completed, the operating performance indicators of the redundant backup nodes are collected, and the resource allocation parameters of the redundant backup nodes are adjusted according to the operating performance indicators until the service capability of the business system is restored to the target level before the fault injection.

2. The system disaster recovery control method as described in claim 1, characterized in that, The step of extracting the fault type and fault occurrence time corresponding to each historical fault record in the historical fault record set includes: Data format identification is performed on each historical fault record in the historical fault record set to obtain the record format type of each historical fault record; According to the record format type, the corresponding fault information parsing template is matched from the preset fault parsing strategy library. The fault information parsing template includes fault type field positioning rules and fault occurrence time field positioning rules. The location of the fault type field is determined from the historical fault records according to the fault type field location rules, and the field value of the fault type field is extracted from the location of the fault type field to obtain the fault type corresponding to each historical fault record. The location of the fault occurrence time field is determined from the historical fault records according to the fault occurrence time field location rules, and the field value of the fault occurrence time field is extracted according to the location of the fault occurrence time field to obtain the fault occurrence time corresponding to each historical fault record.

3. The system disaster recovery control method as described in claim 1, characterized in that, The step of constructing a fault injection model based on the fault type and fault occurrence time corresponding to all the historical fault records includes: Obtain the fault type and fault occurrence time corresponding to all historical fault records, and group all historical fault records according to the fault type to obtain multiple fault type groups; Based on the fault occurrence time in each fault type group, calculate the time interval sequence between adjacent fault occurrence times in the fault type group, and fit the time interval probability distribution function corresponding to the fault type based on the time interval sequence; Based on the time interval probability distribution function and the first fault occurrence time in the fault type group, generate a fault occurrence time prediction function corresponding to the fault type; The model parameter structure of the fault injection model is constructed based on the fault type identifier of all fault type groups and the corresponding fault occurrence time prediction function. The model parameter structure is instantiated into an executable fault injection model.

4. The system disaster recovery control method as described in claim 1, characterized in that, The step of generating the service link graph of the preset service based on the dependency relationship includes: Obtain all preset services in the business system and the dependencies between each preset service; Each of the preset services is taken as a service node, and any two of the service nodes are selected as the source service node and the target service node, respectively. Construct corresponding directed edges based on the dependency relationship between the source service node and the target service node; Using the source service node as the starting node and the target service node as the ending node, a service link graph of all the preset services obtained in the service system is constructed based on all the starting nodes, all the ending nodes, and all the directed edges.

5. The system disaster recovery control method as described in claim 1, characterized in that, The step of determining the target node and redundant backup node for each preset service based on the service link graph includes: Obtain the business node corresponding to each preset business in the business service link graph, and extract the node attribute information of each business node; Each of the preset service nodes is taken as the node to be analyzed. Based on the number of predecessor nodes and successor nodes associated with the node to be analyzed in the service link graph, the topological importance feature value of the node to be analyzed is calculated. Based on the node attribute information of the node to be analyzed, calculate the current processing capability feature value of the node to be analyzed, and combine the topological importance feature value and the current processing capability feature value of the node to be analyzed to calculate the comprehensive feature value of the node to be analyzed. Select a preset number of nodes with the highest comprehensive feature value as the target nodes of the preset service, and select one or more nodes with the highest comprehensive feature value among the neighboring nodes that are directly adjacent to the target node in the service service link graph as preliminary candidate backup nodes. Determine whether the preliminary candidate backup node is in the same physical fault domain as the target node; If the preliminary candidate backup node and the target node are in the same physical fault domain, the preliminary candidate backup node is removed and a new preliminary candidate backup node is selected from the remaining neighboring nodes. If the preliminary candidate backup node and the target node are not in the same physical fault domain, the preliminary candidate backup node will be directly determined as the redundant backup node for the preset service.

6. The system disaster recovery control method as described in claim 1, characterized in that, The step of injecting a corresponding fault signal into the target node according to the fault injection model includes: Obtain the node identifier of the target node, and determine the deployment location of the target node in the business system based on the node identifier; The fault injection time point is generated according to the fault occurrence time prediction function in the fault injection model, and the fault signal template corresponding to the fault type to be injected is matched from the preset fault signal library according to the fault injection model. Based on the fault signal format and fault signal strength parameters in the fault signal template, a fault signal conforming to the target node communication protocol specification is generated, and a signal transmission channel is established between the target node and the preset fault injector. The fault signal is sent to the target node via the signal transmission channel based on the deployment location and the fault injection time.

7. The system disaster recovery control method as described in claim 1, characterized in that, The step of adjusting the resource allocation parameters of the redundant backup nodes according to the operational performance indicators until the service capability of the business system is restored to the target level before the fault injection includes: Obtain the target level of the service capability of the business system before fault injection, and calculate the performance deviation value between each of the operational performance indicators and each indicator in the target level. The resource adjustment direction of the redundant backup node is determined based on the performance deviation value; According to the resource adjustment direction, the resource allocation parameters of the redundant backup nodes are gradually adjusted with a preset adjustment step size, and the updated operating performance indicators of the redundant backup nodes are collected again after each adjustment. Determine whether the updated performance indicators after each adjustment meet or exceed the target level; If all the updated performance metrics after each adjustment reach or exceed the target level, then stop adjusting the resource allocation parameters; If any of the updated performance metrics after each adjustment fails to reach the target level, the resource allocation parameter adjustment will continue until the service capability of the business system is restored to the target level before the fault injection.

8. A system disaster recovery control device, characterized in that, The device includes: The fault information extraction module is used to obtain a set of historical fault records of the business system to be tested, and to extract the fault type and fault occurrence time corresponding to each historical fault record in the set of historical fault records. The fault model construction module is used to construct a fault injection model based on the fault type and fault occurrence time corresponding to all the historical fault records. The relationship graph generation module is used to obtain multiple preset services in the business system, identify the dependency relationships between each preset service, and generate a business service link graph of the preset services based on the dependency relationships. The mapping relationship establishment module is used to determine the target node and redundant backup node of each preset service based on the service link graph, and to establish a traffic switching mapping relationship between the target node and the redundant backup node. The fault signal injection module is used to inject corresponding fault signals into the target node according to the fault injection model, and to monitor the response status data of the business system in real time after each fault signal injection. The disaster recovery control module is used to switch the service traffic of the target node to the corresponding redundant backup node according to the traffic switching mapping relationship when the response status data meets the preset disaster recovery switching trigger conditions. The disaster recovery module is used to collect the operating performance indicators of the redundant backup nodes after the business traffic switching is completed, and adjust the resource allocation parameters of the redundant backup nodes according to the operating performance indicators until the service capability of the business system is restored to the target level before the fault injection.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the system disaster recovery control method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the system disaster recovery control method as described in any one of claims 1 to 7.

Images