A distributed system disaster recovery fault drill method and device based on chain tracking

Abstract

The embodiment of the present invention provides a method, device, electronic device and readable medium for disaster recovery fault drill of distributed system based on chain tracking, which relates to the field of cloud computing technology. The method includes: inserting a custom observation probe into the source code; monitoring the dynamic context information of the program during operation through the custom observation probe; exporting the dynamic context information as a log file; capturing the changes of the log file by deploying an agent in the distributed system, and performing fault injection; recording the flow data of the production environment and saving it as mirrored flow; replaying the mirrored flow in the disaster recovery environment to generate a panoramic view of the business logic link; the panoramic view of the business logic link includes a normal business logic link panoramic view and an abnormal business logic link panoramic view; comparing the normal business logic link panoramic view with the abnormal business logic link panoramic view to perform fault diagnosis. The present invention can improve the fault coverage and the accuracy of fault discovery.

Description

Technical Field

[0001] The present invention relates to the field of cloud computing technology, and in particular to a distributed system disaster recovery fault drill method based on chain tracing, a distributed system disaster recovery fault drill device based on chain tracing, an electronic device and a computer-readable medium. Background Art

[0002] With the development of cloud computing and microservices and the increase in business demand, more software systems are split into distributed system architectures. The complexity of distributed systems has increased, and fault location and debugging have become complicated and difficult. Fault drills simulate various possible fault conditions in the system to test the response and recovery capabilities of the disaster recovery system, ensuring that data backup and recovery can be performed in a timely and accurate manner when a fault actually occurs, thereby ensuring business continuity and stability.

[0003] Existing chaos engineering technologies such as ChaosMesh can be used to simulate and test common faults in distributed systems. They mainly use larger-grained fault injection methods such as killing processes and unplugging network cables. However, this fault injection method has the problem of too large an impact range and incomplete coverage of scenarios. It is also unable to discover problems and fault tolerance in the system from a more fine-grained perspective. In addition, online systems are constantly evolving and changing, so fault drills cannot solve the problem once and for all. Summary of the invention

[0004] The embodiments of the present invention provide a distributed system disaster recovery fault drill method based on chain tracing, a distributed system disaster recovery fault drill device based on chain tracing, an electronic device and a computer-readable storage medium to solve the problems that the existing technology has a large impact range, incomplete coverage of scenarios, and cannot discover problems and fault tolerance in the system from a finer-grained perspective. In addition, the online system is in a process of continuous evolution and change, and fault drills cannot solve the problem once and for all.

[0005] The embodiment of the present invention discloses a distributed system disaster recovery fault drill method based on chain tracing, comprising:

[0006] S1: insert custom observation probes into the source code;

[0007] S2: monitoring the dynamic context information of the program during runtime through the custom observation probe;

[0008] S3: exporting the dynamic context information as a log file;

[0009] S4: By deploying an agent program in a distributed system, changes in the log file are captured and fault injection is performed;

[0010] S5: Record the traffic data of the production environment and save it as mirror traffic;

[0011] S6: replay the mirrored traffic in the disaster recovery environment to generate a panoramic view of the business logic link; the panoramic view of the business logic link includes a normal business logic link panoramic view and an abnormal business logic link panoramic view;

[0012] S7: Compare the normal business logic link panorama with the abnormal business logic link panorama to perform fault diagnosis.

[0013] The embodiment of the present invention discloses a distributed system disaster recovery fault drill device based on chain tracking, comprising:

[0014] Insertion module for inserting custom observation probes into source code;

[0015] A monitoring module, used to monitor the dynamic context information of the program during runtime through the custom observation probe;

[0016] An export module, used to export the dynamic context information as a log file;

[0017] An injection module, used to capture changes in the log files and perform fault injection by deploying an agent program in a distributed system;

[0018] The recording module is used to record the traffic data of the production environment and save it as mirrored traffic;

[0019] A generation module, used to replay the mirrored traffic in a disaster recovery environment to generate a panoramic view of the business logic link; the panoramic view of the business logic link includes a normal business logic link panoramic view and an abnormal business logic link panoramic view;

[0020] The diagnosis module is used to compare the normal business logic link panorama with the abnormal business logic link panorama to perform fault diagnosis.

[0021] The embodiment of the present invention further discloses an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0022] The memory is used to store computer programs;

[0023] The processor is used to implement the distributed system disaster recovery fault drill method based on chain tracing as described in an embodiment of the present invention when executing the program stored in the memory.

[0024] The embodiment of the present invention further discloses one or more computer-readable media on which instructions are stored, which, when executed by one or more processors, enable the processors to execute the distributed system disaster recovery fault drill method based on chain tracing as described in the embodiment of the present invention.

[0025] The embodiments of the present invention include the following advantages:

[0026] The present invention deploys an agent program in a distributed system, captures changes in log files, and performs fault injection, thereby achieving a more fine-grained and accurate fault injection method, while improving fault coverage, and thus discovering problems and fault tolerance in the system from a more fine-grained perspective. Based on link tracking technology, a business logic link panorama is generated, which dynamically describes changes in business logic links and has good adaptability to the evolution of online systems. Fault diagnosis is then performed by comparing the normal business logic link panorama with the abnormal business logic link panorama, thereby improving the accuracy of fault discovery. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a structural diagram of a same-city active-standby disaster recovery system provided in an embodiment of the present invention;

[0028] Figure 2 It is a flowchart of the steps of a distributed system disaster recovery failure drill method based on chain tracing provided in an embodiment of the present invention;

[0029] Figure 3 is a functional schematic diagram of an agent program provided in an embodiment of the present invention;

[0030] Figure 4 is a schematic diagram of a traffic recording provided in an embodiment of the present invention;

[0031] Figure 5 It is a block diagram of a distributed system disaster recovery fault drill device based on chain tracking provided in an embodiment of the present invention;

[0032] Figure 6 is a block diagram of an electronic device provided in an embodiment of the present invention;

[0033] Figure 7 is a schematic diagram of a computer-readable medium provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0035] Reference Figure 1, showing a schematic structural diagram of a same-city active-standby disaster recovery system provided in an embodiment of the present invention.

[0036] After the disaster recovery system plan is confirmed and implemented, fault drills are required to verify the effectiveness and robustness of the disaster recovery system plan. After repeated fault injections, the fault problems are located, thereby continuously improving and strengthening the disaster recovery system.

[0037] In order to better illustrate the purpose of the present invention, Figure 1 Taking the same-city active-standby disaster recovery system as an example, the pain points in the current disaster recovery system solution formulation process are explained. In the same-city active-standby disaster recovery system, services are deployed in the primary and standby availability zones, where the primary zone receives user requests and provides services to the outside world, and the standby zone does not receive user requests, but performs data backup at a certain period. When a failure occurs in the primary zone, such as power outages, fires, and other disasters, the primary zone cannot provide services normally to the outside world. At this time, the traffic is switched to the standby zone, and the user requests are processed by the standby zone resources to ensure as few interruptions to the business as possible. However, the types of failures include not only natural disasters such as power outages and fires, but also many unexpected failures. In order to ensure that the disaster recovery system can cope with as many failure scenarios as possible, a disaster recovery drill plan with as comprehensive coverage as possible is essential.

[0038] As part of the disaster recovery system plan, fault drills need to fully consider the system architecture characteristics and business logic links, create faults from multiple angles and aspects, and inject them into the system to verify and improve the disaster recovery plan.

[0039] The present invention proposes a distributed system disaster recovery fault drill method based on chain tracing, which focuses on helping disaster recovery personnel to better formulate fault injection plans to enhance the robustness of the disaster recovery system.

[0040] Reference Figure 2 , shows a step flow chart of a distributed system disaster recovery fault drill method based on chain tracking provided in an embodiment of the present invention.

[0041] The method for disaster recovery failure drill of a distributed system based on chain tracking may specifically include the following steps:

[0042] S1: Insert custom observation probes into the source code.

[0043] It should be noted that a custom observation probe proposed based on the present invention can automatically identify Web type interfaces and RPC type interfaces in distributed systems, and automatically implant observation probes in the interface methods of distributed applications to collect distributed system business link call logs and provide basic data for analyzing business logic link diagrams.

[0044] Among them, Web type interfaces mainly refer to service communication interfaces based on HTTP / HTTPS protocols. Usually, they are implemented through specifications such as RESTful API, GraphQL or SOAP, and are widely used interface types in modern distributed systems.

[0045] Among them, the RPC (Remote Procedure Call) type interface is a distributed communication mechanism that allows programs to call functions on remote servers just like calling local functions. This method hides the underlying network transmission details and simplifies development.

[0046] In a possible implementation, S1 specifically includes sub-steps S101 to S103:

[0047] S101: Convert source code into an abstract syntax tree.

[0048] Among them, the Abstract Syntax Tree (AST) is a tree-like data structure used to represent the grammatical structure of program code while removing details irrelevant to the program syntax. It is an abstract representation of the source code, in which each node corresponds to a grammatical structure in the source code (such as a statement, expression, function, etc.).

[0049] It should be noted that the abstract syntax tree parses the source code into a hierarchical tree that can clearly show the type and relationship of each code element (such as function, interface, condition, etc.). Based on AST, the target node (such as Web interface and RPC interface) can be identified efficiently and accurately, avoiding errors or omissions caused by manual operations.

[0050] S102: Traverse the abstract syntax tree and query tree nodes of the Web type interface and the RPC type interface according to the node type information.

[0051] S103: Insert custom observation probes at the beginning and end of the tree node.

[0052] It should be noted that by inserting probes at the entry and exit of method execution, dynamic context information such as request parameters, method running time, and return value can be fully recorded. Combined with the log information at the beginning and end of the method, the dependencies of the call chain can be analyzed to help build a complete call chain diagram.

[0053] In the present invention, the customized observation probe can significantly improve the log collection quality and link tracking capability of the distributed system, while reducing the maintenance cost, and provides a solid technical foundation for fault drills and disaster recovery system optimization.

[0054] S2: Monitor the dynamic context information of the program during runtime through custom observation probes.

[0055] Among them, monitoring items often include instruction monitoring, call monitoring, error detection, etc.

[0056] Specifically, monitoring logic can be dynamically inserted into key nodes of the program (such as method entry and exit) to collect call link information at runtime, including context data such as function name, parameters, return value, execution time, exception information, etc. These probes are embedded in the code in a non-invasive way without changing the business logic. They output information in real time through log files, monitoring platforms or distributed tracking systems, support applications such as link tracking, troubleshooting and performance analysis, and help to fully understand the running status and dependencies of the program.

[0057] S3: Export dynamic context information as log files.

[0058] Optionally, the link monitoring information items in the log file are as shown in Table 1.

[0059] Table 1 Link monitoring information items

[0060]

[0061] S4: By deploying agents in distributed systems, we can capture changes in log files and perform fault injection.

[0062] Reference Figure 3 , showing a functional schematic diagram of an agent program provided in an embodiment of the present invention.

[0063] Specifically, the Agent program is uploaded to all operating systems of the distributed system cluster for installation, and the Agent program includes versions of the Windows operating system and the Linux operating system. The Agent is deployed in the distributed system architecture.

[0064] Furthermore, the output logs of the custom observation probes are reported to the log processing module through unified integration for log analysis and visualization operations. By configuring the Agent conf file, the Agent listens to the specified log file, and when the log file changes, it can actively capture and push it to the log processing module.

[0065] Furthermore, the Agent program can proxy host-level and application-level fault injection, such as injecting downtime (poweroff / shutdown, etc.) into the server, modifying network parameters (ufw deny http), killing service processes (kill-9pid), closing container processes (docker stop containerid), and other operation instructions.

[0066] In a possible implementation, the agent program is also used to perform scene recognition.

[0067] Furthermore, the agent can identify scenarios such as operating systems, cloud-native application scenarios, and application types. For example, the agent can read the current operating system type and kernel version through the uname -a command; and can distinguish whether it is currently in a container environment through the cat / proc / 1 / cgroup command.

[0068] It should be noted that, compared with the traditional log agent, the agent in the present invention can not only be used to upload monitoring logs, but also be used to identify distributed scenarios and proxy the issuance of fault instructions.

[0069] In the present invention, by deploying the Agent program, not only can real-time monitoring and log capture be achieved, but also multi-level faults can be flexibly injected to accurately simulate system failure scenarios. At the same time, combined with scenario recognition and cross-platform adaptation capabilities, it helps distributed systems improve fault tolerance, recovery capabilities and availability, and ultimately optimizes the system's operating stability and reliability.

[0070] S5: Record the traffic data of the production environment and save it as mirror traffic.

[0071] Reference Figure 4 , showing a schematic diagram of a traffic recording provided in an embodiment of the present invention.

[0072] In a possible implementation, S5 specifically includes: deploying a network sniffing tool at the service cluster gateway to copy the user request data and the return data and save them as mirror traffic.

[0073] Among them, the sniffer tool is a network analysis tool used to capture and monitor data packets in the network and analyze the communication content. By deploying it at the gateway of the service cluster, the sniffer tool can replicate user requests and return data in real time and save it as mirror traffic for traffic analysis, troubleshooting, performance optimization and other scenarios. This tool can help you gain a deep understanding of the behavior and patterns of network communications and provide key data support for system operation status and problem location.

[0074] It should be noted that saving the traffic recording of the online environment can be used for traffic playback in the disaster recovery environment in subsequent steps.

[0075] Furthermore, compared with the traditional interface testing method, this method has the advantages of low testing cost and simulation of the traffic patterns of the real online environment, and can more completely cover the scenario business and business logic call graph.

[0076] S6: Replay the mirrored traffic in the disaster recovery environment to generate a panoramic view of the business logic link.

[0077] Optionally, the business logic link panorama includes: an interface unit, a link unit, and a node unit.

[0078] Among them, the interface unit is the smallest functional module in the system, usually composed of an interface or a method, completing an independent function. It stores the parameters and return information of the interface, as well as the dependencies with other units.

[0079] Furthermore, in addition to data such as ID, Name, and Tag, the metadata of the interface unit also includes a weight field to identify the importance of the interface unit. The higher the weight, the more traffic flows through it, and the more important the node unit is.

[0080] Among them, the link unit is a link composed of several node units through which a requested session flows. It can describe a certain independent request service. The set of all link units can describe the service-level functions that the entire system can complete. It stores the start and end time of a complete call link and all interface units on the complete link.

[0081] Among them, the node unit is a physical node composed of several microservices and bases, usually a physical server, a database cluster, or a pod composed of service containers with the same function.

[0082] The business logic link panorama is an intuitive visual representation of the service call relationship and business process in the distributed system. It fully reflects the business logic and operation architecture of the system by showing the dependency relationship between interface units, link units and node units. It not only includes call paths, service dependencies and key nodes, but also presents information such as traffic distribution and node weights, helping to identify system bottlenecks, optimize business processes, and quickly locate the scope of fault impact. It is an important tool for understanding and analyzing the operation status of distributed systems.

[0083] Furthermore, all traffic-accessible paths in the business logic link panorama are the selection range for fault injection, and node units and path units with higher weights need to be paid special attention to and tested.

[0084] Among them, the business logic link panorama includes a normal business logic link panorama and an abnormal business logic link panorama.

[0085] In a possible implementation, S6 specifically includes sub-steps S601 and S602:

[0086] S601: Before fault injection, extract metadata information of each element in the mirrored traffic and dependencies between the elements, and draw a panoramic view of the normal business logic link through a visualization UI tool based on the metadata information of each element in the mirrored traffic and dependencies between the elements.

[0087] Specifically, you can use a sniffing tool or proxy service to record network traffic in a real environment. Decode the captured traffic data and extract key fields related to the protocol (such as HTTP, gRPC). Structure the parsed traffic information, such as interface name, request method, parameters, return value, etc. Extract the following metadata information (interface unit metadata information, link unit metadata information, node unit metadata information) of each element from the parsed traffic, and record the upstream and downstream dependencies between interfaces based on the call order, dependent protocol and flow direction. A unique link identifier can be generated to associate all service nodes of the same call. After that, the extracted metadata information is organized into a graph structure, with nodes representing interface units or service nodes. Edges represent the dependency between two nodes in the call chain, marked as "call" or "return", and metadata information is added as attributes of the node or edge (such as time consumption, traffic size, etc.). Generate a panoramic view of the business logic link through tools (such as D3.js, Cytoscape). Use modern visualization frameworks (such as ECharts, GraphQL) to provide interactive functions, such as clicking on nodes to view detailed metadata information and filtering links by weight.

[0088] Optionally, a graph algorithm (such as force-directed layout) is used to optimize the positions of nodes and edges to make the graph structure more intuitive.

[0089] S602: After the fault is injected, the metadata information of each element in the mirrored traffic and the dependencies between the elements are extracted, and a panoramic view of the abnormal business logic link is drawn through a visual UI tool based on the metadata information of each element in the mirrored traffic and the dependencies between the elements.

[0090] In the present invention, by generating a panoramic view of normal and abnormal business logic links and combining it with visual analysis methods, not only can the business logic and dependencies of the system be clearly displayed, but also the scope of fault impact can be quickly and accurately identified and the effectiveness of the disaster recovery plan can be verified, thereby improving the reliability and fault tolerance of the distributed system.

[0091] In the process of fault drill, in addition to injecting faults into the distributed system, another very important process is to locate the system anomalies and monitor the error information after the fault injection. For example, when a node unit in the link unit fails, the length of the entire link unit will change, or a certain interface unit will return a parameter error, etc. Therefore, reasonable monitoring indicators and methods can help to discover the scope of fault impact.

[0092] S7: Compare the normal business logic link panorama with the abnormal business logic link panorama to perform fault diagnosis.

[0093] In a possible implementation, S7 specifically includes sub-steps S701 to S704:

[0094] S701: Compare the number of interface units and upper and lower dependencies in the normal business logic link panorama and the abnormal business logic link panorama to see if they are consistent. If so, proceed to the next step. Otherwise, determine that there is a fault.

[0095] S702: Compare the number and length of link units and the types of interface units in the normal business logic link panorama and the abnormal business logic link panorama to see if they are consistent. If so, proceed to the next step. Otherwise, determine that there is a fault.

[0096] S703: Compare the number of node units in the normal business logic link panorama and the abnormal business logic link panorama to see if they are consistent. If so, proceed to the next step. Otherwise, determine that there is a fault.

[0097] The above judgment logic outputs inconsistent conclusions as assertions. In addition to the above link logic inconsistencies, the following indicators are monitored at the same time to locate the impact of the fault.

[0098] In the present invention, by comparing the changes of the interface unit, link unit and node unit, the specific location and scope of the fault can be accurately identified, thereby quickly locating the root cause of the problem.

[0099] S704: Perform fault diagnosis through unit error response analysis, crash detection, and resource performance monitoring.

[0100] Unit error response analysis refers to error response analysis by capturing Exception, Error, and Panic logs in each unit, including service logs and link tracking information logs captured by the Agent.

[0101] Among them, crash detection refers to real-time detection of the health status of each unit through heartbeat transmission, and an alarm is immediately triggered once a unit becomes unavailable or the response times out.

[0102] Among them, resource performance monitoring refers to the use of stress testing tools and performance testing tools to constantly monitor the resource usage of node units. In addition to monitoring basic indicators such as memory usage, CPU utilization, disk IO, etc., it also includes specific business indicators, such as database throughput, service QPS, etc.

[0103] Furthermore, not only can we determine whether the link changes are abnormal, but we can also fully understand the impact of the failure on system stability and performance by monitoring error logs, node health status and resource usage.

[0104] The embodiments of the present invention include the following advantages:

[0105] The present invention deploys an agent program in a distributed system, captures changes in log files, and performs fault injection, thereby achieving a more fine-grained and accurate fault injection method, while improving fault coverage, and thus discovering problems and fault tolerance in the system from a more fine-grained perspective. Based on link tracking technology, a business logic link panorama is generated, which dynamically describes changes in business logic links and has good adaptability to the evolution of online systems. Fault diagnosis is then performed by comparing the normal business logic link panorama with the abnormal business logic link panorama, thereby improving the accuracy of fault discovery.

[0106] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0107] In addition, refer to Figure 5 , shows a block diagram of a distributed system disaster recovery fault drill device based on chain tracking provided in an embodiment of the present invention.

[0108] The embodiment of the present invention discloses a distributed system disaster recovery fault drill device 20 based on chain tracing, comprising:

[0109] Insertion module 201, used to insert a custom observation probe into the source code;

[0110] A monitoring module 202, configured to monitor the dynamic context information of the program during runtime through the custom observation probe;

[0111] An export module 203, configured to export the dynamic context information as a log file;

[0112] The injection module 204 is used to capture the changes of the log files and perform fault injection by deploying an agent program in the distributed system;

[0113] The recording module 205 is used to record the traffic data of the production environment and save it as mirror traffic;

[0114] A generating module 206 is used to replay the mirrored traffic in a disaster recovery environment to generate a panoramic view of the business logic link; the panoramic view of the business logic link includes a normal business logic link panoramic view and an abnormal business logic link panoramic view;

[0115] The diagnosis module 207 is used to compare the normal business logic link panorama with the abnormal business logic link panorama to perform fault diagnosis.

[0116] In a possible implementation manner, the insertion module 201 is specifically used for:

[0117] Convert the source code into an abstract syntax tree;

[0118] Traversing the abstract syntax tree, and querying tree nodes of the Web type interface and the RPC type interface according to the node type information;

[0119] Insert custom observation probes at the beginning and end of the tree node.

[0120] In a possible implementation manner, the agent program is also used to perform scene recognition.

[0121] In a possible implementation manner, the recording module 205 is specifically configured to:

[0122] By deploying a network sniffing tool at the service cluster gateway, the user request data and the return data are copied and saved as the mirror traffic.

[0123] In a possible implementation, the generating module 206 is specifically configured to:

[0124] Before fault injection, the metadata information of each element in the mirrored traffic and the dependency relationship between the elements are extracted, and according to the metadata information of each element in the mirrored traffic and the dependency relationship between the elements, a panoramic view of the normal business logic link is drawn through a visual UI tool;

[0125] After the fault is injected, the metadata information of each element in the mirrored traffic and the dependencies between the elements are extracted, and based on the metadata information of each element in the mirrored traffic and the dependencies between the elements, a panoramic view of the abnormal business logic link is drawn through a visual UI tool.

[0126] In a possible implementation, the service logic link panorama includes: an interface unit, a link unit, and a node unit.

[0127] In a possible implementation, the diagnosis module 207 is specifically used to:

[0128] Compare the number of interface units and upper and lower dependencies of the normal business logic link panorama with the abnormal business logic link panorama to see if they are consistent; if so, proceed to the next step; otherwise, determine that there is a fault;

[0129] Compare the number, length and type of interface units of the link units in the normal business logic link panorama and the abnormal business logic link panorama to see if they are consistent; if so, proceed to the next step; otherwise, determine that there is a fault;

[0130] Compare the number of node units in the normal business logic link panorama with that in the abnormal business logic link panorama to see if they are consistent; if so, proceed to the next step; otherwise, determine that there is a fault;

[0131] Perform fault diagnosis through unit error response analysis, crash detection, and resource performance monitoring.

[0132] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0133] The embodiments of the present invention include the following advantages:

[0134] The present invention deploys an agent program in a distributed system, captures changes in log files, and performs fault injection, thereby achieving a more fine-grained and accurate fault injection method, while improving fault coverage, and thus discovering problems and fault tolerance in the system from a more fine-grained perspective. Based on link tracking technology, a business logic link panorama is generated, which dynamically describes changes in business logic links and has good adaptability to the evolution of online systems. Fault diagnosis is then performed by comparing the normal business logic link panorama with the abnormal business logic link panorama, thereby improving the accuracy of fault discovery.

[0135] In addition, refer to Figure 6 , shows a block diagram of an electronic device provided in an embodiment of the present invention. The embodiment of the present invention also provides an electronic device, including a processor 1301, a communication interface 1302, a memory 1303 and a communication bus 1304, wherein the processor 1301, the communication interface 1302, and the memory 1303 communicate with each other through the communication bus 1304,

[0136] Memory 1303, used for storing computer programs;

[0137] The processor 1301 is used to implement the following steps when executing the program stored in the memory 1303:

[0138] S1: insert custom observation probes into the source code;

[0139] S2: monitoring the dynamic context information of the program during runtime through the custom observation probe;

[0140] S3: exporting the dynamic context information as a log file;

[0141] S4: By deploying an agent program in a distributed system, changes in the log file are captured and fault injection is performed;

[0142] S5: Record the traffic data of the production environment and save it as mirror traffic;

[0143] S6: replay the mirrored traffic in the disaster recovery environment to generate a panoramic view of the business logic link; the panoramic view of the business logic link includes a normal business logic link panoramic view and an abnormal business logic link panoramic view;

[0144] S7: Compare the normal business logic link panorama with the abnormal business logic link panorama to perform fault diagnosis.

[0145] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0146] The communication interface is used for communication between the above terminal and other devices.

[0147] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0148] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0149] Reference Figure 7, showing a schematic diagram of a computer-readable medium provided in an embodiment of the present invention. In another embodiment provided by the present invention, a computer-readable storage medium 1401 is also provided, in which instructions are stored, and when the instructions are executed on a computer, the computer executes the distributed system disaster recovery fault drill method based on chain tracking described in the above embodiment.

[0150] In another embodiment provided by the present invention, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the distributed system disaster recovery fault drill method based on chain tracing described in the above embodiment.

[0151] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0152] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0153] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0154] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A distributed system disaster recovery failure drill method based on chain tracking, characterized in that: include: S1: insert custom observation probes into the source code; S2: monitoring the dynamic context information of the program during runtime through the custom observation probe; S3: exporting the dynamic context information as a log file; S4: By deploying an agent program in a distributed system, changes in the log file are captured and fault injection is performed; S5: Record the traffic data of the production environment and save it as mirror traffic; S6: replay the mirrored traffic in the disaster recovery environment to generate a panoramic view of the business logic link; the panoramic view of the business logic link includes a normal business logic link panoramic view and an abnormal business logic link panoramic view; S7: Compare the normal business logic link panorama with the abnormal business logic link panorama to perform fault diagnosis.

2. The method for disaster recovery failure drill of a distributed system based on chain tracing according to claim 1 is characterized in that: The S1 specifically includes: S101: Convert the source code into an abstract syntax tree; S102: Traversing the abstract syntax tree, and querying tree nodes of Web type interfaces and RPC type interfaces according to node type information; S103: Inserting custom observation probes at the beginning and end of the tree node.

3. The method for disaster recovery failure drill of a distributed system based on chain tracking according to claim 1 is characterized in that: The agent is also used to perform scene recognition.

4. The method for disaster recovery failure drill of a distributed system based on chain tracing according to claim 1, characterized in that: The S5 is specifically: By deploying a network sniffing tool at the service cluster gateway, the user request data and the return data are copied and saved as the mirror traffic.

5. The method for disaster recovery failure drill of a distributed system based on chain tracing according to claim 1, characterized in that: The S6 specifically includes: S601: before the fault injection, extract the metadata information of each element in the mirrored traffic and the dependency relationship between the elements, and draw a panoramic view of the normal business logic link through a visual UI tool according to the metadata information of each element in the mirrored traffic and the dependency relationship between the elements; S602: After the fault is injected, the metadata information of each element in the mirrored traffic and the dependencies between the elements are extracted, and according to the metadata information of each element in the mirrored traffic and the dependencies between the elements, a panoramic view of the abnormal business logic link is drawn through a visual UI tool.

6. The method for disaster recovery failure drill of a distributed system based on chain tracing according to claim 1, characterized in that: The business logic link panorama includes: an interface unit, a link unit and a node unit.

7. The method for disaster recovery failure drill of a distributed system based on chain tracing according to claim 6 is characterized in that: The S7 specifically includes: S701: Compare the number of interface units and upper and lower dependencies of the normal business logic link panorama and the abnormal business logic link panorama to see if they are consistent; if so, proceed to the next step; otherwise, determine that there is a fault; S702: Compare the number, length and type of interface units of the link units in the normal business logic link panorama and the abnormal business logic link panorama to see if they are consistent; if so, proceed to the next step; otherwise, determine that there is a fault; S703: Compare the number of node units in the normal business logic link panorama with that in the abnormal business logic link panorama to see if they are consistent; if so, proceed to the next step; otherwise, determine that there is a fault; S704: Perform fault diagnosis through unit error response analysis, crash detection, and resource performance monitoring.

8. A distributed system disaster recovery fault drill device based on chain tracking, characterized in that: include: Insertion module for inserting custom observation probes into source code; A monitoring module, used to monitor the dynamic context information of the program during runtime through the custom observation probe; An export module, used to export the dynamic context information as a log file; An injection module, used to capture changes in the log files and perform fault injection by deploying an agent program in a distributed system; The recording module is used to record the traffic data of the production environment and save it as mirrored traffic; A generation module, used to replay the mirrored traffic in a disaster recovery environment to generate a panoramic view of the business logic link; the panoramic view of the business logic link includes a normal business logic link panoramic view and an abnormal business logic link panoramic view; The diagnosis module is used to compare the normal business logic link panorama with the abnormal business logic link panorama to perform fault diagnosis.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the distributed system disaster recovery fault drill method based on chain tracing as described in any one of claims 1 to 7 when executing the program stored in the memory.

10. A computer-readable medium, characterized in that Instructions are stored thereon, which, when executed by one or more processors, enable the processors to execute the distributed system disaster recovery fault drill method based on chain tracing as described in any one of claims 1-7.

Images