Fault processing method and device of service system, equipment and storage medium

Abstract

The embodiment of the invention provides a fault processing method and device for a service system, equipment and a storage medium. The method comprises the following steps: acquiring link tracking data corresponding to the service system; generating a link tracking topology corresponding to the service system according to the link tracking data corresponding to the service system; according to the link tracking topology corresponding to the service system, determining whether a fault exists in the service system; if the business system has the fault, generating a fault processing scheme for ensuring the normal operation of the business system; and processing the fault in the service system by adopting the fault processing scheme. According to the embodiment of the invention, through the link tracking data and the link tracking topology, whether the fault exists in the service system can be quickly identified, and the corresponding fault processing scheme is automatically generated to quickly process the fault in the service system, so that the problem that the system cannot operate due to fault diffusion is avoided, and the efficient and stable operation of the service system is ensured.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method and device for handling faults in a business system, an electronic device, and a storage medium. Background Art

[0002] With the rapid promotion of new products and the sharp increase in business volume, the complexity of business systems has also increased significantly. The calling relationships between centers have become intricate, and the difficulty of troubleshooting has increased exponentially. In addition, with the continuous increase in the number of customers and the uncertainty of the external environment, the stability risks faced by business systems have become increasingly prominent. When facing external attacks and internal system failures, the current business system mainly relies on preset alarm strategies for problem troubleshooting. This method is not only time-consuming and slow to respond, but also lacks timeliness, flexibility, and fault tolerance.

[0003] Although there are a variety of link tracking tools (such as SkyWalking, Eagle Eye, Zipkin, etc.) and circuit breaker or traffic management tools (such as Hystrix, Sentinel, etc.) on the market, most of these tools are designed for a single scenario and cannot ensure that the business system can operate normally even when encountering a failure.

[0004] Therefore, in the face of an increasingly complex business environment, how to ensure that business systems can operate normally even when encountering failures, thereby improving the stability of business systems and user satisfaction, is an issue that needs to be urgently addressed in this field. Summary of the invention

[0005] The embodiment of the present application provides a fault handling method for a business system to solve the problem that the business system cannot operate normally when encountering a fault.

[0006] Correspondingly, the embodiments of the present application also provide a fault handling device for a business system, an electronic device and a storage medium to ensure the implementation and application of the above method.

[0007] In order to solve the above problems, an embodiment of the present application discloses a fault handling method for a business system, the method comprising:

[0008] Obtaining link tracking data corresponding to the business system;

[0009] Generate a link tracking topology corresponding to the business system according to the link tracking data corresponding to the business system;

[0010] Determining whether there is a fault in the business system according to the link tracking topology corresponding to the business system;

[0011] If there is a fault in the business system, a fault handling solution is generated to ensure the normal operation of the business system;

[0012] The fault handling solution is used to handle the fault in the business system.

[0013] The embodiment of the present application also discloses a fault handling device for a business system, the device comprising:

[0014] A link tracking data module, used to obtain link tracking data corresponding to the business system;

[0015] A link tracking topology module, used to generate a link tracking topology corresponding to the business system according to the link tracking data corresponding to the business system;

[0016] A fault determination module, used to determine whether there is a fault in the business system according to the link tracking topology corresponding to the business system;

[0017] A fault existence module is used to generate a fault handling solution for ensuring the normal operation of the business system if a fault exists in the business system;

[0018] A fault handling module is used to handle faults in the business system using the fault handling solution.

[0019] An embodiment of the present application also discloses an electronic device, including: a processor; and a memory, on which executable code is stored. When the executable code is executed, the processor executes a fault handling method of a business system as described in any one of the embodiments of the present application.

[0020] The embodiments of the present application also disclose one or more machine-readable media on which executable codes are stored. When the executable codes are executed, the processor executes the fault handling method of the business system as described in any one of the embodiments of the present application.

[0021] Compared with the prior art, the embodiments of the present application have the following advantages:

[0022] In an embodiment of the present application, the link tracking data corresponding to the business system is obtained; based on the link tracking data corresponding to the business system, a link tracking topology corresponding to the business system is generated; based on the link tracking topology corresponding to the business system, it is determined whether there is a fault in the business system; if there is a fault in the business system, a fault handling solution is generated to ensure the normal operation of the business system; and the fault handling solution is used to handle the fault in the business system. Through link tracking data, the embodiment of the present application can monitor the interaction between various components in the business system in real time to ensure a comprehensive understanding of the operating status of the system; based on the generated link tracking topology, it can quickly identify whether there is a fault in the business system, discover the problem in time, and avoid the problem of the system being unable to operate due to the spread of the fault; once a fault is detected, the system can automatically generate a corresponding fault handling solution to reduce manual intervention and improve processing efficiency, and use the generated fault handling solution to quickly handle the fault in the business system to ensure that the system operation will not be affected by the fault, thereby ensuring the efficient and stable operation of the business system. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a system architecture diagram of a distributed system of the present application;

[0024] Figure 2 This is a flowchart of a method for troubleshooting a business system according to an embodiment of the present application;

[0025] Figure 3 It is a schematic diagram of the link tracking topology of the order placement interface at the initial moment;

[0026] Figure 4 It is a schematic diagram of the link tracking topology before and after the order placement interface is updated at the next moment;

[0027] Figure 5 This is a schematic diagram of the link tracking topology before and after the order placement interface is updated at the next moment;

[0028] Figure 6 It is a structural block diagram of an embodiment of a fault handling device of a business system of the present application;

[0029] Figure 7 It is a schematic diagram of the structure of a device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0031] In related technologies, predictions can often only be made for applications, and users are alerted when deviations occur between predicted values ​​and actual observed values ​​to remind them that there are abnormalities in the application. However, automatic service fault tolerance is not achieved when there are abnormalities in the application.

[0032] Service fault tolerance means that in a distributed system, a service becomes unavailable due to an increase in the number of services, network reasons, high concurrency, etc. However, due to the dependencies between services, the failure will spread, causing the entire system to fail or even paralyze. Therefore, service fault tolerance is required to ensure that when a single service fails, it will not affect the normal operation of other services.

[0033] In order to achieve automatic service fault tolerance in distributed systems, it is necessary to prevent the spread of faults through service fault tolerance strategies. This can be simply understood as the need to use a fault handling method to achieve automatic service fault tolerance in distributed systems in order to improve system stability and thus improve user satisfaction.

[0034] In the distributed system of the embodiment of the present application, a business system and a fault handling system are included, and the business system and the fault handling system are communicatively connected.

[0035] Reference Figure 1 , is a system architecture diagram of a distributed system of the present application, including a presentation layer, a business layer and a data layer.

[0036] 1) The display layer sources include the business system interface and the fault handling system interface.

[0037] 2) The fault handling solution execution module and the fault handling solution generation module are run in the business layer. The fault handling solution execution module is divided into a reporting module, an update module, and an execution module; the fault handling solution generation module is divided into a generation module, a solution module, a push module, and a display module.

[0038] In actual applications, the fault handling solution execution module runs in the business system, while the fault handling solution generation module runs in the fault handling system.

[0039] 3) The data layer is implemented using a MySQL (a relational database management system) cluster to store link tracking data and fault handling solutions.

[0040] Reference Figure 2 , is a flowchart of an embodiment of a method for troubleshooting a business system of the present application, comprising the following steps:

[0041] Step 201: Obtain link tracking data corresponding to the business system;

[0042] In an embodiment of the present application, the business system in the distributed system obtains link tracking data through a tool (such as Zipkin, Jaeger, SkyWalking, etc.) specifically used to collect and analyze call link data between various services in the distributed system.

[0043] After the business system obtains the link tracking data corresponding to the business system, it will Figure 1 The reporting module in reports the link tracking data corresponding to the business system to the fault handling system, so that the fault handling system processes the link tracking data corresponding to the business system and generates a fault handling solution.

[0044] To ensure timely response, the business system will report link tracking data once per second.

[0045] Step 202: Generate a link tracking topology corresponding to the business system according to the link tracking data corresponding to the business system;

[0046] When the reporting module of the business system reports the link tracking data corresponding to the business system to the fault handling system, specifically to the generation module in the fault handling system, the generation module generates the link tracking topology corresponding to the business system based on the collected link tracking data.

[0047] Step 203: Determine whether there is a fault in the business system according to the link tracking topology corresponding to the business system;

[0048] When the link tracking topology corresponding to the business system is generated in the generation module, it will be transmitted to the solution module so that the solution module can analyze the link tracking topology and determine whether there is a fault in the business system.

[0049] Step 204: If the business system has a fault, generate a fault handling solution for ensuring the normal operation of the business system;

[0050] If the solution module analyzes the link tracking topology and determines that there is a fault in the business system, a fault handling plan will be developed.

[0051] Specifically, Figure 1 The push module in the system will send update messages of the latest fault handling solutions, which are implemented using MQ (Message Queue) clusters (for example, ctg-mq). The update module will periodically poll or consume update messages to pull fault handling solutions from the push module. Periodic polling is implemented through a scheduled task scheduling system (for example, using xxl-job). As a general capability, fault handling must be strictly controlled for access, which is why access control is required (for example, using the IAM center). Furthermore, through Figure 1In the display module, you can query the fault handling solutions of each business system in the link tracking topology.

[0052] Step 205: Use the fault handling solution to handle the fault in the business system.

[0053] When a fault handling module is defined in the solution module, the update module in the business system is allowed to pull the fault handling solution from the solution module and update the fault handling solution in the business system.

[0054] Subsequently, the execution module in the business system will execute the fault handling solution in the update module to handle the fault in the business system, avoid the problem of the business system being unable to operate due to the spread of the fault, and improve the efficient and stable operation of the business system.

[0055] In an embodiment of the present application, the fault handling system will automatically formulate a fault handling plan based on the operating status of the business system. The business system executes the latest fault handling plan, realizes fault handling of the business system, and then realizes automatic service fault tolerance of the distributed system, thereby improving the stability of the business system.

[0056] In one embodiment of the present application, the link tracking data has a corresponding data dimension, and generating a link tracking topology corresponding to the business system according to the link tracking data corresponding to the business system includes:

[0057] Analyze the link tracking data corresponding to the business system under different data dimensions, and generate a plurality of topological nodes under different data dimensions for storing the link tracking data and a topological relationship between the plurality of topological nodes;

[0058] According to the multiple topological nodes under different data dimensions and the topological relationships between the multiple topological nodes, a link tracking topology corresponding to the business system under different data dimensions is generated.

[0059] In a distributed system, the business system is deployed on the server and provides services to the outside world through interfaces (such as HTTP interface, RPC interface, etc.).

[0060] Therefore, in an embodiment of the present application, the link tracking data reported by the business system to the fault handling system is divided into a server dimension, a business dimension, and an interface dimension.

[0061] The link tracking data of the server dimension includes: server IP (Internet Protocol, Internet Protocol address), CPU (Central Processing Unit) usage, CPU load, memory usage, disk space usage, disk I / O (Input / Output, input / output), network traffic, number of system processes, business system name, etc.;

[0062] The link tracking data of the business dimension includes: business system name, server IP, called business system name, number of calls, and average response time;

[0063] The link tracking data in the interface dimension includes: business system name, interface URL (Uniform Resource Locator), TraceID (Trace Identifier), SpanID (Span Identifier), start time, end time, response time (end time - start time), request IP, request user information, etc.

[0064] Furthermore, the interface dimension is further divided into a TraceID dimension and a statistical window (eg, 1s) dimension.

[0065] Since the link tracking data has multiple dimensions, the link tracking topology under the dimension will be generated based on the link tracking data of different dimensions, that is, the link tracking topology will also be divided into server dimension, business dimension and interface dimension.

[0066] Specifically, the topological node of the server dimension is the server, the topological node of the business dimension is the business system, and the topological node of the interface dimension is the interface. Each topological node contains the link tracking data under this dimension.

[0067] Furthermore, in each dimension, the topological relationship between multiple topological nodes is analyzed. Specifically, in the server dimension, the topological relationship between servers is obtained through the dependency relationship between servers. In the business dimension, the topological relationship between business systems is obtained through the calling relationship between business systems. In the interface dimension, the topological relationship between interfaces is obtained through the calling relationship between interfaces.

[0068] In different dimensions, a link tracking topology in the dimension is generated based on the topological nodes and topological relationships in the dimension.

[0069] In one embodiment of the present application, the topological node has a corresponding utilization rate, the topological relationship has a corresponding performance state, and the determining whether there is a fault in the business system according to the link tracking topology corresponding to the business system includes:

[0070] Determine a first fault level corresponding to the topological node according to the utilization rate corresponding to the topological node; the first fault level is one of normal, warning and abnormal;

[0071] Determine, according to the performance state corresponding to the topological relationship, a second fault level corresponding to the topological relationship; the second fault level is one of smooth, busy and congested;

[0072] If the first fault level is normal and the second fault level is smooth, determining that there is no fault in the business system;

[0073] If the first fault level is the warning or the exception, and the second fault level is the busy or the congestion, it is determined that a fault exists in the business system.

[0074] In the embodiment of the present application, whether it is the server dimension, the business dimension, or the interface dimension, the topological nodes have three fault levels: normal, warning, and abnormal, and the topological relationship also has three fault levels: smooth, busy, and congested.

[0075] Specifically, when generating the link tracking topology, the fault levels of the topological nodes and topological relationships are determined according to the preset ranges of the three fault levels.

[0076] Regardless of the dimension, the fault level corresponding to the topological node is determined based on the utilization of the topological node. For example, in the server dimension, the preset variables that determine the fault level of the server include the utilization corresponding to the CPU, memory, hard disk, network, and number of system processes.

[0077] In addition, the fault level corresponding to the topological relationship is determined based on the performance status of the topological relationship. For example, in the server dimension, the preset variables that determine the fault level of the topological relationship include the performance status of the network. In the business dimension, the preset variables that determine the fault level of the topological relationship include the number of calls and the average response time. In the interface dimension, the preset variables that determine the fault level of the topological relationship include the number of calls and the average response time.

[0078] In one embodiment of the present application, when the data dimension is a server dimension, the topological node is a server for implementing the operation of the business system, the server has a corresponding inlet network process and an outlet network flow, and if the business system fails, a fault handling solution is generated to ensure the normal operation of the business system, including:

[0079] If the first fault level is the warning, generating a first warning solution for limiting the inlet network traffic to a first traffic and suspending operation of a first number of business systems;

[0080] If the first fault level is the abnormality, a first abnormality solution is generated for limiting the inlet network traffic to a second traffic and suspending the operation of a second number of business systems; the first traffic is greater than the second traffic, and the first number is less than the second number;

[0081] If the second failure level is the busy level, generating a first busy solution for limiting the egress network traffic to a third traffic level;

[0082] If the second fault level is the congestion, generating a first congestion solution for limiting the egress network flow to a fourth flow; the third flow is greater than the fourth flow;

[0083] Using the first warning solution or the first abnormal solution as a first processing solution;

[0084] using the first busy solution or the first congested solution as a second processing solution;

[0085] The first processing solution and the second processing solution are combined to obtain a first fault processing solution for normal operation of the business system in the server dimension.

[0086] The fault handling solutions corresponding to the server dimension in the embodiment of the present application include isolation, downgrade and current limiting. For the server, isolation refers to isolating the core business system from the non-core business system and using different servers; downgrade refers to suspending non-essential business systems (log systems, automated testing systems, daily patrol tasks, etc.), cleaning up outdated logs, etc.; current limiting refers to limiting the traffic of abnormal networks.

[0087] In the embodiment of the present application, the fault handling solution for the server dimension is referred to as the “first fault handling solution”, the fault handling solution for the service dimension is referred to as the “second fault handling solution”, and the fault handling solution for the interface dimension is referred to as the “third fault handling solution”.

[0088] In the server dimension, the first fault handling solution is formulated according to the generated link tracking topology in the server dimension as follows:

[0089] 1) When the server failure level is normal, no operation is required; when it is a warning, use the current limiting and degradation strategy to limit the inlet network traffic and suspend some non-essential business systems (the first warning solution that limits the inlet network traffic to the first traffic and suspends the operation of the first number of business systems); when it is abnormal, use the current limiting and degradation strategy to further limit the inlet network traffic, suspend all non-essential business systems, and suspend some core business systems (the first abnormal solution that limits the inlet network traffic to the second traffic and suspends the operation of the second number of business systems).

[0090] When the situation improves, restore the operations corresponding to the fault handling solution. For example, from warning -> normal, restore all operations of the current limiting and degradation strategies.

[0091] 2) The failure level of the topological relationship of the server is smooth, and no operation is required; when it is busy, a current limiting strategy is used to limit the export network traffic to the downstream node (the first busy solution that limits the export network traffic to the third traffic); when it is congested, a current limiting strategy is used to further limit the export network traffic to the downstream node (the first congested solution that limits the export network traffic to the fourth traffic).

[0092] When the situation improves, resume the operations corresponding to the troubleshooting solution.

[0093] In an embodiment of the present application, when the data dimension is a business dimension, the topological node is the business system, the business system is divided into a core business system and a non-core business system, the business system includes a non-core interface and a downstream node, and if the business system has a fault, a fault handling solution for ensuring the normal operation of the business system is generated, including:

[0094] If the first fault level is the warning, generating a second warning scheme for limiting the request amount of the non-core business system and the request amount of the non-core interface;

[0095] If the first fault level is the exception, generating a second exception solution for suspending the request of the non-core business system and the use of the non-core interface, and limiting the request amount of the core business system;

[0096] If the second failure level is the busy level, generating a second busy solution for limiting the request amount of the downstream node to the first request amount;

[0097] If the second fault level is the congestion, generating a second congestion solution for limiting the request amount of the downstream node to a second request amount; the first request amount is greater than the second request amount;

[0098] Using the second warning solution or the second abnormal solution as a third processing solution;

[0099] using the second busy solution or the second congested solution as a fourth processing solution;

[0100] The third processing solution and the fourth processing solution are combined to obtain a second fault processing solution for normal operation of the business system in the business dimension.

[0101] The fault handling solutions for the business dimension in the embodiments of the present application include isolation, downgrade and current limiting. For the business system, isolation refers to isolating the core interface from the non-core interface and using different resources (thread pool, semaphore, etc.); current limiting refers to limiting the number of requests for a certain business system, giving priority to the requests of the core business system; downgrade refers to reducing the resources of the non-core interface (thread pool, semaphore, etc.) or suspending the use of the non-core interface, giving priority to the resources of the core interface.

[0102] In the business dimension, the second fault handling solution is formulated according to the generated link tracking topology of the business dimension as follows:

[0103] 1) When the fault level of the service is normal, no operation is required; when it is a warning, a flow limiting strategy is used to limit the request volume of non-core business systems and the request volume of non-core interfaces (a second warning solution to limit the request volume of the non-core business systems and the request volume of the non-core interfaces); when it is an abnormality, a flow limiting and degradation strategy is used to suspend requests from non-core business systems, limit the request volume of core businesses, and suspend the use of non-core interfaces (a second abnormal solution to suspend requests from non-core business systems and the use of non-core interfaces, and limit the request volume of the core business systems).

[0104] When the situation improves, resume the operations corresponding to the troubleshooting solution.

[0105] 2) The failure level of the service topology relationship is smooth, no operation is required; when it is busy, a current limiting strategy is used to limit the request amount to the downstream node (the second busy solution that limits the request amount to the downstream node to the first request amount); when it is congested, a current limiting strategy is used to further limit the request amount to the downstream node (the second congested solution that limits the request amount to the downstream node to the second request amount).

[0106] When the situation improves, resume the operations corresponding to the troubleshooting solution.

[0107] In one embodiment of the present application, when the data dimension is an interface dimension, the topological node is an interface of the business system, the business system is used to implement core business, the operation of the interface depends on the downstream node, and if the business system fails, a fault handling solution is generated to ensure the normal operation of the business system, including:

[0108] If the first fault level is the warning, generating a third warning scheme for limiting the request amount of the interface to a third request amount;

[0109] If the first fault level is the exception, a third exception solution is generated for limiting the request amount of the interface to a fourth request amount and limiting the request amount of the core service; the third request amount is greater than the fourth request amount;

[0110] If the second failure level is the busy level, generating a third busy solution for reducing the timeout period of the interface request to the first time and limiting the request amount of the downstream node to a fifth request amount;

[0111] If the second fault level is the congestion, a third congestion solution is generated for reducing the timeout time of the interface request to the second time, limiting the request amount of the downstream node to the sixth request amount, and suspending the call of the abnormal interface; the first time is greater than the second time, and the fifth request amount is greater than the sixth request amount;

[0112] Using the third warning solution or the third abnormal solution as the fifth processing solution;

[0113] using the third busy solution or the third congestion solution as a sixth processing solution;

[0114] The fifth processing scheme and the sixth processing scheme are combined to obtain a third fault processing scheme for the normal operation of the business system in the interface dimension.

[0115] The fault handling solutions for the interface dimension in the embodiments of the present application include timeout, current limiting, fuse and fallback. For the interface, timeout refers to setting a timeout (connectTimeout, readTimeout and writeTimeout) for the interface request; current limiting, for the provider, is to limit the number of requests for a certain interface, and for the requester, is to limit the number of requests for a certain interface; fuse means no longer calling a certain abnormal interface, and checking again after a period of time to see if it has recovered; fallback needs to be used in conjunction with timeout, current limiting and fuse, and there are strategies such as returning default data, fault silence and fast failure.

[0116] At the interface level, the third fault handling solution is formulated based on the generated link tracking topology at the interface level as follows:

[0117] 1) When the fault level of the interface is normal, no operation is required; when it is a warning, a current limiting strategy is used to limit the request volume of the interface (the third warning solution that limits the request volume of the interface to the third request volume); when it is abnormal, a current limiting strategy is used to further limit the request volume of the interface (the third abnormal solution that limits the request volume of the interface to the fourth request volume and limits the request volume of the core business).

[0118] When the situation improves, resume the operations corresponding to the troubleshooting solution.

[0119] 2) When the failure level of the topological relationship of the interface is smooth, a timeout strategy is used, and a timeout time is set for the interface request; when it is busy, a timeout and current limiting strategy is used to reduce the timeout time of the interface request and limit the amount of requests to the downstream node (the third busy solution of reducing the timeout time of the interface request to the first time and limiting the amount of requests to the downstream node to the fifth amount); when it is congested, a timeout, current limiting, fuse and fallback strategy is used to reduce the timeout time of the interface request and further limit the amount of requests to the downstream node. If the number of exceptions exceeds a certain number, the fuse is blown and the fallback strategy is used (the third congestion solution of reducing the timeout time of the interface request to the second time, limiting the amount of requests to the downstream node to the sixth amount and suspending the call of the abnormal interface).

[0120] When the situation improves, resume the operations corresponding to the troubleshooting solution.

[0121] In the embodiment of the present application, the formulation of the fault handling solutions in the three dimensions are independent of each other, and the specific operations are also processed according to the actions configured in the fault handling solutions.

[0122] In one embodiment of the present application, after the fault handling solution is used to handle the fault in the business system, the method includes:

[0123] A fault handling report corresponding to the business system is generated according to the link tracking topology corresponding to the business system, the fault existing in the business system and the fault handling solution.

[0124] In the embodiment of the present application, the solution module in the fault handling system will formulate a fault handling solution and generate a fault handling report.

[0125] The contents of the troubleshooting report include:

[0126] 1) Server Dimension

[0127] From the failure level of one server to the failure level of another server, the first failure handling solution used is the link tracking topology in the server dimension.

[0128] 2) Business Dimension

[0129] From the fault level of one business system to the fault level of another business system, a second fault handling solution is used, and a link tracking topology in the business dimension.

[0130] 3) Interface Dimension

[0131] From the fault level of one interface to the fault level of another interface, a third fault handling scheme is used, the link tracking topology of the interface dimension.

[0132] In the embodiment of the present application, although the formulation of the fault handling plan and the generation of the fault handling report are both performed in the solution module, they are processed separately, thereby speeding up the response speed of the fault handling.

[0133] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the following is an exemplary description of a fault handling solution of an order placement interface:

[0134] An order request involves the ordering interface of the order center, the user information query interface of the user center, the product information query interface of the product center, and the order payment interface of the payment center. In addition, the attention interface of the activity center will also call the user information query interface of the user center. The order center reports the interface data of the statistical window dimension of the ordering interface to the fault handling system, and the activity center reports the interface data of the statistical window dimension of the attention interface to the fault handling system. The fault handling system will generate the following based on the collected link tracking data: Figure 3 The link tracking topology diagram of the order interface at the initial moment is shown.

[0135] At the initial moment, it can be seen that the fault level of each interface is normal, the topological relationship of the interface is smooth, and the fault handling plan formulated is: 1) The fault handling plan of the interface is no operation required; 2) The topological relationship service fault tolerance strategy of the interface is a timeout of 10s.

[0136] At the next moment, the number of calls to the user information query interface collected by the fault handling system increased from 20 / s to 200 / s (in addition to the order interface and the attention interface, there are other interface calls), an increase of 10 times, and the average response time increased from 10ms to 100ms, also an increase of 10 times; the number of calls to the attention interface increased from 10 / s to 80 / s, and the average response time increased from 100ms to 200ms due to the increase in the average response time of the user information query interface; the number of calls to the order interface, the product information query interface, and the order payment interface increased from 10 / s to 80 / s, and the average response time of the product information query interface and the order payment interface remained unchanged. The order interface increased from 300ms to 400ms due to the increase in the average response time of the user information query interface, and the link tracking topology of the order interface was updated as follows: 1) The fault level of the attention interface and the user information query interface changed to warning; 2) The fault level of the topological relationship of the attention interface, the order interface, and the user information query interface changed to busy; 3) Others remained unchanged.

[0137] Reference Figure 4 , is a schematic diagram of the link tracking topology before and after the order placement interface is updated at the next moment.

[0138] The developed troubleshooting plan is updated as follows: 1) A current limiting policy is added to the attention interface, and the number of calls is limited to 20 / s; 2) A current limiting policy is added to the user information query interface, and the number of calls is limited to 100 / s; 3) The topological relationship between the attention interface and the user information query interface, the timeout is changed to 100ms, and the current limit is 20 / s; 4) Others remain unchanged.

[0139] After executing the fault handling solution, the average response time of the query user interface information interface and the attention interface returned to normal to ensure that the query user interface would not be overwhelmed. In addition, the fault level of the topological relationship between the order placement interface and the query user information interface also returned to normal to ensure that the capacity of the core interface is not reduced, thereby improving user satisfaction.

[0140] At the next moment, the average response time of the order payment interface collected by the fault handling system increased from 100ms to 2000ms, and the average response time of the order interface increased from 300ms to 2000ms. The link tracking topology of the order interface was updated as follows: 1) The fault level of the order interface and the order payment interface became abnormal; 2) The topological relationship between the order interface and the order payment interface became congested.

[0141] Reference Figure 5 , is a schematic diagram of the link tracking topology before and after the order interface is updated at the next moment.

[0142] The developed troubleshooting plan is updated as follows: 1) A flow limiting strategy is added to both the order placement interface and the order payment interface, limiting the number of requests to 10 / s; 2) A flow limiting strategy is added to the topological relationship between the order placement interface and the order payment interface, limiting the flow to 10 / s, and changing the timeout to 2s; 3) Other changes remain unchanged.

[0143] After executing the fault handling plan, the average response time of the order interface and the order payment interface returned to normal. Although the processing capacity was limited, the business functions were guaranteed to be used normally. In addition, the fault level of the order interface and the order payment interface became a warning, and the fault level of the topological relationship between the order interface and the user information query interface became busy.

[0144] When the request volume decreases and the interface capacity returns to normal, the reverse operation can be performed and will not be repeated here.

[0145] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0146] On the basis of the above embodiments, this embodiment further provides a fault handling device for a business system, which is applied to electronic devices such as terminal devices and servers.

[0147] Reference Figure 6 , shows a structural block diagram of an embodiment of a fault handling device for a business system of the present application, which may specifically include the following modules:

[0148] The link tracking data module 601 is used to obtain the link tracking data corresponding to the business system;

[0149] The link tracking topology module 602 is used to generate a link tracking topology corresponding to the business system according to the link tracking data corresponding to the business system;

[0150] A fault determination module 603 is used to determine whether there is a fault in the business system according to the link tracking topology corresponding to the business system;

[0151] A fault existence module 604 is used to generate a fault handling solution for ensuring the normal operation of the business system if a fault exists in the business system;

[0152] The fault handling module 605 is used to handle the fault in the business system by using the fault handling solution.

[0153] The embodiment of the present application also provides a non-volatile readable storage medium, which stores one or more modules (programs). When the one or more modules are applied to a device, the device can execute instructions (instructions) of each method step in the embodiment of the present application.

[0154] The present application embodiment provides one or more machine-readable media on which instructions are stored, and when executed by one or more processors, an electronic device executes one or more of the methods described in the above embodiments. In the present application embodiment, the electronic device includes various types of devices such as terminal devices and servers (clusters).

[0155] The embodiments of the present disclosure may be implemented as a device configured as desired using any appropriate hardware, firmware, software, or any combination thereof, and the device may include electronic devices such as terminal devices and servers (clusters). Figure 7 An exemplary apparatus 700 that can be used to implement various embodiments described in this application is schematically illustrated.

[0156] For one embodiment, Figure 7 An exemplary apparatus 700 is shown having one or more processors 702, a control module (chip set) 704 coupled to at least one of the processor(s) 702, a memory 706 coupled to the control module 704, a non-volatile memory (NVM) / storage device 708 coupled to the control module 704, one or more input / output devices 710 coupled to the control module 704, and a network interface 712 coupled to the control module 704.

[0157] The processor 702 may include one or more single-core or multi-core processors, and the processor 702 may include any combination of general-purpose processors or special-purpose processors (such as graphics processors, application processors, baseband processors, etc.). In some embodiments, the device 700 can be used as a terminal device, server (cluster), etc. described in the embodiments of the present application.

[0158] In some embodiments, the apparatus 700 may include one or more computer-readable media (e.g., memory 706 or NVM / storage device 708) having instructions 714 and one or more processors 702 combined with the one or more computer-readable media and configured to execute the instructions 714 to implement a module to perform the actions described in the present disclosure.

[0159] For one embodiment, the control module 704 may include any suitable interface controller to provide any suitable interface to at least one of the processor(s) 702 and / or any suitable device or component in communication with the control module 704 .

[0160] The control module 704 may include a memory controller module to provide an interface to the memory 706. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0161] The memory 706 may be used, for example, to load and store data and / or instructions 714 for the device 700. For one embodiment, the memory 706 may include any suitable volatile memory, such as a suitable DRAM. In some embodiments, the memory 706 may include double data rate type four synchronous dynamic random access memory (DDR4 SDRAM).

[0162] For one embodiment, control module 704 may include one or more input / output controllers to provide an interface to NVM / storage device 708 and input / output device(s) 710 .

[0163] For example, NVM / storage 708 may be used to store data and / or instructions 714. NVM / storage 708 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable non-volatile storage device(s) (e.g., one or more hard disk drives (HDDs), one or more compact disk (CD) drives, and / or one or more digital versatile disk (DVD) drives).

[0164] NVM / storage device 708 may include storage resources that are physically part of the device on which apparatus 700 is installed, or it may be accessible to the device without being part of the device. For example, NVM / storage device 708 may be accessed via input / output device(s) 710 over a network.

[0165] (One or more) input / output devices 710 may provide an interface for the apparatus 700 to communicate with any other appropriate device, and the input / output device 710 may include a communication component, an audio component, a sensor component, etc. The network interface 712 may provide an interface for the apparatus 700 to communicate through one or more networks, and the apparatus 700 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, for example, accessing a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, 5G, etc., or a combination thereof for wireless communication.

[0166] For one embodiment, at least one of the processor(s) 702 may be packaged together with the logic of one or more controllers (e.g., a memory controller module) of the control module 704. For one embodiment, at least one of the processor(s) 702 may be packaged together with the logic of one or more controllers of the control module 704 to form a system-in-package (SiP). For one embodiment, at least one of the processor(s) 702 may be integrated on the same die with the logic of one or more controllers of the control module 704. For one embodiment, at least one of the processor(s) 702 may be integrated on the same die with the logic of one or more controllers of the control module 704 to form a system-on-chip (SoC).

[0167] In various embodiments, the apparatus 700 may be, but is not limited to, a terminal device such as a server, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet computer, a netbook, etc.). In various embodiments, the apparatus 700 may have more or fewer components and / or a different architecture. For example, in some embodiments, the apparatus 700 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touch screen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0168] Among them, the main control chip can be used as a processor or control module in the detection device, sensor data, location information, etc. are stored in a memory or NVM / storage device, the sensor group can be used as an input / output device, and the communication interface may include a network interface.

[0169] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0170] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0171] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable business system fault processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable business system fault processing terminal device generate instructions for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0172] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable business system fault processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0173] These computer program instructions may also be loaded onto a computer or other programmable business system fault processing terminal device, so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions executed on the computer or other programmable terminal device for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0174] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0175] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0176] The above is a detailed introduction to a fault handling method and device for a business system, an electronic device and a storage medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for handling a fault in a business system, characterized in that: The method comprises: Obtaining link tracking data corresponding to the business system; Generate a link tracking topology corresponding to the business system according to the link tracking data corresponding to the business system; Determining whether there is a fault in the business system according to the link tracking topology corresponding to the business system; If there is a fault in the business system, a fault handling solution is generated to ensure the normal operation of the business system; The fault handling solution is used to handle the fault in the business system.

2. The method according to claim 1, characterized in that The link tracking data has a corresponding data dimension, and generating a link tracking topology corresponding to the business system according to the link tracking data corresponding to the business system includes: Analyze the link tracking data corresponding to the business system under different data dimensions, and generate a plurality of topological nodes under different data dimensions for storing the link tracking data and a topological relationship between the plurality of topological nodes; According to the multiple topological nodes under different data dimensions and the topological relationships between the multiple topological nodes, a link tracking topology corresponding to the business system under different data dimensions is generated.

3. The method according to claim 2, characterized in that The topological nodes have corresponding utilization rates, the topological relationships have corresponding performance states, and the determining whether there is a fault in the business system according to the link tracking topology corresponding to the business system includes: Determine a first fault level corresponding to the topological node according to the utilization rate corresponding to the topological node; the first fault level is one of normal, warning and abnormal; Determine, according to the performance state corresponding to the topological relationship, a second fault level corresponding to the topological relationship; the second fault level is one of smooth, busy and congested; If the first fault level is normal and the second fault level is smooth, determining that there is no fault in the business system; If the first fault level is the warning or the exception, and the second fault level is the busy or the congestion, it is determined that a fault exists in the business system.

4. The method according to claim 3, characterized in that When the data dimension is a server dimension, the topological node is a server for implementing the operation of the business system, and the server has a corresponding inlet network process and an outlet network flow. If the business system fails, a fault handling solution is generated to ensure the normal operation of the business system, including: If the first fault level is the warning, generating a first warning solution for limiting the inlet network traffic to a first traffic and suspending operation of a first number of business systems; If the first fault level is the abnormality, a first abnormality solution is generated for limiting the inlet network traffic to a second traffic and suspending the operation of a second number of business systems; the first traffic is greater than the second traffic, and the first number is less than the second number; If the second failure level is the busy level, generating a first busy solution for limiting the egress network traffic to a third traffic level; If the second fault level is the congestion, generating a first congestion solution for limiting the egress network flow to a fourth flow; the third flow is greater than the fourth flow; Using the first warning solution or the first abnormal solution as a first processing solution; using the first busy solution or the first congested solution as a second processing solution; The first processing solution and the second processing solution are combined to obtain a first fault processing solution for normal operation of the business system in the server dimension.

5. The method according to claim 3, characterized in that: When the data dimension is a business dimension, the topological node is the business system, the business system is divided into a core business system and a non-core business system, the business system includes a non-core interface and a downstream node, and if the business system has a fault, a fault handling solution for ensuring the normal operation of the business system is generated, including: If the first fault level is the warning, generating a second warning scheme for limiting the request amount of the non-core business system and the request amount of the non-core interface; If the first fault level is the exception, generating a second exception solution for suspending the request of the non-core business system and the use of the non-core interface, and limiting the request amount of the core business system; If the second failure level is the busy level, generating a second busy solution for limiting the request amount of the downstream node to the first request amount; If the second fault level is the congestion, generating a second congestion solution for limiting the request amount of the downstream node to a second request amount; the first request amount is greater than the second request amount; Using the second warning solution or the second abnormal solution as a third processing solution; using the second busy solution or the second congested solution as a fourth processing solution; The third processing solution and the fourth processing solution are combined to obtain a second fault processing solution for normal operation of the business system in the business dimension.

6. The method according to claim 3, characterized in that When the data dimension is an interface dimension, the topological node is an interface of the business system, the business system is used to implement core business, the operation of the interface depends on the downstream node, and if the business system fails, a fault handling solution is generated to ensure the normal operation of the business system, including: If the first fault level is the warning, generating a third warning scheme for limiting the request amount of the interface to a third request amount; If the first fault level is the exception, a third exception solution is generated for limiting the request amount of the interface to a fourth request amount and limiting the request amount of the core service; the third request amount is greater than the fourth request amount; If the second failure level is the busy level, generating a third busy solution for reducing the timeout period of the interface request to the first time and limiting the request amount of the downstream node to a fifth request amount; If the second fault level is the congestion, a third congestion solution is generated for reducing the timeout time of the interface request to the second time, limiting the request amount of the downstream node to the sixth request amount, and suspending the call of the abnormal interface; the first time is greater than the second time, and the fifth request amount is greater than the sixth request amount; Using the third warning solution or the third abnormal solution as the fifth processing solution; using the third busy solution or the third congestion solution as a sixth processing solution; The fifth processing scheme and the sixth processing scheme are combined to obtain a third fault processing scheme for the normal operation of the business system in the interface dimension.

7. The method according to claim 1, characterized in that After the fault handling solution is used to handle the fault in the business system, the method includes: A fault handling report corresponding to the business system is generated according to the link tracking topology corresponding to the business system, the fault existing in the business system and the fault handling solution.

8. A fault handling device for a business system, characterized in that: The device comprises: A link tracking data module, used to obtain link tracking data corresponding to the business system; A link tracking topology module, used to generate a link tracking topology corresponding to the business system according to the link tracking data corresponding to the business system; A fault determination module, used to determine whether there is a fault in the business system according to the link tracking topology corresponding to the business system; A fault existence module is used to generate a fault handling solution for ensuring the normal operation of the business system if a fault exists in the business system; A fault handling module is used to handle faults in the business system using the fault handling solution.

9. An electronic device, characterized in that: include: processor; and A memory having executable codes stored thereon, which, when executed, causes the processor to execute the fault handling method for the business system as described in any one of claims 1 to 7.

10. One or more machine-readable media having executable codes stored thereon, which, when executed, cause a processor to execute the fault handling method for a business system as claimed in any one of claims 1 to 7.

Images