Fault alarm self-healing system and method

Abstract

The embodiment of the invention provides a fault alarm self-healing system and method, and the system comprises a micro-service target unit, a management unit, a monitoring collection unit, a data calculation unit, and a plan processing unit, and the management unit is used for determining a to-be-added monitoring collection task and a to-be-deleted monitoring collection task according to the monitoring information carried by each metadata; the monitoring acquisition unit is used for acquiring monitoring index data according to a to-be-added monitoring acquisition task and sending the monitoring index data to the data calculation unit; the data calculation unit is used for detecting the monitoring index data in real time according to a preset index detection rule and generating time sequence index data according to a first detection result; the monitoring index data and the time sequence index data are detected in real time according to a preset anomaly detection rule, and an anomaly event is generated according to a second detection result; and the plan processing unit is used for retrieving a fault self-healing scheme matched with each abnormal event and carrying out fault self-healing processing on the abnormal events according to the fault self-healing scheme.

Description

Technical Field

[0001] The present application relates to the field of big data computing technology, and in particular to a fault alarm self-healing system and method. Background Art

[0002] Microservices are applications composed of many smaller, loosely coupled services, which is the opposite of the overall approach of large, tightly coupled applications. A large service platform may be composed of hundreds, thousands, or even tens of thousands of microservices. For each microservice, if the monitoring coverage of the health status of each service or instance, as well as the troubleshooting after a failure, are all handled manually, the cost of monitoring and troubleshooting microservices will be very high. Therefore, the self-healing of microservice faults has become an urgent problem to be solved. Summary of the invention

[0003] The purpose of the embodiment of the present application is to provide a fault alarm self-healing system and method to achieve fault self-healing of microservices. The specific technical solution is as follows:

[0004] In a first aspect, an embodiment of the present application provides a fault alarm self-healing system, the system comprising a microservice target unit, a management unit, a monitoring collection unit, a data calculation unit, and a plan handling unit.

[0005] The management unit is used to obtain metadata corresponding to the instances of each microservice from the microservice target unit; determine the monitoring collection tasks to be added and the monitoring collection tasks to be deleted according to the monitoring information carried by each metadata; and send the monitoring collection tasks to be added and the monitoring collection tasks to be deleted to the monitoring collection unit;

[0006] The monitoring collection unit is used to receive the monitoring collection tasks to be added and the monitoring collection tasks to be deleted sent by the management unit, collect monitoring indicator data according to the monitoring collection tasks to be added, and send the monitoring indicator data to the data calculation unit; and delete the corresponding monitoring collection tasks according to the monitoring collection tasks to be deleted; wherein the monitoring collection tasks to be added carry an identifier, a name, a task type, and a task running configuration content;

[0007] The data calculation unit is used to receive the monitoring indicator data sent by the monitoring collection unit, monitor the monitoring indicator data in real time according to the preset indicator detection rule to obtain a first detection result; generate time series indicator data according to the first detection result; and perform real-time detection on the monitoring indicator data and the time series indicator data according to the preset abnormality detection rule to obtain a second detection result; generate an abnormal event according to the second detection result; and send the abnormal event to the plan handling unit;

[0008] The plan handling unit is used to receive the abnormal events sent by the data computing unit, retrieve the fault self-healing plan matching each abnormal event, and perform fault self-healing processing on the abnormal events according to the fault self-healing plan.

[0009] In some embodiments, the system further includes a data storage unit, a plan learning unit, a plan recommendation and management unit, and the fault self-healing solution is obtained by:

[0010] The plan learning unit is used to perform machine learning processing based on the monitoring indicator data and the time series indicator data to obtain a handling action for the abnormal event; and send the handling action to the plan recommendation and management unit;

[0011] The plan recommendation and management unit is used to receive the treatment action sent by the plan learning unit, optimize the treatment action to obtain an optimized treatment action; and send the optimized treatment action to the data storage unit;

[0012] The data storage unit is used to receive the optimized handling action, and associate the optimized handling action with the abnormal event to obtain abnormal event configuration data;

[0013] The plan handling unit is used to obtain the abnormal event configuration data from the data storage unit, and perform structured processing on the abnormal event configuration data to obtain a structure; according to the structure, determine the fault self-healing plan matching each abnormal event.

[0014] In some embodiments, the system further includes an event alarm unit, and the sending of the abnormal event to the plan handling unit includes:

[0015] The data calculation unit sends the abnormal event to the event alarm unit;

[0016] The event alarm unit receives the abnormal event sent by the event data calculation unit, classifies the abnormal event to obtain a target abnormal event; and sends the target abnormal event to the emergency plan handling unit;

[0017] The abnormal events are classified and processed to obtain target abnormal events, including:

[0018] For the abnormal event, if multiple identical abnormal events are received within a first preset time, one of the multiple identical abnormal events is used as a target abnormal event; and / or,

[0019] For the abnormal event, if there are multiple abnormal events with the same attributes or rules within the second preset time, merge them according to the preset merging rules to obtain the target abnormal event; and / or,

[0020] For the abnormal event, if there are multiple abnormal events with similar characteristics or business attributes within the third preset time, the abnormal events with similar characteristics or business attributes are merged and converged to obtain a target abnormal event.

[0021] In some embodiments, the event alarm unit is further used to:

[0022] For the abnormal event, if no recovery event of the abnormal event is received within a fourth preset time, the abnormal event is regarded as an upgraded abnormal event.

[0023] In some embodiments, the system further includes an alarm sending unit and an alarm receiving entity.

[0024] The alarm sending unit is used to obtain the target abnormal event from the event alarm unit; and send the target abnormal event to the alarm receiving subject according to the alarm rule or subscription rule;

[0025] The alarm receiving entity is used to receive the target abnormal event sent by the alarm sending unit.

[0026] In some embodiments, the data calculation unit is further used to perform real-time detection on the monitoring collection index data and the time series index data according to the preset abnormality detection rule to obtain a third detection result, generate a recovery event according to the third detection result; and send the recovery event to the event alarm unit;

[0027] The event alarm unit is further used to receive the recovery event sent by the data calculation unit.

[0028] In some embodiments, the system further includes a configuration center unit, and the acquiring metadata corresponding to the instance of each microservice from the microservice target unit includes:

[0029] The microservice target unit registers service information of each microservice instance to the configuration center unit, wherein the service information includes a service name, a monitoring collection address, a monitoring collection port, and metadata of the instance;

[0030] The configuration center unit generates a service instance list according to each of the service information;

[0031] The management unit obtains the service instance list from the configuration center unit, and reads metadata corresponding to the instances of each microservice from the service instance list.

[0032] In some embodiments, there is a communication mechanism between the instances of each microservice and the configuration center unit.

[0033] The configuration center unit is further configured to delete the target instance from the service instance list if there is a target instance that has not communicated with the configuration center unit within a fifth preset time.

[0034] In some embodiments, the system further comprises a data storage unit, a data display unit, and a data transmission unit.

[0035] The data storage unit is used to acquire and store system data, and send the system data to the data display unit; the system data includes the metadata, monitoring indicator data, time series indicator data, abnormal events, recovery events, preset indicator detection rules, preset abnormal detection rules, subscription rules, and fault self-healing solutions;

[0036] The data display unit is used to receive the system data sent by the data storage unit and display the system data;

[0037] The sending the monitoring indicator data to the data calculation unit includes:

[0038] The monitoring collection unit sends the monitoring indicator data to the data transmission unit;

[0039] The data transmission unit sends the monitoring indicator data to the data calculation unit.

[0040] In a second aspect, an embodiment of the present application further provides a fault alarm self-healing method, which is applied to a fault alarm self-healing system, wherein the fault alarm self-healing system includes a microservice target unit, a management unit, a monitoring and acquisition unit, a data calculation unit, and a plan disposal unit, and the method includes:

[0041] The management unit obtains metadata corresponding to the instances of each microservice from the microservice target unit; determines the monitoring collection tasks to be added and the monitoring collection tasks to be deleted according to the monitoring information carried by each metadata; and sends the monitoring collection tasks to be added and the monitoring collection tasks to be deleted to the monitoring collection unit;

[0042] The monitoring collection unit receives the monitoring collection tasks to be added and the monitoring collection tasks to be deleted sent by the management unit, collects monitoring indicator data according to the monitoring collection tasks to be added, and sends the monitoring indicator data to the data calculation unit; and deletes the corresponding monitoring collection tasks according to the monitoring collection tasks to be deleted; wherein the monitoring collection tasks to be added carry an identifier, a name, a task type, and a task running configuration content;

[0043] The data calculation unit receives the monitoring index data sent by the monitoring collection unit, performs real-time detection on the monitoring index data according to a preset index detection rule to obtain a first detection result; generates time series index data according to the first detection result; and performs real-time detection on the monitoring index data and the time series index data according to a preset alarm detection rule to obtain a second detection result; generates an abnormal event according to the second detection result; and sends the abnormal event to the plan handling unit;

[0044] The plan handling unit receives the abnormal events sent by the data computing unit, retrieves fault self-healing plans matching each of the abnormal events, and performs fault self-healing processing on the abnormal events according to the fault self-healing plans.

[0045] In some embodiments, the system further includes a data storage unit, a plan learning unit, a plan recommendation and management unit, and the fault self-healing solution is obtained by:

[0046] The plan learning unit performs machine learning processing according to the monitoring indicator data and the time series indicator data to obtain a handling action for the abnormal event; and sends the handling action to the plan recommendation and management unit;

[0047] The plan recommendation and management unit receives the treatment action sent by the plan learning unit, optimizes the treatment action to obtain an optimized treatment action; and sends the optimized treatment action to the data storage unit;

[0048] The data storage unit receives the optimized handling action, and associates the optimized handling action with the abnormal event to obtain abnormal event configuration data;

[0049] The plan handling unit obtains the abnormal event configuration data from the data storage unit, and performs structured processing on the abnormal event configuration data to obtain a structure; and determines a fault self-healing plan matching each abnormal event according to the structure.

[0050] In some embodiments, the system further includes an event alarm unit, and the sending of the abnormal event to the plan handling unit includes:

[0051] The data calculation unit sends the abnormal event to the event alarm unit;

[0052] The event alarm unit receives the abnormal event sent by the event data calculation unit, classifies the abnormal event to obtain a target abnormal event; and sends the target abnormal event to the emergency plan handling unit;

[0053] The abnormal events are classified and processed to obtain target abnormal events, including:

[0054] For the abnormal event, if multiple identical abnormal events are received within a first preset time, one of the multiple identical abnormal events is used as a target abnormal event; and / or,

[0055] For the abnormal event, if there are multiple abnormal events with the same attributes or rules within the second preset time, merge them according to the preset merging rules to obtain the target abnormal event; and / or,

[0056] For the abnormal event, if there are multiple abnormal events with similar characteristics or business attributes within the third preset time, the abnormal events with similar characteristics or business attributes are merged and converged to obtain a target abnormal event.

[0057] In some embodiments, for the abnormal event, if no recovery event of the abnormal event is received within a fourth preset time, the event alarm unit regards the abnormal event as an upgraded abnormal event.

[0058] In some embodiments, the system further includes an alarm sending unit and an alarm receiving entity.

[0059] The alarm sending unit obtains the target abnormal event from the event alarm unit; and sends the target abnormal event to the alarm receiving subject according to the alarm rule or subscription rule;

[0060] The alarm receiving entity receives the target abnormal event sent by the alarm sending unit.

[0061] In some embodiments, the data calculation unit is further used to perform real-time detection on the monitoring collection index data and the time series index data according to the preset abnormality detection rule to obtain a third detection result, generate a recovery event according to the third detection result; and send the recovery event to the event alarm unit;

[0062] The event alarm unit receives the recovery event sent by the data calculation unit.

[0063] In some embodiments, the system further includes a configuration center unit, and the acquiring metadata corresponding to the instance of each microservice from the microservice target unit includes:

[0064] The microservice target unit registers service information of each microservice instance to the configuration center unit, wherein the service information includes a service name, a monitoring collection address, a monitoring collection port, and metadata of the instance;

[0065] The configuration center unit generates a service instance list according to each of the service information;

[0066] The management unit obtains the service instance list from the configuration center unit, and reads metadata corresponding to the instances of each microservice from the service instance list.

[0067] In some embodiments, there is a communication mechanism between the instances of each microservice and the configuration center unit.

[0068] If there is a target instance that has not communicated with the configuration center unit within a fifth preset time, the configuration center unit deletes the target instance from the service instance list.

[0069] In some embodiments, the system further comprises a data storage unit, a data display unit, and a data transmission unit.

[0070] The data storage unit acquires and stores system data, and sends the system data to the data display unit; the system data includes the metadata, monitoring indicator data, time series indicator data, abnormal events, recovery events, preset indicator detection rules, preset abnormal detection rules, subscription rules, and fault self-healing solutions;

[0071] The data display unit receives the system data sent by the data storage unit and displays the system data;

[0072] The sending the monitoring indicator data to the data calculation unit includes:

[0073] The monitoring collection unit sends the monitoring indicator data to the data transmission unit;

[0074] The data transmission unit sends the monitoring indicator data to the data calculation unit.

[0075] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0076] Memory, used to store computer programs;

[0077] The processor is used to implement any of the above-mentioned fault alarm systems when executing the program stored in the memory.

[0078] Beneficial effects of the embodiments of the present application:

[0079] In the technical solution provided in the embodiment of the present application, the management unit obtains metadata corresponding to the instances of each microservice from the microservice target unit, and determines the monitoring collection tasks to be added and the monitoring collection tasks to be deleted according to the monitoring information carried by each metadata; thereafter, the management unit can send the generated monitoring collection tasks to be added and the monitoring collection tasks to be deleted to the monitoring collection unit, and the monitoring collection unit can add corresponding collection tasks based on the monitoring collection tasks to be added, and stop corresponding collection tasks according to the monitoring collection tasks to be deleted, so as to realize the deployment and adjustment of the monitoring tasks, and then obtain the monitoring indicator data collected by the monitoring collection tasks.

[0080] Furthermore, the monitoring collection unit can send the monitoring indicator data to the data calculation unit; the data calculation unit monitors the monitoring indicator data in real time according to the preset indicator detection rules, obtains the first detection result, and generates the timing indicator data according to the first detection result; thereafter, the monitoring indicator data and the timing indicator data are detected in real time according to the preset anomaly detection rules, and the second detection result is obtained, and an abnormal event is generated according to the second detection result, thereby determining the abnormal events existing in the instance operation process of each microservice.

[0081] For each abnormal event, the data computing unit can send each abnormal event to the plan handling unit. After receiving each abnormal event, the plan handling unit can retrieve the fault self-healing plan matching each abnormal event, and process the corresponding abnormal event according to the fault self-healing plan. In this way, the abnormal events existing in the instance operation of each microservice can be automatically resolved by the system, realizing the self-healing of microservice faults.

[0082] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0084] Figure 1 A flow chart of the fault alarm self-healing system provided in the embodiment of the present application;

[0085] Figure 2A structural schematic diagram of a fault alarm self-healing system provided in an embodiment of the present application;

[0086] Figure 3 A schematic diagram of a process of service registration and service discovery provided in an embodiment of the present application;

[0087] Figure 4 A schematic diagram of a process for issuing monitoring tasks and collecting monitoring data provided in an embodiment of the present application;

[0088] Figure 5 A flowchart of an abnormality detection and alarm event sending process provided in an embodiment of the present application;

[0089] Figure 6 A schematic diagram of a process for handling a plan triggered by an alarm event provided in an embodiment of the present application;

[0090] Figure 7 A flowchart of the plan learning and recommended configuration process provided in the embodiment of the present application;

[0091] Figure 8 A schematic diagram of a flow chart of a fault alarm self-healing method provided in an embodiment of the present application;

[0092] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0093] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0094] Below, the nouns appearing in this application are explained.

[0095] Internet: also known as the international network, refers to a huge network connected by networks. These networks are connected by a set of common protocols to form a logically single huge international network.

[0096] The Internet originated from the ARPANET in the United States in 1969. Usually, the Internet refers to the Internet in general, while the Internet refers specifically to the Internet. This method of connecting computer networks to each other can be called "network interconnection". The Internet refers to the global Internet network covering the whole world developed on this basis, that is, the network structure connected to each other. The Internet is not the same as the World Wide Web. The World Wide Web is just a global system based on hypertext links, which can be understood as a service that the Internet can provide.

[0097] Internet of Things (IoT): refers to the use of various information sensors, radio frequency identification technology, global positioning system, infrared sensor, laser scanner and other technologies and equipment to collect any objects or processes that need to be monitored, connected and interacted in real time, such as collecting various required information such as sound, light, heat, electricity, mechanics, chemistry, biology, location, etc., and to achieve ubiquitous connection between objects and objects and between objects and people through various possible network access, and to achieve intelligent perception, identification and management of objects and processes. The Internet of Things is an information carrier based on the Internet, traditional telecommunications networks, etc., which allows all ordinary physical objects that can be independently addressed to achieve interconnection and interoperability.

[0098] Microservices: Also known as microservice architecture, refers to a cloud-native architecture approach that includes many loosely coupled and independently deployable small components or services in a single application. These small components or services usually have their own technology stack, including databases and data management models; communicate with each other through a presentation layer state conversion application programming interface (REST API), event streams, and message brokers; and are organized according to business capabilities, with service separation lines usually called bounded contexts.

[0099] The characteristic of microservices is that the code is easier to update, new features or functions can be added directly without updating the entire application, and the team can use different technology stacks and different programming languages ​​for different components. Components can be expanded independently of each other, thus reducing the waste and cost associated with having to expand the entire application (because a single function may face excessive load).

[0100] Big Data: Also known as massive data, it refers to the amount of data involved that is so large that it cannot be captured, managed, processed, and organized into information that helps companies make more proactive business decisions within a reasonable time through mainstream software tools.

[0101] Cloud computing: It is a type of distributed computing, which means breaking down huge data computing programs into countless small programs through the network "cloud", and then processing and analyzing these small programs through a system composed of multiple servers to obtain results and return them to users. In other words, cloud computing refers to a system with extremely strong computing power formed through a computer network (mostly the Internet), which can store and aggregate related resources, and can be configured on demand to provide personalized services to users.

[0102] In the early days of cloud computing, simply put, it was simple distributed computing, solving task distribution and merging computing results. Therefore, cloud computing is also called grid computing. Through this technology, tens of thousands of data can be processed in a very short time (a few seconds), thereby achieving powerful network services. However, the current cloud service is not just a kind of distributed computing, but the result of the hybrid evolution and leap of computer technologies such as distributed computing, utility computing, load balancing, parallel computing, network storage, hot backup redundancy and virtualization.

[0103] Stream computing and streaming computing: In the traditional data processing process, data is always collected first, and then placed in the database. When the user needs it, the data is queried through the database to get the answer or perform related processing. Although this seems very reasonable, the result is very compact, especially for some specific problems in some real-time search application environments, offline processing similar to the map-reduce method cannot solve the problem well. This leads to a new data computing structure, namely the stream computing method. It can well analyze large-scale flow data in real time during the ever-changing movement process, capture potentially useful information, and send the results to the next computing node.

[0104] Machine learning: refers to a multi-disciplinary interdisciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It aims to study how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. It is the core of artificial intelligence and the fundamental way to make computers intelligent.

[0105] Fault self-healing: refers to real-time alarm detection, pre-diagnosis analysis, automatic fault recovery, and connecting peripheral systems to achieve a closed loop of the entire process. Fault self-healing is divided into passive self-healing and active self-healing. Passive self-healing only treats the symptoms but not the root cause, such as automatic capacity expansion after a disk alarm; active healing requires the cooperation of root cause analysis to directly deal with the cause of the alarm. Among them, root cause analysis (RCA) is a systematic problem-solving method that aims to find the root cause of the problem and eliminate the source of the problem.

[0106] Service registration: refers to the microservice instance registering its own service information (i.e. service instance information) to the registration center (the registration center can be understood as a function of the configuration center unit described below). This part of the service information can include the host address (Internet Protocol Address, IP) where the service is located and the port (port) providing the service, as well as information such as the exposed service status and access protocol.

[0107] Service discovery: refers to the service instance requesting the registration center to obtain the dependent service information. Specifically, the service instance obtains the registered service instance information through the registration center, and uses this information to request the service to be provided. Service registration and service discovery need to focus on issues such as monitoring the running status of service instances.

[0108] Microservice monitoring: refers to the real-time and comprehensive detection and management of the performance, status and security of each instance in the microservice architecture. By collecting, analyzing and displaying various data during the operation of the instance, microservice monitoring can help operation and maintenance personnel and developers to promptly discover potential problems, locate the cause of failures, and optimize system performance, thereby ensuring the stability and availability of the entire microservice architecture.

[0109] In a microservice architecture, the dependencies between services are complex, and the failure of any service may trigger a chain reaction, leading to the collapse of the entire system. Therefore, through microservice monitoring, operation and maintenance personnel and developers can grasp the operating status and performance indicators of each instance in the system in real time, discover and handle abnormal situations in a timely manner, and avoid fault expansion and escalation. At the same time, microservice monitoring can also provide historical data and trend analysis to help the operation and maintenance team predict system capacity and performance bottlenecks, and provide strong support for business decisions.

[0110] Apache Flink Stateful StreamProcessing SQL Engine (Flink SQL): Apache Flink is a framework and distributed processing engine for performing stateful computations on unbounded and bounded data streams; Structured Query Language (SQL) is a database query and programming language used to access data and query, update, and manage relational database systems. Flink SQL is built on top of Apache Flink and uses Flink's powerful processing capabilities to allow users to use SQL statements for stream and batch data processing. Flink SQL supports both real-time stream data processing and bounded batch data processing.

[0111] With the rapid development of computer technology, all aspects of society and all walks of life have experienced exponential growth in business data. The amount of data storage and computing is very large. The real test of storage and computing is the massive data, as well as the data types and structures tend to be more complex. The structures and types are even more diverse, including commonly used structured data, semi-structured data, and unstructured data, such as audio, video, etc., as well as streaming data, vector or graph data, etc. This has brought challenges to the storage and computing of Internet resources, and users currently participating in Internet services have put forward higher requirements for services, such as recommending products or resources that are more in line with user wishes, requiring higher real-time user experience, and users are becoming more and more sensitive to information and data.

[0112] At the same time, with the rapid development of the new Internet of Things, including smart device Internet of Things and vehicle Internet of Things, when facing user needs, enterprises not only communicate and interact with users through web pages or handheld terminal devices as in the Internet era. In the new Internet of Things era, the interaction between users and devices generates device time series data. Compared with the Internet era, this type of data has experienced another explosive growth.

[0113] In the era of the Internet of Things, the time series data of various devices are synchronized in real time (or near real time) with the local area network or the cloud (or edge), and are calculated in real time together with big data. Business results are output in real time through calculation, and different business needs are responded to by users. Therefore, in this scenario, new demands and challenges are also put forward for big data computing. With the emergence of new demands and challenges, the requirements for big data practitioners will become higher and higher. This further increases the professional and technical requirements for entering the big data industry and raises the entry threshold. In the long run, the development of an industry requires more talents to participate, and it needs to be practiced in more business scenarios to verify its technology and tools. Therefore, the increase in the entry threshold of the big data industry is not conducive to the development of the big data industry.

[0114] In the field of big data computing, with the continuous development of business, the collection, calculation and analysis of business data has become a rigid demand. Previously, the commonly used technical solutions in the field of big data were batch processing, with big data technical engineers writing corresponding batch processing tasks to complete the collection, calculation and analysis of business data. However, for this type of technical solution, whenever the mining demand changes, the batch processing tasks have to be modified and redeployed. In other words, in response to changes in business needs, relevant technical personnel are always required to make corresponding adjustments and deployments, and the investment cost is relatively large. At present, with the rise of real-time streaming technology, it has taken a step forward in big data mining and analysis. In essence, it is an iterative update of the previous batch processing technology. Compared with the previous batch processing, it does a better job in data fidelity and responding to changes in business needs. The data time window can be pulled smaller and smaller, and the real-time nature of data calculation results is also higher.

[0115] In the digital age, with the rapid growth of business needs and the continuous evolution of technical architecture, microservice architecture has gradually become the first choice for enterprises to build complex application systems due to its flexibility, scalability, high cohesion and low coupling. However, the fragmentation, dynamism and complexity of microservice architecture also bring unprecedented challenges to the stability, security and performance of the system.

[0116] In the related art, the following two solutions are usually used to solve the problem of service failure self-healing. The following describes the two methods in the related art respectively:

[0117] Solution 1, receiving real-time alarm information sent by the real-time monitoring system, obtaining the alarm type of the real-time alarm information based on the real-time alarm information, and then obtaining the corresponding fault self-healing processing solution based on the alarm type, and performing self-healing processing on the fault corresponding to the real-time alarm information based on the fault self-healing processing solution.

[0118] It is understandable that although Solution 1 performs self-healing of the fault, from the perspective of the self-healing closed loop, Solution 1 may only achieve a closed loop of fault handling, and does not mention how the related fault handling solution is closed. In the environment of large-scale microservices and massive big data, the closed loop of the handling solution is actually very important. If it only relies on manual configuration and improvement, it obviously cannot meet the increasing number of services and fault scenarios.

[0119] Solution 2: Configure a log word frequency analysis strategy based on the application service log information collected from the application log file, perform application log word frequency analysis based on the log word frequency analysis strategy, obtain word frequency monitoring indicators, train a word frequency detection model based on the word frequency monitoring indicators, perform anomaly detection through the word frequency detection model, and perform fault self-healing based on the detection results.

[0120] It is understandable that although Solution 2 also performs self-healing and has a good self-healing effect in the scenario of log application service logs, application logs alone may have shortcomings in some scenarios, such as service dependencies, scenarios such as multiple microservices in series, or complex scenarios such as middleware and databases. Monitoring indicators and log data in other dimensions may be needed to enrich abnormal scenarios, and in the process of handling faults in complex scenarios such as multiple services, multiple systems, and multiple dependencies based on abnormal scenarios, it is necessary to integrate and analyze the monitoring indicators, service logs, multi-link data, service dependency diagrams, and other data of each service, and automatically recommend and associate corresponding fault handling plans for these complex scenarios.

[0121] To solve the above problems, the embodiment of the present application provides a fault alarm self-healing system. The fault alarm self-healing system provided by the embodiment of the present application can be applied to electronic devices. In specific applications, the electronic device can be a server or a terminal device. In practical applications, the terminal device can be a smart phone, a tablet computer, a desktop computer, etc.

[0122] In response to the above-mentioned problem that the explosive growth of data has led to challenges in big data computing, the fault alarm self-healing system provided in the embodiment of the present application can reduce the cost and price of using big data, real-time streaming computing technologies and tools in various industries. By real-time processing, analysis and calculation of monitoring indicator data and time series indicator data, and real-time detection of monitoring indicator data and time series indicator data according to preset anomaly detection rules, abnormal events are generated according to the detection results, and then the machine learning algorithm is used to learn and train the source data and historical fault data, and different fault self-healing solutions are recommended, so that abnormal events caused by microservice failures can be associated with different fault self-healing solutions, so as to achieve microservice fault self-healing.

[0123] Compared with the above-mentioned solution 1, the present application has been more perfectly designed and implemented. The fault self-healing system provided in the embodiment of the present application can support the configuration of the statically configured fault handling plan (i.e., fault self-healing plan) through the experience of operation and maintenance experts, and test and drill the configured plan. Expert experience can also be added to machine learning, as a link of supervised learning in machine learning to optimize and supplement the effect of machine learning. In addition, the fault self-healing system provided in the embodiment of the present application can also learn and predict existing data and historical fault data, as well as some conventional fault self-healing plan data through machine learning algorithms, and recommend the predicted results. After that, the recommended results are screened and filtered by the operating personnel, and after the configuration is activated, a new fault self-healing plan is obtained, and then the new fault self-healing plan is tested and drilled to achieve a positive learning plan, so that the fault self-healing plan can be self-closed in terms of richness.

[0124] This application involves many fields such as Internet big data computing, Internet or Internet of Things real-time stream computing, Internet or Internet of Things batch computing, Internet of Things device access time series data real-time processing and aggregation computing, fault self-healing, artificial intelligence (AI), etc. In addition, this application is based on the large framework of real-time stream computing, based on Flink SQL technology, and is designed to adapt to the ever-changing needs and structures of business data structures through dynamic configuration. It uses a set of systems and services to quickly adapt and realize data analysis and mining needs, without the need for big data development engineers and programmers to constantly repeat the development of programs, and there is no need to constantly redeploy, and dynamically support business needs in this regard. Users only need to understand SQL to complete related operations and uses. In other words, this application can quickly support changing business needs and reduce technical cost investment.

[0125] The technical solution provided by this application is intended to quickly adapt to business needs and solve the stream batch processing of big data real-time data at a low cost. Reduce the threshold for practitioners to use professional technologies such as big data computing and processing, so as to help more operators who do not have professional knowledge in the field of big data computing or have less basic knowledge in the field of big data computing. They can also use tools and systems in the field of big data to complete the needs of business analysis and calculation. Through the technical solution provided by this application, big data real-time stream computing can be promoted and applied in more fields. In this way, business analysts can focus more on business analysis and processing. Furthermore, the technical solution provided by this application is intended to help personnel engaged in development and operation maintenance (development operations, DevOps) quickly discover, monitor and deal with common problems of microservices, rapid discovery and alarm of faults, diagnosis and analysis, and automatic recovery of faults and opening up the surrounding systems to realize the closed loop of the entire process. In this way, the availability of microservices is improved to ensure the continuous and stable operation of the business.

[0126] Specifically, the fault alarm self-healing system provided by this application provides multiple aspects of fault handling and healing closed loops based on microservice monitoring. The first aspect is the closed loop obtained by microservice monitoring and fault handling. Through the fault self-healing solution, the microservice can be matched with the corresponding handling link after the fault occurs, so that the microservice can be handled after the fault occurs; the second aspect is the closed loop of the richness of the plan, which is first associated with historical faults and monitoring indicator data, monitoring abnormal event data, etc., and then trained through machine learning algorithms, which can ensure the richness of the fault self-healing plan and avoid the situation of insufficient types of fault self-healing plans as much as possible; the third aspect is that after the failure of the fault self-healing plan, after the manual intervention and handling for replay analysis, the defective plan is processed and optimized, which optimizes and repairs the effectiveness of the plan.

[0127] In order to solve the challenges brought by the fragmentation, dynamics and complexity of microservice architecture to the stability, security and performance of the system, the present application provides a method for completing the real-time dynamic distribution of monitoring and collection tasks to be added and the monitoring and collection tasks to be deleted through service registration and service discovery, according to the metadata corresponding to the instances of each registered microservice, and the monitoring and collection unit (i.e., the monitoring and collection agent (Agent) or collector (exporter) component) completes the real-time collection of the monitoring indicator data of each instance and reports it to the data transmission unit; the data calculation unit can obtain the monitoring indicator data from the data transmission unit, and generate the timing indicator data according to the monitoring indicator data and the preset indicator detection rules, and then perform real-time detection on the monitoring indicator data and the timing indicator data according to the preset anomaly detection rules, and then generate abnormal events and recovery events, encapsulate the abnormal events and recovery events and send them to the event alarm unit, and the event alarm unit calls the alarm channel service to complete the notification of abnormal events and recovery events, and notifies the relevant alarm subscribers, completing the abnormal general service health status monitoring and alarm event notification processing. Among them, the event alarm unit supports event upgrade processing, event subscription processing, event readiness and disposal processing, event filtering, convergence and aggregation processing. At the same time, in the embodiment of the present application, abnormal events can be divided into common events, alarm events, fault events, etc. Furthermore, according to different service levels, they can also be divided into basic layer events, middleware and database events, service basic events, business events, service level objectives (SLO) availability events, etc.

[0128] At the same time, the present application also provides a mapping process between abnormal events and their corresponding fault self-healing solutions. The fault self-healing solutions include fault self-healing solutions recommended by machine learning and static fault self-healing solutions configured by operation and maintenance experts. When an abnormal event is triggered, the plan processing unit retrieves the associated disposal plan tree corresponding to each event in the system, and calls the corresponding plan (i.e., fault self-healing plan) for disposal based on the association relationship and weight information, and notifies the status of the completed plan result. The success and failure status of the plan must be notified to the relevant plan disposal persons, including standby (SRE OnCall) duty personnel and business R&D personnel related to business services.

[0129] In addition, the embodiments of the present application also provide abnormal event training of real-time indicator data through machine learning, learning of fault self-healing solutions, etc., in which the machine learning service instance learns and trains abnormal events in real time, and recommends fault self-healing solutions related to abnormal events based on commonly used machine learning algorithms. Operation and maintenance experts and related operating personnel add the recommended fault self-healing solutions to the data storage unit, and can add permissions and execution condition relationships to multiple fault self-healing solutions. According to the machine learning algorithms and recommendations, the fault self-healing solutions of the entire fault self-healing system can be positively supplemented and optimized, thereby enabling the fault self-healing solutions to continuously undergo closed-loop learning and optimization and improvement.

[0130] In summary, the embodiment of the present application collects various monitoring indicator data of each instance through the monitoring collection unit, and after real-time aggregation of the time series indicator data of each instance through real-time big data calculation, the monitoring indicator data and the time series indicator data are monitored in real time according to the preset abnormal detection rules to obtain abnormal events. After that, according to the fault self-healing scheme configured for the abnormal event association, the abnormal event jointly triggers the plan disposal, so as to realize the automatic disposal of abnormal events in the microservice. Common fault self-healing schemes can be automatic cleanup event disposal for insufficient disks, automatic switching of Domain Name System (Domain Name System, DNs) domain name resolution when the entrance is blocked or the entrance detection fails, and automatic restart of the service after the service isok (a health detection mechanism of an interface or service, used to detect whether the interface or service is operating normally) detects that the interface exceeds the threshold and fails, etc.

[0131] The following is a detailed description of the fault alarm self-healing system provided by the embodiment of the present application through specific examples.

[0132] Figure 1 A process interaction diagram of the fault alarm self-healing system provided in an embodiment of the present application includes: a microservice target unit, a management unit, a monitoring and collection unit, a data calculation unit, and a plan disposal unit. The specific interaction steps are as follows:

[0133] Step S11: the management unit obtains metadata corresponding to each microservice instance from the microservice target unit;

[0134] Step S12: The management unit determines the monitoring collection tasks to be added and the monitoring collection tasks to be deleted according to the monitoring information carried by each metadata;

[0135] Step S13: the management unit sends the monitoring collection tasks to be added and the monitoring collection tasks to be deleted to the monitoring collection unit;

[0136] Step S14: the monitoring collection unit collects monitoring indicator data according to the monitoring collection tasks to be added, and deletes the corresponding monitoring collection tasks according to the monitoring collection tasks to be deleted;

[0137] Step S15: The monitoring collection unit sends the monitoring index data to the data calculation unit;

[0138] Step S16: the data calculation unit performs real-time detection on the monitoring indicator data according to the preset indicator detection rule to obtain a first detection result, and generates time series indicator data according to the first detection result; and performs real-time monitoring on the monitoring indicator data and the time series indicator data according to the preset abnormality detection rule to obtain a second detection result, and generates an abnormal event according to the second detection result;

[0139] Step S17: The data calculation unit sends the generated abnormal event to the emergency plan handling unit.

[0140] Step S18: The plan handling unit retrieves the fault self-healing plan that matches each abnormal event, and performs fault self-healing processing on the abnormal event according to the fault self-healing plan.

[0141] In the technical solution provided in the embodiment of the present application, the management unit obtains metadata corresponding to the instances of each microservice from the microservice target unit, and determines the monitoring collection tasks to be added and the monitoring collection tasks to be deleted according to the monitoring information carried by each metadata; thereafter, the management unit can send the generated monitoring collection tasks to be added and the monitoring collection tasks to be deleted to the monitoring collection unit, and the monitoring collection unit can add corresponding collection tasks based on the monitoring collection tasks to be added, and stop corresponding collection tasks according to the monitoring collection tasks to be deleted, so as to realize the deployment and adjustment of the monitoring tasks, and then obtain the monitoring indicator data collected by the monitoring collection tasks.

[0142] Furthermore, the monitoring collection unit can send the monitoring indicator data to the data calculation unit; the data calculation unit monitors the monitoring indicator data in real time according to the preset indicator detection rules, obtains the first detection result, and generates the timing indicator data according to the first detection result; thereafter, the monitoring indicator data and the timing indicator data are detected in real time according to the preset anomaly detection rules, and the second detection result is obtained, and an abnormal event is generated according to the second detection result, thereby determining the abnormal events existing in the instance operation process of each microservice.

[0143] For each abnormal event, the data computing unit can send each abnormal event to the plan handling unit. After receiving each abnormal event, the plan handling unit can retrieve the fault self-healing plan matching each abnormal event, and process the corresponding abnormal event according to the fault self-healing plan. In this way, the abnormal events existing in the instance operation of each microservice can be automatically resolved by the system, realizing the self-healing of microservice faults.

[0144] In the above step S11, the microservice target unit generally refers to a service subject object based on a traditional model or a cloud model, such as a subject instance object of one or more running nodes of a microservice implemented in Java (an object-oriented programming language), C++ (a system-level programming language), Go (an open source programming language) or other languages. The microservice target unit is the target subject in this application, and the health status of its monitoring indicator data in each dimension can reflect the health status of this target subject in real time. Its impact path is from a single node to a service cluster, then from a service cluster to a service unit, from a service unit to the entire business line, etc., and the impact range radius ranges from small to large.

[0145] The management unit is an important console management system in this application. Most of the management configuration actions related to the functions of this application (such as adding, modifying and deleting functional operations) rely on the functional modules of the management unit. The management unit can obtain the metadata corresponding to the instance of each microservice from the microservice target unit, where the structure of the metadata is shown in Table 1 below:

[0146] Table 1. Metadata schema

[0147]

[0148] The meaning of the second row of data in Table 1 is that the field type of the field named "id" is "String", and its description (i.e., specific meaning) is "microservice index number Id, generally generated by UUID". In some embodiments, the microservice index number Id can also be generated by using the database auto-increment field (auto_increment); the meaning of the third row of data in Table 1 is that the field type of the field named "name" is "String", and its description is "microservice name". The meanings of other rows in Table 1 refer to the description of the second and third rows above, which will not be repeated here.

[0149] The structure of the payload carried in the metadata shown in Table 1 above is shown in Table 2 below.

[0150] Table 2. Metadata payload structure

[0151]

[0152] The meaning of the fourth row of data in Table 2 is that the field named "name" has a field type of "String", and its description (i.e., specific meaning) is "used when multiple tasks need to be merged, this name can be mapped with the name running in the Flink cluster, and the prefix indicates that it can be distinguished from a single task"; the meaning of the fifth row of data in Table 2 is that the field named "metadata" has a field type of "JSON Object", and its description is "when the task runs in single mode, this name is directly used to correspond to the task name in Flink". The meaning of other rows in Table 2 can be found in the description of the fourth and fifth rows above, which will not be repeated here.

[0153] It can be understood that the instances in the embodiments of the present application refer to IP nodes in the traditional deployment mode, or Pods (the smallest deployment unit in K8s) in the Kubernetes (K8s) deployment mode, etc. A microservice may have multiple instances.

[0154] In some embodiments, the fault alarm self-healing system provided by the embodiments of the present application also includes a configuration center unit, and the microservice target unit can register the service information of each microservice instance to the configuration center unit; then, the configuration center unit generates a service instance list based on the service information; and then the management unit can obtain the service instance list from the configuration center unit, and obtain the metadata corresponding to each microservice instance from the service instance list. Among them, the service information includes the service name, monitoring collection address, monitoring collection port, and instance metadata; the list consisting of each service and its corresponding deployed instance node is called the service instance list, that is, the service instance list.

[0155] Specifically, the microservice target unit can register its own service information (including service name, IP, port and Metadata instance metadata) to the configuration center unit when each instance is started, and the configuration center unit stores the service information in the form of a service instance list. The management unit can actively pull the instance and service instance list from the configuration center unit regularly, or the management unit can subscribe to the push event of the configuration center, and the configuration center actively pushes the instance and service instance list to the management unit in real time. Here, the main function of the management center to obtain the service instance list is to monitor the instance status of each instance, monitor the status of each monitoring collection task, configure indicator calculation strategies and rules (such as the preset indicator detection rules below), configure anomaly detection strategies and rules (such as the preset anomaly detection rules below), meet the needs of monitoring and event data display, etc.

[0156] In some embodiments, there is a communication mechanism between the instance of each microservice and the configuration center unit. If there is a target instance that has not performed a heartbeat detection with the configuration center within the fifth preset time, the configuration center unit can delete the target instance from the service instance list. Among them, the fifth preset time is the time pre-set by the user according to actual needs, which can be 1 minute (min), 2min, 3min, etc.; the target instance is an instance that has not communicated with the configuration center within the fifth preset time; the method of communication in the embodiment of the present application can be heartbeat detection, for example, the microservice target unit regularly initiates a heartbeat request to the configuration center unit (or the configuration center unit regularly issues a heartbeat detection). If there is an instance that has not responded within the fifth preset time, the example can be regarded as a target example, and the configuration center unit deletes the instance from the service instance list.

[0157] It can be understood that the service instance list is located in the configuration center unit. When the state of the instance changes, it can be automatically reported to the configuration center unit, or it can be detected and discovered by the configuration center unit through the communication mechanism. The service instance list in the configuration center unit is updated in real time, and the management unit only synchronizes data (such as instances and service instance lists) from the configuration center unit for query or display by other units. It can be understood as a backup data of the instance list in the configuration center unit;

[0158] Based on the description in the above embodiments of the present application, it can be known that the service instance list in the configuration center unit will be added, updated or deleted as the status of each instance changes, and there will be no deletion processing in the management unit. Only the service instance list data will be added or updated. For instances deleted by the configuration center unit, their status will be modified in the management unit. Only the status of this type of instance will be updated to deleted, and its data will not be deleted.

[0159] In some embodiments, the fault alarm self-healing system provided by the present application also includes a data storage unit. After the management unit regularly synchronizes each instance and each metadata from the configuration center unit, it can send each instance and each metadata to the data storage unit through an interface. The data storage unit will uniformly and persistently store each instance and each metadata for query and use by other units (such as the data calculation unit, data display unit, etc. mentioned below).

[0160] Among them, in addition to supporting the basic functions of microservice registration and service discovery, the configuration center unit also has the function of centralized and unified management of microservice configuration. In the process of deploying, starting and maintaining many microservice instances, if the configuration is managed manually, the cost is high. Therefore, it is also very important to use the configuration center unit to manage each microservice and configuration, which can greatly reduce labor costs. There are many implementation schemes for the configuration center unit, such as Eureka (a service discovery framework), Consul (a tool for service discovery and configuration sharing), Zookeeper (a distributed coordination service) and other technical solutions to implement the configuration center unit.

[0161] The management unit can support the generation and distribution of monitoring and collection tasks (such as monitoring and collection tasks to be added and monitoring and collection tasks to be deleted), the synchronous storage and processing of service information of each instance, and the configuration modification and verification of preset indicator detection rules, preset anomaly detection rules, event filtering and upgrading rules (see the description of the event alarm unit below), and other commonly used rules. At the same time, the management unit also provides some data operation and statistical functions in this application, and provides data operation, function auditing, and other abnormal function handling tools for each functional module.

[0162] In the above step S12, after the management unit obtains each metadata, it can determine the monitoring collection tasks to be added and the monitoring collection tasks to be deleted according to the monitoring information carried in the metadata. Among them, the monitoring information can be understood as the monitoring types supported by each metadata obtained by the management unit from the metadata of each instance; the monitoring collection tasks to be added can be understood as the monitoring collection tasks corresponding to the monitoring collection types supported by each metadata, and the monitoring collection tasks to be deleted can be understood as the monitoring collection tasks corresponding to the monitoring collection types not supported by each metadata; among them, whether each metadata supports a certain monitoring collection type can be understood as whether each metadata reports the monitoring collection type.

[0163] For example, when each instance is started, the management unit analyzes the metadata of each instance and finds that the supported monitoring types reported by the metadata are central processing unit (CPU) indicators and memory indicators. This indicates that the monitoring and collection tasks to be added are the monitoring and collection tasks corresponding to the CUP indicators and the monitoring and collection tasks corresponding to the Memory indicators. Except for the monitoring and collection tasks corresponding to the CUP indicators and the monitoring and collection tasks corresponding to the Memory indicators, all other tasks can be regarded as monitoring and collection tasks to be deleted.

[0164] In the subsequent process, after the management unit updates the metadata of each instance by actively acquiring or passively pushing, it is found that the current metadata supports the CPU indicators and the disk usage indicators. At this time, the monitoring and collection tasks to be added are the collection tasks corresponding to the CPU indicators and the disk usage indicators, and the monitoring and collection tasks to be deleted are the collection tasks other than the collection tasks corresponding to the CPU indicators and the disk usage indicators. It should be noted that the collection tasks corresponding to the CPU indicators have been deployed when the instance is started. Therefore, although the collection tasks corresponding to the CPU indicators here are also the collection tasks to be added, in the actual operation process, the collection tasks corresponding to the CPU indicators can no longer be deployed in the monitoring and collection unit, and only the collection tasks corresponding to the disk usage indicators can be deployed. At the same time, since the monitoring and collection tasks corresponding to the Memory indicators are also deployed when each instance is started, the monitoring and collection unit should stop the monitoring and collection tasks corresponding to the Memory indicators based on the monitoring and collection tasks to be deleted this time.

[0165] Among them, the monitoring types that can be currently supported can be divided into five major layers according to the basic layer, middleware layer, application layer, service layer, user layer, etc., and each major layer has multiple types of indicators. For example: the basic layer may include system CPU usage, system Memory usage, system Memory total, usage and total amount of each major mount disk path, etc.; the middleware layer will have different indicators for different types of middleware. For example, for Kafka (an open source stream processing platform) type middleware, the supported indicators may include Kafka memory usage, Kafka memory application total, etc.

[0166] In the above step S13, the management unit can send the determined monitoring and collection tasks to be added and the monitoring and collection tasks to be deleted to the monitoring and collection unit, and also support the synchronization and query of the status of each monitoring and collection task in the monitoring and collection unit, and the management of abnormal status of the monitoring and collection tasks. For example, the management unit provides task start, task stop, task interruption and other services to the operator, and the abnormal status is managed through the operation of the operator.

[0167] The management unit mentioned above supports synchronization and query of the status of each monitoring and collection task in the monitoring and collection unit. In some embodiments, the monitoring and collection unit will synchronize the monitoring and collection tasks and status to the management unit at a regular interval, so as to update the survival activities of the monitoring and collection unit and the status data of the monitoring and collection tasks to the management center in real time, which is convenient for operation and management. In addition, each collection unit and the collection tasks assigned to it and the status of the collection data results will be synchronized and saved to the data storage unit by the management unit at a regular interval.

[0168] In addition, the monitoring and collection tasks to be added sent by the management unit to the monitoring and collection unit carry the identification, name, task type, and task operation configuration content of the monitoring and collection task. Among them, the task type is used to indicate whether the monitoring and collection task is a one-time task or a scheduled task, etc.; the collection task configuration content can be a script, or a Shell command (referring to a series of instructions entered through a command line interface or a command interpreter), and the monitoring and collection task content can be divided into collection telnet (remote terminal protocol) tasks, HTTP (hypertext transfer protocol) type tasks, etc. The configuration of the monitoring and collection task may be different due to different technical solutions adopted, such as using filebeat (a lightweight log collection and processing tool), ELK (a solution for log collection, processing and display), logkit (a log collection and processing tool) secondary development, etc.

[0169] In the above step S14, after the management unit sends the monitoring collection tasks to be added and the monitoring collection tasks to be deleted to the monitoring collection unit, the monitoring collection unit can deploy new monitoring collection tasks according to the monitoring collection tasks to be added, and stop the corresponding monitoring collection tasks according to the monitoring collection tasks to be deleted, thereby realizing the collection of monitoring data.

[0170] Continuing with the embodiment shown in the above step S12 as an example, the monitoring and collection tasks to be added that the management unit sends to the monitoring and collection unit when each instance is started are the monitoring and collection tasks corresponding to the CUP class indicators and the monitoring and collection tasks corresponding to the Memory class indicators, and the monitoring and collection tasks to be deleted are the monitoring and collection tasks other than the monitoring and collection tasks corresponding to the CUP class indicators and the monitoring and collection tasks corresponding to the Memory class indicators. At this time, if the monitoring and collection tasks corresponding to the CUP class indicators and the monitoring and collection tasks corresponding to the Memory class indicators do not exist in the monitoring and collection unit, the monitoring and collection tasks corresponding to the CUP class indicators and the monitoring and collection tasks corresponding to the Memory class indicators need to be deployed, and if at this time there are monitoring and collection tasks other than the monitoring and collection tasks corresponding to the CUP class indicators and the monitoring and collection tasks corresponding to the Memory class indicators in the monitoring and collection unit, these monitoring and collection tasks need to be stopped.

[0171] In the subsequent process, the management unit updates the metadata of each instance and sends the monitoring and collection tasks to be added and the monitoring and collection tasks to be deleted again. The monitoring and collection tasks to be added are the collection tasks corresponding to the CPU indicators and the collection tasks corresponding to the disk usage indicators; the monitoring and collection tasks to be deleted are the collection tasks other than the collection tasks corresponding to the CPU indicators and the collection tasks corresponding to the disk usage indicators. At this time, since the collection tasks corresponding to the CPU indicators have been added to the monitoring and collection unit when each instance is started, the monitoring and collection unit only needs to add the collection tasks corresponding to the disk usage indicators this time. At the same time, since the monitoring and collection tasks corresponding to the Memory indicators have been added to the monitoring and collection unit when each instance is started, the monitoring and collection unit needs to stop the monitoring and collection tasks corresponding to the Memory indicators this time.

[0172] It can be understood that the management unit performs unified management of monitoring and collection tasks only for the convenience of operation, and the actual monitoring and collection tasks are distributed to instances on each distributed node. For example, in this application, there is mainly an Agent process, which is deployed on a physical machine node (Node). The Node collects data from services deployed on this Node, or services in containers on this Node, or K8s Pods deployed on this Node.

[0173] In the embodiment of the present application, the monitoring collection unit is mainly composed of multiple technologies and multiple components, which is mainly used to collect multi-level and multi-dimensional monitoring indicator data from hardware to software, from network to application. The collection capability of monitoring indicator data includes hardware level such as host temperature, network card information, network bandwidth, etc., operating system level such as central processing unit (CPU), memory, disk, etc., various business indicator data of database and middleware, service business custom buried point indicator data, etc.

[0174] After receiving the monitoring collection task sent by the management unit, the monitoring collection unit can collect the monitoring indicator data of the specified dimension once or multiple times according to the task, where the dimension can be understood as the indicator type (in the embodiment of the present application, the dimension mainly refers to the dimension used to express the time series indicator data after the indicator modeling). In addition, the monitoring collection unit and the management unit have real-time communication capabilities. The management unit can send the collection task to each monitoring collection unit component from time to time. After each monitoring collection unit component completes the collection of the monitoring indicator data, the monitoring indicator data is reported to the data transmission unit in real time.

[0175] Among them, in addition to completing the monitoring coverage of each instance, the monitoring component (i.e., the monitoring collection unit component) also needs to cover the calling relationship and dependency relationship between each instance. In this way, after a failure occurs in one or more instances, the affected dependent services can be quickly analyzed, the scope of impact can be evaluated, and the calling relationship and fault impact relationship can be intuitively displayed in the fault tree.

[0176] There are many implementation schemes for monitoring collection units. The commonly used one is collection based on Agent technology, and the collected monitoring indicator data is reported to the data transmission unit in real time for transmission. Another scheme is collection based on Exporter component (a component that collects monitoring data and provides data to the outside through Prometheus monitoring specification, where Prometheus is an open source service monitoring system and time series database). In Agent, a scheme based on HTTP interface is enabled to expose monitoring indicator data, and the monitoring indicator data is pulled uniformly by the collection center. This scheme has fewer usage scenarios, and its main problem is that unified pulling will increase the pressure on the collection center. Therefore, this technical scheme is generally not adopted. After the Agent collects the data, it is uniformly adopted to upload or report it to the data transmission channel to complete the collection and reporting of monitoring indicator data.

[0177] There are also some special scenarios in which different solutions may be used in the monitoring collection unit. For example, for collection based on network hardware, it is necessary to complete the collection and processing of monitoring indicator data by calling the interface provided by the hardware service provider. Based on the operating system level, there will be multiple solutions to implement it, and based on middleware and database, facing different middleware or databases, the collection solutions are also different. And the cloud-native solutions under the new technology environment are obviously different. Therefore, the collection technology solutions for various scenarios and environments are inconsistent, and multiple solutions need to be adopted for compatibility and adaptation.

[0178] In the above step S15, the monitoring collection unit may send the collected monitoring index data to the data calculation unit in addition to sending the collected monitoring index data to the management unit.

[0179] In some embodiments, the fault alarm self-healing system provided in the embodiments of the present application also includes a data transmission unit. The monitoring and acquisition unit can send the monitoring indicator data to the data transmission unit, and then the data transmission unit sends the monitoring indicator data to the data calculation unit, or the data calculation unit can consume the monitoring indicator data from the data transmission unit in real time.

[0180] Among them, the data transmission unit in this application mainly plays the role of a data service bus, such as realizing the reporting and transmission functions of monitoring index data, that is, the monitoring collection unit can report the collected multi-dimensional and multi-level monitoring index data to the data transmission unit, and the data calculation unit and the plan learning unit can realize real-time consumption of these data.

[0181] The data transmission unit also supports the function of transmitting and consuming abnormal events between multiple services. For example, the event alarm unit sends the abnormal event to the data transmission unit, which is then consumed by the plan handling unit and the plan learning unit to complete the plan handling and plan learning. For more information about this part, please refer to the subsequent description.

[0182] At present, the mainstream technical solution for implementing the data transmission unit is the distributed data stream processing platform (Apache Kafka) technology stack. In addition to the Apache Kafka technology stack, other technology stacks can also be used, such as: Apache ActiveMQ (the most popular open source message bus produced by Apache), RabbitMQ (open source message broker and queue server developed in Erlang language, used to achieve communication between systems in a lightweight way), ZeroMQ (high-performance asynchronous message library), RocketMQ (distributed message middleware) and other technical solutions.

[0183] In the above step S16, after the data calculation unit obtains the monitoring indicator data, it can perform real-time detection on the monitoring indicator data according to the preset indicator detection rules to obtain a first detection result, generate time series indicator data based on the first detection result, and store the generated time series indicator data in the data storage unit. Among them, the dimensions of the generated time series indicator data include: service name (serviceName); indicator type (Metric), which can be iaa, app, etc., which represent the indicator types of the basic layer and application layer respectively; the monitoring type (MonitorType), such as db for database indicators, kafka for kafka middleware indicators, etc.; indicator name (MetricName), such as CpuUseRate for Cpu usage; instance (Endpoint), such as 192.168.1.100, or the name of the Pod, etc.

[0184] In an embodiment of the present application, the preset indicator detection rules include static indicator detection rules and dynamic indicator detection rules. Below, the principle of generating time series indicator data according to the preset indicator detection rules and monitoring indicator data in an embodiment of the present application is explained with a specific example.

[0185] For static indicator detection rules, for example, the monitoring indicator data is the memory usage and total memory of the host node, the data calculation unit can perform a secondary calculation on the obtained memory usage and total memory of the host node to obtain the memory usage rate of the host node per minute; for another example, the monitoring indicator data is the disk usage and total disk capacity of a certain path, the data calculation unit can perform a secondary calculation on the obtained disk usage and total disk capacity of a certain path to obtain the disk usage rate of the host node under a certain path per minute.

[0186] For dynamic indicator detection rules, for example, the error request quantity indicator data of the interface error code is calculated and counted for each service dimension and interface dimension every minute; for another example, the error quantity indicator data obtained by dividing the interface error quantity and the total number of requests is calculated and counted for each service dimension and interface dimension every minute. It can be understood that this is only an example of static indicator detection rules and dynamic indicator detection rules, and does not represent all static indicator detection rules and dynamic indicator detection rules in this application.

[0187] In addition, after the data calculation unit generates time series indicator data based on the first detection result, it can perform real-time detection on the indicator data (i.e., monitoring indicator data and time series indicator data) according to the preset anomaly detection rules to obtain a second detection result, and generate an abnormal event based on the second detection result.

[0188] Specifically, the data calculation unit uses a preset anomaly detection rule to perform real-time detection on the indicator data. When the indicator data meets the preset anomaly detection rule, a corresponding abnormal event can be generated, that is, the abnormality indicated by the abnormal event can be understood as the content of the preset anomaly detection rule that it meets, or the abnormal event can be used to indicate the reason for meeting the preset anomaly detection rule. The preset anomaly detection rules include static anomaly detection rules and dynamic anomaly detection rules. Below, the principle of generating abnormal events according to the preset anomaly detection rules and indicator data in the embodiment of the present application is explained with a specific example.

[0189] For static anomaly detection rules, for example, the static anomaly detection rules include that the amount of interface request errors configured within 3 minutes is continuously greater than 100. When the indicator data meets the condition that the amount of interface request errors configured within 3 minutes is continuously greater than 100, an abnormal event is generated. For another example, the static anomaly detection rules include that the amount of errors is continuously greater than 50 within 5 minutes. When the indicator data meets the condition that the amount of errors is continuously greater than 50 within 5 minutes, an abnormal event is generated.

[0190] For dynamic anomaly detection rules, for example, the dynamic anomaly detection rules include that the request throughput of a certain interface fluctuates by more than 10%. When the indicator data indicates that the request throughput of a certain interface fluctuates by more than 10%, an abnormal event is generated, which can be used to indicate that the interface request is abnormal, or the interface is attacked, the service is unavailable, etc.; for another example, the dynamic anomaly detection rules include that the request throughput of a certain interface fluctuates by less than 10%. When the indicator data indicates that the request throughput of a certain interface fluctuates by less than 10%, an abnormal event is generated, which can be used to indicate that there is an availability problem with the interface service.

[0191] At present, the most important technical solutions for data computing units in the field of big data real-time computing include MapReduce (distributed computing model) and Apache Spark (memory-based distributed computing framework) based on batch processing mode, Storm and Apache Flink based on real-time stream processing mode, etc. In the embodiments of this application, Apache Flink is tended to be used to implement the data computing unit.

[0192] It is understandable that compared with Spark and Storm, Flink solves the problems of Spark and Storm in stream processing, state management and fault tolerance, time and watermark processing mechanism. Specifically, first of all, Flink provides a true stream-by-stream processing model, achieving lower latency. Compared with Spark Streaming (converting streams into small batches for processing), Flink can provide closer to real-time data processing capabilities. Flink's processing model enables it to provide higher timeliness when processing streaming data, meeting the real-time requirements of a large number of global businesses. Secondly, in terms of state management and fault tolerance, Flink's design makes it more efficient when processing stateful operations. Flink's state management is designed to be pluggable and provides built-in keyed state and operator state. Furthermore, regarding the time and watermark processing mechanism, Flink provides a more powerful and flexible time and watermark processing mechanism, which can better handle event time and process events, which is particularly important when processing late or out-of-order data caused by factors such as network delays. By introducing the concepts of watermark (mainly used to deal with out-of-order events) and lateness (mainly used to deal with data latency issues), Flink can handle these problems more effectively, which is what Spark and Storm lacked in early versions. Therefore, Flink solves the shortcomings of Spark and Storm in terms of stream processing timeliness, state management, fault tolerance, and time processing by providing a true stream-by-stream processing model, efficient state management and fault tolerance mechanisms, and powerful time and watermark processing capabilities. Therefore, compared with Spark and Storm, using Apache Flink to implement data computing units is more advantageous.

[0193] In the above step S17 , the data computing unit may send the generated abnormal event to the plan handling unit, or the plan handling unit may consume the abnormal event generated by the data computing unit in real time.

[0194] In the above step S18: after the plan handling unit obtains the abnormal event, it can retrieve the fault self-healing plan (also called plan) matching each abnormal event, and perform fault self-healing processing on the abnormal event according to the fault self-healing plan.

[0195] In some embodiments, the fault alarm self-healing system provided by the present application further includes an event alarm unit, and the sending of abnormal events to the plan handling unit shown in the above step S17 includes: the data calculation unit sends the abnormal event to the event alarm unit, and the event alarm unit classifies the abnormal event to obtain the target abnormal event, and then the event alarm unit can send the target event to the plan handling unit. Among them, the way of classifying and processing abnormal events includes any one or more of the following four classification methods:

[0196] Classification method 1: For abnormal events, if multiple identical abnormal events are received within the first preset time, one of the multiple identical abnormal events is used as the target abnormal event. That is, the event alarm processing unit filters abnormal events according to the rules. For the same abnormal event, if multiple repeated events are sent within a specified time period, the event alarm unit needs to filter such repeated abnormal events to achieve the function of deduplication.

[0197] Classification method 2: For abnormal events, if there are multiple abnormal events, other events, and fault events with the same attributes or rules within the second preset time, they are merged according to the preset merging rules to obtain the target abnormal event; that is, the event alarm unit can merge the received abnormal events according to the preset merging rules, and the abnormal events that need to be merged mainly refer to abnormal events with similar characteristics or business attributes. After aggregating such abnormal events according to the preset merging rules, a target abnormal event and notification of an explanatory nature can be generated.

[0198] Classification method three: For abnormal events, if there are multiple abnormal events with similar characteristics or business attributes within the third preset time, these abnormal events with similar characteristics or business attributes are merged and converged to obtain the target abnormal event. That is, the event alarm unit can use the same rule to merge and converge the abnormal events. The abnormal events that need to be merged and converged are multiple abnormal events with similar characteristics or business attributes within the third preset time. After merging and converging these abnormal events, the target abnormal event can be obtained.

[0199] After the emergency plan handling unit obtains the target abnormal event, it can retrieve the fault self-healing plan that matches the target abnormal event. Specifically, the emergency plan handling unit can extract relevant features based on the content of the target abnormal event, and then match the corresponding fault self-healing plan based on the features, wherein the extracted relevant features can be the ID of the target abnormal event, the content keywords of the target abnormal event, etc.

[0200] In addition, the plan handling unit can obtain the fault self-healing plan in the following ways to achieve the matching of the target abnormal event with the fault self-healing plan:

[0201] The plan learning unit can consume the indicator data in the data transmission unit and the target abnormal event in the event alarm unit, and based on the machine learning algorithm, it can obtain the disposal action for the target abnormal event, and then send the disposal action for the target abnormal event to the plan recommendation and management unit; after the plan management unit receives the disposal action sent by the plan learning unit, it can optimize the disposal action to obtain the optimized disposal action, and send the optimized disposal action to the data storage unit; after the data storage unit receives the optimized disposal action, it can associate the optimized disposal action with the abnormal event to obtain the abnormal event configuration data; the plan configuration unit can obtain the abnormal event configuration data from the data storage unit, and perform structured processing on the abnormal event configuration data to obtain a structure; determine the fault self-healing plan for each abnormal event according to the structure.

[0202] In the embodiment of the present application, the role of the plan recommendation and management unit in the present application is mainly to filter the plans recommended after machine learning by relevant operation staff, and configure them to the associated abnormal events. After the plan is activated, the next time a related abnormal event occurs, the associated plan processing will be triggered. When the operation staff chooses to filter the plans recommended by machine learning, they can perform fault rehearsals through the recommended plans and test events, and detect the effectiveness of the plans through fault rehearsals. They can also allow the entire plan disposal to be run in advance once or multiple times in a real environment to test whether the results of the plan execution are truly in line with expectations.

[0203] In some embodiments, the abnormal event configuration data in the data storage unit may also include: some possible abnormal event handling methods pre-configured by experts based on experience, and abnormal event configuration data obtained by associating these handling methods with corresponding abnormal events. Alternatively, the plan learning unit may be used to perform machine learning on historical indicator data and historical abnormal events in advance to obtain handling actions corresponding to historical abnormal events, and the plan recommendation and management unit may optimize the handling actions to obtain optimized handling actions. After that, the historical abnormal events may be understood as possible abnormal events, and the abnormal event configuration data stored in the data storage unit may be obtained by associating the optimized handling actions with historical abnormal events.

[0204] It can be understood that when each abnormal event is sent directly to the plan handling unit without being processed by the event alarm unit, the principle of obtaining the fault self-healing solution is the same as the principle of obtaining the fault self-healing solution when the target abnormal event is sent to the plan handling unit, and will not be repeated here.

[0205] In some embodiments, when the system is initialized, the plan handling unit can fully synchronize the abnormal event configuration data from the data storage unit (including the abnormal event configuration data obtained through the expert configuration mentioned above, the abnormal event configuration data obtained after the plan learning unit and the plan recommendation and management unit process the indicator data and abnormal events in real time, and the abnormal event configuration data obtained after the plan learning unit and the plan recommendation and management unit process the historical indicator data and historical abnormal events), and then regularly synchronize the incremental abnormal event configuration data (that is, the plan learning unit and the plan recommendation and management unit will process the indicator data and abnormal events in real time, and regularly send the obtained optimized handling actions to the data storage unit to constitute the abnormal event configuration data), and perform structured processing on the configuration data. Among them, system initialization generally means that the service will load some configuration data from the database during the startup phase. At the same time, some systems may load data from the database at regular intervals (such as once every hour) during operation to initialize the values ​​of some system variables.

[0206] In the embodiment of the present application, structured processing means that the plan configuration may be of various types, such as calling an interface, executing a script, etc., and each plan must be associated with an abnormal event. The association relationship here will strongly associate the id of the abnormal event with the id of the plan task configuration when the system is initialized or the data is preheated, and the content of the plan configuration will serve as the value of the structured data.

[0207] The process of obtaining a fault self-healing plan for an abnormal event is explained using an example. For example, the plan processing unit receives an abnormal event with a CPU usage rate exceeding 80%. The plan processing unit can query whether the corresponding abnormal event has corresponding abnormal event configuration data (or structure). If the query hits, the handling action in the abnormal event configuration data (or structure) can be called and executed. If the hit is a restart plan, the plan handling unit can call the corresponding plan to complete the restart of the problematic node.

[0208] It is understandable that an abnormal event allows multiple plans to be associated. For example, when the CPU usage of a node exceeds 80%, its associated plans can also include: deleting log files in the specified directory, restarting the service, calling the interface to complete the supplier switch, etc.

[0209] The plan learning unit consists of two parts, online and offline. It uses machine learning to classify faults through original logs, indicator data and historical fault-related data, and selects models for the classified faults, adjusts the ratio of positive and negative samples, adjusts algorithm parameters, etc., and evaluates the prediction effect. After that, based on the evaluation results, it recommends them from high to low to the operation and maintenance personnel for judgment and filtering selection, and synchronizes the evaluation results to the plan recommendation and management unit.

[0210] In some embodiments, when the fault self-healing solution handles an abnormal event, the results of the plan handling action will also be recorded. For each process of the plan handling process, the handling event will be saved, and the plan handling process will be saved to ensure that each step of the plan handling has a historical record that can be queried and reviewed.

[0211] The plan recommendation and management unit also includes the manual addition, modification and deletion of plan functions. This function is mainly provided to expert operation and maintenance engineers, who are mainly responsible for completing the configuration verification and activation of historical experience plans. These plans do not need to be recommended through machine learning, but are relatively fixed and reusable experience accumulation. For example, if the disk usage rate exceeds 80%, the files in the specified directory need to be deleted.

[0212] In addition, another important function of the plan recommendation and management unit is to manually score and label the plans recommended by machine learning, so that the scored plans can be sent back to the machine learning module to further optimize the effect of machine learning. This makes the results of plan recommendations more accurate. At the same time, after the plan is manually triggered, the results of the plan execution can also be returned to the machine learning. If it is successful, it will be added as a positive sample, and if it fails, it can be added as a negative sample.

[0213] In some embodiments, the event alarm unit in the embodiment of the present application can also be used to perform silent processing on certain abnormal events according to business rules. Specifically, for some abnormal events, according to business rules, it may be required not to send an alarm notification within the rule time range, but it is necessary to record the alarm event, so silent processing can be used for such abnormal events.

[0214] In addition, the event alarm unit can classify and upgrade abnormal event processing according to specified rules, and the upgraded abnormal event is generated and notified according to predetermined rules. In some embodiments, if a recovery event of the abnormal event is not received within a fourth preset time, such as after the abnormal event alarm, if a recovery event of the abnormal event is not received for more than 10 minutes, the abnormal event can be upgraded to obtain an upgraded abnormal event, and the upgraded abnormal event can be notified to the duty manager and the product line business manager, etc. For example, the event alarm unit sends the upgraded abnormal time to the alarm sending unit, and the alarm sending unit notifies the duty manager and the product line business manager of the upgraded abnormal event.

[0215] It can be understood that after the data calculation unit performs real-time detection on the indicator data according to the preset anomaly detection rules, a third detection result can be obtained, a recovery event can be generated based on the third detection result, and the data calculation unit can also send the recovery event to the event alarm unit. Specifically, it can be understood that when the indicator data does not meet the preset anomaly detection rules, a recovery event can be generated. For example, the preset anomaly detection rules include that the number of erroneous requests of the interface continues to exceed 500 within 5 minutes, and the data calculation unit determines that the current indicator data indicates that there is a situation where the number of erroneous requests of the interface continues to exceed 500 within 5 minutes, so an abnormal event can be generated; after a period of time, the data calculation unit obtains new indicator data and determines that the new indicator data indicates that there is no situation where the number of erroneous requests of the interface continues to exceed 500 within 5 minutes, then a recovery event for the aforementioned abnormal event can be generated.

[0216] Furthermore, the event alarm unit can send different levels of alarm event notifications to the designated recipients according to the specified rules or subscription rules. In addition, the event alarm unit can send the received abnormal events to the transmission channel of the data transmission unit, which is used as a backup channel for alarm events, other events and fault events, and also as a bypass channel for other business logics.

[0217] In some embodiments, the fault alarm self-healing system provided in the embodiment of the present application also includes a data storage unit, which is all systems and components for persistently storing the data in the present application (including metadata related to service registration and discovery, monitoring indicator data, time series indicator data, abnormal events, recovery events, monitoring and acquisition tasks, preset abnormal detection rules, alarm strategies and rules, abnormal event configuration data, etc.). Some of the data are implemented using a relational database solution, such as monitoring and acquisition tasks, preset abnormal detection rules, alarm strategies and rules, abnormal event configuration data, etc. These data can be directly stored in a relational database system, such as a MySQL database cluster. And indicator data, abnormal abnormal events, etc. can be stored in a time series database or a non-relational database, such as a distributed search and analysis engine (Elasticsearch), a time series database (InfluxDB), etc.

[0218] In some embodiments, the fault alarm self-healing system provided in the embodiments of the present application also includes a data display unit, which is mainly used for filtering, screening and viewing of indicator data (mainly referring to time series indicator data here), filtering, screening and viewing of abnormal events, screening and viewing of fault self-healing solutions, etc., wherein the basis for screening and viewing is determined by the user. For example, the data storage unit stores indicator data based on a fixed model of multiple dimensions, and the user may need to view indicator data of various dimensions based on actual needs. At this time, the data display unit is required to provide functions for screening and filtering according to user instructions, and display the obtained results. Other functions such as monitoring and collection task viewing, alarm strategy and preset abnormal detection rules viewing are also within its scope. In this application, open source Grafana can be used as a data display component to display, query, and filter indicator data, abnormal events, etc.

[0219] In some embodiments, the fault alarm self-healing system provided by the embodiment of the present application also includes an alarm sending unit, which can notify the corresponding receiving object according to the abnormal events of different channels and different levels generated by the event alarm unit, and complete batch push, single push, retransmission processing, etc. of abnormal events through different technical solutions. And some targeted compensation processing is performed according to the result returned by calling the application programming interface (Application Programming Interface, API).

[0220] For example, for notifications of abnormal events such as WeChat, the alarm sending unit can call the WeChat API interface to complete the push processing of the WeChat abnormal event notification, and the corresponding WeChat public account subscription object can receive the abnormal event notification through the WeChat public account push. Among them, the WeChat abnormal event push function requires the user who receives the SMS push to follow and subscribe to the public account in advance before receiving the alarm event push processing from the WeChat public account.

[0221] Notification of SMS abnormal events refers to the notification of SMS abnormal events that needs to be completed through multiple SMS sending channels to complete the notification of SMS abnormal events to the specified mobile phone number. Among them, multiple SMS sending channels can be understood as each user is connected to multiple SMS providers. In this way, each provider provides a channel, that is, multiple channels are realized. Based on this, if one or more channels are unavailable, the availability of SMS alarm channels can be guaranteed by timely switching or closing channels, that is, the success rate of SMS notifications can be guaranteed. That is, when pushing SMS, if it is detected that the channel is unavailable, the unavailable channels need to be filtered and processed in time. Specifically, this can be achieved by pre-setting certain detection rules. For example, if a channel fails to send three users in a row, the channel needs to be closed; and for abnormal events that fail to send, other SMS channels need to be switched to complete the sending of abnormal events, that is, switching channels.

[0222] At the same time, the SMS alarm channel can add a sensitive word filtering function. The alarm channel service also needs to subscribe to and process the status codes returned by each operator or gateway service interface, including retransmission of certain retransmittable status codes.

[0223] Notification of telephone abnormal events refers to the notification of telephone abnormal events (notification of abnormal events with very high real-time requirements). It is necessary to complete the notification of telephone voice abnormal events to the specified mobile phone number users through multiple voice sending gateway channels. If one or more of the channels are unavailable, it is necessary to switch or close the channel in time to ensure the availability of the voice abnormal event notification channel. The voice alarm event notification channel needs to subscribe to and process the sending status returned by the API provided by each voice operator, and process the sending status. If it is a sending status that needs to be resent, it is necessary to switch the channel in time or delay the resending process. For resending, it is necessary to set an upper limit on the number of resending times. Until the upper limit is reached, no more resending will be performed, indicating that the sending of this notification failed.

[0224] In some embodiments, the fault alarm self-healing system provided by the embodiments of the present application also includes an alarm receiving subject, which is the object that receives the alarm event. In the microservice availability assurance system, the objects that need to receive abnormal event notifications generally include SRE OnCall duty personnel, SRE OnCall duty leaders, microservice-related R&D personnel, etc. In some embodiments, the receiving subject may also include a receiving method that can be adopted, such as: through WeChat or DingTalk, SMS, phone, etc.

[0225] Next, through Figure 2 The structural diagram of the fault alarm self-healing system shown in the figure illustrates the information interaction between the units in the fault self-healing system provided in the embodiment of the present application. It can be understood that Figure 2 This is only an example, and the units included therein are not all the units in the fault alarm self-healing system shown in the present application, and the data transmission between units does not include the data transmission between all the units in the fault alarm self-healing system shown in the present application.

[0226] Depend on Figure 2 It can be seen that the configuration center unit 202 is an important component of service registration and service discovery. When the microservice deployment is started, the microservice target unit 201 needs to register the instances of each microservice and the service information of each instance to the configuration center unit 202, where the service information includes the information of the service itself and other information that the service needs to expose, such as the monitoring collection address and port. The configuration center unit 202 has the function of service registration and service discovery. For details, please refer to the description in the above step S11.

[0227] After completing the registration, the configuration center unit 202 can store the registered service information, and at the same time, the configuration center unit 202 can detect the health status of each registered instance. If the detection fails after a preset number of times (i.e., the configured upper limit of health detection times, such as N times), it is considered that the instance is offline, and it can be removed from the configuration center unit 202, and the service discovery client is notified at the same time. For details, please refer to the description in the above step S11.

[0228] The management unit 204 can periodically and synchronously pull (or passively subscribe to) service registration and deregistration events from the configuration center unit 202, and read service information; at the same time, according to the preset indicator detection rules and the preset anomaly detection rules, the monitoring collection tasks to be added and the monitoring collection tasks to be deleted are generated, and sent to the designated monitoring collection unit 203 to implement task delivery. At the same time, the management unit 204 can also synchronously save the service information synchronized from the configuration center unit 202 to the data storage unit 207 to implement the storage and query of the service information. For details, please refer to the description in the above steps S12 and S13.

[0229] After the monitoring and collection unit 203 passively receives the monitoring and collection tasks (including the monitoring and collection tasks to be added and the monitoring and collection tasks to be deleted) issued from the management unit 204, it completes the collection of monitoring index data according to the issued monitoring and collection tasks, and reports the collected monitoring index data to the data transmission unit 205 in real time (i.e., realizes monitoring data reporting). The monitoring and collection unit 203 can realize interface detection and monitoring data collection based on the microservice target unit 201. In addition, the monitoring and collection unit 203 will synchronize the collection tasks and status to the management unit 204 at regular intervals (i.e., report the status) to realize the real-time update of the status and data of the survival activities and collection tasks of the monitoring and collection unit 203 to the management unit 204, which is convenient for operation and processing. Each collection unit and the collection tasks and collection data result status assigned to it will be saved and synchronized by the management unit 204 to the data storage unit 207 at regular intervals, that is, the storage unit can provide storage and query functions, such as the management unit 204 can realize service information storage or query through the data storage unit 207. Among them, it can be understood that the microservice instance node can output monitoring indicators that reflect the health status of the microservice through standard behaviors from the hardware network layer, infrastructure layer, operating system layer, application layer, etc. during operation, including business monitoring indicator output based on many technologies such as Agent, etc., which can directly or indirectly reflect the multi-dimensional monitoring indicator data of the health status of the node microservice. After the monitoring and collection unit regularly pulls and collects these indicator data, it can be collected in real time and uploaded to the cloud collection cluster based on big data, and these monitoring indicator data can be transmitted in real time through the data transmission channel of big data. For details, please refer to the description in the above steps S14 and S15.

[0230] The data storage unit 207 is mainly used to implement all data storage and query processing, as well as data backup and recovery functions. The data transmission unit 205 is mainly used to implement data reporting transmission, consumption pulling and other functions in this application, and is the data bus of the entire device. For details, please refer to the above description of the data storage unit.

[0231] The data calculation unit 206 is used to consume data in real time from the data transmission unit 205 (i.e., perform data consumption processing), and realize data synchronization and rule distribution with the management unit 204. After the data calculation unit 206 performs filtering and judgment calculation according to the preset abnormal detection rules stored in the management unit 204, if the preset abnormal detection rules are met, an abnormal event is generated, if not, a recovery event is generated, and the data calculation unit 206 sends both the abnormal event and the recovery event to the event alarm unit 209 for processing (i.e., event sending). In addition, the data calculation unit 206 can also generate new indicator time series data after secondary calculation according to the preset indicator detection rules. This type of indicator data can also be directly reported to the data transmission unit 205, and then perform abnormal detection judgment, and generate abnormal events or recovery events. For details, please refer to the description in the above step S16. The data calculation unit 206 can also realize data storage through the data storage unit 207.

[0232] The data display unit 208 is mainly used to intuitively display the time series indicator data, abnormal events, fault self-healing solutions and other data, such as displaying them in text or graphics, so as to facilitate users to browse and query, etc. For example, the data display unit 208 can realize data pulling based on the data calculation unit 206 and the data storage unit 207, and realize alarm query based on the event alarm unit 209.

[0233] The plan learning unit 213 is mainly used to complete the machine learning processing of abnormal events, source indicator time series data, historical fault data and other data according to the machine learning algorithm, and recommend the most similar plan according to the algorithm, and then send it to the plan recommendation and management unit 214 (i.e., to implement the plan recommendation), and the relevant personnel will complete the sorting, association and activation and saving processing. In this way, the plan recommended by machine learning is added to the plan handling unit 211 (i.e., to implement the synchronization plan rules). For details, please refer to the description in the above step S18.

[0234] It is understandable that the plan handling unit 211 is only the user of the plan, and is only the subject that executes one or more plans when an abnormal event occurs, while the data storage unit 207 only stores data and does not have management functions. The plan recommendation and management unit 214 is the management subject of the plan. The plan comes from the experience configuration of the operation and maintenance experts, or the plan recommended by machine learning. In theory, machine learning supplements and improves the experience of operation and maintenance experts.

[0235] The event alarm unit 209 is mainly used to receive abnormal events, filter, converge, aggregate, etc., and then call the corresponding channel to complete the push notification processing of the alarm event according to the level, channel type and target recipient of the abnormal event. Figure 2The event alarm unit 209 sends aggregated events to the plan learning unit 213, where the aggregated events refer to the event alarm unit 209 performing aggregate processing on similar abnormal events. For example, there may be an abnormal event in which the CPU usage of the node 192.168.1.100 exceeds 80% every minute for 10 consecutive minutes. In this case, the event alarm unit 209 will not send this event to the plan learning unit 213 every minute. In a possible embodiment, the event alarm unit 209 may aggregate these 10 abnormal events into one abnormal event every 10 minutes and send it to the plan learning unit 213 for processing, because the plan learning unit 213 does not need event details, and it itself also needs to perform dimensionality reduction processing on the events.

[0236] In addition, the event alarm unit 209 can also support the upgrade of alarm events or fault events according to rules, so that the on-duty personnel can prepare, handle and transfer abnormal events. For details, please refer to the above description of the event alarm unit.

[0237] The alarm sending unit 210 is mainly a channel for pushing abnormal event notifications. It can obtain alarm notifications from the event alarm unit 209 and the plan handling unit 211. It can be divided into different types of channels such as WeChat, SMS, phone calls, and self-developed IM channels according to the target receiving type. Each channel will subscribe to the push status according to the pushed channel API, and process resending notifications according to the return status. For details, please refer to the above description of the alarm sending unit.

[0238] The alarm receiving subject 212 mainly refers to the device that needs to receive abnormal event notifications, such as mobile phones, computers, etc. Abnormal events are pushed to mobile phones through various channels to notify the corresponding personnel to handle the pushed abnormal events. Figure 2 As shown, the alarm sending unit 210 can send the alarm to the alarm receiving entity 212.

[0239] The plan handling unit 211 is mainly used to index the rules of the existing plan according to the alarm event. If the rule is hit, the corresponding plan needs to be triggered for processing, and the relevant personnel need to be notified according to the handling result. At the same time, the handling steps and results need to be saved for later review and analysis. For details, please refer to the description in the above step S18. Figure 2 As shown, the plan handling unit 211 calls the plan from the event alarm unit 209. It can be actually understood that the event alarm unit 209 completes the triggering of the abnormal event. When the abnormal event is sent to the plan handling unit 211, the plan handling unit 211 hits the plan configuration according to the mapping relationship (such as the abnormal event configuration data or structure mentioned above), thereby triggering the execution action of the plan task.

[0240] See also Figure 3 , Figure 3 A schematic diagram of a process of service registration and service discovery provided in an embodiment of the present application may include the following steps:

[0241] Step S31: When the service is started, the microservice target unit registers service information with the configuration center unit; for details, please refer to the description in the above step S11.

[0242] Step S32: When the instance changes, update the service information corresponding to the instance to the configuration center unit; for details, please refer to the description in the above step S11.

[0243] Step S33: Each instance communicates with the configuration center unit using a certain mechanism (such as heartbeat). If the configuration center cannot communicate with an instance for a long time, the instance is deregistered;

[0244] The configuration center unit provides functions such as service registration, service discovery, service health check, and service information storage. If the instance health check fails, the instance node is removed; for details, please refer to the description in the above step S11.

[0245] Step S34: The service consumer updates the service information of each instance by actively pulling (or subscribing to push notifications from the configuration center); here, the service consumer can be understood as a management unit; for details, please refer to the description in the above step S11.

[0246] Step S35: The service consumer uses the corresponding load balancing rules to perform call processing when calling the service provider according to the load balancing rules registered with the service;

[0247] In the embodiment of the present application, the service consumer refers to the party that consumes the data provided by the service provider, and the service provider is the party that provides data to the service consumer. For example, if service A calls service B, in microservices, it is generally through the service registration center, and service B first registers to the service registration center, and then service A pulls the registration information of service B from the service registration center, and finally completes the function of service A calling service B. Here, service B is the service provider, and service A is the service consumer. The specific service consumer and service provider are determined according to the actual scenario.

[0248] Step S36: The management unit subscribes to the service information from the service configuration center unit, and analyzes the monitoring collection address and port, monitoring collection rules, etc. For details, please refer to the description in the above steps S11 and S12.

[0249] The technical solution provided in the embodiment of the present application provides a function of automatically sending monitoring collection through microservice registration and service discovery, and registering monitoring collection related configuration information in the service registration.

[0250] Specifically, when a microservice is registered with the configuration center unit, it will register the monitoring indicator collection address and port, monitoring indicator type, etc. that can be exposed by the service instance node. When the registration is completed, the configuration center unit will synchronize the service information of each instance to the management unit; after the management unit obtains the monitoring collection address and port, monitoring indicator dimension type and other information corresponding to the service instance, it will generate the corresponding monitoring collection task according to the established monitoring collection rules. The monitoring collection task can be a single collection task or a task with a fixed collection frequency. After the monitoring collection task is generated, it can be sent to the monitoring collection unit through the collection task channel, and the monitoring collection unit will initiate the monitoring collection task.

[0251] Before the monitoring collection unit initiates a collection task, it will complete the configuration of the collection task inside the monitoring collection unit. The configuration information may include: the address and port of the object to be collected, the collection method, collection frequency, collection times, collection indicator data type, collection protocol method (such as whether the collection is collected in HTTP JSON or Telnet JSON), collection data reporting channel method (Kafka or HTTP API JSON), reporting channel address and other information.

[0252] After the monitoring and collection collection unit initiates the collection task, the corresponding monitoring indicator data can be reported to the data transmission channel, and the data transmission channel will transmit the collected and reported monitoring indicator data in real time to the back-end real-time computing unit, such as the data computing unit, and the computing unit will complete the corresponding calculation processing.

[0253] See also Figure 4 , Figure 4 A flow chart of the monitoring collection task issuance and monitoring indicator data collection provided in the embodiment of the present application may include the following steps:

[0254] Step S41: the management unit synchronizes service information from the configuration center unit, obtains the monitoring address of the service registration, and then generates a monitoring collection task according to the monitoring task generation rules configured by the management unit; for details, please refer to the description in the above steps S11 and S12.

[0255] Step S42: the management unit sends the generated monitoring collection task to the monitoring collection unit, and the monitoring collection unit initiates the collection of monitoring indicator data according to the task rules; for details, please refer to the description in the above step S14.

[0256] Step S43: The monitoring collection unit reports the collected monitoring indicator data to the data transmission unit; for details, please refer to the description in the above step S15.

[0257] Step S44: The monitoring and collection unit periodically reports the task status, the number of collection indicators, the status data of the monitoring and collection unit itself, etc. to the management unit for task management; for details, please refer to the description in the above step S13.

[0258] Step S45: The management center performs node switching processing on the tasks assigned to the monitoring collection units with abnormal collection status to ensure the reliable operation of the collection tasks and perform task migration on the collection nodes with abnormal monitoring in real time. For details, please refer to the description in the above step S13.

[0259] It is understandable that machine learning plays an important role in anomaly detection. With the development of industry and the Internet, large-scale data sets have become the basis for the operation of intelligent systems. These data often contain various abnormal and fault-related signals and data, such as server anomalies, communication interruptions of network devices, database access failures, disk write failures, 100% disk usage, 100% CPU usage, and other abnormal information. Machine learning technology can be used to detect and predict these abnormal information, so as to discover abnormal events or failures that may occur in the system in advance.

[0260] Anomalies can be defined as areas or regions in the data that do not conform to the rules. Specifically, they can be divided into the following three categories: First, anomalies: Anomalies refer to values ​​in the data that are significantly beyond the normal range. For example, in a demographic data set, if the population of a city significantly exceeds that of other cities, it can be considered an anomaly; interval anomalies: interval anomalies refer to values ​​in the data that are significantly beyond the normal range for a continuous period of time. For example, in a temperature data set, if the temperature within a period of time significantly exceeds the highest temperature in history, it can be considered as an interval of abnormal data; mixed anomalies: mixed anomalies refer to data that contains both abnormal points and interval anomalies. For example, in a production data set, if the production volume within a period of time has both single product anomalies and overall production volume anomalies, this can be considered as mixed abnormal data.

[0261] Anomaly detection methods can be classified into the following three categories: Statistical methods: methods that detect anomalies based on the statistical characteristics of data, such as using statistical characteristics such as mean, median, variance, standard deviation, etc. to detect anomalies; Machine learning methods: methods that detect anomalies based on machine learning algorithms, such as using decision trees, random forests, support vector machines and other machine learning algorithms to detect anomalies; Deep learning methods: methods that detect anomalies based on deep learning algorithms, such as using deep learning algorithms such as autoencoders and recurrent neural networks to detect anomalies.

[0262] Anomaly detection through machine learning means finding data points that do not conform to normal patterns from large-scale data sets. These data points may represent abnormal behaviors in the system. Anomaly detection can be achieved through supervised and unsupervised learning. Supervised learning methods require a large amount of training data containing labels about abnormalities and normality, and then train classifiers with these labels to identify abnormal samples. Unsupervised learning does not require labeled data. It finds abnormal samples by reducing the dimensionality of the data and performing clustering or password estimation.

[0263] See also Figure 5 , Figure 5 A flowchart of an abnormality detection and abnormal event sending processing process provided in an embodiment of the present application may include the following steps:

[0264] Step S51: The data computing unit periodically synchronizes service information and monitoring alarm detection rules from the configuration center and storage service;

[0265] In the embodiment of the present application, the storage service can be understood as a data storage unit; the alarm detection rule can be understood as a preset anomaly detection rule.

[0266] Step S52: the data calculation unit consumes monitoring index data from the data transmission unit in real time;

[0267] Step S53: The data calculation unit detects and filters the index data consumed from the data transmission channel according to the alarm rule, and generates an abnormal event for the index data that meets the conditions; for details, please refer to the description in the above step S16.

[0268] Step S54: the data calculation unit filters the indicator data according to the alarm detection rule, and if the opposite condition of the rule is met, a recovery event corresponding to the abnormal event is generated;

[0269] Step S55: Send the generated abnormal events and recovery events to the event alarm unit and the alarm sending unit through the interface to complete the alarm sending.

[0270] See also Figure 6 , Figure 6 A flowchart of a process for handling a plan triggered by an alarm event provided in an embodiment of the present application may include the following steps:

[0271] Step S61: The plan handling unit synchronizes the plan configuration from the management unit at regular intervals;

[0272] Step S62: The emergency plan handling unit consumes the abnormal event transmitted in the data transmission unit in real time;

[0273] Step S63: When the emergency plan associated with the abnormal event is hit, the execution of the emergency plan is triggered;

[0274] Step S64: After the plan is successfully executed, the results of each step of the plan execution are recorded, and the execution results are notified to the relevant contacts;

[0275] Step S65: After the execution of the plan fails, the results of each step of the execution of the plan are recorded, and the execution results are notified to the relevant contacts;

[0276] Step S66: After the execution of the plan fails, it is necessary to notify personnel to take over, close the loop of the plan through manual processing, and generate a work order for review;

[0277] Step S67: after the plan execution fails, manual intervention is performed, and the failure reason and handling steps are filled in;

[0278] Step S68: The review is completed, and the relevant details of the plan are optimized through the review to improve the success rate of the plan.

[0279] See also Figure 7 , Figure 7 A flowchart of a plan learning and recommended configuration process provided in an embodiment of the present application may include the following steps:

[0280] Step S71: The plan learning unit obtains indicator data, full-link data, service logs, service dependency graphs and other related data, historical fault data and other data from the data transmission channel;

[0281] Step S72: the plan learning unit analyzes and classifies the data and extracts relevant features according to the machine learning algorithm;

[0282] Step S73: training and testing according to relevant features and configuration models;

[0283] Step S74: configuring the model that meets the test criteria online, and performing fault prediction and detection on real-time data based on online real-time learning;

[0284] Step S75: Push the recommended predicted faults and corresponding disposal plans to the maintenance plan operator, who will filter and configure them into the plan disposal service and activate them;

[0285] Step S76: After the handling plan is triggered by the abnormal alarm event and the corresponding fault is handled successfully, the relevant personnel or the on-duty personnel will score or label the plan;

[0286] Step S77: The plan learning unit optimizes and learns the labeled plan. If it is a successful plan, it will be scored accordingly, and if it is a failed plan, it will be deducted;

[0287] Step S78: Review the operation and maintenance OnCall weekly meeting to adjust and optimize the plan.

[0288] In the embodiments of the present application, fault prediction refers to predicting possible future faults through historical data. Fault prediction can help enterprises take appropriate measures before a fault occurs, perform repairs or replace equipment in advance, thereby reducing downtime and reducing costs. Machine learning can use effective supervised learning and unsupervised learning in fault prediction business scenarios.

[0289] In the technical solution provided in the embodiment of the present application, relevant plans are recommended through machine learning, and then the plans are configured and activated by manual operation. When an abnormal event is reported, the plan association and handling system detects and judges the correlation between the event and the plan. If the rules and conditions are met, the corresponding emergency plan is triggered for handling, thereby shortening the handling time before the fault occurs and allowing fault handling to intervene in advance, thereby achieving the goal of self-healing of the fault.

[0290] In addition, the embodiments of the present application can also manually label and determine the results of the plan disposal, and then go back to optimize the plan disposal process, and manually optimize and process the inappropriate or inappropriate processes and steps in the disposal process, so as to achieve further refined optimization of the plan. To make up for the aspects that machine learning may not be more humane. Therefore, the present device actually achieves the method of machine learning plus manual operation, so as to achieve monitoring anomaly judgment and plan recommendation and configuration, more refined associated plans for faults, and accurate plan disposal. Multiple steps such as replay learning optimization have achieved a self-closed loop and operation from monitoring to anomaly detection, fault linkage plan to plan learning.

[0291] The embodiment of the present application also provides a fault alarm self-healing method, which is applied to any of the above-mentioned fault alarm self-healing systems, the fault alarm self-healing system includes a microservice target unit, a management unit, a monitoring and collection unit, a data calculation unit, and a plan disposal unit. Figure 8 , the method comprises the following steps:

[0292] Step S81: the management unit obtains metadata corresponding to each microservice instance from the microservice target unit; determines the monitoring collection tasks to be added and the monitoring collection tasks to be deleted according to the monitoring information carried by each metadata; and sends the monitoring collection tasks to be added and the monitoring collection tasks to be deleted to the monitoring collection unit;

[0293] Step S82: The monitoring collection unit receives the monitoring collection tasks to be added and the monitoring collection tasks to be deleted sent by the management unit, collects monitoring indicator data according to the monitoring collection tasks to be added, and sends the monitoring indicator data to the data calculation unit; and deletes the corresponding monitoring collection tasks according to the monitoring collection tasks to be deleted; wherein the monitoring collection tasks to be added carry an identifier, a name, a task type, and a task running configuration content;

[0294] Step S83: the data calculation unit receives the monitoring index data sent by the monitoring collection unit, performs real-time detection on the monitoring index data according to the preset index detection rule, and obtains a first detection result; generates time series index data according to the first detection result; and performs real-time detection on the monitoring index data and the time series index data according to the preset alarm detection rule, and obtains a second detection result; generates an abnormal event according to the second detection result; and sends the abnormal event to the plan handling unit;

[0295] Step S84: the plan handling unit receives the abnormal events sent by the data computing unit, retrieves the fault self-healing plan matching each abnormal event, and performs fault self-healing processing on the abnormal event according to the fault self-healing plan.

[0296] In the technical solution provided in the embodiment of the present application, the management unit obtains metadata corresponding to the instances of each microservice from the microservice target unit, and determines the monitoring collection tasks to be added and the monitoring collection tasks to be deleted according to the monitoring information carried by each metadata; thereafter, the management unit can send the generated monitoring collection tasks to be added and the monitoring collection tasks to be deleted to the monitoring collection unit, and the monitoring collection unit can add corresponding collection tasks based on the monitoring collection tasks to be added, and stop corresponding collection tasks according to the monitoring collection tasks to be deleted, so as to realize the deployment and adjustment of the monitoring tasks, and then obtain the monitoring indicator data collected by the monitoring collection tasks.

[0297] Furthermore, the monitoring collection unit can send the monitoring indicator data to the data calculation unit; the data calculation unit monitors the monitoring indicator data in real time according to the preset indicator detection rules, obtains the first detection result, and generates the timing indicator data according to the first detection result; thereafter, the monitoring indicator data and the timing indicator data are detected in real time according to the preset anomaly detection rules, and the second detection result is obtained, and an abnormal event is generated according to the second detection result, thereby determining the abnormal events existing in the instance operation process of each microservice.

[0298] For each abnormal event, the data computing unit can send each abnormal event to the plan handling unit. After receiving each abnormal event, the plan handling unit can retrieve the fault self-healing plan matching each abnormal event, and process the corresponding abnormal event according to the fault self-healing plan. In this way, the abnormal events existing in the instance operation of each microservice can be automatically resolved by the system, realizing the self-healing of microservice faults.

[0299] In some embodiments, the fault alarm self-healing system further includes a data storage unit, a plan learning unit, a plan recommendation and management unit, and the fault self-healing solution is obtained by:

[0300] The plan learning unit performs machine learning processing based on the monitored indicator data and time series indicator data to obtain the handling action for the abnormal event; and sends the handling action to the plan recommendation and management unit;

[0301] The plan recommendation and management unit receives the treatment action sent by the plan learning unit, optimizes the treatment action to obtain the optimized treatment action; and sends the optimized treatment action to the data storage unit;

[0302] The data storage unit receives the optimized processing action, and associates the optimized processing action with the abnormal event to obtain abnormal event configuration data;

[0303] The plan handling unit obtains the abnormal event configuration data from the data storage unit, and performs structured processing on the abnormal event configuration data to obtain a structure; according to the structure, a fault self-healing plan matching each abnormal event is determined.

[0304] In some embodiments, the fault alarm self-healing system further includes an event alarm unit, and the sending of abnormal events to the plan handling unit includes:

[0305] The data calculation unit sends the abnormal event to the event alarm unit;

[0306] The event alarm unit receives the abnormal events sent by the event data calculation unit, classifies and processes the abnormal events to obtain target abnormal events; and sends the target abnormal events to the emergency plan handling unit;

[0307] Among them, the abnormal events are classified and processed to obtain target abnormal events, including:

[0308] For an abnormal event, if multiple identical abnormal events are received within a first preset time, one of the multiple identical abnormal events is used as a target abnormal event; and / or,

[0309] For an abnormal event, if there are multiple abnormal events with the same attributes or rules within the second preset time, the abnormal events are merged according to the preset merging rules to obtain the target abnormal event; and / or,

[0310] For abnormal events, if there are multiple abnormal events with similar characteristics or business attributes within the third preset time, the abnormal events with similar characteristics or business attributes are merged and converged to obtain a target abnormal event.

[0311] In some embodiments, for an abnormal event, if a recovery event of the abnormal event is not received within a fourth preset time, the event alarm unit regards the abnormal event as an upgraded abnormal event.

[0312] In some embodiments, the fault alarm self-healing system further includes an alarm sending unit and an alarm receiving body.

[0313] The alarm sending unit obtains the target abnormal event from the event alarm unit; and sends the target abnormal event to the alarm receiving subject according to the alarm rule or subscription rule;

[0314] The alarm receiving subject receives the target abnormal event sent by the alarm sending unit.

[0315] In some embodiments, the data calculation unit performs real-time detection on the monitoring collection index data and the time series index data according to a preset anomaly detection rule to obtain a third detection result, generates a recovery event according to the third detection result; and sends the recovery event to the event alarm unit;

[0316] The event alarm unit receives the recovery event sent by the data calculation unit.

[0317] In some embodiments, the fault alarm self-healing system further includes a configuration center unit, and the acquiring metadata corresponding to the instance of each microservice from the microservice target unit includes:

[0318] The microservice target unit registers the service information of each microservice instance to the configuration center unit. The service information includes the service name, monitoring collection address, monitoring collection port, and instance metadata.

[0319] The configuration center unit generates a service instance list according to each service information;

[0320] The management unit obtains the service instance list from the configuration center unit, and reads the metadata corresponding to the instance of each microservice from the service instance list.

[0321] In some embodiments, there is a communication mechanism between each microservice instance and the configuration center unit.

[0322] If there is a target instance that has not communicated with the configuration center unit within a fifth preset time, the configuration center unit deletes the target instance from the service instance list.

[0323] In some embodiments, the fault alarm self-healing system further includes a data storage unit, a data display unit, and a data transmission unit.

[0324] The data storage unit acquires and stores system data, and sends the system data to the data display unit; the system data includes metadata, monitoring indicator data, time series indicator data, abnormal events, recovery events, preset indicator detection rules, preset abnormal detection rules, subscription rules, and fault self-healing solutions;

[0325] The data display unit receives the system data sent by the data storage unit and displays the system data;

[0326] The step of sending the monitoring indicator data to the data calculation unit includes:

[0327] The monitoring collection unit sends the monitoring index data to the data transmission unit;

[0328] The data transmission unit sends the monitoring indicator data to the data calculation unit.

[0329] The present application also provides an electronic device, such as Fig. 9 As shown, including:

[0330] Memory 91, used for storing computer programs;

[0331] The processor 92 is used to implement any of the above-mentioned fault alarm self-healing systems when executing the program stored in the memory 91.

[0332] Furthermore, the electronic device may further include a communication bus and / or a communication interface, and the processor 92, the communication interface, and the memory 91 communicate with each other via the communication bus.

[0333] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0334] The communication interface is used for communication between the above electronic device and other devices.

[0335] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0336] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0337] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, any of the above-mentioned fault alarm self-healing systems is implemented.

[0338] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the fault alarm self-healing system described in any one of the above embodiments.

[0339] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0340] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0341] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the method embodiment, since it is basically similar to the system embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0342] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A fault alarm self-healing system, characterized in that: The system includes a microservice target unit, a management unit, a monitoring and acquisition unit, a data calculation unit, and a plan handling unit. The management unit is used to obtain metadata corresponding to each microservice instance from the microservice target unit; determine the monitoring collection tasks to be added and the monitoring collection tasks to be deleted according to the monitoring information carried by each metadata; Sending the monitoring collection task to be added and the monitoring collection task to be deleted to the monitoring collection unit; The monitoring collection unit is used to receive the monitoring collection tasks to be added and the monitoring collection tasks to be deleted sent by the management unit, collect monitoring indicator data according to the monitoring collection tasks to be added, and send the monitoring indicator data to the data calculation unit; and delete the corresponding monitoring collection tasks according to the monitoring collection tasks to be deleted; wherein the monitoring collection tasks to be added carry an identifier, a name, a task type, and a task running configuration content; The data calculation unit is used to receive the monitoring indicator data sent by the monitoring collection unit, monitor the monitoring indicator data in real time according to the preset indicator detection rule to obtain a first detection result; generate time series indicator data according to the first detection result; and perform real-time detection on the monitoring indicator data and the time series indicator data according to the preset abnormality detection rule to obtain a second detection result; generate an abnormal event according to the second detection result; and send the abnormal event to the plan handling unit; The plan handling unit is used to receive the abnormal events sent by the data computing unit, retrieve the fault self-healing plan matching each abnormal event, and perform fault self-healing processing on the abnormal events according to the fault self-healing plan.

2. The system according to claim 1, characterized in that The system further includes a data storage unit, a plan learning unit, a plan recommendation and management unit, and the fault self-healing solution is obtained by: The plan learning unit is used to perform machine learning processing based on the monitoring indicator data and the time series indicator data to obtain a handling action for the abnormal event; and send the handling action to the plan recommendation and management unit; The plan recommendation and management unit is used to receive the treatment action sent by the plan learning unit, optimize the treatment action to obtain an optimized treatment action; and send the optimized treatment action to the data storage unit; The data storage unit is used to receive the optimized handling action, and associate the optimized handling action with the abnormal event to obtain abnormal event configuration data; The plan handling unit is used to obtain the abnormal event configuration data from the data storage unit, and perform structured processing on the abnormal event configuration data to obtain a structure; According to the structure, a fault self-healing solution matching each of the abnormal events is determined.

3. The system according to claim 1, characterized in that The system further includes an event alarm unit, and the abnormal event is sent to the plan handling unit, including: The data calculation unit sends the abnormal event to the event alarm unit; The event alarm unit receives the abnormal event sent by the event data calculation unit, classifies the abnormal event to obtain a target abnormal event; and sends the target abnormal event to the emergency plan handling unit; The abnormal events are classified and processed to obtain target abnormal events, including: For the abnormal event, if multiple identical abnormal events are received within a first preset time, one of the multiple identical abnormal events is used as a target abnormal event; and / or, For the abnormal event, if there are multiple abnormal events with the same attributes or rules within the second preset time, merge them according to the preset merging rules to obtain the target abnormal event; and / or, For the abnormal event, if there are multiple abnormal events with similar characteristics or business attributes within the third preset time, the abnormal events with similar characteristics or business attributes are merged and converged to obtain a target abnormal event.

4. The system according to claim 3, characterized in that The event alarm unit is also used to: For the abnormal event, if no recovery event of the abnormal event is received within a fourth preset time, the abnormal event is regarded as an upgraded abnormal event.

5. The system according to claim 3, characterized in that The system also includes an alarm sending unit and an alarm receiving body. The alarm sending unit is used to obtain the target abnormal event from the event alarm unit; and send the target abnormal event to the alarm receiving subject according to the alarm rule or subscription rule; The alarm receiving entity is used to receive the target abnormal event sent by the alarm sending unit.

6. The system according to claim 4, characterized in that The data calculation unit is further used to perform real-time detection on the monitoring collection index data and the time series index data according to the preset anomaly detection rule to obtain a third detection result, and generate a recovery event according to the third detection result; and sending the recovery event to the event alarm unit; The event alarm unit is further used to receive the recovery event sent by the data calculation unit.

7. The system according to claim 1, characterized in that The system further includes a configuration center unit, and the step of obtaining metadata corresponding to the instance of each microservice from the microservice target unit includes: The microservice target unit registers service information of each microservice instance to the configuration center unit, wherein the service information includes a service name, a monitoring collection address, a monitoring collection port, and metadata of the instance; The configuration center unit generates a service instance list according to each of the service information; The management unit obtains the service instance list from the configuration center unit, and reads metadata corresponding to the instances of each microservice from the service instance list.

8. The system according to claim 7, characterized in that There is a communication mechanism between the instances of each microservice and the configuration center unit. The configuration center unit is further configured to delete the target instance from the service instance list if there is a target instance that has not communicated with the configuration center unit within a fifth preset time.

9. The system according to claim 1, characterized in that The system also includes a data storage unit, a data display unit, and a data transmission unit. The data storage unit is used to acquire and store system data, and send the system data to the data display unit; the system data includes the metadata, monitoring indicator data, time series indicator data, abnormal events, recovery events, preset indicator detection rules, preset abnormal detection rules, subscription rules, and fault self-healing solutions; The data display unit is used to receive the system data sent by the data storage unit and display the system data; The sending the monitoring indicator data to the data calculation unit includes: The monitoring collection unit sends the monitoring indicator data to the data transmission unit; The data transmission unit sends the monitoring indicator data to the data calculation unit.

10. A fault alarm self-healing method, characterized in that: The method is applied to a fault alarm self-healing system, which includes a microservice target unit, a management unit, a monitoring and acquisition unit, a data calculation unit, and a plan disposal unit. The method includes: The management unit obtains metadata corresponding to the instances of each microservice from the microservice target unit; determines the monitoring collection tasks to be added and the monitoring collection tasks to be deleted according to the monitoring information carried by each metadata; and sends the monitoring collection tasks to be added and the monitoring collection tasks to be deleted to the monitoring collection unit; The monitoring collection unit receives the monitoring collection tasks to be added and the monitoring collection tasks to be deleted sent by the management unit, collects monitoring indicator data according to the monitoring collection tasks to be added, and sends the monitoring indicator data to the data calculation unit; and deletes the corresponding monitoring collection tasks according to the monitoring collection tasks to be deleted; wherein the monitoring collection tasks to be added carry an identifier, a name, a task type, and a task running configuration content; The data calculation unit receives the monitoring index data sent by the monitoring collection unit, performs real-time detection on the monitoring index data according to a preset index detection rule to obtain a first detection result; generates time series index data according to the first detection result; and performs real-time detection on the monitoring index data and the time series index data according to a preset alarm detection rule to obtain a second detection result; generates an abnormal event according to the second detection result; and sends the abnormal event to the plan handling unit; The plan handling unit receives the abnormal events sent by the data computing unit, retrieves fault self-healing plans matching each of the abnormal events, and performs fault self-healing processing on the abnormal events according to the fault self-healing plans.

Images