Network system event protection method and system

Abstract

The present application relates to a network system event protection method and system, which receives data from a data source, analyzes the data from the data source, obtains alarm data and link alarm data, and intelligently analyzes the alarm data and link alarm data to reduce useless alarms and improve the quality of alarms. Based on the alarm compression algorithm, the data to be processed is compressed to support intelligent association of relevant indicators and call link data when an alarm is issued. Using the kafka algorithm, the compressed alarms are merged to generate events to be handled. It is determined whether the event to be handled is the first dispatch, and a machine learning algorithm is used to identify problems in applications, services, middleware, databases, and infrastructure that cause most of today's events and interruptions, recommend possible causes of failures, and help operation and maintenance personnel identify the root causes of accidents and downtime in real time. Reduce the interference of alarm storms on data flow in network system events.

Description

Technical Field

[0001] The present application relates to the field of network protection technology, and in particular to a network system event protection method and system. Background Art

[0002] Data flow refers to the process of data transmission in software development. Simply viewing a function or a whole system as the transmission of data is data flow. The data is ultimately transmitted to the database or front-end page, or somewhere else.

[0003] In network protection technology, the trend of alarm events is often closely related to data flow. Traditional network system event protection has a relatively discrete detection system and is difficult to track problem events, which leads to a large interference of alarm storms on data flow. Therefore, it is necessary to propose a network system event protection method and system to address the disadvantage that traditional network system event protection methods are difficult to prevent the interference of alarm storms on data flow. Summary of the invention

[0004] Based on this, it is necessary to propose a network system event protection method and system to address the disadvantage that traditional network system event protection methods are difficult to prevent alarm storms from interfering with data flow.

[0005] The present application provides a network system event protection method, comprising:

[0006] Based on the alarm access port, receive data from the data source;

[0007] Use data verification algorithms to clean the data from the data source to obtain the data to be processed;

[0008] Based on the alarm compression algorithm, the data to be processed is compressed to obtain a compressed alarm;

[0009] Using the Kafka algorithm, compressed alerts are merged to generate events to be processed;

[0010] Select an event to be processed;

[0011] Determine whether the event to be processed is the first dispatch;

[0012] If the event to be handled is assigned for the first time, the risk level of the event to be handled is determined to be level 2 based on the handling algorithm;

[0013] If the event to be handled is not assigned for the first time, the risk level of the event to be handled is determined based on the event determination algorithm;

[0014] Based on the risk level of the event to be handled, alarm notification information is fed back.

[0015] The present application also provides a network system event protection system, comprising:

[0016] A server, used to execute the network system event protection method;

[0017] A memory is communicatively connected to the server.

[0018] The present application relates to a network system event protection method and system, which receives data from a data source, analyzes the data from the data source, obtains alarm data and link alarm data, performs intelligent analysis on the alarm data and link alarm data, reduces useless alarms, and improves the quality of alarms. Based on the alarm compression algorithm, the data to be processed is compressed to support intelligent association of relevant indicators and call link data when the alarm is issued. The compressed alarms are merged using the kafka algorithm to generate events to be handled. Specifically, the alarm messages of the monitoring system are unified on a single platform for management. The alarm source supports multiple alarm source access methods such as REST API and URL callback, as well as third-party alarm sources. Then, the fields and contents of the original alarm are corrected so that the alarm has a unified field, which is convenient for the display interface to sort, search, display, etc., and at the same time, it is convenient for the subsequent process of generating events, unifying the fields of alarm information from different data sources, and facilitating the setting of associations. Using additional data, the alarm message is given more abundant information, such as other information such as the host and service, and the resource information maintained in the CMDB is obtained and associated with the alarm message. Based on intelligent algorithms, alarm compression and alarm merging are performed on massive and continuous redundant messages, which suppresses the number of alarm messages, reduces the frequency of alarm messages, reduces the interference of redundant alarm messages on operation and maintenance work, and improves the efficiency of operation and maintenance work. It determines whether the event to be handled is the first dispatch, uses machine learning algorithms to identify problems in applications, services, middleware, databases, and infrastructure that cause most of today's events and interruptions, recommends possible causes of failures, and helps operation and maintenance personnel identify the root causes of accidents and downtime in real time. It reduces the interference of alarm storms on data flow in network system events. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings constituting a part of this application are used to provide a further understanding of this application, so that other features, purposes and advantages of this application become more obvious. The illustrative embodiment drawings and their descriptions of this application are used to explain this application and do not constitute an improper limitation on this application.

[0020] Figure 1 A method flow chart of a network system event protection method provided in one embodiment of the present application;

[0021] Figure 2 A structural connection diagram of a network system event protection system provided in one embodiment of the present application.

[0022] Reference numerals:

[0023] 100-server; 200-storage. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0025] The present application provides a network system event protection method.

[0026] like Figure 1 As shown, in one embodiment of the present application, a network system event protection method includes:

[0027] S100: Receive data from a data source based on an alarm access port.

[0028] Specifically, the alarm access port accepts requests from the web layer and encapsulates data processing logic. It uses JAVA language to encapsulate request objects and return results. It relies on the web layer and calls the business logic layer.

[0029] S200, using a data verification algorithm to clean the data of the data source to obtain data to be processed.

[0030] Specifically, the data verification algorithm is based on the fields of the alarm compression index, such as filter conditions, logical symbols, field names, field values, etc.

[0031] S300, based on an alarm compression algorithm, compress the data to be processed to obtain a compressed alarm.

[0032] Specifically, the alarm compression algorithm is used to integrate the interface component name, the initiator data format, the time window, etc.

[0033] S400: Using the Kafka algorithm, the compressed alarms are merged to generate events to be processed.

[0034] Specifically, Kafka's partition assignment strategy is an important part of determining how to assign partitions to the business logic layer, the basic service layer, and the timing links between the layers. Each timestamp in the timing links between the layers needs to be responsible for processing a certain number of partitions, and the partition assignment strategy determines how to fairly and effectively assign these partitions to achieve load balancing and high availability.

[0035] S500: Select an event to be processed.

[0036] S600: Determine whether the event to be processed is the first dispatch.

[0037] Specifically, the time correlation of alarms is analyzed based on the ACorFrepm algorithm. Alarms are abstracted into indicator data, and the co-occurrence of alarms is counted using a sliding time window to mine the time correlation of alarms and output the dynamic correlation degree between faults.

[0038] S700: If the event to be handled is assigned for the first time, the risk level of the event to be handled is determined to be level 2 based on the handling algorithm.

[0039] S800: If the event to be handled is not assigned for the first time, the risk level of the event to be handled is determined based on an event determination algorithm.

[0040] Specifically, the processing priority of alarms is determined dynamically by analyzing the current health status of the system. The periodic characteristics, impact range, processing time, severity and other information of the corresponding faults are obtained from the alarms with the same fault or problem, the associated indicators and the associated topology. Finally, the alarm level and the risk level of the event to be handled are determined through machine learning and sorting models.

[0041] S900: Feedback alarm notification information based on the risk level of the event to be handled.

[0042] The present embodiment relates to a network system event protection method, which receives data from a data source, analyzes the data from the data source, obtains alarm data and link alarm data, performs intelligent analysis on the alarm data and link alarm data, reduces useless alarms, and improves the quality of alarms. Based on the alarm compression algorithm, the data to be processed is compressed to support intelligent association of relevant indicators and call link data when an alarm is issued. The compressed alarms are merged using the kafka algorithm to generate events to be handled. Specifically, the alarm messages of the monitoring system are unified on a single platform for management. The alarm source supports multiple alarm source access methods such as REST API and URL callback, as well as third-party alarm sources. Then, the fields and contents of the original alarm are corrected so that the alarm has unified fields, which facilitates sorting, searching, and displaying of the display interface, and facilitates the subsequent process of generating events, unifies the fields of alarm information from different data sources, and facilitates setting associations. Additional data is used to give the alarm message more abundant information, such as other information such as hosts and services, and obtains resource information maintained in the CMDB to associate with the alarm message. Based on intelligent algorithms, alarm compression and alarm merging are performed on massive and continuous redundant messages, which suppresses the number of alarm messages, reduces the frequency of alarm messages, reduces the interference of redundant alarm messages on operation and maintenance work, and improves the efficiency of operation and maintenance work. It determines whether the event to be handled is the first dispatch, uses machine learning algorithms to identify problems in applications, services, middleware, databases, and infrastructure that cause most of today's events and interruptions, recommends possible causes of failures, and helps operation and maintenance personnel identify the root causes of accidents and downtime in real time. It reduces the interference of alarm storms on data flow in network system events.

[0043] In one embodiment of the present application, S100 includes:

[0044] S110, the data source includes one or more of data from an alarm program system, data from an indicator monitoring system, and data from a mail server.

[0045] Specifically, data classification includes alarms, original alarm messages, alarm silence, alarm templates, dispatch logs, dispatch task instances, dispatch rules, events, event logs, merge rules, fault self-healing, etc.

[0046] These types of data come from the alarm system data, indicator monitoring system data, and mail server data.

[0047] The acquisition of rest alarm data, zabbix alarm data, kafka alarm data, indicator alarm data, email alarm, SMS alarm, alarm message, alarm, event, alarm message, alarm, event, disposal, alarm, event, disposal and other alarms can enable the network system event protection algorithm to participate in data flow.

[0048] Simply put, the alarm program system, indicator monitoring system, and mail server participate in data flow based on the network system event protection algorithm.

[0049] In one embodiment of the present application, S100 further includes:

[0050] S121, obtaining data parameters from a data source.

[0051] S122, using an array to summarize the alarm data corresponding to the data parameters.

[0052] Specifically, the usage scenario of array collection parameters: In an HTML form, there is a form item that supports multiple selections (checkboxes), and multiple selected values ​​can be submitted.

[0053] If the request parameter name is the same as the parameter array name and there are multiple request parameters, define an array type parameter to receive the parameters.

[0054] S123, encapsulate data parameters using timestamp.

[0055] S124, receiving request parameters based on the timestamp of the data parameters.

[0056] Specifically, when the server initiates data seeking, the server may transmit some common request data. In the original Web program, it is necessary to pass through the API provided in the Servlet.

[0057] AP1 is HtpServletRequest (request object), which obtains relevant information of the request, such as request parameters. In development, if you encounter different request parameters, you can convert them to keep the format of the request parameters consistent.

[0058] When using simple parameters as a data transmission method, the number of formal parameters in the backend controller method must be the same as the number of request parameters passed by the frontend. If there are many request parameters, it will be cumbersome to receive them one by one in the above method. In this case, we can consider encapsulating the request parameters into an entity class object. The request parameter name is the same as the attribute name of the entity class.

[0059] In one embodiment of the present application, S200 includes:

[0060] S211, calling the timestamp of the data from the data source.

[0061] S212, determine the encapsulated data parameters.

[0062] S213: Determine the corresponding alarm data of the data parameter based on the summarized array.

[0063] S214, determining the data source of the alarm data.

[0064] Specifically, data access is responsible for the maintenance of business data, including adding, deleting, modifying, and querying. Logic processing is responsible for the code of business logic processing. Request processing and response data are responsible for receiving page requests and responding to page data.

[0065] The basic service layer receives requests sent by the front end, processes the requests, and responds with data. Business logic layer. Processes specific business logic. Data access layer (Data Access Object), also known as the persistence layer. Responsible for data access operations, including adding, deleting, modifying, and querying data. The data access layer (Data Access Object) can use the timestamp of the data source of the business front end.

[0066] In one embodiment of the present application, S200 further includes:

[0067] S221, clean the data of the data source based on the data cleaning format.

[0068] S222, determining whether the cleaned data of the data source has necessary fields in the data cleansing format.

[0069] S223: If the data of the cleaned data source has necessary fields of the data cleansing format, the data of the cleaned data source is used as data to be processed.

[0070] S224: If the data of the cleaned data source does not have necessary fields of the data cleansing format, the data of the cleaned data source is converted into fields to generate data to be processed.

[0071] Specifically, the data to be processed includes the interface address, calling method type, request data type and response data type, etc. Such as event name, data status level, alarm source name, check field, merge name, timestamp, etc.

[0072] In one embodiment of the present application, S300 includes:

[0073] S310, calling data to be processed.

[0074] S320, compressing the alarm field in the data to be processed according to the alarm compression rule.

[0075] S330, determining the alarm level of the compressed alarm field.

[0076] S340, sorting each compressed alarm field based on the alarm level.

[0077] S350, returning to the step of compressing the alarm field in the data to be processed according to the alarm compression rule, until the alarm level of the compressed alarm field is determined.

[0078] S360, defining the alarm levels of the compressed alarm fields to be arranged from low to high.

[0079] Specifically, more fault information is obtained from the inside and outside of the alarm to form a normalized alarm message. The real-time alarm message is aggregated into time series data, and the data is stabilized from the perspective of energy by combining difference and sliding average. The bilateral energy of the storm point is calculated to achieve real-time detection of the alarm storm.

[0080] The alarm levels of the compressed alarm fields are arranged from low to high, which can realize the real-time detection of the alarm storm. The alarm levels of the compressed alarm fields are arranged from low to high, which can realize the calculation of the bilateral energy of the storm point.

[0081] In one embodiment of the present application, S400 includes:

[0082] S411, incorporate the business logic layer and basic service layer into the network system event protection layer.

[0083] S412, based on the interface layer, establish a communication connection between the business front end and the business logic layer.

[0084] S413, establish a communication connection between the business logic layer and the basic service layer to form a Kafka algorithm communication link.

[0085] Specifically, the processing priority of alarms is determined dynamically by analyzing the current health status of the system. The periodic characteristics, impact range, processing time, severity and other information of the corresponding faults are obtained from the alarms with the same fault or problem, the associated indicators and the associated topology, and finally the alarm level is determined through machine learning and sorting models.

[0086] S421, based on the timestamp of the data source of the business front end.

[0087] S422, determining the parallelism of the compression alarm on the timing line of the Kafka algorithm communication link.

[0088] S423: Generate an event to be processed based on the parallelism of the compressed alarm.

[0089] Specifically, the time correlation of alarms is analyzed based on the ACorFrepm algorithm. Alarms are abstracted into indicator data, and the co-occurrence of alarms is counted using a sliding time window to mine the time correlation of alarms and output the dynamic correlation degree between faults.

[0090] In one embodiment of the present application, S600 includes:

[0091] S610, calling the timing line of the Kafka algorithm communication link.

[0092] S620: Determine whether there is at least one compressed alarm whose parallelism is greater than 2 among the events to be processed.

[0093] S630: If the parallelism of at least one compressed alarm of the event to be processed is greater than 2, it is determined that the event to be processed is not assigned for the first time.

[0094] S640: If the event to be processed does not have a compression alarm whose parallelism is greater than 2, it is determined that the event to be processed is assigned for the first time.

[0095] Specifically, the timing line of the communication link of the Kafka algorithm is called to find out from the complicated events to be processed that there is at least one event to be processed whose parallelism of compressed alarm is greater than 2.

[0096] This allows the first dispatch of events to be handled with a collection of intrinsically related alarm messages, allowing limited human resources to be assigned to handle them, quickly shortening MTTR and ensuring that the system is always online.

[0097] The present application provides a network system event protection system.

[0098] like Figure 2 As shown, in one embodiment of the present application, a network system event protection system includes:

[0099] The server 100 is used to execute the network system event protection method.

[0100] The communication connection between the storage device 200 and the server 100

[0101] This embodiment relates to a network system event protection system. The server 100 receives data from a data source, analyzes the data of the data source stored in the memory 200, obtains alarm data and link alarm data, and performs intelligent analysis on the alarm data and link alarm data to reduce useless alarms and improve the quality of alarms. Based on the alarm compression algorithm, the data to be processed is compressed to support intelligent association of relevant indicators and call link data when an alarm is issued. The compressed alarms are merged using the kafka algorithm to generate events to be handled. Specifically, the alarm messages of the monitoring system are unified on a single platform for management. The alarm source supports multiple alarm source access methods such as REST API and URL callback, as well as third-party alarm sources. Then, the fields and contents of the original alarm are corrected so that the alarm has unified fields, which facilitates sorting, searching, and displaying of the display interface, and facilitates the subsequent process of generating events, unifies the fields of alarm information from different data sources, and facilitates setting associations. Additional data is used to give the alarm message more abundant information, such as other information such as the host and service, and obtains resource information maintained in the CMDB to associate with the alarm message. Based on intelligent algorithms, alarm compression and alarm merging are performed on massive and continuous redundant messages, which suppresses the number of alarm messages, reduces the frequency of alarm messages, reduces the interference of redundant alarm messages on operation and maintenance work, and improves the efficiency of operation and maintenance work. It determines whether the event to be handled is the first dispatch, uses machine learning algorithms to identify problems in applications, services, middleware, databases, and infrastructure that cause most of today's events and interruptions, recommends possible causes of failures, and helps operation and maintenance personnel identify the root causes of accidents and downtime in real time. It reduces the interference of alarm storms on data flow in network system events.

[0102] The technical features of the above-described embodiments may be arbitrarily combined, and the execution order of the method steps is not limited. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0103] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A network system event protection method, characterized in that: include: Based on the alarm access port, receive data from the data source; Use data verification algorithms to clean the data from the data source to obtain the data to be processed; Based on the alarm compression algorithm, the data to be processed is compressed to obtain a compressed alarm; Using the Kafka algorithm, compressed alerts are merged to generate events to be processed; Select an event to be processed; Determine whether the event to be processed is the first dispatch; If the event to be handled is assigned for the first time, the risk level of the event to be handled is determined to be level 2 based on the handling algorithm; If the event to be handled is not assigned for the first time, the risk level of the event to be handled is determined based on the event determination algorithm; Based on the risk level of the event to be handled, alarm notification information is fed back.

2. The network system event protection method according to claim 1, characterized in that: The receiving data from the data source based on the alarm access port includes: The data source includes one or more of data from an alarm program system, data from an indicator monitoring system, and data from a mail server.

3. The network system event protection method according to claim 2, characterized in that: The receiving data from the data source based on the alarm access port also includes: Get data parameters from the data source; Use arrays to summarize the alarm data corresponding to data parameters; Use timestamps to encapsulate data parameters; Receive request parameters based on the timestamp of the data parameter.

4. The network system event protection method according to claim 3, characterized in that: The data verification algorithm is used to clean the data of the data source to obtain the data to be processed, including: The timestamp of the data in the calling data source; Determine the encapsulated data parameters; Based on the summarized array, determine the corresponding alarm data of the data parameter; Determine the data source of the alarm data.

5. The network system event protection method according to claim 4, characterized in that: The method of using a data verification algorithm to clean the data of the data source to obtain the data to be processed also includes: Clean the data from the data source based on the data cleaning format; Determine whether the data in the cleaned data source has the necessary fields in the data cleansing format; If the data of the cleaned data source contains necessary fields of the data cleansing format, the data of the cleaned data source is used as the data to be processed; If the data of the cleaned data source does not have necessary fields of the data cleansing format, the data of the cleaned data source is converted into fields to generate data to be processed.

6. The network system event protection method according to claim 5, characterized in that: The method of compressing the data to be processed based on the alarm compression algorithm to obtain a compressed alarm includes: Call the data to be processed; Compress the alarm field in the data to be processed according to the alarm compression rule; Determine the alarm level of the compressed alarm field; Sort each compressed alert field based on the alert level; Returning to compressing the alarm field in the data to be processed according to the alarm compression rule until the alarm level of the compressed alarm field is determined; The alarm levels defining the compressed alarm fields are arranged from low to high.

7. The network system event protection method according to claim 6, characterized in that: The Kafka algorithm is used to merge the compressed alarms to generate events to be processed, including: Incorporate the business logic layer and basic service layer into the network system event protection layer; Based on the interface layer, the business front end and the business logic layer form a communication connection; Establish a communication connection between the business logic layer and the basic service layer to form a Kafka algorithm communication link.

8. The network system event protection method according to claim 7, characterized in that: The method of merging compressed alarms by using the Kafka algorithm to generate events to be processed also includes: Timestamp based on the data source of the business front end; Determine the parallelism of compression alerts on the timing line of the Kafka algorithm communication link; Generates events to be processed based on the parallelism of compressed alerts.

9. The network system event protection method according to claim 8, characterized in that: The determining whether the event to be handled is the first dispatch includes: The timing line of calling the Kafka algorithm communication link; Determine whether there is at least one compressed alarm with a parallelism greater than 2 in the event to be processed; If the parallelism of at least one compressed alarm of the event to be processed is greater than 2, it is determined that the event to be processed is not dispatched for the first time; If the parallelism of the event to be processed does not exist and the compression alarm is greater than 2, it is determined that the event to be processed is dispatched for the first time.

10. A network system event protection system, characterized in that: include: A server, configured to execute the network system event protection method according to any one of claims 1 to 9; A memory is communicatively connected to the server.

Images