Network log analysis method, device, electronic device and storage medium

Abstract

The present application provides a network log analysis method, device, electronic device and storage medium, the method comprising: obtaining a network log of a network device; using a distributed event stream platform to buffer the network log data to obtain the buffered network log; using a distributed stream data flow engine to analyze the buffered network log to obtain an analysis result. In the above implementation process, by using a dedicated distributed event stream platform to buffer the streaming data, and using a dedicated distributed stream data flow engine to analyze the buffered streaming data, a sudden and instantaneous large amount of network log data can be effectively used as a streaming data buffer, effectively coping with the situation where a sudden and instantaneous large amount of network log data is sent simultaneously.

Description

Technical Field

[0001] The present application relates to the technical fields of network communication management, automated operation and maintenance, and network log analysis, and more specifically, to a network log analysis method, device, electronic device, and storage medium. Background Art

[0002] The Simple Network Management Protocol (SNMP) forms part of the Internet protocol suite defined by the Internet Engineering Task Force (IETF). The protocol enables network management systems to monitor devices connected to the network for any conditions that require management attention.

[0003] File Transfer Protocol (FTP) refers to a set of standard protocols for transferring files on the Internet. The FTP protocol is the fourth layer of the Transmission Control Protocol model, that is, the application layer, and uses the Transmission Control Protocol instead of the User Datagram Protocol.

[0004] At present, the existing network log collection and analysis usually adopts the SNMP trap (Trap) collection method or the FTP protocol collection method to collect and analyze the network logs. Among them, the SNMP trap (Trap) collection method is an example: the managed device in the SNMP protocol actively sends log information to the SNMP manager, and then the SNMP manager passively receives the log information sent by the managed device in the SNMP protocol. The FTP protocol collection method is an example: the log file download of the remote client is realized through the FTP protocol. However, in the actual practice process, it is found that if the server encounters a sudden and instantaneous situation where a large amount of network log data is sent at the same time, the server receives a large amount of network log data per unit time, which will cause the server's resources to be exhausted instantly. In other words, the server in the existing network log analysis process is difficult to cope with the sudden and instantaneous situation where a large amount of network log data is sent at the same time. Summary of the Invention

[0005] The purpose of the embodiments of the present application is to provide a network log analysis method, device, electronic device and storage medium, which are used to improve the problem of difficulty in coping with the situation where a large amount of network log data is sent simultaneously in an instant.

[0006] An embodiment of the present application provides a network log analysis method, comprising: acquiring network logs of network devices in real time through a collection kernel, and pushing the acquired network logs to a distributed event stream platform in real time through the collection kernel; buffering the network logs using a message queue on the distributed event stream platform to obtain buffered network logs; and analyzing the buffered network logs cached by the distributed event stream platform in real time using a distributed stream data flow engine to obtain analysis results. In the above implementation process, first, the network logs are buffered using the distributed event stream platform, and then the buffered network logs are analyzed using the distributed stream data flow engine to obtain analysis results. That is, by using a dedicated distributed event stream platform to buffer streaming data and using a dedicated distributed stream data flow engine to analyze the buffered streaming data, a sudden burst of large amounts of network log data can be effectively buffered as streaming data. Furthermore, since Flink nodes can consume and process the buffered network log data in Kafka's message queue at a constant speed and in real time, additional Flink nodes can be added according to specific circumstances to increase the real-time processing capability of the network log data, thereby effectively addressing situations where sudden bursts of large amounts of network log data are too large to handle.

[0007] Optionally, in an embodiment of the present application, the acquisition kernel is provided with a distributed acquisition instance and corresponding multiple ports, and the network log of the network device is obtained, including: receiving the network log sent by the network device through a preset protocol through different ports of the distributed acquisition instance, and one port of the distributed acquisition instance corresponds to a network device of a type of manufacturer; marking the network log with the manufacturer type according to the different ports of the distributed acquisition instance, and obtaining the marked network log with the manufacturer type mark. In the above implementation process, the network log sent by the network device through the preset protocol is received through different ports of the distributed acquisition instance, and the network log is marked according to the different ports of the distributed acquisition instance. Since the network device of a type of manufacturer transmits to one port, the above electronic device can perform unified protocol processing on the network log transmitted by the port, thereby unifying the log formats of different manufacturers, effectively improving the problem of inconsistent network log formats generated by devices of different manufacturers.

[0008] Optionally, in an embodiment of the present application, a distributed event stream platform is used to employ a message queue to buffer network log data, including: inputting the network log of a network device into a message queue in the distributed event stream platform; and using the message queue to buffer the network log of the network device. In the above implementation process, by using the message queue of a distributed event stream platform (e.g., Kafka) to buffer the network log of the network device, a sudden burst of large amounts of network log data is effectively buffered as streaming data, ensuring that server resources will not be exhausted when encountering a sudden burst of large amounts of network log data.

[0009] Optionally, in an embodiment of the present application, a distributed stream data flow engine is used to analyze the buffered network log, including: using the distributed stream data flow engine to obtain the buffered network log from the message queue; using regular expressions to match the buffered network log with alarm types to obtain multiple alarm types; respectively counting the number of times multiple alarm types appear in the buffered network log to obtain multiple alarm counts; performing alarm suppression analysis based on the multiple alarm counts to obtain alarm suppression analysis results. In the above implementation process, by using the stream processing program in the distributed stream data flow engine to consume the network log data in the message queue in real time, when encountering a sudden burst of large amounts of network log data, the network log data in the message queue can be consumed effectively and quickly, thereby greatly reducing the probability that the server's resources will be exhausted.

[0010] Optionally, in an embodiment of the present application, after obtaining the alarm suppression analysis results, the method further includes: performing implicit analysis on the alarm suppression analysis results to obtain implicit analysis results. In the above implementation process, by analyzing and predicting alarm trends from network logs before any false alarms are generated, it is possible to analyze implicit problems in the network based on different suppression thresholds, thereby greatly reducing the probability of implicit problems occurring in the server.

[0011] Optionally, in an embodiment of the present application, a distributed streaming data flow engine is used to analyze buffered network logs, including: using the distributed streaming data flow engine to obtain buffered network logs from a message queue; performing a unified field specification analysis on the buffered network logs to obtain log data with standardized field specifications. In the above implementation process, by using the distributed streaming data flow engine to obtain buffered network logs from a message queue and performing a unified field specification analysis on the buffered network logs, standardized network logs from devices of different manufacturers can be merged together to implement functions such as comprehensive queries.

[0012] Optionally, in an embodiment of the present application, after obtaining the log data with standardized field specifications, it also includes: storing the log data with standardized field specifications in a database; reading the log data from the database through an application system program, and analyzing and visually outputting the log data.

[0013] An embodiment of the present application also provides a network log analysis device, including: a network log acquisition module, which is used to acquire the network logs of network devices in real time through an acquisition kernel, and push the acquired network logs to a distributed event stream platform in real time through the acquisition kernel; a network log buffering module, which is used to use a distributed event stream platform to use a message queue to buffer the network log data and obtain the buffered network log; an analysis result acquisition module, which is used to use a distributed stream data flow engine to analyze the buffered network logs cached by the distributed event stream platform in real time and obtain analysis results.

[0014] Optionally, in an embodiment of the present application, the network log acquisition module includes: a network log receiving module, which is used to receive network logs sent by network devices through a preset protocol through different ports of a distributed collection instance, and one port of the distributed collection instance corresponds to a network device of a certain manufacturer type; a network log marking module, which is used to mark the network logs according to the different ports of the distributed collection instance to obtain the marked network logs.

[0015] Optionally, in an embodiment of the present application, the network log buffer module includes: a network log input module, which is used to input the network log of the network device into the message queue in the distributed event stream platform; and a message queue buffer module, which is used to use the message queue to buffer the network log of the network device.

[0016] Optionally, in an embodiment of the present application, the analysis result acquisition module includes: a first log buffer module, which is used to use a distributed streaming data flow engine to obtain the buffered network log from the message queue; an alarm type matching module, which is used to use a regular expression to match the alarm type of the buffered network log to obtain multiple alarm types; an alarm number statistics module, which is used to count the number of times multiple alarm types appear in the buffered network log to obtain multiple alarm numbers; an alarm suppression analysis module, which is used to perform alarm suppression analysis based on multiple alarm numbers to obtain alarm suppression analysis results.

[0017] Optionally, in an embodiment of the present application, the network log analysis device further includes: a log implicit analysis module, which performs implicit analysis on the alarm suppression analysis result to obtain an implicit analysis result.

[0018] Optionally, in an embodiment of the present application, the analysis result acquisition module includes: a second log buffer module, which is used to use a distributed stream data flow engine to obtain buffered network logs from the message queue; a field specification analysis module, which is used to perform unified field specification analysis on the buffered network logs to obtain log data with standardized field specifications.

[0019] Optionally, in an embodiment of the present application, the network log analysis device further includes: a log data storage module for storing log data with standardized field specifications in a database; a log visual output module for reading log data from the database through an application system program, and analyzing and visually outputting the log data.

[0020] An embodiment of the present application further provides an electronic device, including: a processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and when the machine-readable instructions are executed by the processor, the method described above is performed.

[0021] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method described above is executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 A flow chart of a network log analysis method provided by an embodiment of the present application is shown;

[0024] Figure 2 A schematic diagram of a processing flow of a network log provided by an embodiment of the present application is shown;

[0025] Figure 3 A schematic diagram of the structure of a network log analysis device provided by an embodiment of the present application is shown;

[0026] Figure 4 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0028] Before introducing the network log analysis method provided by the embodiment of the present application, some concepts involved in the embodiment of the present application are first introduced:

[0029] Kafka is an open source stream processing platform and a distributed event streaming platform written in Scala and Java. Kafka aims to provide a unified, high-throughput, low-latency platform for processing real-time data. The Kafka persistence layer is essentially a large-scale publish or subscription message queue based on a distributed transaction log architecture.

[0030] Flink is an open source stream processing framework, also known as Apache Flink. Its core is a distributed streaming data flow engine written in Java and Scala. Flink executes arbitrary streaming data programs in a data-parallel and pipeline manner. Flink's pipeline runtime system can execute batch and stream processing programs.

[0031] A regular expression, sometimes called a regular expression, describes a string matching pattern. Regular expressions are often used to retrieve and replace text that matches a pattern.

[0032] It should be noted that the network log analysis method provided in the embodiment of the present application can be executed by an electronic device, where the electronic device refers to a device terminal or server with the function of executing a computer program, such as a smart phone, a personal computer (PC), a tablet computer or a mobile Internet device (MID), etc.; servers include x86 servers and non-x86 servers, and non-x86 servers include mainframes, minicomputers and UNIX servers.

[0033] The following introduces application scenarios where the network log analysis method is applicable. The application scenarios here are not limited to: using the network log analysis method to collect network logs of network devices in a local area network or a cluster network, and performing alarm suppression analysis, implicit analysis, and field specification analysis on the collected network logs, etc., to improve the situation where the server has difficulty in coping with the sudden and simultaneous sending of a large amount of network log data.

[0034] See Figure 1The flowchart of the network log analysis method provided by the embodiment of the present application is shown; the main idea of ​​the network log analysis method is to buffer the streaming data by using a dedicated distributed event stream platform, and analyze the buffered streaming data by using a dedicated distributed streaming data flow engine, so as to effectively use the sudden and instantaneous large amount of network log data as streaming data buffer. Moreover, since the Flink node can consume and process the buffered network log data in the Kafka message queue at a uniform speed, the Flink node can be added according to the specific situation to increase the processing capacity of the network log data, thereby effectively dealing with the sudden and instantaneous large amount of network log data that cannot be processed.

[0035] The above-mentioned network log analysis method may include:

[0036] Step S110: obtaining network logs of network devices in real time through the collection kernel, and pushing the obtained network logs to the distributed event stream platform in real time through the collection kernel.

[0037] There are many implementations of step S110, including but not limited to the following:

[0038] The first implementation method uses an agent pre-installed on a network device to collect network logs from the network device and sends the agent to different ports of a distributed collection instance. For example, this implementation method:

[0039] Step S111: The network device collects network logs through an agent collection program, and uses a preset protocol to send the network logs to different ports of the distributed collection instance in the electronic device.

[0040] The preset protocol refers to the protocol for sending and receiving network logs. Different network device manufacturers may use different protocols to transmit network logs. Since network devices of the same manufacturer transmit to a single port, the electronic device can perform unified protocol processing on the network logs transmitted from that port, thereby achieving a unified log format for different manufacturers. Of course, it is also possible to use only one protocol to transmit network logs. In this embodiment, the Syslog protocol is uniformly used to transmit logs.

[0041] See Figure 2The diagram of the processing flow of the network log provided by the embodiment of the present application is shown; the implementation method of the above step S111 is, for example: an agent collection program is installed on the network device, so the network log of the network device can be collected through the agent collection program, and the agent collection program can first identify the manufacturer type identifier of the network device. For example: assuming that the device manufacturer identifier of the first device manufacturer is 1001, and the receiving port number corresponding to the manufacturer type identifier is 8001 queried from the configuration file or database, the network log can be sent to the 8001 port corresponding to the manufacturer type identifier. Similarly, assuming that the device manufacturer identifier of the second device manufacturer is 1002, and the receiving port number corresponding to the manufacturer type identifier is 8002 queried from the configuration file or database, the network log can be sent to the 8002 port corresponding to the manufacturer type identifier.

[0042] Step S112: The electronic device receives the network log sent by the network device through a preset protocol through different ports of the distributed collection instance.

[0043] In the implementation of step S112, for example, the preset protocol may be a syslog protocol, or may be specifically configured based on the vendor type. The electronic device receives network logs sent by the network device via the preset protocol via different ports of the distributed collection instance. Each port of the distributed collection instance corresponds to a network device of a particular vendor type. The electronic device may perform unified protocol processing on the network logs transmitted by the port, thereby achieving distributed collection and unifying log formats across different vendors.

[0044] Step S113: The electronic device marks the network log with the vendor type according to different ports of the distributed collection instance to obtain the network log marked with the vendor type.

[0045] For example, the implementation method of the above step S113 is as follows: the electronic device marks the network log received from port 8001 as 1001, and marks the network log received from port 8002 as 1002. The specific marking process is also called the labeling process, that is, different labels are added according to different device manufacturers. Specifically, a field can be added to the network log to store the label value, and then the labeled network log is stored as the network log; in the process of collecting network logs, an open source collection kernel can also be used.

[0046] In the second implementation, the vendor type of the network device is manually identified in advance, and then a receiving port to which the network log needs to be sent is configured in the network device. The receiving port corresponds to the vendor type. For example, in this implementation:

[0047] Step S114: the network device sends the network log to the preset port of the distributed collection instance in the electronic device using a preset protocol.

[0048] For example, the implementation method of the above step S114 is as follows: assuming that the device manufacturer identifier of the first device manufacturer is 1001, the network device of the first device manufacturer uses the syslog protocol to send network logs to port 8001 of the distributed collection instance in the electronic device; assuming that the device manufacturer identifier of the second device manufacturer is 1002, the network device of the second device manufacturer uses the syslog protocol to send network logs to port 8002 of the distributed collection instance in the electronic device. In a specific practice, the above network device can be an optical line terminal (OLT) device. Specifically, the OLT device can be used to collect network logs of different alarm types and send network logs of different alarm types through the syslog protocol. The above alarm types include but are not limited to: optical modem power failure alarm, optical modem power failure recovery alarm, passive optical network (PON) port fiber break alarm, PON port fiber break recovery alarm, optical modem fiber break alarm, and optical modem fiber break recovery alarm, etc.

[0049] Step S115: The electronic device receives the network log sent by the network device through the preset protocol through the preset port of the distributed collection instance.

[0050] Step S116: The electronic device marks the network log according to different ports of the distributed collection instance to obtain the marked network log.

[0051] Among them, the implementation principles and implementation methods of steps S115 to S116 are similar to those of steps S112 to S113. Therefore, their implementation principles and implementation methods are no longer explained here. If there are any unclear points, please refer to the description of steps S112 to S113.

[0052] After step S110 , step S120 is executed: using a distributed event stream platform and a message queue to buffer the network log data to obtain a buffered network log.

[0053] The implementation method of the above step S120 may include: after labeling the network log, obtaining the labeled network log, and then inputting the labeled network log into a message queue in a distributed event stream platform (such as Kafka). The network log of the network device is buffered using the message queue to obtain the buffered network log. In the above implementation process, by using the message queue of the distributed event stream platform (such as Kafka) to buffer the network log of the network device, the sudden and large amount of network log data is effectively used as streaming data buffer, which can ensure that the server resources will not be exhausted when encountering a sudden and large amount of network log data.

[0054] After step S120 , step S130 is executed: using a distributed stream data flow engine to analyze the buffered network logs cached by the distributed event stream platform in real time to obtain analysis results.

[0055] A distributed stream data flow engine refers to a distributed stream data flow engine that analyzes network logs. The distributed stream data flow engine can specifically adopt the Flink stream processing framework. The stream processing program in the Flink stream processing framework can consume network log data in Kafka in real time.

[0056] There are many implementations of step S130, including but not limited to the following:

[0057] In a first implementation, an alarm suppression analysis is performed on the network log. The specific process of the alarm suppression analysis may include:

[0058] Step S131: using the distributed stream data flow engine to obtain the buffered network log from the message queue.

[0059] An example of an implementation of the above step S131 is: obtaining the buffered network log from Kafka's message queue through a distributed stream data flow engine (for example, using the Flink stream processing framework); a specific example: after the network log is collected and buffered by Kafka's message queue at a very fast speed, the Flink stream processing framework can be used to uniformly consume and process the network log data buffered in Kafka's message queue. Consumption here refers to uniform acquisition and calculation of the following steps, thereby achieving the effect of uncoupling the storage buffering and processing consumption of the network log. Compared to the traditional method of collecting a network log and consuming and calculating a network log, the Kafka extremely fast collection buffer provided by the embodiment of the present application, coupled with the Flink uniform consumption calculation, effectively separates the data collection process and the calculation process, thereby significantly improving the processing power and speed of the network log.

[0060] Step S132: Use regular expressions to match the buffered network logs with alarm types to obtain multiple alarm types.

[0061] An implementation example of the above-mentioned step S132 is: using regular expressions through the Flink stream processing framework to match the alarm types of the buffered network logs to obtain multiple alarm types; wherein, regular expressions can be used to check whether a string contains a certain substring, replace the matching substring, or extract a substring that meets a certain condition from a string, etc.

[0062] Step S133: Counting the number of times that multiple alarm types appear in the buffered network log to obtain multiple alarm counts.

[0063] An implementation example of the above-mentioned step S133 is as follows: the number of times multiple alarm types appear in the buffered network log is counted separately through the Flink stream processing framework, that is, the number of times the alarm types appear in the buffered network log is accumulated to obtain multiple alarm counts. The alarm counts here can be used for the following alarm suppression analysis.

[0064] Step S134: performing alarm suppression analysis according to multiple alarm times to obtain an alarm suppression analysis result.

[0065] Alarm suppression means that when there are N consecutive alarm messages that need to be sent within a unit of time, only one real alarm message is sent. For example, if 1,000 alarm messages of the same type and for the same device need to be sent within one minute, then after alarm suppression analyzes this situation, only one alarm message is sent to the user.

[0066] An implementation of the above step S134 is, for example: performing alarm suppression analysis based on multiple alarm times through a Flink stream processing framework to obtain an alarm suppression analysis result.

[0067] In the above implementation process, by using the stream processing program in the Flink stream processing framework to consume the network log data in Kafka in real time, when encountering a sudden burst of large amounts of network log data, the network log data in Kafka can be consumed efficiently and quickly, thereby greatly reducing the probability of server resource exhaustion.

[0068] The second implementation method is to perform implicit analysis on the network log. That is, after obtaining the alarm suppression analysis results, implicit analysis can also be performed. The specific process of implicit analysis may include:

[0069] Step S135: Perform implicit analysis on the alarm suppression analysis result to obtain an implicit analysis result.

[0070] For example, in step S135, the alarm suppression analysis results include different suppression thresholds. Hidden network issues are then analyzed based on these different suppression thresholds. Hidden issues here refer to predicting alarm trends by analyzing network logs before any false alarms are generated. For example, an alarm trend might be the increase in the CPU usage of a network device from 30% to 80% over a period of time. In practice, indicator data can also be collected based on the alarm trends. Examples of such indicator data include the CPU usage, memory usage, and bandwidth usage of the network device.

[0071] In a third implementation, field specification analysis is performed on network logs. The specific process of field specification analysis may include:

[0072] Step S136: Use the distributed stream data flow engine to obtain the buffered network log from the message queue.

[0073] Among them, the implementation principle and implementation method of step S136 are similar to those of step S131. Therefore, its implementation principle and implementation method will not be explained here. If there are any unclear points, please refer to the description of step S131.

[0074] Step S137: performing unified field specification analysis on the buffered network log to obtain log data with standardized field specifications.

[0075] For example, step S137 can be implemented by using the Flink stream processing framework to perform unified field specification analysis (i.e., log format standardization) on the buffered network logs to obtain log data with standardized field specifications. Specifically, for example, fields such as the alarm source, alarm time, alarm device, and device manufacturer identifier can be extracted from the buffered network logs. The standardized log data can then be stored in a database (e.g., an Oracle database or a MySQL database).

[0076] Optionally, after the field specification is standardized, the log data with the field specification standardized may be analyzed and visualized. This implementation may include:

[0077] Step S138: Store the log data with standardized field specifications into the database.

[0078] An implementation example of the above-mentioned step S138 is as follows: the database can be understood as a data warehouse for storing log data, and specifically, an in-memory database, a relational database and / or a non-relational database can be used. Therefore, the log data with standardized field specifications can be stored in the in-memory database, the relational database and / or the non-relational database; among them, the in-memory databases that can be used include: Memcached and Redis, etc., the relational databases that can be used include: Mysql, PostgreSQL, Oracle and SQLSever, etc., and the non-relational databases that can be used include: Grakn database, Neo4j graph database, Hadoop subsystem HBase, MongoDB and CouchDB, etc.

[0079] It's understandable that storing log data with standardized field specifications in the database means that standardized network logs from devices from different manufacturers can be merged and stored in the database, enabling comprehensive analysis and query. If network logs from different manufacturers' devices don't have a standardized format, merging them together will be difficult, making comprehensive analysis and query impossible.

[0080] Step S139: Read the log data from the database through the application system program, analyze the log data and output it in a visual manner.

[0081] An example implementation of step S139 is to read log data from a database through an application system program and analyze hidden network issues based on different suppression thresholds. Hidden issues here refer to analyzing and predicting alarm trends from network logs before any false alarms are generated. An example of an alarm trend is an increase in the CPU usage of a network device from 30% to 80% over a period of time. The log data is then analyzed and visualized, for example, by displaying different colors on a Hypertext Markup Language (HTML) web page based on the number of alarms, effectively displaying the alarm trends and severity to the user.

[0082] In the specific practice process, the first to third implementation methods mentioned above can also be combined. After obtaining the log data with field specification standardization, alarm suppression analysis and visualization output can be performed only on the log data with field specification standardization, or alarm suppression analysis, implicit analysis and visualization output can be performed on the log data with field specification standardization.

[0083] In the above implementation process, network logs are first buffered using a distributed event stream platform, and then the buffered network logs are analyzed using a distributed streaming data flow engine to obtain analysis results. In other words, by using a dedicated distributed event stream platform to buffer streaming data and a dedicated distributed streaming data flow engine to analyze the buffered streaming data, large amounts of sudden network log data can be effectively buffered as streaming data. Furthermore, since Flink nodes can consume and process the buffered network log data in Kafka's message queue at a uniform rate, additional Flink nodes can be added to increase network log data processing capabilities based on specific circumstances, effectively addressing situations where sudden, large amounts of network log data are too much to handle.

[0084] See Figure 3 The structure diagram of the network log analysis device provided by the embodiment of the present application is shown; the embodiment of the present application provides a network log analysis device 200, including:

[0085] The network log acquisition module 210 is used to acquire the network logs of the network devices in real time through the acquisition kernel, and push the acquired network logs to the distributed event stream platform in real time through the acquisition kernel.

[0086] The network log buffer module 220 is used to use the distributed event stream platform to adopt a message queue to buffer the network log data and obtain the buffered network log.

[0087] The analysis result obtaining module 230 is used to use the distributed stream data flow engine to analyze the buffered network logs cached by the distributed event stream platform in real time to obtain analysis results.

[0088] Optionally, in an embodiment of the present application, the collection kernel is provided with a distributed collection instance and corresponding multiple ports, and the network log acquisition module includes:

[0089] The network log receiving module is used to receive network logs sent by network devices through a preset protocol through different ports of the distributed collection instance. One port of the distributed collection instance corresponds to a network device of one manufacturer and type.

[0090] The network log marking module is used to mark the network log with the vendor type according to the different ports of the distributed collection instance, and obtain the network log with the vendor type mark.

[0091] Optionally, in an embodiment of the present application, the network log buffer module includes:

[0092] The network log input module is used to input the network logs of network devices into the message queue in the distributed event stream platform.

[0093] The message queue buffer module is used to use the message queue to buffer the network log of the network device.

[0094] Optionally, in an embodiment of the present application, the analysis result obtaining module includes:

[0095] The first log buffer module is used to obtain the buffered network log from the message queue using the distributed stream data flow engine.

[0096] The alarm type matching module is used to use regular expressions to match the alarm types of the buffered network logs to obtain multiple alarm types.

[0097] The alarm count statistics module is used to count the number of times multiple alarm types appear in the buffered network log to obtain multiple alarm counts.

[0098] The alarm suppression analysis module is used to perform alarm suppression analysis based on multiple alarm times and obtain alarm suppression analysis results.

[0099] Optionally, in the embodiment of the present application, the network log analysis device further includes:

[0100] The log implicit analysis module performs implicit analysis on the alarm suppression analysis results to obtain implicit analysis results.

[0101] Optionally, in the embodiment of the present application, the analysis result obtaining module may further include:

[0102] The second log buffer module is used to obtain the buffered network log from the message queue using the distributed stream data flow engine.

[0103] The field specification analysis module is used to perform unified field specification analysis on the buffered network logs to obtain log data with standardized field specifications.

[0104] Optionally, in the embodiment of the present application, the network log analysis device further includes:

[0105] The log data storage module is used to store log data with standardized field specifications into the database.

[0106] The log visual output module is used to read log data from the database through the application system program, and analyze and visualize the log data.

[0107] It should be understood that the device corresponds to the above-mentioned network log analysis method embodiment and can perform each step involved in the above-mentioned method embodiment. The specific functions of the device can be found in the description above. To avoid repetition, a detailed description is omitted here. The device includes at least one software function module that can be stored in the memory in the form of software or firmware or fixed in the operating system (OS) of the device.

[0108] See Figure 4 The electronic device 300 provided in the embodiment of the present application includes a processor 310 and a memory 320, wherein the memory 320 stores machine-readable instructions executable by the processor 310, and when the machine-readable instructions are executed by the processor 310, the method described above is performed.

[0109] The embodiment of the present application further provides a computer-readable storage medium 330 , on which a computer program is stored. When the computer program is run by the processor 310 , the above method is executed.

[0110] Among them, the computer-readable storage medium 330 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, disk or optical disk.

[0111] In the several embodiments provided in the embodiments of the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are only schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible implementation architectures, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and a module, a program segment or a part of a code contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also be different from the order of occurrence marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which is mainly based on the functions involved.

[0112] In addition, the functional modules of each embodiment in the embodiments of the present application can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part.

[0113] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any actual relationship or order between these entities or operations.

[0114] The above description is only an optional implementation method of the embodiment of the present application, but the protection scope of the embodiment of the present application is not limited to this. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in the embodiment of the present application, and they should all be covered by the protection scope of the embodiment of the present application.

Claims

1. A network log analysis method, characterized in that: include: Acquire network logs of network devices in real time through a collection kernel, and push the acquired network logs to a distributed event stream platform in real time through the collection kernel, wherein the distributed event stream platform adopts Kafka; Using the distributed event stream platform and the Kafka message queue to perform data buffering on the network log, obtaining a buffered network log, wherein the Kafka message queue is used to use the network log as a streaming data buffer; Analyze the buffered network logs cached by the distributed event stream platform in real time using a distributed stream data flow engine to obtain analysis results, wherein the distributed stream data flow engine adopts a Flink stream processing framework, and a stream processing program in the Flink stream processing framework is used to consume and analyze the network logs in Kafka; Among them, the use of a distributed streaming data flow engine to analyze the buffered network log cached by the distributed event stream platform in real time includes: using the distributed streaming data flow engine to obtain the buffered network log from the message queue; using a regular expression to match the buffered network log with an alarm type to obtain multiple alarm types; respectively counting the number of times the multiple alarm types appear in the buffered network log to obtain multiple alarm counts; performing alarm suppression analysis based on the multiple alarm counts to obtain alarm suppression analysis results; performing implicit analysis on the alarm suppression analysis results to obtain implicit analysis results.

2. The method according to claim 1, characterized in that The collection kernel is provided with a distributed collection instance and corresponding multiple ports, and the network log of the network device is obtained, including: Receiving the network log sent by the network device through a preset protocol through different ports of the distributed collection instance, wherein one port of the distributed collection instance corresponds to a network device of one manufacturer type; The network log is marked with a vendor type according to different ports of the distributed collection instance to obtain a network log of the network device marked with the vendor type.

3. The method according to claim 2, characterized in that The method of using a distributed event stream platform to buffer data of the network log using a message queue includes: Inputting the network log of the network device into the message queue in the distributed event stream platform; The message queue is used to perform data buffering on the network log of the network device.

4. The method according to claim 3, characterized in that The using a distributed stream data flow engine to analyze the buffered network log includes: Obtaining the buffered network log from the message queue using the distributed streaming data flow engine; The buffered network logs are subjected to a unified field specification analysis to obtain log data with standardized field specifications.

5. The method according to claim 4, characterized in that After obtaining the log data with standardized field specifications, the method further includes: The log data with standardized field specifications is stored in a database; The log data is read from the database through an application system program, and the log data is analyzed and output visually.

6. A network log analysis device, characterized in that: include: A network log acquisition module is used to acquire network logs of network devices in real time through an acquisition kernel, and push the acquired network logs to a distributed event stream platform in real time through the acquisition kernel, wherein the distributed event stream platform adopts Kafka; A network log buffering module, used to use the distributed event stream platform to use the Kafka message queue to perform data buffering on the network log to obtain the buffered network log, wherein the Kafka message queue is used to use the network log as streaming data buffer; An analysis result acquisition module is used to use a distributed stream data flow engine to analyze the buffered network logs cached by the distributed event stream platform in real time to obtain analysis results, wherein the distributed stream data flow engine adopts a Flink stream processing framework, and a stream processing program in the Flink stream processing framework is used to consume and analyze the network logs in Kafka; Among them, the use of a distributed streaming data flow engine to analyze the buffered network log cached by the distributed event stream platform in real time includes: using the distributed streaming data flow engine to obtain the buffered network log from the message queue; using a regular expression to match the buffered network log with an alarm type to obtain multiple alarm types; respectively counting the number of times the multiple alarm types appear in the buffered network log to obtain multiple alarm counts; performing alarm suppression analysis based on the multiple alarm counts to obtain alarm suppression analysis results; performing implicit analysis on the alarm suppression analysis results to obtain implicit analysis results.

7. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and when the machine-readable instructions are executed by the processor, the method according to any one of claims 1 to 5 is performed.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is executed.

Images