Intelligent management method, system and equipment for network and log and readable storage medium

Abstract

The invention discloses a network and log intelligent management method. The method comprises the following steps: deploying a network monitoring tool and a log analysis tool in a target network; defining a network monitoring index, monitoring a target network in real time through a network monitoring tool, and obtaining log data through a log monitoring tool; carrying out fusion analysis on the network monitoring information and the log data, and judging whether the target network is abnormal or not according to an analysis result; and defining an alarm mechanism, and giving an alarm when the target network is abnormal. Abnormalities in a system and a network can be quickly found through monitoring, the processing time is shortened, the availability and stability of the system and the network are improved, and configuration optimization can be carried out on network resources through monitoring; potential problems can be identified and prevented through data analysis; the network monitoring information and the log data are fused and analyzed, so that the anomaly detection capability can be further enhanced; and by establishing a sound monitoring and emergency alarm mechanism, the continuity and stability of the service are ensured.

Description

Technical Field

[0001] The present invention relates to the field of network security, and in particular to a network and log intelligent management method, system, device and readable storage medium. Background Art

[0002] With the rapid development of information technology, the scale of the network continues to expand at an unprecedented speed. From local area networks to wide area networks, and then to the Internet covering the world, each node is closely connected, forming a huge and complex digital ecosystem. At the same time, the amount of data is also showing an explosive growth trend, from social media information and shopping records generated by individuals in daily life, to transaction data and customer information in corporate operations, to scientific research data and government information at the national level, etc. The large amount and variety of these data far exceed the scope of traditional data processing and storage capabilities.

[0003] However, the expansion of network scale and the surge in data volume have also brought unprecedented data security challenges. First, more devices and systems connected to the network means that the potential attack surface has become wider, and hackers and lawless elements have more opportunities to take advantage of. They may use system vulnerabilities, malware, phishing attacks and other means to illegally obtain, tamper with or destroy sensitive data, posing a serious threat to personal privacy, corporate interests and even network security.

[0004] Secondly, the increase in data volume makes data management and protection more complex and difficult. How to accurately identify valuable information in massive data and effectively protect it while ensuring the integrity, availability and confidentiality of the data has become a difficult problem that needs to be solved urgently. In addition, during the transmission, storage and processing of data, it may also suffer loss or leakage due to technical failures, human errors or natural disasters, further exacerbating the risk of data security.

[0005] Enterprise network disaster refers to the serious impact on enterprise information systems and network infrastructure due to natural, technological or human factors. In addition to natural disasters and human factors, its causes also include technical failures, network attacks, etc. Network disasters will affect business continuity and data security, resulting in business terminals, data loss and even damage to reputation. Disasters will directly affect the operation, economic benefits and reputation of enterprises. Therefore, early warning of enterprise network disasters is crucial to the normal operation of enterprise networks. The existing technology for dealing with network enterprise disasters is to monitor network anomalies. However, in reality, enterprise network disasters are not enough to rely solely on the implementation of network monitoring. There is a lack of technical means to prevent them before they happen. Therefore, there is an urgent need for a method that can detect anomalies in a timely manner and predict anomalies in advance through other data. Summary of the invention

[0006] In view of the current problem of being unable to provide early warning of enterprise network disasters, the present invention provides a network and log intelligent management method, which integrates real-time network monitoring and log data for analysis to provide an alarm when a problem occurs, or provides an alarm through log analysis, and integrates real-time monitoring and log analysis to achieve early warning.

[0007] To achieve the above object, the embodiments of the present invention adopt the following technical solutions:

[0008] A network and log intelligent management method, the network and log intelligent management method comprising the following steps:

[0009] Deploy network monitoring tools and log monitoring tools on the target network;

[0010] Define network monitoring indicators, monitor the target network in real time through network monitoring tools, and obtain log data through log monitoring tools;

[0011] The network monitoring information is integrated with the log data for analysis, and the target network is judged to see whether there are any abnormalities based on the analysis results;

[0012] Define an alarm mechanism to generate an alarm when an abnormality occurs in the target network.

[0013] According to one aspect of the present invention, the network monitoring tool is used to monitor the traffic of the target network and the status and performance of the devices in the target network in real time.

[0014] According to one aspect of the present invention, the log monitoring tool is used to collect and analyze the generated logs from the systems, applications and devices of the target network in real time.

[0015] According to one aspect of the present invention, the network monitoring indicators include any one or more of CPU usage, memory usage, and bandwidth usage.

[0016] According to one aspect of the present invention, the log data includes logs of the target network's system, target network-related applications, and target network-related devices; the log monitoring tool includes a log collection tool, a log storage tool, a log analysis tool, and a report generation tool.

[0017] According to one aspect of the present invention, the fusing and analyzing the network monitoring information with the log data and judging whether the target network has an abnormality according to the analysis result includes:

[0018] Obtain network monitoring information and log data for fusion analysis, and pre-process the network monitoring information and log data;

[0019] Extracting features of the network monitoring information and the log data;

[0020] Find relevant features from the extracted features and associate network monitoring information with log data;

[0021] Analyze the associated network monitoring information and log data, and determine whether there are any abnormalities in the target network based on the analysis results.

[0022] According to one aspect of the present invention, the analyzing of the associated network monitoring information and log data further includes: generating a comprehensive network monitoring view based on the network monitoring information and its associated log data.

[0023] According to one aspect of the present invention, the method for defining an alarm mechanism is: obtaining a network monitoring indicator, setting an alarm threshold according to the network monitoring indicator, and sending an alarm notification when the network monitoring indicator exceeds the threshold.

[0024] According to one aspect of the present invention, the method for defining an alarm mechanism also includes: obtaining log data, setting a target state, and sending an alarm notification when the state of any one of the target network's system, the target network-related application, and the target network-related device becomes the target state.

[0025] According to one aspect of the present invention, the network and log intelligent management method further includes: generating a network and log analysis report according to the analysis result and the network anomaly judgment result.

[0026] A network and log intelligent management system, comprising:

[0027] Monitoring management module, used to deploy monitoring tools and log monitoring tools on the target network;

[0028] The monitoring and collection module defines network monitoring indicators, which are used to monitor the target network in real time through network monitoring tools and obtain log data through log monitoring tools;

[0029] The data processing module is used to integrate and analyze the network monitoring information and log data, and determine whether there is an abnormality in the target network based on the analysis results;

[0030] The abnormal alarm module defines an alarm mechanism for generating an alarm when an abnormality occurs in the target network.

[0031] A network and log intelligent management device, comprising:

[0032] Memory for storing computer programs;

[0033] The processor is used to implement the steps of the network and log intelligent management method as described above when executing the computer program.

[0034] A readable storage medium for intelligent management of a network and logs, wherein a computer program is stored on the readable storage medium, and the computer program is used to execute the steps of the method for intelligent management of a network and logs as described above.

[0035] Advantages of the implementation of the present invention: The network and log intelligent management method described in the present invention includes the following steps: deploying network monitoring tools and log analysis tools in the target network; defining network monitoring indicators, monitoring the target network in real time through network monitoring tools, and obtaining log data through log monitoring tools; fusing and analyzing network monitoring information with log data, and judging whether the target network has abnormalities based on the analysis results; defining an alarm mechanism, and alarming when abnormalities exist in the target network. By implementing monitoring, abnormalities in the system and network can be quickly discovered, processing time can be reduced, and the availability and stability of the system and network can be improved. Network resources can also be configured and optimized through monitoring; potential problems can be identified and prevented through data analysis; fusing and analyzing network monitoring information with log data can further enhance abnormality detection capabilities; and by establishing a sound monitoring and emergency alarm mechanism, business continuity and stability can be guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 This is a flow chart of the network and log intelligent management method according to the first embodiment of the present invention;

[0038] Figure 2 This is a flow chart of the network and log intelligent management method according to the second embodiment of the present invention;

[0039] Figure 3 This is a structural diagram of the network and log intelligent management system described in Embodiment 3 of the present invention;

[0040] Figure 4 This is a structural diagram of the network and log intelligent management device described in Example 4 of the present invention. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] Embodiment 1

[0043] like Figure 1 As shown, a network and log intelligent management method includes the following steps:

[0044] Step S1: deploy network monitoring tools and log monitoring tools on the target network;

[0045] Network monitoring tools are used to monitor the traffic of the target network, the status and performance of the devices in the target network in real time; by monitoring the real-time network traffic, it is possible to promptly detect traffic anomalies and identify potential security threats; monitor the performance of devices in the network such as routers, switches, firewalls, etc.

[0046] The network monitoring tool can use Nagios, which is an open source network monitoring tool that can monitor network devices, services, and applications. You can also consider Zabbix, which is also an open source monitoring solution and supports multiple monitoring types and complex alarm mechanisms; or PRTG Network Monitor, which integrates multiple network monitoring functions and supports real-time traffic analysis and alarm notifications.

[0047] Using network monitoring tools to monitor the network requires configuring network devices. By configuring the Simple Network Management Protocol (SNMP) of network devices, the monitoring tool can obtain device status information. After the network device configuration is completed, the network monitoring tool is deployed on the target network. Specifically, the network monitoring tool is installed and configured on the server of the target network.

[0048] Deploying network equipment includes the following steps:

[0049] Enable Simple Network Management Protocol (SNMP); log in to the management interface of the network device (such as router, switch); find the SNMP setting option and enable the SNMP service; configure the SNMP community string (such as public, private) and set permissions; set the IP address of the SNMP Trap receiver to receive device alarm information.

[0050] Configure the device management information base (MIB); confirm the MIB files supported by the device, which can be obtained from the device manufacturer; import the device MIB files in the monitoring tool; determine the object identifiers (OIDs) that need to be monitored, such as CPU usage, memory usage, interface status, etc.

[0051] The SNMP protocol uses a simple request-response model, where the manager sends a request to the agent and the agent returns a response. SNMP v1 / v2 uses community strings as a simple authentication mechanism, and SNMP v3 supports more advanced security features (such as encryption and authentication).

[0052] MIB is a tree structure that contains a series of management objects, each of which has a unique OID. SNMP accesses and modifies management objects through OIDs.

[0053] To deploy a network monitoring tool, first select a suitable monitoring tool from Nagios, Zabbix, PRTG Network Monitor, etc., download and install it; configure the server environment of the monitoring tool, such as the operating system, network connection, etc.; configure network device information and monitoring parameters in the tool's management interface; add network devices to the monitoring tool, configure the device IP address and SNMP community string; define the parameters that need to be monitored, such as CPU usage, bandwidth usage, interface status, etc.; finally, set the data collection frequency, such as once a minute, and collect data from the device regularly through a sampling algorithm.

[0054] In addition, use a relational database or time series database to store monitoring data and perform necessary preprocessing, and use time series data storage and analysis methods to store and analyze data.

[0055] Log monitoring tools are used to collect and analyze logs generated from systems, applications, and devices on the target network in real time.

[0056] Log monitoring tools include log collection tools, log storage tools, log analysis tools, and report generation tools;

[0057] Among them, the log collection tool uses Logstash or other log collection tools to collect logs of the target network's system, target network-related applications, and target network-related devices; the log storage tool uses Elasticsearch to support efficient log search and query; the log analysis tool uses ELK Stack (Elasticsearch, Logstash, Kibana), as an open source log management and analysis tool, which can support real-time processing and visualization of large-scale log data; Splunk, a commercial log management and analysis platform, provides powerful log search, analysis, and visualization functions; Graylog, an open source log management platform, supports real-time log analysis and alarms; the report generation tool is used to regularly generate log analysis reports to help with system auditing and troubleshooting.

[0058] Methods for deploying log collection tools: deploy log collection tools on each log source system, configure log collection tools to collect designated log files or log streams; set the target address of the log collection tool and ensure the security of log transmission; configure filter plug-ins in the log collection tool, format the log data, and then use Grok or regular expressions to parse the original logs and convert them into a structured format; configure the log transmission pipeline to transfer the logs from the collection agent to the centralized storage platform, and configure the filters and converters in the pipeline to ensure the consistency and integrity of the log format.

[0059] Log transmission uses reliable transport protocols such as TCP, UDP, and HTTP to ensure secure transmission of log data; TLS encryption can also be enabled to further ensure the security of log transmission. Efficient log data stream processing and transmission can be achieved through data stream processing algorithms, such as using stream processing frameworks (such as Apache Kafka).

[0060] Log storage defines the structure and mapping of the index by configuring the index template in Elasticsearch; sets the index sharding and copy strategy to ensure high availability and performance; further configures the log lifecycle management, such as data expiration strategy and archiving strategy; and also needs to regularly clean up old logs to free up storage space.

[0061] Among them, the inverted index algorithm is used to accelerate log search and query in Elasticsearch, and the storage space requirements are reduced through data compression and storage algorithms such as LZ4.

[0062] Step S2: define network monitoring indicators, monitor the target network in real time through network monitoring tools, and obtain log data through log monitoring tools;

[0063] Network monitoring indicators include any one or more of CPU usage, memory usage, bandwidth usage, and interface status. These monitoring indicators are defined for each device in the monitoring tool. CPU usage indicates the CPU usage of the monitored device, and memory usage indicates the memory usage of the monitored device; network bandwidth usage indicates the inbound and outbound bandwidth usage of the monitored interface; and interface status indicates the link status and error rate of the monitored interface.

[0064] Network monitoring indicators are based on performance monitoring algorithms and trend analysis algorithms. The performance monitoring algorithm uses statistical analysis algorithms to calculate average values, maximum values, minimum values, etc. to evaluate device performance; the trend analysis algorithm uses moving average, exponential smoothing and other algorithms to analyze data trends and predict future performance.

[0065] The log data includes the logs of the target network's system, target network-related applications, and target network-related devices. The log format and content of the log data also need to be confirmed, such as system logs, application logs, security logs, etc.

[0066] Step S3: merging and analyzing the network monitoring information and log data, and judging whether there is any abnormality in the target network according to the analysis results;

[0067] Step S31: obtaining network monitoring information and log data for fusion analysis, and preprocessing the network monitoring information and log data;

[0068] To query log information, you can use Kibana or similar tools to configure a log dashboard to display log data in real time; by configuring common queries and filter conditions, you can quickly locate problems. Use the full-text search function of Elasticsearch and Lucene to provide efficient log search capabilities; use the aggregation function of Elasticsearch to count and analyze log data.

[0069] Obtain network monitoring information and log data for fusion analysis, and preprocess the network monitoring and log data. The preprocessing includes cleaning, denoising, formatting and other operations to improve data quality and make it meet the requirements of the fusion algorithm.

[0070] Step S32: extracting features of the network monitoring information and the log data;

[0071] Log data can be parsed using regular expressions or Grok patterns to extract key information.

[0072] Step S33: Find the associated features from the extracted features and associate the network monitoring information with the log data;

[0073] Network monitoring information includes network information obtained from devices in the monitoring network, such as routers, switches, firewalls, etc. At the same time, the data sources of log data include the system to which the target network belongs, the target network-related applications, and the target network-related devices. Therefore, data association can be performed with network monitoring information through the data source of log data.

[0074] Step S34: analyzing the associated network monitoring information and log data, and judging whether there is any abnormality in the target network according to the analysis result;

[0075] Network monitoring focuses on network traffic and device status, while log monitoring focuses on the operating status of systems and applications and the event logs they generate. The combination of the two can provide a comprehensive monitoring view. By analyzing the CPU usage, memory usage, bandwidth usage, etc. in the network monitoring information, network analysis results are obtained. On this basis, log data associated with the network monitoring information is found, and the log data is analyzed to obtain log analysis results. The network analysis results and log analysis results are combined for analysis to monitor anomalies in network monitoring information and log data.

[0076] Traffic analysis algorithms and anomaly detection algorithms can be used to determine whether there are anomalies in network monitoring information; traffic analysis algorithms can use NetFlow or sFlow protocols to collect network traffic data, analyze traffic patterns, and identify abnormal traffic; anomaly detection algorithms can use machine learning-based anomaly detection algorithms, such as K-means clustering, isolation forest, etc., to detect abnormal behavior in network traffic.

[0077] Log parsing algorithms, association analysis algorithms, and anomaly detection algorithms can be used to determine whether there are anomalies in log data. Log parsing algorithms can use regular expressions or GroK patterns to parse log formats and extract key information. Association analysis algorithms use association rule algorithms, such as Apriori or FP-Growth, to analyze association patterns in log data and identify complex security incidents. Anomaly detection algorithms use time series analysis algorithms, such as autoregressive integrated moving average models (ARIMA) or long short-term memory networks (LSTM), to detect abnormal patterns in log data.

[0078] You can also perform correlation analysis on different logs, configure log correlation rules, and detect related events across systems and applications; use correlation analysis algorithms to identify complex security events and abnormal behaviors. The correlation rule algorithm uses Apriori or FP-Growth algorithms to analyze correlation patterns in log data.

[0079] Step S35: Generate a comprehensive network monitoring view based on the network monitoring information and its associated log data.

[0080] The comprehensive network monitoring view uses specific technologies and tools to conduct comprehensive, real-time monitoring and display of network traffic, device status, security events, etc., to help network administrators quickly discover and solve problems in the network and ensure the stable operation and security of the network. In this embodiment, a comprehensive network monitoring view is generated by obtaining network monitoring information of various dimensions and its associated log data.

[0081] Step S4: define an alarm mechanism to generate an alarm when an abnormality occurs in the target network;

[0082] Obtain network monitoring indicators, set alarm thresholds based on the network monitoring indicators, and send alarm notifications when the network monitoring indicators exceed the thresholds;

[0083] Get log data, set the target state, and send an alarm notification when the state of any of the target network's system, target network-related applications, and target network-related devices changes to the target state; or send an alarm when specific error logs, failed attempts, etc. appear in the log data. Log alarms can use rule engines, such as Elasticsearch's Watcher, to trigger alarms based on preset rules, and use Bayesian networks to evaluate the relevance and severity of alarms to reduce false alarms.

[0084] In actual applications, the thresholds of network monitoring indicators need to be set reasonably based on historical data and equipment specifications. The thresholds are further divided into multiple levels according to the degree of impact, and corresponding policies are set. The threshold levels can be warning, serious, fatal, etc.

[0085] Alarm notifications can be triggered based on preset rules through rule engines such as Drools, and the relevance and severity of alarms can be evaluated through Bayesian networks to reduce false alarms.

[0086] In actual applications, unified alarms can be achieved through a shared alarm mechanism for network monitoring and log monitoring to ensure that any abnormal events can be discovered and handled in a timely manner.

[0087] The alarm also needs to configure the receiver of the alarm, such as the administrator's email, SMS, phone, etc., and set the alarm notification method and frequency to ensure that the alarm information can be responded to in a timely manner. The notification uses a notification algorithm to optimize the alarm notification strategy based on priority and time window to reduce duplicate alarms.

[0088] It is also possible to use log analysis algorithms and log data to analyze the causes and impacts of alarm triggering and provide detailed alarm reports. By integrating network monitoring data and log data, it is possible to extract features of common network anomalies and issue early warnings when similar abnormalities appear in network monitoring information or log data.

[0089] This embodiment provides a network and log intelligent management method, including the following steps: deploying network monitoring tools and log analysis tools in the target network; defining network monitoring indicators, monitoring the target network in real time through network monitoring tools, and obtaining log data through log monitoring tools; fusing and analyzing network monitoring information with log data, and judging whether the target network has abnormalities based on the analysis results; defining an alarm mechanism, and alarming when abnormalities exist in the target network. By implementing monitoring, abnormalities in the system and network can be quickly discovered, processing time can be reduced, and the availability and stability of the system and network can be improved. Network resources can also be configured and optimized through monitoring; potential problems can be identified and prevented through data analysis; fusing and analyzing network monitoring information with log data can further enhance abnormality detection capabilities; and by establishing a sound monitoring and emergency alarm mechanism, business continuity and stability can be guaranteed.

[0090] Embodiment 2

[0091] like Figure 2 As shown, a network and log intelligent management method includes the following steps:

[0092] Step S1: deploy network monitoring tools and log monitoring tools on the target network;

[0093] Network monitoring tools are used to monitor the traffic of the target network, the status and performance of the devices in the target network in real time; by monitoring the real-time network traffic, it is possible to promptly detect traffic anomalies and identify potential security threats; monitor the performance of devices in the network such as routers, switches, firewalls, etc.

[0094] The network monitoring tool can use Nagios, which is an open source network monitoring tool that can monitor network devices, services, and applications. You can also consider Zabbix, which is also an open source monitoring solution and supports multiple monitoring types and complex alarm mechanisms; or PRTG Network Monitor, which integrates multiple network monitoring functions and supports real-time traffic analysis and alarm notifications.

[0095] Using network monitoring tools to monitor the network requires configuring network devices. By configuring the Simple Network Management Protocol (SNMP) of network devices, the monitoring tool can obtain device status information. After the network device configuration is completed, the network monitoring tool is deployed on the target network. Specifically, the network monitoring tool is installed and configured on the server of the target network.

[0096] Deploying network equipment includes the following steps:

[0097] Enable Simple Network Management Protocol (SNMP); log in to the management interface of the network device (such as router, switch); find the SNMP setting option and enable the SNMP service; configure the SNMP community string (such as public, private) and set permissions; set the IP address of the SNMP Trap receiver to receive device alarm information.

[0098] Configure the device management information base (MIB); confirm the MIB files supported by the device, which can be obtained from the device manufacturer; import the device MIB files in the monitoring tool; determine the object identifiers (OIDs) that need to be monitored, such as CPU usage, memory usage, interface status, etc.

[0099] The SNMP protocol uses a simple request-response model, where the manager sends a request to the agent and the agent returns a response. SNMP v1 / v2 uses community strings as a simple authentication mechanism, and SNMP v3 supports more advanced security features (such as encryption and authentication).

[0100] MIB is a tree structure that contains a series of management objects, each of which has a unique OID. SNMP accesses and modifies management objects through OIDs.

[0101] To deploy a network monitoring tool, first select a suitable monitoring tool from Nagios, Zabbix, PRTG Network Monitor, etc., download and install it; configure the server environment of the monitoring tool, such as the operating system, network connection, etc.; configure network device information and monitoring parameters in the tool's management interface; add network devices to the monitoring tool, configure the device IP address and SNMP community string; define the parameters that need to be monitored, such as CPU usage, bandwidth usage, interface status, etc.; finally, set the data collection frequency, such as once a minute, and collect data from the device regularly through a sampling algorithm.

[0102] In addition, use a relational database or time series database to store monitoring data and perform necessary preprocessing, and use time series data storage and analysis methods to store and analyze data.

[0103] Log monitoring tools are used to collect and analyze logs generated from systems, applications, and devices on the target network in real time.

[0104] Log monitoring tools include log collection tools, log storage tools, log analysis tools, and report generation tools;

[0105] Among them, the log collection tool uses Logstash or other log collection tools to collect logs of the target network's system, target network-related applications, and target network-related devices; the log storage tool uses Elasticsearch to support efficient log search and query; the log analysis tool uses ELK Stack (Elasticsearch, Logstash, Kibana), as an open source log management and analysis tool, which can support real-time processing and visualization of large-scale log data; Splunk, a commercial log management and analysis platform, provides powerful log search, analysis, and visualization functions; Graylog, an open source log management platform, supports real-time log analysis and alarms; the report generation tool is used to regularly generate log analysis reports to help with system auditing and troubleshooting.

[0106] Methods for deploying log collection tools: deploy log collection tools on each log source system, configure log collection tools to collect designated log files or log streams; set the target address of the log collection tool and ensure the security of log transmission; configure filter plug-ins in the log collection tool, format the log data, and then use Grok or regular expressions to parse the original logs and convert them into a structured format; configure the log transmission pipeline to transfer the logs from the collection agent to the centralized storage platform, and configure the filters and converters in the pipeline to ensure the consistency and integrity of the log format.

[0107] Log transmission uses reliable transport protocols such as TCP, UDP, and HTTP to ensure secure transmission of log data; TLS encryption can also be enabled to further ensure the security of log transmission. Efficient log data stream processing and transmission can be achieved through data stream processing algorithms, such as using stream processing frameworks (such as Apache Kafka).

[0108] Log storage defines the structure and mapping of the index by configuring the index template in Elasticsearch; sets the index sharding and copy strategy to ensure high availability and performance; further configures the log lifecycle management, such as data expiration strategy and archiving strategy; and also needs to regularly clean up old logs to free up storage space.

[0109] Among them, the inverted index algorithm is used to accelerate log search and query in Elasticsearch, and the storage space requirements are reduced through data compression and storage algorithms such as LZ4.

[0110] Step S2: define network monitoring indicators, monitor the target network in real time through network monitoring tools, and obtain log data through log monitoring tools;

[0111] Network monitoring indicators include any one or more of CPU usage, memory usage, bandwidth usage, and interface status. These monitoring indicators are defined for each device in the monitoring tool. CPU usage indicates the CPU usage of the monitored device, and memory usage indicates the memory usage of the monitored device; network bandwidth usage indicates the inbound and outbound bandwidth usage of the monitored interface; and interface status indicates the link status and error rate of the monitored interface.

[0112] Network monitoring indicators are based on performance monitoring algorithms and trend analysis algorithms. The performance monitoring algorithm uses statistical analysis algorithms to calculate average values, maximum values, minimum values, etc. to evaluate device performance; the trend analysis algorithm uses moving average, exponential smoothing and other algorithms to analyze data trends and predict future performance.

[0113] The log data includes the logs of the target network's system, target network-related applications, and target network-related devices. The log format and content of the log data also need to be confirmed, such as system logs, application logs, security logs, etc.

[0114] Step S3: merging and analyzing the network monitoring information and log data, and judging whether there is any abnormality in the target network according to the analysis results;

[0115] Step S31: obtaining network monitoring information and log data for fusion analysis, and preprocessing the network monitoring information and log data;

[0116] To query log information, you can use Kibana or similar tools to configure a log dashboard to display log data in real time; by configuring common queries and filter conditions, you can quickly locate problems. Use the full-text search function of Elasticsearch and Lucene to provide efficient log search capabilities; use the aggregation function of Elasticsearch to count and analyze log data.

[0117] Obtain network monitoring information and log data for fusion analysis, and preprocess the network monitoring and log data. The preprocessing includes cleaning, denoising, formatting and other operations to improve data quality and make it meet the requirements of the fusion algorithm.

[0118] Step S32: extracting features of the network monitoring information and the log data;

[0119] Log data can be parsed using regular expressions or Grok patterns to extract key information.

[0120] Step S33: Find the associated features from the extracted features and associate the network monitoring information with the log data;

[0121] Network monitoring information includes network information obtained from devices in the monitoring network, such as routers, switches, firewalls, etc. At the same time, the data sources of log data include the system to which the target network belongs, the target network-related applications, and the target network-related devices. Therefore, data association can be performed with network monitoring information through the data source of log data.

[0122] Step S34: analyzing the associated network monitoring information and log data, and judging whether there is any abnormality in the target network according to the analysis result;

[0123] Network monitoring focuses on network traffic and device status, while log monitoring focuses on the operating status of systems and applications and the event logs they generate. The combination of the two can provide a comprehensive monitoring view. By analyzing the CPU usage, memory usage, bandwidth usage, etc. in the network monitoring information, network analysis results are obtained. On this basis, log data associated with the network monitoring information is found, and the log data is analyzed to obtain log analysis results. The network analysis results and log analysis results are combined for analysis to monitor anomalies in network monitoring information and log data.

[0124] Traffic analysis algorithms and anomaly detection algorithms can be used to determine whether there are anomalies in network monitoring information; traffic analysis algorithms can use NetFlow or sFlow protocols to collect network traffic data, analyze traffic patterns, and identify abnormal traffic; anomaly detection algorithms can use machine learning-based anomaly detection algorithms, such as K-means clustering, isolation forest, etc., to detect abnormal behavior in network traffic.

[0125] Log parsing algorithms, association analysis algorithms, and anomaly detection algorithms can be used to determine whether there are anomalies in log data. Log parsing algorithms can use regular expressions or GroK patterns to parse log formats and extract key information. Association analysis algorithms use association rule algorithms, such as Apriori or FP-Growth, to analyze association patterns in log data and identify complex security incidents. Anomaly detection algorithms use time series analysis algorithms, such as autoregressive integrated moving average models (ARIMA) or long short-term memory networks (LSTM), to detect abnormal patterns in log data.

[0126] You can also perform correlation analysis on different logs, configure log correlation rules, and detect related events across systems and applications; use correlation analysis algorithms to identify complex security events and abnormal behaviors. The correlation rule algorithm uses Apriori or FP-Growth algorithms to analyze correlation patterns in log data.

[0127] Step S35: Generate a comprehensive network monitoring view based on the network monitoring information and its associated log data.

[0128] The comprehensive network monitoring view uses specific technologies and tools to conduct comprehensive, real-time monitoring and display of network traffic, device status, security events, etc., to help network administrators quickly discover and solve problems in the network and ensure the stable operation and security of the network. In this embodiment, a comprehensive network monitoring view is generated by obtaining network monitoring information of various dimensions and its associated log data.

[0129] Step S4: define an alarm mechanism to generate an alarm when an abnormality occurs in the target network;

[0130] Obtain network monitoring indicators, set alarm thresholds based on the network monitoring indicators, and send alarm notifications when the network monitoring indicators exceed the thresholds;

[0131] Get log data, set the target state, and send an alarm notification when the state of any of the target network's system, target network-related applications, and target network-related devices changes to the target state; or send an alarm when specific error logs, failed attempts, etc. appear in the log data. Log alarms can use rule engines, such as Elasticsearch's Watcher, to trigger alarms based on preset rules, and use Bayesian networks to evaluate the relevance and severity of alarms to reduce false alarms.

[0132] In actual applications, the thresholds of network monitoring indicators need to be set reasonably based on historical data and equipment specifications. The thresholds are further divided into multiple levels according to the degree of impact, and corresponding policies are set. The threshold levels can be warning, serious, fatal, etc.

[0133] Alarm notifications can be triggered based on preset rules through rule engines such as Drools, and the relevance and severity of alarms can be evaluated through Bayesian networks to reduce false alarms.

[0134] In actual applications, unified alarms can be achieved through a shared alarm mechanism for network monitoring and log monitoring to ensure that any abnormal events can be discovered and handled in a timely manner.

[0135] The alarm also needs to configure the receiver of the alarm, such as the administrator's email, SMS, phone, etc., and set the alarm notification method and frequency to ensure that the alarm information can be responded to in a timely manner. The notification uses a notification algorithm to optimize the alarm notification strategy based on priority and time window to reduce duplicate alarms.

[0136] It is also possible to use log analysis algorithms and log data to analyze the causes and impacts of alarm triggering and provide detailed alarm reports. By integrating network monitoring data and log data, it is possible to extract features of common network anomalies and issue early warnings when similar abnormalities appear in network monitoring information or log data.

[0137] Step S5: Generate a network and log analysis report based on the analysis results and the network anomaly judgment results.

[0138] Configure the task of periodically generating network and log analysis reports, which can be generated daily, weekly, or monthly. The reports need to include key indicators, abnormal events, and trend analysis.

[0139] The report generation algorithm uses data aggregation and statistical analysis algorithms to generate detailed log analysis reports, and uses visualization algorithms to display log data using charts or graphs to provide intuitive analysis results. Through the generation of the above network and log analysis reports, enterprises can effectively implement network and log monitoring, ensure the stable operation of systems and applications, and quickly respond to various emergencies.

[0140] Based on the first embodiment, this embodiment configures the tasks of network and log analysis reports, and uses report generation algorithms and visualization algorithms to generate network and log analysis reports, so as to help enterprises effectively implement network and log monitoring, obtain experience in anomaly detection and processing through reports, improve anomaly detection and processing capabilities, and ensure the stable operation of networks, systems and applications.

[0141] Embodiment 3

[0142] like Figure 3 As shown, a network and log intelligent management system includes:

[0143] Monitoring and management module, used to deploy monitoring tools and log analysis tools on the target network;

[0144] The monitoring and collection module defines network monitoring indicators, which are used to monitor the target network in real time through network monitoring tools and obtain log data through log monitoring tools;

[0145] The data processing module is used to integrate and analyze the network monitoring information and log data, and determine whether there is an abnormality in the target network based on the analysis results;

[0146] The abnormal alarm module defines an alarm mechanism for generating an alarm when an abnormality occurs in the target network.

[0147] In addition, the network and log intelligent management system also includes: a report generation module, which is used to generate a network and log analysis report based on the analysis results and network anomaly judgment results.

[0148] Embodiment 4

[0149] like Figure 4 As shown, a network and log intelligent management device includes:

[0150] A memory 100, used for storing computer programs;

[0151] The processor 200 is used to implement the steps of the network and log intelligent management method as described in Embodiment 1 and Embodiment 2 when executing the computer program.

[0152] Embodiment 5

[0153] A readable storage medium for intelligent management of a network and logs, wherein a computer program is stored on the readable storage medium, and the computer program is used to execute the steps of the method for intelligent management of a network and logs as described in Embodiment 1 and Embodiment 2.

[0154] The present invention may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium (or medium) having computer-readable program instructions thereon, the computer-readable program instructions being used to cause a processor to perform aspects of the present invention.

[0155] The readable storage medium is a computer-readable storage medium, which can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer-readable storage medium can be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device (such as a punch card or a raised structure in a groove with instructions recorded thereon), and any suitable combination of the above.

[0156] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device or downloaded to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network). The network can include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in a computer-readable storage medium in the corresponding computing / processing device.

[0157] The computer programs described herein are computer-readable program instructions that can be downloaded from a computer-readable storage medium to a corresponding computing / processing device or downloaded to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network). The network may include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in a computer-readable storage medium in the corresponding computing / processing device.

[0158] The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, an electronic circuit (including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA)) may execute the computer-readable program instructions to personalize the electronic circuit by utilizing the state information of the computer-readable program instructions so as to perform aspects of the present invention.

[0159] Aspects of the present invention are described herein with reference to flowcharts of methods, systems, devices, and computer program products according to embodiments of the present invention and / or block diagrams. It should be understood that each box in the flowchart and / or block diagram and the combination of boxes in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0160] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine that, when executed by a processor of the computer or other programmable data processing apparatus, creates a device for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that can instruct a computer, a programmable data processing apparatus, and / or other device to function in a particular manner, such that a computer-readable storage medium having instructions stored therein includes an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0161] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with the art within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A network and log intelligent management method, characterized in that: The network and log intelligent management method comprises the following steps: Deploy network monitoring tools and log monitoring tools on the target network; Define network monitoring indicators, monitor the target network in real time through network monitoring tools, and obtain log data through log monitoring tools; The network monitoring information is integrated with the log data for analysis, and the target network is judged to see whether there are any abnormalities based on the analysis results; Define an alarm mechanism to generate an alarm when an abnormality occurs in the target network.

2. The network and log intelligent management method according to claim 1, characterized in that: The network monitoring tool is used to monitor the traffic of the target network and the status and performance of the devices in the target network in real time.

3. The network and log intelligent management method according to claim 1, characterized in that: The log monitoring tool is used to collect and analyze the generated logs from the systems, applications and devices of the target network in real time.

4. The network and log intelligent management method according to claim 1, characterized in that: The network monitoring indicators include any one or more of CPU usage, memory usage, and bandwidth usage.

5. The network and log intelligent management method according to claim 4, characterized in that: The log data includes logs of the target network's system, target network-related applications, and target network-related devices; the log monitoring tool includes a log collection tool, a log storage tool, a log analysis tool, and a report generation tool.

6. The network and log intelligent management method according to claim 1, characterized in that: The fusion analysis of network monitoring information and log data, and judging whether the target network has abnormalities according to the analysis results include: Obtain network monitoring information and log data for fusion analysis, and pre-process the network monitoring information and log data; Extracting features of the network monitoring information and the log data; Find relevant features from the extracted features and associate network monitoring information with log data; Analyze the associated network monitoring information and log data, and determine whether there are any abnormalities in the target network based on the analysis results.

7. The network and log intelligent management method according to claim 6, characterized in that: The analyzing of the associated network monitoring information and log data also includes: generating a comprehensive network monitoring view based on the network monitoring information and its associated log data.

8. The network and log intelligent management method according to claim 5, characterized in that: The method for defining the alarm mechanism is: obtaining network monitoring indicators, setting alarm thresholds according to the network monitoring indicators, and sending alarm notifications when the network monitoring indicators exceed the thresholds.

9. The network and log intelligent management method according to claim 8, characterized in that: The method for defining an alarm mechanism also includes: acquiring log data, setting a target state, and sending an alarm notification when the state of any one of a target network-related system, a target network-related application, and a target network-related device changes to the target state.

10. The network and log intelligent management method according to claim 1, characterized in that: The network and log intelligent management method further includes: generating a network and log analysis report according to the analysis result and the network anomaly judgment result.

11. A network and log intelligent management system, characterized in that: The network and log intelligent management system includes: Monitoring management module, used to deploy monitoring tools and log monitoring tools on the target network; The monitoring and collection module defines network monitoring indicators, which are used to monitor the target network in real time through network monitoring tools and obtain log data through log monitoring tools; The data processing module is used to integrate and analyze the network monitoring information and log data, and determine whether there is an abnormality in the target network based on the analysis results; The abnormal alarm module defines an alarm mechanism for generating an alarm when an abnormality occurs in the target network.

12. A network and log intelligent management device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the network and log intelligent management method as described in any one of claims 1 to 10 when executing the computer program.

13. A readable storage medium for intelligent management of network and logs, characterized in that: The readable storage medium stores a computer program, and the computer program is used to execute the steps of the network and log intelligent management method as described in any one of claims 1 to 10.

Images