Network monitoring method, device, equipment and medium

Abstract

The embodiment of the invention provides a network monitoring method and device, equipment and a medium. The method comprises the following steps: receiving a plurality of second-level monitoring original data which are sent by a plurality of network devices; performing abnormal data shielding processing on the plurality of second-level monitoring original data to obtain a plurality of effective original data; performing data preprocessing on the plurality of effective original data to obtain a plurality of preprocessed data; performing calculation processing according to the multiple pieces of preprocessed data to obtain multiple pieces of index data; according to the multiple pieces of index data, the monitoring results of the multiple pieces of network equipment are determined, monitoring of the network equipment indexes is achieved, and the network monitoring precision is improved.

Description

Technical Field

[0001] This application relates to the fields of network security and computer network technologies, and particularly to a network monitoring method, apparatus, device, and medium. Background Art

[0002] Network monitoring refers to the process of real-time monitoring, recording, and controlling network devices, network traffic, network behaviors, video information, etc. within a local area network or a wide area network through network technologies. With the rapid development of the fifth-generation mobile communication technology (5G), network devices have been widely used. Therefore, the security of network devices is crucial, and network monitoring is required to ensure the security of network devices.

[0003] Most traditional network monitoring methods are based on pull-based network monitoring technologies such as the Simple Network Management Protocol (SNMP) and the command-line interface (CLI). If you want to improve the accuracy of network monitoring, you need to increase the frequency of data pulling, which consumes a large amount of resources such as the device's Central Processing Unit (CPU) and memory.

[0004] However, the network monitoring methods of the prior art occupy a large amount of network resources, cannot stably collect network monitoring data, and result in low accuracy of network monitoring. Summary of the Invention

[0005] This application provides a network monitoring method, apparatus, device, and medium to solve the problem that the network monitoring method of the prior art occupies a large amount of network resources, cannot stably collect network monitoring data, and results in low accuracy of network monitoring.

[0006] In a first aspect, this application provides a network monitoring method, including:

[0007] Receiving a plurality of second-level monitoring raw data, where the plurality of second-level monitoring raw data are sent by a plurality of network devices;

[0008] Performing abnormal data masking processing on the plurality of second-level monitoring raw data to obtain a plurality of valid raw data;

[0009] Performing data preprocessing on the plurality of valid raw data to obtain a plurality of preprocessed data;

[0010] Performing calculation processing based on the plurality of preprocessed data to obtain a plurality of index data;

[0011] Determine the monitoring results of the multiple network devices according to the multiple index data.

[0012] This application provides an efficient and accurate network monitoring method, which can monitor multiple network devices simultaneously. Compared with the traditional pull-type network monitoring technology, this application supports connecting to multiple network devices, receiving the second-level monitoring raw data actively sent by multiple network devices, and does not need to frequently initiate requests for collecting data. Without affecting the performance of network devices, a small amount of server resources are used to receive a large amount of second-level data, improving the efficiency and accuracy of obtaining and monitoring raw data. After receiving the multiple second-level monitoring raw data, through operations such as shielding abnormal data and data preprocessing, the preprocessed data that can accurately reflect the state of network devices can be obtained. By calculating the preprocessed data, the monitoring results of multiple network devices can be determined, realizing the monitoring of network device indicators and improving the accuracy of network monitoring.

[0013] Optionally, the determining the monitoring results of the multiple network devices according to the multiple index data includes:

[0014] Add the multiple index data to the message queue of the first distributed stream processing platform;

[0015] Consume the multiple index data in the message queue of the first distributed stream processing platform in real time to determine the monitoring results of the multiple network devices.

[0016] Among them, for the multiple index data obtained by processing, this application adds them to the message queue of the first distributed stream processing platform (Apache Kafka, Kafka), and consumes the multiple index data in the first Kafka message queue in real time to determine the detection results. The Kafka message queue has the characteristics of high throughput, low latency, persistent storage and reliability, which can ensure the stability of network monitoring and further improve the efficiency and accuracy of network monitoring.

[0017] Optionally, the consuming the multiple index data in the message queue of the first distributed stream processing platform in real time to determine the monitoring results of the multiple network devices includes:

[0018] Consume the multiple index data in the message queue of the first distributed stream processing platform in real time, and write the multiple index data into a database, where the database is used to store the multiple index data as the index historical data corresponding to the multiple network devices.

[0019] Here, the present application consumes multiple metric data in the message queue of the first distributed stream processing platform in real time, writes the metric data into a database, and stores the metric historical data in the database for querying and displaying the metric historical data, realizing the monitoring of network devices in multiple periods and improving the comprehensiveness of network monitoring.

[0020] Optionally, the real-time consumption of multiple metric data in the message queue of the first distributed stream processing platform to determine the monitoring results of the multiple network devices includes:

[0021] Consume multiple metric data in the message queue of the first distributed stream processing platform in real time, and determine whether to issue a monitoring metric alarm according to a preset metric alarm threshold.

[0022] Here, the present application consumes multiple metric data in the message queue of the first distributed stream processing platform in real time, automatically compares the metric data with a preset metric alarm threshold, can perform metric alarm judgment, and can issue a monitoring metric alarm based on the judgment result, improving the timeliness of network device alarms and the security of network devices.

[0023] Optionally, the abnormal data masking process for the multiple second-level monitoring raw data to obtain multiple valid raw data includes:

[0024] Perform unmanaged device data masking process and / or keyword masking process on the multiple second-level monitoring raw data to obtain multiple valid raw data.

[0025] Before processing the second-level monitoring raw data, the present application first performs abnormal data masking. Specifically, it includes unmanaged device data masking process and keyword masking process. The unmanaged device data masking is used to automatically mask the data sent by unmanaged network devices to ensure that only the data from trusted devices is received, guaranteeing the security of data reception. The keyword masking process is used to handle scenarios such as sudden bursts of abnormal data volume or masking of metric data that does not need to be concerned, only receiving the valid metric content of managed devices, preventing the impact caused by abnormal situations, and ensuring the data reception efficiency. The above data masking process further improves the stability and security of network monitoring.

[0026] Optionally, the data preprocessing of the multiple valid raw data to obtain multiple preprocessed data includes:

[0027] Perform at least one of multi-level nested data flattening process, data metric classification process, and data metric enrichment process on the multiple valid raw data to obtain multiple preprocessed data.

[0028] The data preprocessing operations of this application include multi-level nested data flattening, data indicator classification, and data indicator enrichment. Among them, since the effective original data is in a multi-level nested format, for convenient subsequent processing, it is necessary to flatten the data and split it into multiple lines. Data indicator classification splits the data into different indicator classifications, such as interface traffic indicator classification, interface optical power indicator classification, device CPU memory indicator classification, etc.; data indicator enrichment enriches each indicator classification, which is mainly used for subsequent data calculation processes. Through the above data preprocessing, it is convenient to determine the results of network monitoring. The preprocessed data obtained after data preprocessing can be directly used as indicator data or used as indicator data after calculation. The above data preprocessing can obtain intuitive and easy-to-process data, improving the efficiency of network monitoring.

[0029] Optionally, the calculating and processing based on the multiple preprocessed data to obtain multiple indicator data includes:

[0030] Performing at least one of cumulative value calculation difference, indicator unit conversion, and utilization rate indicator calculation on the multiple preprocessed data to obtain multiple indicator data.

[0031] Here, through methods such as cumulative value calculation difference, indicator unit conversion, and utilization rate indicator calculation, this application can obtain indicator data that accurately reflects the status of network devices. Cumulative value calculation difference can obtain actual indicator data, and indicator unit conversion can visually and accurately unify the data, facilitating the monitoring of network device status through the obtained indicator data. Utilization rate indicator calculation can accurately reflect the utilization status of each network device, realizing the accuracy and comprehensiveness of network monitoring.

[0032] In a second aspect, this application provides a network monitoring device, including:

[0033] A receiving module, configured to receive multiple second-level monitoring original data, where the multiple second-level monitoring original data are sent by multiple network devices;

[0034] A shielding processing module, configured to perform abnormal data shielding processing on the multiple second-level monitoring original data to obtain multiple effective original data;

[0035] A preprocessing module, configured to perform data preprocessing on the multiple effective original data to obtain multiple preprocessed data;

[0036] A calculating and processing module, configured to perform calculating and processing based on the multiple preprocessed data to obtain multiple indicator data;

[0037] A determining module, configured to determine the monitoring results of the multiple network devices based on the multiple indicator data.

[0038] Optionally, the determining module includes:

[0039] An adding sub-module, configured to add the multiple metric data to a message queue of a first distributed stream processing platform;

[0040] A consuming sub-module, configured to consume in real time the multiple metric data in the message queue of the first distributed stream processing platform to determine monitoring results of the multiple network devices.

[0041] Optionally, the consuming sub-module is specifically configured to:

[0042] Consume in real time the multiple metric data in the message queue of the first distributed stream processing platform, and write the multiple metric data into a database, where the database is configured to store the multiple metric data as metric historical data corresponding to the multiple network devices.

[0043] Optionally, the consuming sub-module is specifically configured to:

[0044] Consume in real time the multiple metric data in the message queue of the first distributed stream processing platform, and determine whether to issue a monitoring metric alarm according to a preset metric alarm threshold.

[0045] Optionally, the masking processing module is specifically configured to:

[0046] Perform unmanaged device data masking processing and / or keyword masking processing on the multiple second-level monitoring raw data to obtain multiple valid raw data.

[0047] Optionally, the preprocessing module is specifically configured to:

[0048] Perform at least one of multi-level nested data flattening processing, data metric classification processing, and data metric enrichment processing on the multiple valid raw data to obtain multiple preprocessed data.

[0049] Optionally, the calculation processing module is specifically configured to:

[0050] Perform at least one of cumulative value calculation difference, metric unit conversion, and utilization rate metric calculation on the multiple preprocessed data to obtain multiple metric data.

[0051] In a third aspect, the present application provides a network monitoring device, including: a memory, a processor;

[0052] The memory stores computer execution instructions;

[0053] The processor executes the computer-executable instructions stored in the memory, such that the processor performs the above first aspect and / or various possible implementation manners of the first aspect.

[0054] In a fourth aspect, the present application provides a computer-readable storage medium, in which computer-executable instructions are stored, and when the computer-executable instructions are executed by a processor, they are used to implement the above first aspect and / or various possible implementation manners of the first aspect.

[0055] The network monitoring method, device, equipment and medium provided by the present application support connection with multiple network devices, receive the second-level monitoring raw data actively sent by the multiple network devices, do not need to frequently initiate requests for collecting data, and use less server resources to receive a large amount of second-level data without affecting the performance of the network devices, improving the efficiency and accuracy of obtaining and monitoring the raw data. After receiving the multiple second-level monitoring raw data, through operations of shielding abnormal data and data preprocessing, preprocessed data that accurately reflects the state of the network devices can be obtained. By calculating the preprocessed data, the monitoring results of the multiple network devices can be determined, realizing the monitoring of the network device metrics and improving the accuracy of network monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The drawings herein are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application.

[0057] Figure 1 FIG. is a schematic diagram of an application scenario of a network monitoring method provided by an embodiment of the present application;

[0058] Figure 2 FIG. is a schematic flowchart of a network monitoring method provided by an embodiment of the present application;

[0059] Figure 3 FIG. is a schematic diagram of a process of a network monitoring processing method provided by an embodiment of the present application;

[0060] Figure 4 FIG. is a schematic structural diagram of a network monitoring device provided by an embodiment of the present application;

[0061] Figure 5 FIG. is a schematic structural diagram of a network monitoring device provided by an embodiment of the present application.

[0062] Through the above drawings, the clear embodiments of the present application have been shown, and there will be more detailed descriptions hereinafter. These drawings and the text description are not intended to limit the scope of the concept of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0063] Exemplary embodiments will be described in detail herein, and examples thereof are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of apparatuses and methods consistent with some aspects of the present application as detailed in the appended claims.

[0064] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use, and processing of relevant data need to comply with relevant laws, regulations, and standards, and corresponding operation entrances are provided for users to select authorization or rejection.

[0065] The following introduces the terms in the embodiments of the present application:

[0066] SNMP: A network management standard protocol widely used in Transmission Control Protocol / Internet Protocol (TCP / IP) networks. This protocol can support network management systems to monitor whether there are any situations that cause management concerns for devices connected to the network. SNMP adopts a polling mechanism, provides the most basic function set, and is carried by User Datagram Protocol (UDP) packets. Therefore, it is supported by the vast majority of devices, and at the same time ensures the transmission of management information between any two points, facilitating administrators to retrieve information at any node on the network for troubleshooting.

[0067] CLI: The command-line interface was the most widely used user interface before the popularization of the graphical user interface. It usually does not support the mouse, and users input commands through the keyboard. After the computer receives the commands, it executes them. It can be used for command-line interaction with network devices.

[0068] Telemetry: It is a new generation of network monitoring technology, a technology that can remotely and rapidly collect data from devices. Compared with traditional pull-based network monitoring technologies such as SNMP and CLI, telemetry actively sends device CPU / memory, interface traffic, etc. information to the collector periodically through the "push mode", and can achieve push at the second level or even sub-second level, providing more real-time, faster, and more accurate network monitoring functions.

[0069] Kafka: An open-source stream processing platform. Kafka is a high-throughput distributed publish-subscribe messaging system that can handle all action stream data of consumers in websites. This data is usually solved by processing logs and log aggregation due to throughput requirements.

[0070] High-performance Remote Procedure Call Protocol (gRPC): A modern open-source high-performance remote procedure call framework that can run in any environment. It can efficiently connect services within and across data centers, support load balancing, tracing, health checks, and authentication. It is also suitable for the last mile of distributed computing, connecting devices, mobile applications, and browsers to backend services.

[0071] Binary Data Encoding Format (Google Protocol Buffers, GPB): A data description language that can serialize structured data and can be used for data storage, communication protocols, etc., and is widely used for storing and exchanging various structured information. It is language- and platform-independent and highly scalable, supporting multiple programming languages.

[0072] Flink: An open-source stream processing framework whose core is a distributed data stream engine. Flink executes any stream data program in a data-parallel and pipelined manner. Flink's pipelined runtime system can execute batch processing and stream processing programs. In addition, Flink's runtime itself also supports the execution of iterative algorithms.

[0073] syslog: Often referred to as system log or system record, it is a standard for transmitting log messages on the Internet Protocol (TCP / IP) network.

[0074] Traditional network monitoring methods are difficult to achieve second-level monitoring and have the following problems: Frequently collecting data from devices such as routers and switches in the network at the second level will consume a large amount of device resources such as CPU and memory, resulting in a decline in device performance and affecting the services they carry; moreover, network devices generally only support minute-level data collection, and the device's underlying counters use a caching mechanism. When collecting data at the second level or shorter intervals, the data will not change, resulting in invalid collected data; at the same time, the collection tool also has resource consumption problems. Collecting a large amount of high-frequency data requires the collection end to have sufficient server memory and CPU resources to cache and process the data. If the resources are insufficient, problems such as data loss or collection jams will occur, and it is difficult to ensure stable second-level data collection.

[0075] The network monitoring methods of the prior art occupy a large amount of network resources, cannot stably collect network monitoring data, and result in low accuracy of network monitoring.

[0076] To solve the above problems, embodiments of the present application provide a network monitoring method, device, medium, and product, which support connection with multiple network devices, receive second-level monitoring raw data actively sent by multiple network devices, and can obtain preprocessed data that accurately reflects the status of network devices through operations of shielding abnormal data and data preprocessing. By calculating the preprocessed data, the monitoring results of multiple network devices can be determined, realizing stable monitoring of network device metrics.

[0077] Optionally, applying the second-level data acquisition capability of telemetry technology to the network monitoring and alerting scenario can, without affecting the performance of network devices, use a very small amount of server resources compared with the traditional method to receive a large amount of second-level monitoring data, and analyze, process, and perform abnormal alerting on the monitoring data in real time.

[0078] Optionally, Figure 1 FIG. is a schematic diagram of an application scenario of a network monitoring method provided by an embodiment of the present application. This application scenario includes multiple network devices and multiple collectors, such as Figure 1 shown, including a first switch 101, a second switch 102, a third switch 103, a first router 104, a second router 105, a first collector 106, a second collector 107, a third collector 108, and a performance metric database 109.

[0079] Among them, Figure 1 the first switch 101, the second switch 102, the third switch 103, the first router 104, and the second router 105 in are network devices, and the above-mentioned multiple network devices can come from different manufacturers.

[0080] Among them, the first collector 106, the second collector 107, and the third collector 108 can also be referred to as collection components.

[0081] It can be understood that Figure 1 is only schematic, Figure 1 and the quantity and structure of any component in are not specifically limited.

[0082] Currently, mainstream data center network switch and router manufacturers all support telemetry. Network devices can configure and enable corresponding telemetry monitoring metrics as needed to push telemetry data. The method involved in the embodiments of the present application requires the device to configure and enable telemetry metric pushing, and the metric sending period is second-level. The device supports a sending period of 30s or faster. The shorter the sending period, the higher the timely rate of abnormal alerting for monitoring metrics.

[0083] The embodiments of this application support multi-vendor network devices. The applicable network types of the method include but are not limited to data center networks. As long as the network devices support the configuration to enable telemetry metric push, they can be adapted and supported.

[0084] Optionally, Figure 1 Any collector in Figure 1 is a general data collection component that supports multi-vendors. Each collection component can receive the telemetry second-level monitoring raw data sent by multi-vendor network devices. The collection component is stateless and supports single-unit independent deployment or multi-unit cluster deployment. Load balancing technology is used for horizontal expansion to ensure high service availability, meet the network usage requirements of various scales, and ensure that the telemetry second-level monitoring data can be received normally.

[0085] Optionally, the embodiments of this application provide a network monitoring system, which may include Figure 1 the first collector 106, the second collector 107, the third collector 108, and the performance metric database 109 in Figure 1 .

[0086] Optionally, the network monitoring system further includes a processing device.

[0087] The processing device can receive the second-level monitoring raw data actively sent by multiple network devices. Through operations such as shielding abnormal data and data preprocessing, it can obtain the preprocessed data that accurately reflects the status of the network devices. By calculating the preprocessed data, it can determine the monitoring results of multiple network devices, realizing stable monitoring of network device metrics.

[0088] It should be understood that the above-mentioned processing device can be implemented by a processor reading and executing instructions in a memory, or can also be implemented by a chip circuit.

[0089] In addition, the network architecture and business scenarios described in the embodiments of this application are for more clearly explaining the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those of ordinary skill in the art know that with the evolution of the network architecture and the emergence of new business scenarios, the technical solutions provided by the embodiments of this application are equally applicable to similar technical problems.

[0090] The following uses specific embodiments to elaborate in detail on the technical solutions of this application and how the technical solutions of this application solve the above technical problems. These several specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The following will describe the embodiments of this application with reference to the accompanying drawings.

[0091] Optionally, Figure 2The flowchart shows a network monitoring method provided by an embodiment of this application. The execution subject of the embodiment of this application can be a network monitoring system or a processing device or a processor of the network monitoring system, and the specific execution subject can be determined according to the actual application scenario. As Figure 2 shown, the method includes the following steps:

[0092] S201: Receive multiple raw second-level monitoring data.

[0093] Among them, the multiple raw second-level monitoring data are sent by multiple network devices.

[0094] Optionally, the multiple network devices here can come from the same or different manufacturers.

[0095] Optionally, the raw second-level monitoring data is the telemetry data pushed by the network device according to the configured telemetry monitoring metrics as needed.

[0096] Optionally, the sending of the network device and the receiving of the raw second-level monitoring data can be based on the UDP protocol or the gRPC protocol.

[0097] Among them, most manufacturer devices support sending data to the general data collection component through the gRPC protocol. The gRPC protocol is the currently mainstream transmission protocol, with higher data sending security than the UDP protocol, and high data compression ratio and sending efficiency.

[0098] Optionally, the data encoding format for device sending uses GPB. The specific implementation details are different for different manufacturers. The general collection component supports the currently mainstream data encoding formats for data center network devices. The telemetry metric content sent by network devices of different manufacturers is also different. The metric content supported by the general collection component includes but is not limited to device performance metrics, device interface performance metrics, device interface queue performance metrics, routing metrics, Address Resolution Protocol (ARP) metrics, Media Access Control Address (MAC) metrics, etc.

[0099] S202: Perform abnormal data masking processing on the multiple raw second-level monitoring data to obtain multiple valid raw data.

[0100] Optionally, performing abnormal data masking processing on the multiple raw second-level monitoring data to obtain multiple valid raw data includes:

[0101] Perform unmanaged device data masking processing and / or keyword masking processing on the multiple raw second-level monitoring data to obtain multiple valid raw data.

[0102] Specifically, the embodiments of the present application support shielding abnormal telemetry second-level monitoring raw data, saving the qualified raw data to the message queue of the second distributed stream processing platform (raw data message queue, which can be a Kafka message queue), and the shielding methods include: shielding unmanaged device data and shielding raw data metrics of device monitoring according to keywords.

[0103] In a possible implementation, devices are managed using information such as network device names, device computer communication identification addresses (Internet Protocol, IP) addresses, and serial numbers. Devices within the management scope are trusted devices, and all telemetry data sent by unmanaged devices will be automatically shielded to ensure that only telemetry data of trusted devices is received, guaranteeing the security of component data reception. At the same time, it supports shielding some content of the raw data of managed device monitoring according to keywords, mainly used to cope with scenarios such as sudden bursts of abnormal data volume or shielding data metrics that do not need to be concerned about, only receiving valid metric content of managed devices, and preventing the impact caused by abnormal situations on the acquisition components, ensuring data reception efficiency.

[0104] The raw data message queue for telemetry second-level monitoring stores the most original data of the received device telemetry without any processing.

[0105] Before processing the second-level monitoring raw data, the embodiments of the present application first perform abnormal data shielding. Specifically, it includes unmanaged device data shielding processing and keyword shielding processing. Unmanaged device data shielding is used to automatically shield data sent by unmanaged network devices, ensuring that only data of trusted devices is received and guaranteeing the security of data reception. Keyword shielding processing is used to cope with scenarios such as sudden bursts of abnormal data volume or shielding data metrics that do not need to be concerned about, only receiving valid metric content of managed devices, preventing the impact caused by abnormal situations, and ensuring data reception efficiency. The above data shielding processing further improves the stability and security of network monitoring.

[0106] S203: Perform data preprocessing on multiple valid raw data to obtain multiple preprocessed data.

[0107] Optionally, performing data preprocessing on multiple valid raw data to obtain multiple preprocessed data includes:

[0108] Performing at least one of multi-level nested data flattening processing, data metric classification processing, and data metric enrichment processing on multiple valid raw data to obtain multiple preprocessed data.

[0109] In a possible implementation, Flink is used to consume the raw data message queue data in real time, and the raw data for second-level monitoring is preprocessed. The preprocessing content includes: flattening multi-level nested data, classifying data metrics, and enriching data metrics, etc.

[0110] The content of the raw data is in a multi-level nested format. For the convenience of subsequent processing, it is necessary to flatten the data and split it into multiple lines; the data metric classification is based on the sensor path of the telemetry data, and the data is split into different metric classifications, such as interface traffic metric classification, interface optical power metric classification, device CPU memory metric classification, etc.; the data metric enrichment is to enrich each metric classification, which is mainly used for subsequent data calculation processes. For example, to calculate the interface bandwidth utilization rate, there are interface in-out traffic fields in the metric classification, but there is no bandwidth field. It is necessary to supplement the corresponding interface bandwidth field data to the metric classification, and then in the data calculation process, calculate the interface bandwidth utilization rate according to the in-out traffic fields and the bandwidth.

[0111] For the preprocessed telemetry metric data, some of the metric data is already directly usable, and some data needs to be calculated later to obtain the finally usable metric data.

[0112] The data preprocessing operation of this application embodiment includes multi-level nested data flattening processing, data metric classification processing, and data metric enrichment processing. Among them, since the effective raw data is in a multi-level nested format, for the convenience of subsequent processing, it is necessary to flatten the data and split it into multiple lines. The data metric classification splits the data into different metric classifications, such as interface traffic metric classification, interface optical power metric classification, device CPU memory metric classification, etc.; the data metric enrichment is to enrich each metric classification, which is mainly used for subsequent data calculation processes. Through the above data preprocessing, it is convenient to determine the results of network monitoring. The preprocessed data obtained after data preprocessing can be directly used as metric data, or used as metric data after calculation. The above data preprocessing can obtain intuitive and easy-to-process data, improving the efficiency of network monitoring.

[0113] S204: Perform calculation processing based on multiple preprocessed data to obtain multiple metric data.

[0114] It should be noted that some of the preprocessed data can be directly used as metric data, such as CPU utilization rate. Another part of the preprocessed data needs to be calculated to obtain metric data.

[0115] Optionally, performing calculation processing based on multiple preprocessed data to obtain multiple metric data includes:

[0116] Based on multiple preprocessed data, perform at least one of the calculations of calculating the difference of the cumulative value, converting the metric unit, and calculating the utilization rate metric to obtain multiple metric data.

[0117] In a possible implementation, perform calculation operations on some of the metric data of the preprocessed data, and write the final metric data after processing into the message queue of the first distributed stream processing platform (also known as the metric data message queue, which can also be the telemetry second-level monitoring metric data Kafka message queue). The calculation processing content includes: calculating the difference of the cumulative value, converting the metric unit, calculating the utilization rate metric, etc.

[0118] Specifically, the Telemetry monitoring metrics will include some cumulative value metrics. Each time a value is taken, an increasing cumulative counter value will be obtained. Each time the metric data is calculated, the difference between the current value and the previous data collection value will be calculated to obtain the actual metric data for the current data cycle. The metric unit conversion is mainly to convert the original numerical values of metrics such as traffic or bandwidth, for example, converting to KB, MB, Kb, and Mb, etc. The utilization rate metric calculation will calculate and generate new utilization rate metrics based on the original metric data. For example, dividing the interface in-out traffic metric by the interface bandwidth and multiplying by 100% to obtain the interface in-out bandwidth utilization rate, etc.

[0119] Among them, the telemetry second-level monitoring metric data Kafka message queue stores the final metric data, which is used for subsequent monitoring metric data warehousing and monitoring metric alarm judgment and notification.

[0120] Here, the embodiments of the present application can obtain metric data that accurately reflects the status of network devices through methods such as calculating the difference of the cumulative value, converting the metric unit, and calculating the utilization rate metric. Calculating the difference of the cumulative value can obtain the actual metric data, and converting the metric unit can intuitively and accurately unify the data, facilitating the monitoring of the status of network devices through the obtained metric data. Calculating the utilization rate metric can accurately reflect the utilization status of each network device, realizing the accuracy and comprehensiveness of network monitoring.

[0121] S205: Determine the monitoring results of multiple network devices according to multiple metric data.

[0122] The present application provides an efficient and accurate network monitoring method, which can monitor multiple network devices simultaneously. Compared with the traditional pull-type network monitoring technology, the embodiments of the present application support connecting to multiple network devices, receiving the second-level monitoring raw data actively sent by multiple network devices, and without frequently initiating requests for collecting data. On the premise of not affecting the performance of network devices, less server resources are used to receive a large amount of second-level data, improving the efficiency and accuracy of obtaining and monitoring raw data. After receiving multiple second-level monitoring raw data, through operations of shielding abnormal data and data preprocessing, the preprocessed data that can accurately reflect the status of network devices can be obtained. By calculating the preprocessed data, the monitoring results of multiple network devices can be determined, realizing the monitoring of network device indicators and improving the accuracy of network monitoring.

[0123] The following is a specific explanation of step S205:

[0124] Optionally, determining the monitoring results of multiple network devices according to multiple index data includes: adding the multiple index data to the message queue of the first distributed stream processing platform; and consuming the multiple index data in the message queue of the first distributed stream processing platform in real time to determine the monitoring results of multiple network devices.

[0125] Among them, for the multiple index data obtained through processing, the embodiments of the present application add them to the first Kafka message queue, and consume the multiple index data in the first Kafka message queue in real time to realize the determination of the detection results. The Kafka message queue has the characteristics of high throughput, low latency, persistent storage and reliability, which can ensure the stability of network monitoring and further improve the efficiency and accuracy of network monitoring.

[0126] Optionally, consuming the multiple index data in the message queue of the first distributed stream processing platform in real time to determine the monitoring results of multiple network devices includes: consuming the multiple index data in the message queue of the first distributed stream processing platform in real time and writing the multiple index data into a database, where the database is used to store the multiple index data as index historical data corresponding to multiple network devices.

[0127] In a possible implementation manner, Flink is used to consume the index data in the Kafka message queue of the telemetry second-level monitoring index data in real time and write the index data into the database. The database stores the telemetry second-level monitoring index historical data for querying and displaying the index historical data.

[0128] Here, embodiments of the present application consume multiple metric data in the message queue of the first distributed stream processing platform in real time, write the metric data into a database, and store historical metric data in the database for querying and displaying historical metric data, realizing the monitoring of network devices in multiple periods and improving the comprehensiveness of network monitoring.

[0129] Optionally, consuming multiple metric data in the message queue of the first distributed stream processing platform in real time to determine the monitoring results of multiple network devices includes: consuming multiple metric data in the message queue of the first distributed stream processing platform in real time and determining whether to issue a monitoring metric alarm according to a preset metric alarm threshold.

[0130] In a possible implementation manner, while the data is written into the database in real time, Flink is used to consume the metric data in the Kafka message queue of the telemetry second-level monitoring metrics in real time for metric alarm judgment. According to the configured metric alarm threshold, if the current metric value does not meet the threshold condition, a monitoring metric alarm is issued and the alarm is sent to the network management or other external alarm components.

[0131] Alarm push supports notification methods such as syslog logs and text messages. The second-level monitoring metric alarm is immediately pushed to the network management system through the syslog log method; the second-level monitoring metric alarm is timely pushed to the network operation and maintenance duty personnel in the form of text messages.

[0132] Here, embodiments of the present application consume multiple metric data in the message queue of the first distributed stream processing platform in real time, automatically compare the metric data with a preset metric alarm threshold, can perform metric alarm judgment, and can issue a monitoring metric alarm based on the judgment result, improving the timeliness of network device alarms and the security of network devices.

[0133] Optionally, Figure 3 is a schematic diagram of the process of a network monitoring processing method provided by an embodiment of the present application, as Figure 3As shown in the figure, the method of the embodiment of the present application can achieve the following functions: implement a general data collection component that supports multiple vendors, receive telemetry second-level monitoring data sent by network devices of multiple vendors, and the collection component supports single-machine and cluster-mode deployment and uses load balancing technology for horizontal expansion to ensure that all telemetry second-level monitoring raw data can be received normally; support shielding abnormal telemetry second-level monitoring raw data, save the qualified raw data to the telemetry second-level monitoring raw data Kafka message queue (raw data message queue), and the shielding methods include: shielding unmanaged device data, shielding raw data metrics of devices according to keywords; consume the data in the telemetry second-level monitoring raw data Kafka message queue (raw data message queue) in real time, and preprocess the second-level monitoring raw data. The preprocessing content includes: flattening multi-level nested data, classifying data metrics, and enriching data metrics, etc.; perform calculation operations on some metric data of the preprocessed data, and write the processed final metric data into the telemetry second-level monitoring metric data Kafka message queue (metric data message queue). The calculation and processing content includes: calculating the difference of cumulative values, converting metric units, calculating utilization rate metrics, etc.; consume the metric data in the telemetry second-level monitoring metric data Kafka message queue in real time, and write the metric data into the database; at the same time, consume the metric data in the telemetry second-level monitoring metric data Kafka message queue in real time, perform metric alarm judgment, and according to the configured metric alarm threshold, if the current metric value does not meet the threshold condition, perform monitoring metric alarm and send the alarm to the network management or other external alarm components.

[0134] Figure 4 The following is a schematic structural diagram of a network monitoring device provided by an embodiment of the present application, as Figure 4 shown, the network monitoring device provided in this embodiment includes: a receiving module 401, a shielding processing module 402, a preprocessing module 403, a calculation processing module 404, and a determination module 405. The network monitoring device here can be a processing device, a processor, the server itself, or a chip or integrated circuit that implements the functions of the processor. It should be noted here that the division of the receiving module 401, the shielding processing module 402, the preprocessing module 403, the calculation processing module 404, and the determination module 405 is only a logical function division, and physically the two can be integrated or independent.

[0135] Among them, the receiving module is used to receive multiple second-level monitoring raw data, where the multiple second-level monitoring raw data are sent by multiple network devices;

[0136] The shielding processing module is used to perform abnormal data shielding processing on the multiple second-level monitoring raw data to obtain multiple valid raw data;

[0137] A preprocessing module for preprocessing multiple valid original data to obtain multiple preprocessed data;

[0138] A calculation processing module for performing calculation processing based on multiple preprocessed data to obtain multiple index data;

[0139] A determination module for determining the monitoring results of multiple network devices based on multiple index data.

[0140] Optionally, the determination module includes:

[0141] An adding sub-module for adding multiple index data to the message queue of the first distributed stream processing platform;

[0142] A consuming sub-module for consuming in real time multiple index data in the message queue of the first distributed stream processing platform to determine the monitoring results of multiple network devices.

[0143] Optionally, the consuming sub-module is specifically used for:

[0144] Consuming in real time multiple index data in the message queue of the first distributed stream processing platform and writing the multiple index data into a database, where the database is used to store the multiple index data as index historical data corresponding to multiple network devices.

[0145] Optionally, the consuming sub-module is specifically used for:

[0146] Consuming in real time multiple index data in the message queue of the first distributed stream processing platform and determining whether to issue a monitoring index alarm according to a preset index alarm threshold.

[0147] Optionally, the shielding processing module is specifically used for:

[0148] Performing unmanaged device data shielding processing and / or keyword shielding processing on multiple second-level monitoring original data to obtain multiple valid original data.

[0149] Optionally, the preprocessing module is specifically used for:

[0150] Performing at least one of multi-level nested data flattening processing, data index classification processing, and data index enrichment processing on multiple valid original data to obtain multiple preprocessed data.

[0151] Optionally, the calculation processing module is specifically used for:

[0152] Performing at least one of cumulative value calculation difference, index unit conversion, and utilization rate index calculation on multiple preprocessed data to obtain multiple index data.

[0153] Reference Figure 5 shows a schematic structural diagram of a network monitoring device 500 suitable for implementing the embodiments of the present disclosure. The network monitoring device 500 may be a terminal device or a server. Among them, the terminal device may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (PADs), portable media players (PMPs), in-vehicle terminals (such as in-vehicle navigation terminals), etc., and fixed terminals such as digital TVs and desktop computers. Figure 5 The network monitoring device shown is only an example and should not impose any limitations on the functions and usage scopes of the embodiments of the present disclosure.

[0154] As Figure 5 shown, the network monitoring device 500 may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 501, which may perform various appropriate actions and processes according to the programs stored in a read-only memory (ROM) 502 or the programs loaded from a storage device 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the network monitoring device 500 are also stored. The processing device 501, the ROM 502, and the RAM 503 are connected to each other through a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0155] Generally, the following devices may be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 509. The communication device 509 may allow the network monitoring device 500 to communicate with other devices wirelessly or wiredly to exchange data. Although Figure 5 the network monitoring device 500 with various devices is shown, it should be understood that it is not required to implement or include all the shown devices. Instead, more or fewer devices may be implemented or included.

[0156] In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, an embodiment of the present disclosure includes a computer program product that includes a computer program carried on a computer-readable medium, and the computer program includes program code for performing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by a processing device 501, the above-described functions defined in the methods of the embodiments of the present disclosure are performed.

[0157] It should be noted that the above computer-readable medium in the present disclosure can be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. And in the present disclosure, a computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, and the computer-readable signal medium can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted by any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0158] The above computer-readable medium can be included in the above network monitoring; or it can exist separately and not be assembled into the network monitoring.

[0159] The above computer-readable medium carries one or more programs, and when the one or more programs are executed by the network monitoring, the network monitoring is caused to perform the methods shown in the above embodiments.

[0160] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may execute entirely on the user's computer, partially on the user's computer, execute as a stand-alone software package, execute partially on the user's computer and partially on a remote computer, or execute entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0161] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a portion of code that contains one or more executable instructions for implementing the specified logical function. It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur in a different order than noted in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, may be implemented by a dedicated hardware-based system that performs the specified functions or operations, or may be implemented by a combination of dedicated hardware and computer instructions.

[0162] The units described in the embodiments of the present disclosure may be implemented in software or in hardware. Among them, the name of the unit does not constitute a limitation to the unit itself in some cases. For example, the first acquisition unit may also be described as "the unit for acquiring at least two Internet protocol addresses".

[0163] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGA), application specific integrated circuits (ASIC), application specific standard products (ASSP), system on a chip (SOC), complex programmable logic devices (CPLD), and so on.

[0164] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0165] The network monitoring method of the embodiments of the present application can be used to implement the technical solutions in the foregoing method embodiments of the present application. The implementation principles and technical effects are similar and will not be elaborated herein.

[0166] The embodiments of the present application also provide a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, are used to implement the network monitoring method of any one of the foregoing.

[0167] The embodiments of the present application also provide a computer program product including a computer program that, when executed by a processor, is used to implement the network monitoring method of any one of the foregoing.

[0168] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of the present invention and include common general knowledge or conventional technical means in the technical field not disclosed in the present invention. It is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present invention is only limited by the appended claims.

Claims

1. A network monitoring method, characterized in that: include: Receiving a plurality of second-level monitoring raw data, wherein the plurality of second-level monitoring raw data are sent by a plurality of network devices; Performing abnormal data shielding processing on the plurality of second-level monitoring raw data to obtain a plurality of valid raw data; Performing data preprocessing on the multiple valid original data to obtain multiple preprocessed data; Performing calculations based on the plurality of pre-processed data to obtain a plurality of indicator data; Determine monitoring results of the multiple network devices based on the multiple indicator data.

2. The method according to claim 1, characterized in that Determining monitoring results of the plurality of network devices according to the plurality of indicator data includes: Adding the plurality of indicator data to the first distributed stream processing platform message queue; The plurality of indicator data in the message queue of the first distributed stream processing platform is consumed in real time to determine the monitoring results of the plurality of network devices.

3. The method according to claim 2, characterized in that The real-time consumption of multiple indicator data in the first distributed stream processing platform message queue to determine the monitoring results of the multiple network devices includes: The plurality of indicator data in the message queue of the first distributed stream processing platform are consumed in real time, and the plurality of indicator data are written into a database, wherein the database is used to store the plurality of indicator data as indicator historical data corresponding to the plurality of network devices.

4. The method according to claim 2, characterized in that: The real-time consumption of multiple indicator data in the first distributed stream processing platform message queue to determine the monitoring results of the multiple network devices includes: Consume multiple indicator data in the message queue of the first distributed stream processing platform in real time, and determine whether to issue a monitoring indicator alarm based on a preset indicator alarm threshold.

5. The method according to any one of claims 1 to 4, characterized in that: The performing abnormal data shielding processing on the plurality of second-level monitoring raw data to obtain a plurality of valid raw data includes: The multiple second-level monitoring raw data are subjected to unmanaged device data shielding processing and / or keyword shielding processing to obtain multiple valid raw data.

6. The method according to any one of claims 1 to 4, characterized in that: The performing data preprocessing on the plurality of valid original data to obtain a plurality of preprocessed data includes: At least one of multi-level nested data flattening processing, data indicator classification processing and data indicator enrichment processing is performed on the multiple valid original data to obtain multiple pre-processed data.

7. The method according to any one of claims 1 to 4, characterized in that: The step of performing calculations based on the plurality of pre-processed data to obtain a plurality of indicator data includes: At least one of cumulative value calculation difference, indicator unit conversion and utilization rate indicator calculation is performed according to the plurality of pre-processed data to obtain a plurality of indicator data.

8. A network monitoring device, characterized in that: include: A receiving module, used for receiving a plurality of second-level monitoring raw data, wherein the plurality of second-level monitoring raw data are sent by a plurality of network devices; A shielding processing module, used for performing abnormal data shielding processing on the plurality of second-level monitoring raw data to obtain a plurality of valid raw data; A preprocessing module, used for performing data preprocessing on the plurality of valid original data to obtain a plurality of preprocessed data; A calculation processing module, used for performing calculation processing according to the plurality of pre-processed data to obtain a plurality of indicator data; A determination module is used to determine the monitoring results of the multiple network devices based on the multiple indicator data.

9. A network monitoring device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

Images