Network element data processing method, electronic device, computer readable medium and product
By converting network element data into time-series data and storing it in a database, and then analyzing it using Prometheus and PromQL tools, the problem of scattered data collection in traditional methods is solved, and effective detection of the changing trends of network element indicators is achieved.
Patent Information
- Application Number
- CN202410986034.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-22
- Publication Date
- 2026-01-23
AI Technical Summary
Traditional methods of collecting network element performance metrics data are scattered, lack historical data, and make it difficult to analyze the trend of metric changes, resulting in difficulty in detecting potential risks and problems.
The network element's operational data is converted into time-series data in a preset data format and stored in a time-series database. Data is collected and analyzed using tools such as Prometheus or VictoriaMetrics, and metrics are queried and analyzed using PromQL or MetricsQL.
It enables unified storage and analysis of network element indicator data, and can detect risks and problems based on changes in time sequence, thereby improving operation and maintenance efficiency and detection capabilities.
Smart Images
Figure CN121396830A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, specifically to a network data processing method, electronic device, computer-readable medium, and product. Background Technology
[0002] Faced with increasingly complex network structures and growing network security issues, network administrators need to understand the basic status of the network in a timely manner, such as traffic, hardware, and software process errors. They typically detect the status and performance of network elements' hardware, software, and network by collecting data on hardware, software, and network performance indicators in real time.
[0003] However, traditional methods of collecting network element performance data output data as soon as a request is made, and the output data is often scattered, usually lacking historical data, making it difficult to analyze and judge the trend of the indicators, and thus difficult to detect potential risks and problems of network elements. Summary of the Invention
[0004] This application provides a network data processing method, an electronic device, a computer-readable medium, and a product.
[0005] This application provides a method for processing network data, which includes:
[0006] Obtain operational data of network elements;
[0007] The operational data is converted into time series data in a preset data format. The time series data includes indicators, corresponding data values of the indicators, labels of the indicators, and timestamps.
[0008] The time series data is output to a time series database for storage;
[0009] Query the time series data corresponding to the indicator to be analyzed in the time series database;
[0010] The indicator to be analyzed is analyzed based on the time series data corresponding to the indicator to be analyzed.
[0011] This application provides an electronic device, including: one or more processors; and a memory storing one or more computer programs thereon, wherein when the one or more computer programs are executed by the one or more processors, the one or more processors implement any of the network data processing methods in this application.
[0012] This application provides a computer-readable medium storing a computer program that, when executed by a processor, implements any of the network data processing methods described in this application.
[0013] This application provides a computer program product, which includes a computer program that, when executed by a processor, implements any of the network data processing methods described in this application.
[0014] According to the network data processing method, electronic device, computer-readable medium, and product provided in the embodiments of this application, the collected network element operation data is processed by data conversion to obtain time series data in a preset data format, and stored in a time series database. When indicator analysis is required, the time series data of the indicator to be analyzed is queried from the time series database and the indicator analysis is performed. Since the network element indicator data are all stored in a unified preset data format, when performing indicator analysis, the time series data of the indicator changing in chronological order can be queried from the time series database. This is beneficial for judging the trend of indicator changes based on the time series data of the indicator changing in chronological order, and thus beneficial for detecting potential risks and existing problems of the network element.
[0015] Further details regarding the above embodiments and other aspects of this application, as well as their implementations, are provided in the accompanying drawings, detailed description, and claims. Attached Figure Description
[0016] In the accompanying drawings of the embodiments of this application:
[0017] Figure 1 This diagram illustrates the system architecture of a network data processing system provided in an embodiment of this application.
[0018] Figure 2 This diagram illustrates a flowchart of a network data processing method provided in an embodiment of this application.
[0019] Figure 3 This illustration shows a schematic diagram of time-series data in a preset data format carried through a web page, as shown in an embodiment of this application.
[0020] Figure 4 This is a block diagram illustrating the composition of an electronic device provided in an embodiment of this application. Detailed Implementation
[0021] To enable those skilled in the art to better understand the technical solutions of this disclosure, the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings.
[0022] The present disclosure will be described more fully below with reference to the accompanying drawings; however, the embodiments shown may be embodied in different forms, and the present disclosure should not be construed as limited to the embodiments set forth below. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will enable those skilled in the art to fully understand the scope of the disclosure.
[0023] The accompanying drawings of the embodiments disclosed herein are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the detailed embodiments to explain this disclosure and do not constitute a limitation thereof. The above and other features and advantages will become more apparent to those skilled in the art from the description of the detailed embodiments with reference to the accompanying drawings.
[0024] This disclosure may be described with reference to plan and / or cross-sectional views using the ideal schematic diagrams of this disclosure. Therefore, the example illustrations may be modified according to manufacturing techniques and / or tolerances.
[0025] Where there is no conflict, the various embodiments of this disclosure and the features thereof in the embodiments may be combined with each other.
[0026] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to limit the disclosure. The term "and / or" as used in this disclosure includes any and all combinations of one or more of the associated enumerated entries. The singular forms "a" and "the" as used in this disclosure are also intended to include the plural forms, unless the context clearly indicates otherwise. The terms "comprising," "made of," etc., as used in this disclosure specify the presence of the stated feature, integral, step, operation, element, and / or component, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof.
[0027] Unless otherwise specified, all terms used in this disclosure (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined in this disclosure.
[0028] This disclosure is not limited to the embodiments shown in the accompanying drawings, but includes modifications to the configuration based on the manufacturing process. Therefore, the areas illustrated in the drawings are schematic, and the shapes of the areas shown illustrate specific shapes of the areas of an element, but are not intended to be limiting.
[0029] In related technologies, real-time indicator data of network elements are usually collected through command line interface (CLI). Although real-time indicator data can be collected, such as network packet loss, port signal-to-noise ratio, buffer utilization, and data packet transmission time of network interface, the indicator data output by command line interface is relatively scattered, lacks historical indicator data, and is difficult to analyze and judge the trend of indicator changes, thus making it difficult to detect potential risks and problems of network elements.
[0030] In related technologies, to improve the operation and maintenance level of network elements, performance indicator data of network elements can usually be collected through methods such as developing CLI commands, developing inspection tools, and managing network management interfaces. However, while developing CLI commands can achieve the output and display of various sets of operation and maintenance data of network elements, the development workload of CLI commands is large, and various commands are easily piled up, increasing the interface complexity of network elements and reducing the operation and maintenance efficiency of network elements. Developing inspection tools, which uses various scripts to obtain CLI output data and perform programming analysis, has the drawback that it can only perform case-by-case analysis, and the programming workload is large and the methods are not universal. The problem with managing through network management interfaces is that network management interfaces are relatively fixed and not easy to expand, and the network management system and network elements are not necessarily provided by the same vendor, and in many scenarios, there is no network management system.
[0031] Therefore, embodiments of this application provide a network data processing method, electronic device, computer-readable medium, and product, which aim to effectively improve the technical problems existing in the above-mentioned related technologies.
[0032] Please see Figure 1 , Figure 1 This illustration shows a system architecture diagram of a network data processing system provided in an embodiment of this application, such as... Figure 1 As shown, the network data processing system 100 may include a network element 101 and a network data processing device, which includes a data converter 102, a data collector 103 and a data analyzer 104.
[0033] Among them, network element 101 refers to an element or device in the network, also known as a network element (NE). Network element 101 can be a real physical network device or communication device, or a virtualized network application process, network service or microservice, etc. Network element 101 is responsible for outputting running data to data converter 102.
[0034] The data converter 102 is responsible for obtaining the operating data of the network element 101 from the network element 101 and converting the operating data of the network element 101 into time series data in a preset data format. The time series data includes indicators, the data values corresponding to the indicators, the labels of the indicators, and the timestamps.
[0035] The data collector 103 is responsible for acquiring the format-converted time series data from the data converter 102 and outputting the time series data to the time series database D for storage.
[0036] In some application scenarios, the data collector 103 can be a time-series data collection tool such as Prometheus or VictoriaMetrics application services. Prometheus is an open-source service monitoring system and time-series database, while VictoriaMetrics (VM) is a monitoring solution and time-series database that supports high availability, cost-effectiveness and scalability.
[0037] Data analyzer 104 is responsible for querying the time series data corresponding to the indicator to be analyzed in the time series database D, and performing indicator analysis on the indicator to be analyzed based on the time series data corresponding to the indicator to be analyzed. For example, it calculates the average rate at which a certain network interface sends data packets per second in the past 5 minutes.
[0038] In some application scenarios, the data analyzer 104 can use PromQL (Prometheus Query Language) or MetricsQL (Metrics Query Language) tools to query and analyze indicator data. Both PromQL and MetricsQL are query languages used for querying and analyzing time-series data; PromQL is provided by Prometheus, and MetricsQL is provided by VictoriaMetrics. Furthermore, the data analyzer 104 can also be used to display the queried and analyzed indicator data, set alarm detection, etc., for example, through a human-computer interface via a web page or an application (APP).
[0039] In some embodiments, one or more of the data converter 102, data collector 103 and data analyzer 104 described above may also be deployed in network element 101.
[0040] Based on the aforementioned network data processing system 100, this application proposes a network data processing method. Figure 2 This illustration shows a flowchart of a network data processing method provided in an embodiment of this application. Figure 2 As shown, the network data processing method in this application embodiment includes, but is not limited to, the following steps.
[0041] Step S21: Obtain the operating data of the network element.
[0042] In this embodiment of the application, the network element's operating data can be obtained by sending a request to the network element. The network element's operating data may include, but is not limited to, data generated by the network element's hardware, network, software, etc. during operation.
[0043] In some embodiments, the data converter can periodically initiate requests to network elements to obtain their operational data. The data acquisition cycle can be set according to actual needs. The data converter can connect to network elements through various connection methods, including Telnet-based, SSH-based, or other methods. Telnet is a member of the TCP / IP protocol suite and is the standard protocol and primary method for Internet remote login services. Secure Shell (SSH) is a protocol used for secure remote login and other secure network services on insecure networks.
[0044] Step S22: Convert the running data into time series data in a preset data format. The time series data includes indicators, the data values corresponding to the indicators, the labels of the indicators, and the timestamps.
[0045] The metrics can include hardware performance metrics, network performance metrics, software performance metrics, or other operational performance metrics of network elements that need to be collected. Metrics can be represented by a metric name. The corresponding data value of the metric is the measurement value that reflects the metric. The label of the metric is the identification information used to identify and index the data of the metric. The timestamp can be the collection timestamp of the operational data of the network element.
[0046] In this embodiment, the operational data of network elements can be processed to extract indicator-related data, including indicators, corresponding data values, and indicator tags. The collected indicator-related data and timestamps are converted into time-series data (also known as time series data) in a preset data format and stored according to the preset data format. The preset data format is a data format used to describe time-series data; that is, it defines how to export and use time-series data.
[0047] In some embodiments, after the data converter obtains the network element's operating data from the network element, it converts the network element's operating data into time series data in a preset data format.
[0048] Step S23: Output the time series data to the time series database for storage.
[0049] Time series databases (also known as time-series databases) are mainly used to process data with time labels (data that changes in chronological order, i.e., time-series data). Data with time labels is also called time series data.
[0050] In some embodiments, the data collector obtains time-series data in a preset data format from the data converter and stores it in a preset time-series database. In some embodiments, the data collector may periodically obtain time-series data in a preset data format from the data converter.
[0051] In some embodiments, the data converter outputs time-series data in a preset data format obtained through data conversion processing to the data collector, and the data collector stores the time-series data in a preset time-series database.
[0052] Step S24: Query the time series data corresponding to the indicator to be analyzed in the time series database.
[0053] Step S25: Perform indicator analysis on the indicator to be analyzed based on the time series data corresponding to the indicator to be analyzed.
[0054] In this embodiment of the application, when it is necessary to analyze a certain indicator of a network element, the time series data corresponding to the indicator to be analyzed can be queried from the time series database, and the indicator to be analyzed can be analyzed based on the time series data corresponding to the indicator. The indicator analysis is used to perform data analysis, statistics and processing on the data values corresponding to the indicator in the time series data corresponding to the indicator, so as to obtain the indicator analysis results of the required indicator to be analyzed.
[0055] For example, there can be multiple time series data corresponding to the indicator to be analyzed, each corresponding to multiple timestamps. In order to analyze the trend of the indicator, the trend of the indicator can be obtained by analyzing the data values of the indicator in the multiple time series data corresponding to the indicator.
[0056] In some embodiments, when it is necessary to analyze a certain indicator of a network element, the data analyzer queries the time series data corresponding to the indicator to be analyzed in the time series database, and performs indicator analysis on the indicator to be analyzed based on the time series data corresponding to the indicator to be analyzed.
[0057] According to the network data processing method of this application embodiment, the collected network element operation data is processed by data conversion to obtain time series data in a preset data format, and stored in a time series database. When indicator analysis is required, the time series data of the indicator to be analyzed is queried from the time series database and the indicator analysis is performed. Since the network element indicator data are all stored in a unified preset data format, the time series data of the indicator changing in chronological order can be queried from the time series database when performing indicator analysis. This is beneficial to analyze and judge the trend of indicator changes based on the time series data of the indicator changing in chronological order, and thus is beneficial to detect potential risks and existing problems of the network element.
[0058] In some embodiments, the step of obtaining the network element's operational data, i.e. step S21, may further include: sending a data acquisition command to the network element through a preset network management interface; and obtaining the operational data output by the network element through the network management interface.
[0059] In some embodiments, the network management interface includes any one of the following: Command Line Interface (CLI), SNMP (Simple Network Management Protocol) protocol interface, gRPC (Google Remote Procedure Call) protocol interface, and YANG model.
[0060] The command-line interface (CLI) is an interface used to manage and configure network devices. Through the CLI, administrators can interact with network devices using text commands to perform various operations, such as configuring network interfaces, viewing device status, and diagnosing problems. The CLI typically provides a rich set of commands, allowing administrators to flexibly manage and control devices.
[0061] The SNMP protocol is a set of network management protocols defined by the Internet Engineering Task Force (IETF). Using SNMP, a management workstation can remotely manage all network elements that support this protocol, including detecting network status, modifying network element configurations, and receiving network event warnings.
[0062] gRPC is a modern, open-source, high-performance remote procedure call (RPC) framework developed under the leadership of Google. It uses the HTTP / 2 protocol, supports multiple languages, and can efficiently connect to different services.
[0063] YANG (Yet Another Next Generation) is a data modeling language used to define hierarchical structures of data, particularly suitable for operations based on network configuration management protocols, including configuration, status data, remote procedure calls, and notifications.
[0064] Taking the command-line interface as an example, the aforementioned data acquisition command can be a SHOW command or a Diag command, or other commands predefined by the command-line interface. The SHOW command is used to view and display information such as the configuration, status, and logs of network elements. The Diag command is a commonly used diagnostic tool used to check the status of network elements and connections. By executing the Diag command, key information about the network element's hardware, software, interfaces, and protocols can be obtained. In some embodiments, the SHOW command, Diag command, or other predefined commands can be input through the network element's command-line interface to obtain the network element's operational data output by the command-line interface.
[0065] In some embodiments, the operation data of network elements can be collected through command line interface, SNMP protocol interface, gRPC protocol interface or YANG model. Data collection can be performed based on commands developed from the existing network management interface of the network element, without increasing the development workload of the network management interface itself.
[0066] In some embodiments, the step of converting the running data into time series data in a preset data format, i.e., step S22, may further include: extracting indicator data from the running data, wherein the indicator data includes an indicator, the corresponding data value of the indicator, the label of the indicator, and the timestamp; and converting the indicator data into time series data in a preset data format.
[0067] To facilitate data conversion and meet the requirements of a preset data format, in some embodiments, before data conversion, after obtaining the network element's operating data, indicator data is extracted from the operating data. The indicator data includes the indicator, the corresponding data value, the indicator's label, and the timestamp. The extracted indicator data is then converted into time series data in a preset data format.
[0068] In some embodiments, based on the obtained network element operation data, by selecting the indicator name of one or more indicators to be extracted, defining the indicator label as the indicator index, the indicator data of each indicator is extracted from the operation data, and then the extracted indicator data is converted into time series data in a preset data format.
[0069] In some embodiments, the preset data format may include the following information: information representing the indicator name, information representing the indicator type, information representing the indicator label, and information representing the data collection timestamp. The preset data format may be represented as follows:
[0070] #TYPE metric_name metric_type
[0071] metric_name{label1="val1",...}value timestamp
[0072] Wherein, metric_name represents the metric name, metric_type represents the metric type, label represents the metric label, value represents the metric data value, and timestamp represents the timestamp.
[0073] For example, the data format of time series data for an indicator can be represented as follows:
[0074] #HELP interface_status Interface status(1=up,0=down)
[0075] #TYPE interface_status gauge
[0076] interface_status{interface="GigabitEthernet0 / 0"}1
[0077] In this context, GigabitEthernet0 / 0 represents a GigabitEthernet interface of the network element. A GigabitEthernet interface is an interface used to connect devices, enabling data transmission and communication. It is numbered 0 / 0, with the first 0 indicating slot 0 and the second 0 indicating port 0. Of course, network elements have various network interfaces, such as Ethernet (1 / 2 / 3), etc. The numbers may represent racks, sub-cards, or ports, etc., and there are many types, which will not be listed here.
[0078] `interface_status` indicates the interface status, representing the metric name. "gauge" indicates the metric type. The GigabitEthernet interface number "GigabitEthernet0 / 0" represents the metric label, and "1" represents the metric's data value.
[0079] For example, the data format of time series data for an indicator can be represented as follows:
[0080] #HELP interface_inputrate Interface input rate over 5minutes
[0081] #TYPE interface_inputrate gauge
[0082] interface_inputrate{interface="GigabitEthernet0 / 0"}10000
[0083] Here, interface_inputrate is the interface input rate, represents the indicator name, "gauge" represents the indicator type, and the gauge type is an indicator type whose value can fluctuate arbitrarily, "GigabitEthernet0 / 0" represents the indicator label, and "10000" represents the indicator data value.
[0084] For example, the data format of time series data for an indicator can be represented as follows:
[0085] #HELP interface_outpackets Total packets transmitted
[0086] #TYPE interface_outpackets counter
[0087] interface_outpackets{interface="GigabitEthernet0 / 0"}1000
[0088] Here, interface_outpackets represents the number of data packets sent by the interface, indicates the metric name, "counter" indicates the metric type, the counter type is a cumulative data metric type, which represents a monotonically increasing counter, "GigabitEthernet0 / 0" represents the metric label, and "1000" represents the metric data value.
[0089] It should be noted that the embodiments of this application do not impose special restrictions on the representation of each data in the data format of time series data. It can be defined according to time needs. For example, the above-mentioned "GigabitEthernet0 / 0" format can also be represented as ""GigabitEthernet", slot="0", port="0".
[0090] In some embodiments, the same label can be used to identify and index the indicator data of the same network interface of the collected network element.
[0091] In some embodiments, the default data format includes the OpenMetrics data format. OpenMetrics is an open metric format and protocol for collecting, transmitting, and displaying system performance metrics and data. It aims to standardize metric collection and transmission in applications and system management, providing scalability, reliability, and interoperability. It is a project proposed and supported by the Cloud Native Computing Foundation (CNCF), which aims to provide a common, open standard for monitoring in cloud-native environments. OpenMetrics defines a unified data text format, a universal structured format that can represent various types of metric data. Its main components include metric name, label, value, and metric type, enabling different systems to record and transmit metric data in the same way.
[0092] In some embodiments, the step of outputting time series data to a time series database for storage, i.e. step S23, may further include: outputting time series data to a time series database for storage through a preset transmission method, wherein the preset method includes any one or more of the following: standard output method, message queue output method, pull output method, and push output method.
[0093] In some embodiments, time-series data of an indicator in a preset data format can be transmitted via a web page based on the HTTP protocol or as a static file.
[0094] Figure 3 This illustration shows a schematic diagram of time-series data in a preset data format carried through a web page, as shown in some embodiments of this application. After the data converter transforms the network element's operational data into time-series data in the preset data format, such as... Figure 3 As shown, the data converter can act as an HTTP server, i.e. a web server, providing a constantly refreshed web page to carry time-series data in a preset data format, which can be provided and presented whenever other applications or services access it.
[0095] In some embodiments, after the data converter converts the network element's operating data into time-series data in a preset data format, the time-series data can be stored as a static file.
[0096] In some embodiments, the time series database includes: the Prometheus database, or the VictoriaMetrics database, or any time series database that supports a preset data format.
[0097] In some embodiments, the step of querying the time series data corresponding to the indicator to be analyzed in the time series database, i.e., step S24, may further include: using the label of the indicator to be analyzed as an index to retrieve the time series data corresponding to the indicator to be analyzed in the time series database.
[0098] In some embodiments, a preset indicator analysis tool is used to retrieve the time series data corresponding to the indicator to be analyzed from the time series database, using the label of the indicator to be analyzed as an index.
[0099] In some embodiments, the step of performing indicator analysis on the indicator to be analyzed based on the time series data corresponding to the indicator to be analyzed, i.e. step S25, may further include: using a preset indicator analysis tool to perform indicator analysis on the indicator to be analyzed based on the time series data corresponding to the indicator to be analyzed.
[0100] Using the preset indicator analysis tools, you can query and perform advanced analysis on the time series data of the indicators to be analyzed in the time series database to obtain the required analysis results.
[0101] For example, the data format of time series data for an indicator can be represented as follows:
[0102] #HELP http_requests_total Total number of HTTP requests
[0103] #TYPE http_requests_total counter
[0104] http_requests_total{method="GET",status="200"}1345
[0105] http_requests_total{method="POST",status="200"}546
[0106] http_requests_total{method="GET",status="404"}23
[0107] Here, `http_requests_total` is the name of a metric that reflects the total number of HTTP requests. `method` and `status` are the metric labels, which can be used to indicate the status of HTTP requests. The following values represent the metric's data value. Metric analysis tools can be used to query and analyze the total number of HTTP requests. For example, to query and analyze the total number of successful HTTP requests: `sum(http_requests_total{status="200"})`.
[0108] For example, the data format of time series data for an indicator can be represented as follows:
[0109] #HELP cpu_temperature Current CPU temperature
[0110] #TYPE cpu_temperature gauge
[0111] cpu_temperature{core="0"}45.6
[0112] cpu_temperature{core="1"}46.2
[0113] cpu_temperature{core="2"}43.8
[0114] Here, `cpu_temperature` is the name of an indicator that reflects the temperature of each CPU core. This value can be high or low. `core` is the indicator's label, representing the number of different CPU cores. The following numbers represent the indicator's data value. Indicator analysis tools can be used to query and analyze CPU core temperature data. For example, to query and analyze the number of CPU cores with temperatures above a certain threshold: `count(cpu_temperature>50)`.
[0115] In some embodiments, the metrics analysis tool may include: PromQL tool, or MetricsQL tool, or other tools with advanced metrics analysis capabilities.
[0116] In some embodiments, querying and analyzing time-series data of the indicators to be analyzed in a time-series database using indicator analysis tools such as PromQL or MetricsQL enables advanced analysis of the indicators. This eliminates the need to develop inspection tools that require extensive programming, is easily extensible, has high development efficiency, high versatility, and does not require the deployment of network management interfaces, thus having low resource requirements. Furthermore, indicator analysis tools such as PromQL or MetricsQL can also visualize the raw time-series data and the analyzed data, and trigger alarms.
[0117] In some embodiments, after performing indicator analysis on the indicator to be analyzed based on the time series data corresponding to the indicator to be analyzed, the network data processing method further includes: visually displaying the query results and / or analysis results of the indicator to be analyzed through a visualization tool.
[0118] The following example uses PromQL as an indicator analysis tool to illustrate a scenario where advanced analysis is performed on one or more given indicators.
[0119] Scenario 1: Calculate the average rate at which the network interface labeled interface="GigabitEthernet", slot="0", and port="0" sends data packets per second over the past 5 minutes.
[0120] Using the PromQL tool, query the time series data of the indicators to be analyzed corresponding to the above labels, and perform the following indicator analysis operation: rate(interface_outpackets{interface="GigabitEthernet",slot="0",port="0"}[5m]), thereby analyzing the average rate at which the network interface labeled interface="GigabitEthernet",slot="0",port="0" sends data packets per second in the past 5 minutes.
[0121] Analysis Scenario 2: Calculate the instantaneous packet loss rate per second for the network interface labeled interface="GigabitEthernet", slot="0", port="0" in the past minute.
[0122] Using the PromQL tool, query the time series data of the indicators to be analyzed corresponding to the above labels, and perform the following indicator analysis operation: deriv(interface_droppackets{interface="GigabitEthernet",slot="0",port="0"}[1m]), thereby analyzing the instantaneous packet loss rate per second of the network interface with the labels interface="GigabitEthernet",slot="0",port="0" in the past minute.
[0123] Analysis Scenario 3: Based on the memory buffer usage data of the network interface labeled interface="GigabitEthernet", slot="0", port="0" in the last 10 minutes, a linear regression model is used to predict the memory buffer usage of the network interface in the next hour.
[0124] Using the PromQL tool, query the time series data of the indicators to be analyzed corresponding to the above labels, and perform the following indicator analysis operation: predict_linear(memory_buffer{interface="GigabitEthernet",slot="0",port="0"}[10m],3600), so that the memory buffer usage of the network interface with the label interface="GigabitEthernet",slot="0",port="0" can be predicted in the next hour.
[0125] Analysis Scenario 4: Joint analysis of multiple indicators. For example, for a network interface with the label interface="GigabitEthernet", slot="0", and port="0", the indicators to be analyzed for this network interface are interface_outpackets (the number of data packets sent by the interface) and interface_droppackets (the number of data packets dropped by the interface), and correlation analysis is performed.
[0126] Using the PromQL tool, query the time series data of the indicators to be analyzed corresponding to the above labels, and perform the following indicator analysis operations:
[0127] rate(interface_outpackets{interface="GigabitEthernet", slot="0", port="0"}[5m])
[0128] / rate(interface_droppackets{interface="GigabitEthernet", slot="0", port="0"}[5m])
[0129] This yields the ratio of the rate of change in the number of data packets sent to the rate of change in the number of data packets dropped for the network interface labeled "interface="GigabitEthernet", slot="0", port="0" over the past 5 minutes.
[0130] As can be seen from the above examples, after converting the collected network element data using a preset data format (such as the OpenMetrics data format), high-level language tools such as PromQL or MetricsQL can be used to perform multi-dimensional and advanced indicator analysis on the time series data in the preset data format. This fully utilizes the original time series data in the preset data format, explores the data potential, and improves the level of detecting and managing the status and performance of network elements.
[0131] It should be clarified that this application is not limited to the specific configurations and processes described in the above embodiments and shown in the figures. For the sake of convenience and brevity, detailed descriptions of known methods are omitted here and will not be repeated.
[0132] Figure 4 This is a block diagram illustrating the composition of an electronic device provided in an embodiment of this application.
[0133] like Figure 4 As shown, the electronic device includes: at least one processor 401, at least one memory 402, and one or more I / O interfaces 403. The one or more I / O interfaces 403 are connected between the processor 401 and the memory 402. The memory 402 stores one or more computer programs, which are executed by the at least one processor 401 to enable the at least one processor 401 to implement any of the network data processing methods described in the above embodiments.
[0134] Among them, the processor is a device with data processing capabilities, including but not limited to the central processing unit (CPU); the memory is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and flash memory (FLASH); the I / O interface (read-write interface) is connected between the processor and the memory, enabling information exchange between the memory and the processor, including but not limited to the data bus (Bus).
[0135] This application also provides a computer-readable medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements any of the network data processing methods described in the above embodiments.
[0136] This application also provides a computer program product, which includes a computer program that, when executed by a processor, implements any of the network data processing methods described in the above embodiments.
[0137] Those skilled in the art will understand that all or some of the steps, systems, and devices disclosed above, as functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof.
[0138] In hardware implementations, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be executed by several physical components working together.
[0139] Some or all of the physical components may be implemented as software executed by a processor, such as a central processing unit (CPU), digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technique for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH) or other disk storage; read-only optical disc (CD-ROM), digital versatile disc (DVD) or other optical disc storage; magnetic cartridges, magnetic tapes, disk storage or other magnetic storage; and any other media that can be used to store desired information and can be accessed by a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.
[0140] This disclosure has disclosed exemplary embodiments, and although specific terminology has been used, it is for general illustrative purposes only and should not be construed as limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in conjunction with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in conjunction with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this disclosure as set forth by the appended claims.
Claims
1. A method for processing network data, comprising: Obtain operational data of network elements; The operational data is converted into time series data in a preset data format. The time series data includes indicators, corresponding data values of the indicators, labels of the indicators, and timestamps. The time series data is output to a time series database for storage; Query the time series data corresponding to the indicator to be analyzed in the time series database; The indicator to be analyzed is analyzed based on the time series data corresponding to the indicator to be analyzed.
2. The network data processing method according to claim 1, wherein, The acquisition of network element operational data includes: Data acquisition commands are sent to the network element through a preset network management interface; Obtain the operational data output by the network element through the network management interface.
3. The network data processing method according to claim 2, wherein, The network management interface includes any one of the following: command line interface, SNMP protocol interface, gRPC protocol interface, and YANG model.
4. The network data processing method according to claim 1, wherein, The step of converting the running data into time series data in a preset data format includes: Extract indicator data from the operational data, wherein the indicator data includes the indicator, the corresponding data value of the indicator, the label of the indicator, and the timestamp; The indicator data is converted into time series data in a preset data format.
5. The network data processing method according to claim 4, wherein, The preset data format includes the OpenMetrics data format.
6. The network data processing method according to claim 1, wherein, The step of outputting the time series data to a time series database for storage includes: The time series data is output to the time series database for storage using a preset transmission method, which includes any one or more of the following: standard output method, message queue output method, pull output method, and push output method.
7. The network data processing method according to claim 1, wherein, The time series databases include: the Prometheus database, or the VictoriaMetrics database.
8. The network data processing method according to claim 1, wherein, The querying of the time series data corresponding to the indicator to be analyzed in the time series database includes: Using the label of the indicator to be analyzed as an index, the time series data corresponding to the indicator to be analyzed is retrieved from the time series database.
9. The network data processing method according to claim 1, wherein, The step of performing indicator analysis on the indicator to be analyzed based on the time series data corresponding to the indicator to be analyzed includes: Using a preset indicator analysis tool, the indicator to be analyzed is analyzed based on the time series data corresponding to the indicator to be analyzed.
10. The network data processing method according to claim 9, wherein, The indicator analysis tools include: PromQL or MetricsQL.
11. The network data processing method according to claim 1, wherein, After performing indicator analysis on the indicator to be analyzed based on the time series data corresponding to the indicator to be analyzed, the network data processing method further includes: The query results and analysis results of the indicators to be analyzed are visualized using visualization tools.
12. An electronic device, wherein, include: One or more processors; A memory having stored thereon one or more computer programs that, when executed by one or more processors, cause the one or more processors to implement the method as described in any one of claims 1 to 11.
13. A computer-readable medium, wherein, The computer-readable medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 11.
14. A computer program product comprising a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 11.