Intelligent computing application deployment-oriented network topology three-dimensional feature map extraction method

Abstract

The invention belongs to the technical field of multi-mode intelligent computing networks, and particularly relates to an intelligent computing application deployment-oriented network topology three-dimensional feature map extraction method, which comprises the following steps of: deploying a monitoring agent on each node; deploying a main control system on the main node, and setting storage, transfer and calculation three-dimensional monitoring indexes; each monitoring agent samples monitoring data of the nodes and reports the sampled data to the main control system; the master control system summarizes the data, constructs an application feature graph, and visually displays the performance conditions of each node in three dimensions of network conversion and storage computation; and the main control system analyzes the application feature map and identifies key features. According to the method, key features of specific applications are extracted according to different service scenes, including feature information of three dimensions of network, storage and forwarding, and application feature analysis is performed by adopting an advanced feature extraction technology, so that more accurate and more comprehensive application feature information is provided for network management personnel, and the efficiency of application feature management and analysis can be remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of multimodal intelligent computing networks, and in particular, relates to a method for extracting three-dimensional feature graphs of network topology for intelligent computing application deployment. Background Art

[0002] With the continuous advancement of the construction of Digital China, simply relying on network speed-up can no longer meet the diversified and high-quality network carrying requirements of vertical industry applications for ubiquitous interconnection of people, machines and things. The existing digital infrastructure has three major pain points when facing vertical industry applications. The first is the contradiction between the demand for integrated services and the coexistence of multiple network systems. The second is the contradiction between the demand for elastic data processing and the solidified bandwidth service. The third is the contradiction between the extreme performance demand and the discrete supply of resources. Traditional networks are difficult to guarantee the intelligent computing needs of applications. It is necessary to build a new modality-based intelligent computing network based on the underlying network architecture to release the potential of network services.

[0003] The new intelligent computing network is no longer composed of a single network mode, but is jointly constructed by multiple network modes. Different modes serve different application scenarios and industries. Their characteristics are different, making it difficult to conduct unified control and resource management. In this context, in view of the differentiated and diversified demand characteristics of intelligent computing applications in vertical industries, in order to break through the bottleneck of single solidification of traditional network services and network structures, it is urgent to study the extraction method of network topology three-dimensional feature maps for intelligent computing application deployment, lay the foundation for the construction of optimized adaptation of mapping differentiated application features to modal space, and form a multi-modal intelligent computing elastic networking architecture that supports deep integration of cloud and network through the perception and measurement of cloud network / computing network resources, so as to ensure the provision of intelligent, efficient and stable network service capabilities for diversified vertical industries.

[0004] The field of multimodal networks has only been around for a short time. At present, traditional application feature extraction methods are often limited by the ability to extract features in a single dimension when faced with complex and diverse network applications and services, and are unable to adapt to the extraction needs of differentiated features between different modalities. There are relatively few existing intelligent computing application feature extraction methods for storage, transfer, and computing three-dimensional features, and most of them only focus on network forwarding dimension features, such as network service quality, bandwidth, latency, etc., without paying attention to other dimensional features such as storage and computing. The extracted features are one-sided and cannot fully characterize the characteristics of various types of modalities. In addition, the final output results of many existing methods are in text form, and no visual structured output is provided. The modal feature information is not intuitive enough, making it difficult to provide network managers with intuitive, accurate, and comprehensive application features. Summary of the invention

[0005] The technical problem to be solved by the present invention is to overcome the one-sidedness and non-intuitiveness of the prior art in feature extraction, and propose a three-dimensional feature map extraction method for network topology for intelligent computing application deployment. By extracting features in three dimensions of storage, forwarding and calculation, a comprehensive and accurate description of application features is achieved, and feature information is made easier to understand and apply through visualization.

[0006] In order to solve the above technical problems, the present invention proposes a method for extracting a three-dimensional feature map of network topology for intelligent computing application deployment, comprising the following steps:

[0007] Step 1: Deploy a monitoring agent on each infrastructure node in the cloud or cluster, set the monitoring agent as a daemon, and configure the monitoring frequency and data transmission protocol of each monitoring agent;

[0008] Step 2: Select a node as the master node, deploy the main control system on the master node, set monitoring indicators through the main control system, the monitoring indicators include node network forwarding indicators, node storage indicators, node computing indicators, and then send the monitoring indicators to each monitoring agent;

[0009] Step 3: Each monitoring agent samples monitoring data of the node according to the settings of the monitoring indicators, and reports the basic information of the node and the monitoring indicator data to the main control system;

[0010] Step 4: The main control system summarizes the data, and after data cleaning and preprocessing, constructs an application feature graph to intuitively display the performance of each node in the three dimensions of network forwarding, storage, and computing.

[0011] Preferably, after step 4, there is further provided step 5: the main control system analyzes the application feature graph, identifies key features, and displays them on the application feature graph, wherein the key features are reference data for optimizing network service performance.

[0012] Specifically, step 4 includes the following sub-steps:

[0013] Step 4-1: The master node aggregates the monitoring data received from all nodes, and performs preliminary data cleaning and preprocessing to remove blank data;

[0014] Step 4-2: For the cleaned and preprocessed data, based on the basic information of each node, including node name, geographic location, and node type, use the adjacency matrix to build the application feature graph node graph, and build edges in the graph between node pairs with traffic;

[0015] Step 4-3: On the application feature node graph, the corresponding storage and calculation dimension indicator data is used as the dynamically changing attribute of the node, and the indicator data of the network forwarding dimension is used as the dynamically changing attribute of the edge.

[0016] Specifically, step 5 includes the following sub-steps:

[0017] Step 5-1: The main control system obtains monitoring data of all nodes from the database, including historical data of various indicators;

[0018] Step 5-2: The main control system selects a certain time window and calculates the average value of the historical data of each indicator within the time window;

[0019] Step 5-3: For each indicator, the main control system calculates the difference between its current value and the average value, that is, the standard deviation;

[0020] Step 5-4: Compare the standard deviation with the preset threshold. If the standard deviation exceeds the threshold, the indicator is considered abnormal.

[0021] Step 5-5: Based on step 5-4, if an abnormal indicator is detected, the characteristic dimension to which the indicator belongs is listed as a key characteristic dimension, and the abnormal indicator and the key characteristic dimension are presented to the operation and maintenance personnel to provide information support for the operation and maintenance personnel to perform application maintenance.

[0022] The beneficial effects of the present invention are as follows:

[0023] This patent proposes a method for extracting three-dimensional feature maps of network topology for intelligent computing application deployment, setting a variety of different types of measurement indicators to extract key features of specific applications according to different service scenarios, including feature information in three dimensions: network, storage, and forwarding. This method is no longer limited to feature extraction in a single network dimension, but performs storage-forwarding and computing three-dimensional feature analysis and extraction based on specific applications, thereby providing network managers with more accurate and comprehensive application feature information, which can significantly improve the efficiency of application feature management and analysis.

[0024] The present invention also designs a visual structured output for the extracted feature information, that is, displaying the application feature information in the form of an application feature graph to help technical personnel intuitively understand the focus of the feature.

[0025] The feature extraction framework designed in the present invention has good scalability, can adapt to changes in future network technologies and application scenarios, and is convenient for subsequent technology upgrades and function expansions.

[0026] The present invention has good real-time performance, can respond to changes in application features in real time, accelerate the process of feature recognition and matching, and improve system availability. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0028] Figure 1This is an overall flow chart of the network topology three-dimensional feature map extraction method for intelligent computing application deployment of the present invention.

[0029] Figure 2 A topological diagram of the application characteristic graph generated for monitoring in the present invention.

[0030] Figure 3 Schematic diagram of the adjacency matrix used to construct the application feature graph.

[0031] Figure 4 Schematic diagram of key feature identification of application feature graph. DETAILED DESCRIPTION

[0032] Combination Figure 1 As shown, the present invention proposes a method for extracting a three-dimensional feature graph of network topology for intelligent computing application deployment, comprising the following steps:

[0033] Step 1: Deploy a monitoring agent on each infrastructure node in the cloud or cluster, set the monitoring agent as a daemon, and configure the monitoring frequency and data transmission protocol of each monitoring agent;

[0034] Step 2: Select a node as the master node, deploy the main control system on the master node, set monitoring indicators through the main control system, the monitoring indicators include node network forwarding indicators, node storage indicators, node computing indicators, and then send the monitoring indicators to each monitoring agent;

[0035] Step 3: Each monitoring agent samples monitoring data of the node according to the settings of the monitoring indicators, and reports the basic information of the node and the monitoring indicator data to the main control system;

[0036] Step 4: The main control system summarizes the data, and after data cleaning and preprocessing, builds an application feature graph to intuitively display the performance of each node in the three dimensions of network forwarding, storage, and computing;

[0037] Step 5: The main control system analyzes the application feature graph, identifies key features, and displays them on the application feature graph. The key features are reference data for optimizing network service performance.

[0038] Specifically, step 1 includes the following sub-steps:

[0039] Step 1-1: Obtain administrator privileges for the cloud or cluster where the vertical industry application is deployed in advance, prepare the monitoring agent software in advance, and ensure that it can run as a daemon process on each infrastructure node and automatically start the monitoring agent software when the node is restarted;

[0040] Step 1-2: Install and configure the monitoring agent on each infrastructure node to ensure that it can collect and transmit data normally;

[0041] Step 1-3: Configure the monitoring frequency and data transmission protocol of the monitoring agent. In theory, the higher the monitoring frequency, the better the real-time performance. However, too high a monitoring frequency will occupy node resources and affect node performance. Therefore, the present invention selects a monitoring frequency of seconds, which is neither too high to occupy node resources nor sufficient to meet real-time requirements. Therefore, the present invention has good real-time performance, can respond to changes in application features in real time, accelerate the process of feature identification and matching, and improve system availability. At the same time, for the data transmission protocols of each monitoring agent, the present invention does not require the use of a unified data transmission protocol, so the present invention has scalability, can adapt to changes in future network technologies and application scenarios, and is convenient for subsequent technical upgrades and functional expansions.

[0042] The monitoring indicators set in step 2 include node network forwarding indicators, node storage indicators, and node computing indicators, including:

[0043] Step 2-1: Set monitoring indicators for network forwarding dimensions and monitor the traffic throughput indicators of each node, including node-to-node bandwidth, request sending rate, packet loss rate, average response delay, and long-tail delay;

[0044] Step 2-2: Set monitoring indicators for storage dimensions, including storage capacity utilization and I / O operation frequency;

[0045] Step 2-3: Set the monitoring indicators of the computing dimension, including CPU utilization, GPU utilization, and memory usage.

[0046] Step 4 includes the following sub-steps:

[0047] Step 4-1: The master node aggregates the monitoring data received from all nodes and performs preliminary data cleaning and preprocessing. Since each monitoring agent transmits data using different transmission protocols, the resulting monitoring data files are in different formats, such as json, protobuf, xml, etc. They need to be uniformly converted and saved in the local time series database influxDB. For example, the storage and forwarding feature data table is shown in Table 1, and the network forwarding feature data table is shown in Table 2.

[0048] Table 1 Store and forward feature data table

[0049]

[0050] Table 2 Network forwarding characteristics data table

[0051]

[0052] Due to network fluctuations and insufficient synchronization accuracy of node sampling time, some indicators may not have data sampled, and this part of the data is marked as zero. For example, the GPU utilization of node B in Table 1 is marked as zero.

[0053] Step 4-2: For the cleaned and preprocessed data, based on the basic information of each node, including node name, geographic location, and node type, use the adjacency matrix to store the graph node data and node traffic relationship. Build the application feature graph node graph based on the adjacency matrix, and build the edges in the graph between the node pairs with traffic. The topological diagram of the application feature graph is as follows: Figure 2 The corresponding adjacency matrix is ​​shown as Figure 3 As shown. Figure 2 In the figure, each node is marked with a box, which contains basic information about the node, including the node name, geographical location (i.e., the availability zone where the node is located), and node type (i.e., the specification of the node, ecs represents the cloud server, and u1 is the specification number of the cloud server). The edges in the figure are marked with arrows, and the edge numbers are displayed near them. Figure 3 The adjacency matrix in contains a vertex table and an edge table. The vertex table stores node information. The fields node, nodedata, and firstarc represent the node name, other node information, and the position of the first neighbor node, respectively. The edge table stores edge information. The fields adjvex, edgedata, and nextarc represent the sequence number of the neighbor node, the number information of the edge, and the position of the next neighbor node, respectively.

[0054] Step 4-3: On the application feature node graph, use the corresponding storage and calculation dimension indicator data as the dynamically changing attribute of the node, and use the network forwarding dimension indicator data as the dynamically changing attribute of the edge. Click a node or edge on the application feature graph to view its attributes.

[0055] Step 5: The main control system analyzes the application feature graph, identifies key features, and displays them on the application feature graph. The key features are reference data for optimizing network service performance.

[0056] The main control system analyzes the application feature graph, uses an abnormal point recognition algorithm to analyze the historical data of various indicators, identifies abnormal indicators, and uses the characteristic dimension to which the abnormal indicators belong as the key feature. There are many options for specific abnormal point recognition algorithms, such as threshold-based standard deviation detection methods, machine learning or deep learning-based methods. The threshold-based standard deviation monitoring method detects anomalies by calculating the average value over a period of time and comparing the difference between the current value and the average value with the threshold. The machine learning or deep learning-based method predicts future values ​​by using models such as long short-term memory networks and detects the deviation between the actual value and the predicted value.

[0057] This specific implementation method provides a specific implementation sub-step of step 5, including:

[0058] Step 5-1: Based on step 4-1, the main control system obtains monitoring data of all nodes from the database, including historical data of various indicators;

[0059] Step 5-2: The main control system selects a time window, such as a five-minute time window, for the historical data of each indicator and calculates the average value of each indicator within the time window. For example, calculate the average value of the CPU utilization indicator in the past five minutes;

[0060] Step 5-3: The main control system calculates the difference between the current value and the average value, i.e. the standard deviation, for the historical data of each indicator;

[0061] Step 5-4: Compare the standard deviation with the preset threshold. If the standard deviation exceeds the threshold, the indicator is considered abnormal. For example, set the percentage abnormal threshold to 100%, the time window length to five minutes, and calculate the average CPU utilization in the time window to be 30%. For example, the latest CPU utilization monitored is 90%, the standard deviation is 90%-30%=60%, which is greater than the CPU utilization threshold of 30%*100%=30%, then the CPU utilization indicator is considered abnormal.

[0062] Step 5-5: Based on step 5-4, if an abnormal indicator is detected, refer to step 2 to list the characteristic dimension of the indicator as the key characteristic dimension, and present the abnormal indicator and key characteristic dimension to the operation and maintenance personnel to provide information support for maintenance, such as Figure 4 As shown, the network forwarding bandwidth between node A and node B is abnormal.

[0063] These key features can provide a reference for operation and maintenance personnel to optimize the network. Based on the feedback from the application feature graph, operation and maintenance personnel can continuously adjust monitoring strategies and technical solutions to continuously optimize network service performance.

[0064] Finally, it should be noted that the above specific implementation methods are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

Claims

1. A method for extracting three-dimensional feature graphs of network topology for intelligent computing application deployment, characterized in that: The following steps are involved: Step 1: Deploy a monitoring agent on each infrastructure node in the cloud or cluster, set the monitoring agent as a daemon, and configure the monitoring frequency and data transmission protocol of each monitoring agent; Step 2: Select a node as the master node, deploy the main control system on the master node, set monitoring indicators through the main control system, the monitoring indicators include node network forwarding indicators, node storage indicators, node computing indicators, and then send the monitoring indicators to each monitoring agent; Step 3: Each monitoring agent samples monitoring data of the node according to the settings of the monitoring indicators, and reports the basic information of the node and the monitoring indicator data to the main control system; Step 4: The main control system summarizes the data, and after data cleaning and preprocessing, constructs an application feature graph to intuitively display the performance of each node in the three dimensions of network forwarding, storage, and computing.

2. The method for extracting a three-dimensional feature graph of network topology for intelligent computing application deployment according to claim 1, characterized in that: After step 4, there is also step 5: The main control system analyzes the application feature graph, identifies key features, and displays them on the application feature graph. The key features are reference data for optimizing network service performance.

3. The method for extracting a three-dimensional feature map of network topology for intelligent computing application deployment according to claim 1 or 2, characterized in that: The step 4 includes the following sub-steps: Step 4-1: The master node aggregates the monitoring data received from all nodes, and performs preliminary data cleaning and preprocessing to remove blank data; Step 4-2: For the cleaned and preprocessed data, based on the basic information of each node, including node name, geographic location, and node type, use the adjacency matrix to build the application feature graph node graph, and build edges in the graph between node pairs with traffic; Step 4-3: On the application feature node graph, the corresponding storage and calculation dimension indicator data is used as the dynamically changing attribute of the node, and the indicator data of the network forwarding dimension is used as the dynamically changing attribute of the edge.

4. According to claim 3, the method for extracting three-dimensional feature graphs of network topology for intelligent computing application deployment is characterized in that: The step 5 comprises the following sub-steps: Step 5-1: The main control system obtains monitoring data of all nodes from the database, including historical data of various indicators; Step 5-2: The main control system selects a certain time window and calculates the average value of the historical data of each indicator within the time window; Step 5-3: For each indicator, the main control system calculates the difference between its current value and the average value, that is, the standard deviation; Step 5-4: Compare the standard deviation with the preset threshold. If the standard deviation exceeds the threshold, the indicator is considered abnormal. Step 5-5: If an abnormality is detected in a certain indicator, the characteristic dimension to which the indicator belongs is listed as a key characteristic dimension, and the abnormal indicator and the key characteristic dimension are presented to the operation and maintenance personnel to provide information support for the operation and maintenance personnel to perform application maintenance.

Images