Graph-based heterogeneous network ICT resource analysis method

Abstract

The invention discloses a graph-based heterogeneous network ICT resource analysis method, and belongs to the technical field of information resource analysis, and the method comprises the steps: grouping ICT resources based on different types of the ICT resources, and constructing a graph-based resource model; installing software; according to different resource types, deploying an acquisition plug-in to a host in the heterogeneous network; iCT resource information is automatically collected and stored in a time sequence database by running a collection plug-in; performing data conversion and cleaning on the acquired data information; storing the cleaned data information in a relational database, and establishing a data management system for management and maintenance; on the basis of the resource model and the collected data, ICT resources are analyzed and optimized by using a graph computing technology; iCT resources on a heterogeneous network are detected in real time through a network scanning tool, and the resources are automatically found and managed. The data conversion and cleaning technology is adopted, and the accuracy and reliability of collected data are ensured.

Description

Technical Field

[0001] The present invention belongs to the technical field of information resource analysis, and in particular relates to a graph-based heterogeneous network ICT resource analysis method. Background Art

[0002] With the rapid development of information and communications technology (ICT), the network environment is becoming increasingly complex. The network resources on the project exist in private clouds, public clouds and hybrid clouds at the same time. Heterogeneous networks have become an important part of modern communication systems. However, the ICT resources (such as computing resources, storage resources, network resources, etc.) in heterogeneous networks are of various types and complex structures, which brings huge challenges to the management and analysis of resources. How to effectively and automatically collect information about these resources has become an urgent problem to be solved. Traditional resource analysis methods are often unable to cope with the complexity and dynamics of heterogeneous networks and require manual intervention, resulting in low resource utilization and high management costs.

[0003] And the prior art also has the following defects:

[0004] 1. Low collection efficiency: Traditional ICT resource data collection is done manually or semi-automatically. Manual binding of the relationship between resources requires a lot of manpower and time. Especially when dealing with large-scale and complex heterogeneous network environments, manual collection is prone to omissions and errors, and is less efficient.

[0005] 2. No support for cross-network environment collection: Traditional collection tools have limited support for complex heterogeneous network environments and can only collect resource data in the same network environment, making it difficult to meet the business needs of network resource collection in complex environments.

[0006] 3. Lack of automatic update mechanism: The ICT network environment is dynamically changing, and ICT resources and relationships are frequently updated as network topology is adjusted and business needs change. Traditional collection methods often make it difficult to track and update resource network topology maps in a timely manner, resulting in inconsistency between resource network topology maps and actual conditions.

[0007] In view of the above existing business needs and technical problems, the existing collection methods are difficult to meet the large-scale and dynamically changing ICT resource management needs. Therefore, it is necessary to develop a method that can automatically and efficiently collect and analyze ICT resource information in a heterogeneous network environment. Summary of the invention

[0008] The present invention aims at solving the problems in the prior art and provides a graph-based heterogeneous network ICT resource analysis method.

[0009] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0010] A graph-based heterogeneous network ICT resource analysis method includes the following steps: step 1, based on different types of ICT resources, ICT resources are grouped and a graph-based resource model is constructed; step 2, software is installed; step 3, according to different resource types, collection plug-ins are deployed on hosts in the heterogeneous network; step 4, data information of ICT resources is collected by running the collection plug-in and stored in a time series database; step 5, data conversion and cleaning are performed on the collected data information; step 6, the cleaned data information is stored in a relational database, and a data management system is established for management and maintenance; step 7, based on the resource model and the collected data, the graph computing technology is used to analyze and optimize the ICT resources; step 8, the ICT resources on the heterogeneous network are detected in real time by a network scanning tool, and the resources are automatically discovered and brought into management.

[0011] Based on the above technical solution, further, in step 1, a graph database is used to store the resource model, and the ICT resources in the heterogeneous network are abstracted into nodes and edges in the graph resource model, wherein the nodes represent various ICT resources; and the edges represent the connection relationships and dependency relationships between various ICT resources.

[0012] Based on the above technical solution, further, in step 2, the installed software includes Agent and Proxy.

[0013] Based on the above technical solution, further, in step 3, the collection plug-in includes but is not limited to: NodeExporter, used to collect host hardware information and operating system information; Nginx Exporter, used to collect Nginx indicator data; Redis Exporter, used to collect various performance indicators of Redis, SNMP Exporter, used to collect SNMP client data through SNMP protocol.

[0014] Based on the above technical solution, further, in step 5, ETL technology is used for data conversion and cleaning, and DataX tool is used. DataX includes but is not limited to the following processing: Data mapping: supports custom data mapping rules, and can convert data from one form to another according to needs; Field calculation: complex calculation logic can be defined to process and convert data; Data aggregation: provides data aggregation function, which can merge and summarize data from multiple data sources according to specific rules; Outlier detection and processing: provides a variety of outlier detection methods, such as statistical threshold detection, clustering-based outlier identification, etc.; Data deduplication: provides data deduplication function, which can identify and delete duplicate records based on specific fields (such as primary keys) to avoid data redundancy from interfering with analysis results; Format standardization: data in different formats can be uniformly converted into a standard format to ensure data consistency and comparability.

[0015] Based on the above technical solution, further, in step 6, the data management system at least includes operations of data backup, recovery and update.

[0016] Based on the above technical solution, further, in step 7, different nodes and connection relationships constitute a complex graph structure, and these nodes and connection relationships are stored and managed in the form of a graph database.

[0017] Based on the above technical solution, further, in step 7, the heterogeneous network ICT resource analysis process is: using a graph traversal algorithm to traverse and extract nodes and edges, wherein the graph traversal algorithm includes a breadth-first search and a depth-first search.

[0018] Based on the above technical solution, further, in step 8, the ICT resources on the network are scanned by the nmap network scanning tool. When new ICT resources are detected, the key features of the resources are matched with the feature library to identify potential resources and include them in asset management; for ICT resources whose key features are not identified, they are classified as unknown resources by default and manually checked and verified.

[0019] Based on the above technical solution, further, security and privacy protection operations are performed during the entire ICT resource analysis process, and the security and privacy protection operations at least include data encryption transmission, access control and anonymization processing.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] The present invention builds a graph-based resource model that is applicable to ICT resources of various types and relationships, enhancing its applicability and flexibility in complex heterogeneous network environments; and adopts data conversion and cleaning technology to ensure the accuracy and reliability of collected data; it also analyzes ICT resources through graph computing technology, identifies key resources, bottleneck resources, redundant resources, etc. in the network, and formulates corresponding resource optimization strategies; it also automatically discovers resources through network scanning tools, identifies potential resources and incorporates them into asset management; and it also adopts OAuth2 to ensure the security of ICT resource information and the protection of user privacy. The present invention as a whole involves automated collection that significantly reduces manual intervention and improves the efficiency and accuracy of resource information collection; and it has good flexibility and scalability, and can be customized and optimized according to actual needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 The present invention is a flow chart of the analytical method;

[0023] Figure 2 A schematic diagram of ICT resources and relationships of the present invention;

[0024] Figure 3 A schematic diagram of installing software for the present invention;

[0025] Figure 4 This is an ICT resource topology diagram in an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The present invention is further described and illustrated below in conjunction with the accompanying drawings and specific embodiments. The technical features of each embodiment of the present invention can be combined accordingly without conflicting with each other.

[0027] In order to make the above-mentioned purpose, features and advantages of the present invention more obvious and easy to understand, the specific implementation mode of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The technical features in each embodiment of the present invention can be combined accordingly without conflicting with each other.

[0028] Example

[0029] Combination Figure 1 As shown, this embodiment provides a graph-based heterogeneous network ICT resource analysis method, including the following steps:

[0030] Step 1: Group ICT resources based on their different types and build a graph-based resource model; and manage these resource models through the configuration platform.

[0031] In this embodiment, the Neo4j graph database storage resource model is used to abstract the ICT resources in the heterogeneous network into nodes and edges in the graph model. Nodes represent various ICT resources, such as servers, storage devices, switches, multiple databases, and middleware; edges represent the connection relationship and dependency relationship between resources. The relationship of edges includes but is not limited to belonging to, composed of, deployed on, connected to, and running on. Edges have directionality to reflect the direction of data flow. In this way, the complex heterogeneous network structure can be simplified into a clear graph structure.

[0032] For example, taking the ICT resource management of a certain project as an example, the resources on the project include hosts, switches, firewalls, databases, etc. Different resource types have different corresponding attributes, as shown in Table 1 below.

[0033] Table 1

[0034] Resource Type property Host Name, manufacturer, IP address, operating system, number of CPU cores, memory size switch Asset number, name, manufacturer, device model, IP address middleware Middleware type, instance name, IP address, port number, version Firewall Asset number, name, manufacturer, device model, IP address

[0035] Establish connections between different ICT resources. For example, the middleware Nginx runs on the host. That is, the relationship between Nginx and the host is "running on". Figure 2 As shown, for Figure 2 The ICT resources and relationships are stored in the graph resource model, and the nodes and edges are designed accordingly. For example, the design of the node mainly includes the following attributes: node ID, label Lable, node name Name, where the label is used to identify the same ICT resource type. It should be noted that the storage structure corresponding to Table 2 below can be expressed as: (Node{ID:1, Label:"Host", Name:"Host1"}.

[0036] Table 2

[0037] ID Label Name 1 Host Host 1 2 switch Switch 1 3 Firewall Firewall 1 4 middleware Nginx1

[0038] For another example, for edge design, the edge represents the relationship between nodes, and the relationship between ICT resources includes belonging to, composition, topological composition, running on, etc. Among them, the attributes of the edge mainly include: start node StartNode, end node EndNode, relationship name RelationName, weight Weight, as shown in Table 3 below. It should be noted that the storage structure corresponding to Table 3 below can be expressed as: (Edge{ID:1,StartNode:"Host 1",EndNode:"Switch 1",RelationName:"Connection",Weight:"0.5"}).

[0039] Table 3

[0040] ID StartNode EndNode RelationName Weight 1 Host 1 Switch 1 connect 0.5 2 Nginx1 Host 1 Run on 0.6

[0041] Step 2, install the software; specifically, the installed software includes Agent and Proxy, where Agent is an agent program deployed on the target host. After installing Agent on the target host, the host can be managed and controlled through the main program, which can include file distribution, job execution, data reporting, basic information collection, etc. In the non-directly connected control area, you need to install Proxy so that the main program can connect to the host in this area for management and control. Proxy is a transit service program used to connect the transit service programs of two non-directly connected control areas. It should be noted that the host where the Proxy is located needs to have a public IP. If you consider load balancing and disaster recovery, you can also install multiple Proxies in one control area.

[0042] In this embodiment, in a complex network environment, when it is necessary to manage and control the isolated network area hosts in a private cloud or public cloud, the management can be performed through a one-level or multi-level proxy cascade method, such as Figure 3 As shown, the main application program is deployed in cloud region A, but needs to manage the host in cloud region C. There is a situation where cloud region A and cloud region C cannot communicate with each other. In this case, it is necessary to install a multi-level proxy transfer service program for communication.

[0043] Step 3: Deploy the collection plug-in to the host in the heterogeneous network according to different resource types, where the host can be a physical machine or a virtual machine.

[0044] In this embodiment, the collection tool is Prometheus, which is an open source monitoring collection tool with multiple built-in collection plug-ins and supports custom collection plug-ins. The collection plug-ins include but are not limited to: Node Exporter, which is used to collect host hardware information and operating system information; Nginx Exporter is used to collect Nginx indicator data; Redis Exporter is used to collect various performance indicators of Redis, and SNMP Exporter is used to collect SNMP client data (such as switches, routers, firewalls, etc.) through the SNMP protocol. It has at least the following core features:

[0045] 1. Multidimensional data model: Prometheus uses a time series database as storage, which can flexibly store multi-dimensional data.

[0046] 2. Flexible query language: Prometheus uses the powerful PromQL query language, which can query and aggregate time series data in real time.

[0047] 3. Pull-based collection: Prometheus periodically captures the status of monitored components through the HTTP protocol instead of receiving push data through the port.

[0048] 4. Service discovery: Prometheus supports various service discovery mechanisms and can automatically discover monitoring targets. If there are relatively few services to be monitored, static configuration can also be used.

[0049] 5. Visualization component: Use Grafana for visualization to display monitoring data.

[0050] 6. Alarm management: The alarm function is implemented through Alertmanager. You can use email or customize the alarm processing method through Webhook.

[0051] Step 4: Automatically collect ICT resource information and store it in a time series database by running the collection plug-in. Specifically, different collection plug-ins collect corresponding types of resource data, and it is necessary to save the collected data at different times. This technology uses the InfluxDB time series database, which is mainly used to process data with time tags (changes in time sequence, i.e., time serialization).

[0052] Step 5: Convert and clean the collected data information; specifically, because the data formats and structures of different data sources may be different, data conversion is required to convert the collected data into a standard format for storage. At the same time, the data needs to be cleaned and filtered to remove duplicate, erroneous or invalid data. The main reasons for invalid data include incorrect data format, missing necessary information, outdated data, deleted or replaced data, etc. For example, incorrect data format, input of an invalid IP address or incorrect subnet mask, these invalid data may mislead the data analysis results, so filtering and processing are required before data analysis to ensure the accuracy and reliability of the collected data.

[0053] In this embodiment, ETL technology is used to convert and clean the data. ETL stands for extraction, transformation and loading, and is a data processing technology. The DataX tool is used. DataX is an ETL tool. DataX supports data conversion and cleaning operations, such as field type conversion, data filtering, data formatting, etc. Specifically, 1. Data mapping: supports custom data mapping rules, and can convert data from one form to another according to needs. 2. Field calculation: complex calculation logic can be defined to process and convert data. 3. Data aggregation: provides data aggregation function, and can merge and summarize data from multiple data sources according to specific rules. 4. Outlier detection and processing: provides a variety of outlier detection methods, such as statistical threshold detection, clustering-based outlier identification, etc. 5. Data deduplication: provides data deduplication function, and can identify and delete duplicate records based on specific fields (such as primary keys) to avoid data redundancy from interfering with the analysis results. 6. Format standardization: data in different formats can be uniformly converted into a standard format to ensure data consistency and comparability.

[0054] Step 6: Store the cleaned data information in a relational database, and establish a data management system for management and maintenance; Specifically, the cleaned data is stored in a relational database, and a data management system is established for management and maintenance. The data management system may include functions such as data backup, recovery, and update to ensure data security and integrity.

[0055] Step 7: Based on the resource model and collected data, use graph computing technology to analyze and optimize ICT resources.

[0056] In this embodiment, according to the resource model and the collected data, ICT resources are used as nodes, and the relationships between nodes are represented by edges. Different nodes and connection relationships constitute a complex graph structure, and these nodes and connection relationships are stored and managed in the form of a graph database. In the process of analyzing resources, graph traversal algorithms such as breadth-first search and depth-first search are used to achieve rapid traversal and extraction of nodes and edges in the graph. Furthermore, breadth-first search (BFS) starts from a specified node, traverses the network layer by layer, and collects information about adjacent nodes. It is suitable for scenarios that need to quickly cover a large number of nodes. Depth-first search (DFS) is to explore a branch in depth until the termination condition is reached (such as reaching the maximum depth, finding duplicate nodes, etc.) and then backtrack. It is suitable for scenarios that require in-depth exploration of specific paths or areas.

[0057] Taking searching for resources of a certain project as an example, we use the breadth-first search algorithm to find the shortest path. We take one of the resources as the starting node (such as a switch), and visit all its adjacent nodes (such as hosts, firewalls, etc.) according to the connection relationship between resources (represented by edges in the figure). Then, we start from these adjacent nodes and visit their respective unvisited adjacent nodes, and so on, until all nodes in the figure are visited. We can get the entire resource topology of this project, as shown in the figure below: Figure 4 shown.

[0058] The execution process of its breadth-first search algorithm is:

[0059] (1) Initialization: Create a queue to store the nodes to be visited, mark the starting node as visited, and add it to the queue.

[0060] (2) Iteration process: When the queue is not empty, take a node from the queue and call it the current node; traverse all adjacent nodes of the current node. If the adjacent node has not been visited, mark it as visited and add it to the queue.

[0061] (3) Termination condition: When the queue is empty, the algorithm ends, and all nodes in the graph that are connected to the starting node have been visited.

[0062] Based on the resource network topology obtained through the above graph traversal algorithm analysis, we can identify key resources, bottleneck resources, redundant resources and potential optimization space in the network, and formulate corresponding resource optimization strategies.

[0063] For example, Figure 4 Take the resource network topology diagram as an example:

[0064] If several host resources are associated with the same switch at the same time and the switch fails, other associated resources will also be inaccessible, then the switch is a critical resource and can be protected as a priority; if multiple applications are found to be running on the same host resource, such as mysql, nginx, and tomcat applications running on the same host, then the host may have bottlenecks due to too many applications being deployed, combined with the host's CPU, memory and other performance information, and it is recommended to optimize resources, increase resources or expand capacity to improve the performance and resource utilization of the entire network.

[0065] Step 8. Use network scanning tools to detect ICT resources on heterogeneous networks in real time, automatically discover resources and bring them into management. Specifically, select nmap network scanning tool. nmap is an open source network scanning tool. nmap supports multiple protocols, including but not limited to TCP, UDP, and ICMP. By scanning resources on the network with nmap, it can be used to discover hosts and services on the network. nmap has the following functions: 1. Host detection: nmap can detect hosts on the network, list hosts that respond to TCP and ICMP requests, and help detect which hosts are online; 2. Port scanning: Analyze open ports on the target host by sending detection packets such as TCP SYN scan, TCP Connect scan, and UDP scan; 3. System detection: The operating system used by the host can be inferred.

[0066] Use the nmap network scanning tool to scan resources on the network. When new resources are detected, match the key features of the resources with the feature library, identify potential resources and include them in asset management; for resources whose key features are not identified, they are classified as "unknown resources" by default and require manual inspection and verification. When the host detection or port scan does not respond, the resource can be judged to be offline, and the online status of the resource is updated at the same time. The automatic discovery mechanism can meet the large-scale and dynamically changing ICT resource management needs.

[0067] Key features refer to the unique identifiers of resources, such as object classification, manufacturer, model, port, etc., which need to be pre-set in the configuration platform before matching. The following table lists the features of different resource types 4.

[0068] Table 4

[0069] Resource Classification Manufacturer Model / Version port switch Cisco cisco2512 80 Host DELL PowerEdge T150 22 Load Balancing F5 BIG-LTM-1600-4G-R 80 database Mysql 8.0 3306

[0070] In this embodiment, security policies are implemented throughout the entire method, including data encryption transmission, access control, and anonymization processing, to ensure the security of ICT resource information and the protection of user privacy. For example, the perfect authorization mechanism of OAuth2 is used to better manage the client and ensure data security. Among them, OAuth 2.0 is an open authorization standard that allows users to authorize third-party applications to access private resources (such as photos, videos, contact lists) stored on a website without providing usernames and passwords to third-party applications. OAuth 2.0 provides four authorization modes, including authorization code mode, simplified mode, password mode, and client mode, to adapt to different application scenarios and security requirements.

[0071] Finally, it should be noted that the above content is only used to illustrate the technical solution of the present invention, rather than to limit the scope of protection of the present invention. Simple modifications or equivalent substitutions of the technical solution of the present invention by ordinary technicians in this field do not deviate from the essence and scope of the technical solution of the present invention.

Claims

1. A graph-based heterogeneous network ICT resource analysis method, characterized in that: The following steps are involved: Step 1: Group ICT resources based on their different types and build a graph-based resource model. Step 2: Install the software; Step 3: Deploy the collection plug-in to the host in the heterogeneous network according to different resource types; Step 4: Collect data information of ICT resources by running the collection plug-in and store it in the time series database; Step 5: Convert and clean the collected data information; Step 6: Store the cleaned data information in a relational database and establish a data management system for management and maintenance; Step 7: Based on the resource model and collected data, use graph computing technology to analyze and optimize ICT resources; Step 8: Use network scanning tools to detect ICT resources on heterogeneous networks in real time, automatically discover resources, and bring them under management.

2. The graph-based heterogeneous network ICT resource analysis method according to claim 1, characterized in that: In step 1, a graph database is used to store the ICT resource model, and the ICT resources in the heterogeneous network are abstracted into nodes and edges in the graph resource model, where nodes represent various ICT resources; edges represent the connection relationships and dependency relationships between various ICT resources.

3. The graph-based heterogeneous network ICT resource analysis method according to claim 1, characterized in that: In step 2, the installed software includes Agent and Proxy.

4. The graph-based heterogeneous network ICT resource analysis method according to claim 1, characterized in that: In step 3, the collection plug-ins include but are not limited to: Node Exporter, which is used to collect host hardware information and operating system information; Nginx Exporter, which is used to collect Nginx indicator data; Redis Exporter, which is used to collect various performance indicators of Redis; SNMP Exporter, which is used to collect SNMP client data through the SNMP protocol.

5. The graph-based heterogeneous network ICT resource analysis method according to claim 1, characterized in that: In step 5, ETL technology is used to convert and clean the data, using the DataX tool. DataX performs at least the following processing: Data mapping: support custom data mapping rules; Field calculation: define calculation logic, process and convert data; Data aggregation: Provides data aggregation function to merge and summarize data from multiple data sources according to specific rules; Outlier detection and processing: Provides a variety of outlier detection methods; Data deduplication: Provides data deduplication function, which can identify and delete duplicate records based on specific fields; Format standardization: Convert data in different formats into a standard format.

6. The graph-based heterogeneous network ICT resource analysis method according to claim 1, characterized in that: In step 6, the data management system at least includes operations of data backup, recovery and update.

7. The graph-based heterogeneous network ICT resource analysis method according to claim 1, characterized in that: In step 7, different nodes and connection relationships constitute a graph structure, and these nodes and connection relationships are stored and managed in the form of a graph database.

8. The graph-based heterogeneous network ICT resource analysis method according to claim 1, characterized in that: In step 7, the heterogeneous network ICT resource analysis process is: using a graph traversal algorithm to traverse and extract nodes and edges, where the graph traversal algorithm includes a breadth-first search and a depth-first search.

9. The graph-based heterogeneous network ICT resource analysis method according to claim 1, characterized in that: In step 8, the ICT resources on the network are scanned using the nmap network scanning tool. When new ICT resources are detected, the key features of the resources are matched with the feature library to identify potential resources and include them in asset management. For ICT resources whose key features are not identified, they are classified as unknown resources by default and manually checked and verified.

10. The graph-based heterogeneous network ICT resource analysis method according to claim 1, characterized in that: Security and privacy protection operations are also performed during the entire ICT resource analysis process, which at least include data encryption transmission, access control, and anonymization processing.

Images