Method, system and equipment for automatically drawing network topological graph, and storage medium

Abstract

The invention discloses a method, a system and equipment for automatically drawing a network topological graph, and a storage medium. The method comprises the following steps: acquiring a message of a switch through a port mirror image; the analyzer receives the message and analyzes the equipment information, and the analyzer sends the message and the equipment information to a database ClickHouse through a kafka system; the database ClickHouse performs deduplication and aggregation processing on the equipment information and the message corresponding to the equipment, and stores the information and the message into a graph database; and selecting a time range and any node in the graph database, and generating a network topological graph. Based on port mirror image characteristics of a router and a switch, a message of a network device is sent to a specified monitoring device in a port mirror image mode, a source IP, a source port, a target IP and a target port of the message are analyzed through an analyzer, and after message data are processed through a kafka system and a database ClickHouse, the message is sent to the specified monitoring device. And obtaining related information such as time, frequency and quantity of data exchange of the network equipment, and finally forming a network topological graph.

Description

Technical Field

[0001] This application relates to the field of computer technology, and particularly relates to a method, system, device, and storage medium for automatically drawing a network topology diagram. Background Art

[0002] Currently, almost all products for automatically obtaining network topologies on the market rely on the SNMP protocol. By sending SNMP instructions through the client, the routing information of the device side is obtained. For example, in the patent application No. 201911222941.9, "A Method and Related Device for Generating a Network Topology Diagram of a Network Device", it can only qualitatively obtain whether there is data exchange between devices in a specific network, and cannot quantitatively obtain relevant information such as the time, frequency, and quantity of exchanged data. Summary of the Invention

[0003] To solve the above problems, the present invention adopts the following technical solutions:

[0004] A method for automatically drawing a network topology diagram, the steps including:

[0005] Obtain the packets of the switch through port mirroring;

[0006] The analyzer receives the packets and parses out the device information, where the device information includes time, source IP address, source MAC address, source port number, destination IP address, destination MAC address, destination port number, and protocol type;

[0007] The analyzer sends the packets and device information to the database ClickHouse through the kafka system;

[0008] The database ClickHouse performs deduplication and aggregation processing on the device information and the packets corresponding to the devices, and stores them in the graph database;

[0009] Select a time range and any node in the graph database to generate a network topology diagram.

[0010] As a further solution of the present invention, the database ClickHouse performs deduplication and aggregation processing on the device information and the packets corresponding to the devices, including the steps of: in terms of the time dimension, counting the source IP address, source MAC address, source port number, destination IP address, destination MAC address, destination port number, protocol type, and the number of packets to generate a packet table; storing the packet table in the graph database; the granularity of the time dimension is adjusted according to the storage load situation.

[0011] As a further solution of the present invention, when selecting a time range and any node in the graph database to generate a network topology diagram, the steps include:

[0012] The starting node is composed of the source IP address, source MAC address, and source port number, and the ending node is composed of the destination IP address, destination MAC address, and destination port number. Using the number of packets as the weight, there is a communication relationship from the starting node to the ending node;

[0013] Select the time range and any node, and generate a network topology diagram within the time range according to the device information of the nodes and the communication relationships between the nodes.

[0014] As a further solution of the present invention, the graph database uses the Neo4j graph database; the CREATE command is used to create nodes, the MATCH command is used to create connection relationships, and the Cypher command is used to query any time range and node range to generate a network topology diagram.

[0015] As a further solution of the present invention, when selecting the time range and any node in the graph database to generate a network topology diagram, the steps further include: using the Cypher command to query any time range and node range to generate a graph format supported by the Graphviz graphics drawing library, and using the Graphviz library to load the graph data to generate a network topology diagram.

[0016] An automated network topology diagram drawing system includes: a port mirroring module, a packet analysis module, and a network topology diagram generation module;

[0017] The port mirroring module is used to configure the switch port and copy the packets of the port to the specified observation port;

[0018] The packet analysis module is used to analyze the packets collected by the port mirroring module and determine the communication relationships between the nodes; specifically, it includes receiving the packets and parsing out the device information, where the device information includes time, source IP address, source MAC address, source port number, destination IP address, destination MAC address, destination port number, and protocol type; performing deduplication and aggregation processing on the device information and the packets corresponding to the devices, and storing them in the graph database;

[0019] The network topology diagram generation module selects the time range and any node in the graph database to generate a network topology diagram.

[0020] As a further solution of the present invention, it further includes: a network topology diagram display module, which is used to display the network topology diagram and provide a human-computer interaction interface.

[0021] As a further solution of the present invention, the step of performing deduplication and aggregation processing on the device information and the packets corresponding to the devices includes the steps:

[0022] Statistically analyze the source IP address, source MAC address, source port number, destination IP address, destination MAC address, destination port number, protocol type, and the number of packets in terms of time dimension to generate a packet table; store the packet table in a graph database; the granularity of the time dimension is adjusted according to the storage load situation;

[0023] Select a time range and any node in the graph database to generate a network topology graph, including the steps:

[0024] Use the source IP address, source MAC address, and source port number together to form a starting node, use the target IP address, target MAC address, and target port number together to form an ending node, use the number-of-packets metric as the weight, and there is a communication relationship from the starting node to the ending node;

[0025] Select a time range and any node, and generate a network topology graph within the time range according to the device information of the nodes and the communication relationships between the nodes.

[0026] A network device includes a memory and a processor;

[0027] The memory stores computer execution instructions;

[0028] At least one of the processors executes the computer execution instructions stored in the memory, so that at least one of the processors executes the automated network topology graph drawing method described in any one of the above.

[0029] A computer-readable storage medium stores computer execution instructions therein, and when the computer execution instructions are executed, the automated network topology graph drawing method described in any one of the above is implemented.

[0030] The beneficial effects of the present invention:

[0031] An automated network topology graph drawing method, system, device, and storage medium, based on the port mirroring characteristics of routers and switches, send the packets of network devices to a specified monitoring device through port mirroring, parse the source IP, source port, target IP, and target port of the packets through an analyzer, and after processing the packet data through the kafka system and the ClickHouse database, obtain relevant information such as the time, frequency, and quantity of data exchange of network devices, and finally form a network topology graph. Description of the Drawings

[0032] Figure 1 is a flowchart of an automated network topology graph drawing method of the present invention;

[0033] Figure 2 is a structural diagram of an automated network topology graph drawing method of the present invention;

[0034] Figure 3 It is a schematic structural diagram of a system for automatically drawing network topology diagrams according to the present invention. Specific implementation manners

[0035] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. It should be understood that the present application is not limited by the exemplary embodiments disclosed herein. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the scope of protection of the present application.

[0036] In this application document, there are the following professional terms, which are explained separately herein:

[0037] The port mirroring function is to forward the data traffic of one or more source ports to a specified port on a switch or router to monitor the network. The specified port is called the "mirror port". Without seriously affecting the normal throughput of the source port, the network traffic can be monitored and analyzed through the mirror port. In an enterprise, the mirroring function can be used to monitor and manage the internal network data of the enterprise well. When a network failure occurs, the failure can be quickly located.

[0038] A message is a data unit exchanged and transmitted in a network, that is, a data block that a site needs to send at one time. A message contains the complete data information to be sent, and its length varies greatly, with no limit and being variable.

[0039] The Kafka system is a high-performance and scalable message queue system for processing real-time data streams. Its design goal is to provide reliable and high-throughput message delivery, and support persistent storage and fault tolerance mechanisms.

[0040] The ClickHouse database, ClickHouse is a columnar database management system (DBMS) for online analytical processing (OLAP). In a traditional row-based database system, the data in the same row is always physically stored together. In a columnar database system, the data from the same column is stored together.

[0041] OLAP (On-line Analytical Processing) is a computer processing technology used for analyzing and querying large-scale data sets. OLAP technology is mainly used for multi-dimensional data analysis and data mining. By providing a multi-dimensional data model and multi-dimensional query capabilities, it helps users analyze and query data from different perspectives and levels, with a focus on analytical decision-making.

[0042] The Neo4j graph database is a graph database management system that is specifically used for storing, managing, and querying graph data. Graph databases use graph structures to represent and store data, where nodes represent entities and edges represent relationships between nodes. Neo4j provides efficient storage and query mechanisms, making it easier and more efficient to process complex relational data.

[0043] The Graphviz graph drawing library is an open-source graph visualization toolkit used for drawing graphs and network diagrams. It provides a set of drawing tools and libraries that can convert graph data into a visual graph representation. Graphviz supports multiple graph layout algorithms and can automatically layout nodes and edges to generate aesthetically pleasing graph outputs.

[0044] Embodiment 1

[0045] As shown in the figure, a method for automatically drawing a network topology diagram includes the following steps:

[0046] S1: Obtain the packets of the switch through port mirroring;

[0047] S2: The analyzer receives the packets and parses out the device information, which includes time, source IP address, source MAC address, source port number, destination IP address, destination MAC address, destination port number, and protocol type;

[0048] S3: The analyzer sends the packets and device information to the ClickHouse database through the kafka system;

[0049] S4: The ClickHouse database performs deduplication and aggregation processing on the device information and the packets corresponding to the devices, and stores them in the graph database;

[0050] S5: Select a time range and any node in the graph database to generate a network topology diagram.

[0051] In this embodiment, in step S1, obtaining the packets of the switch through port mirroring specifically means copying the packets of one port to an observation port, and the observation port is connected to the monitoring device to collect the packets of the switch. In a specific embodiment, the methods for configuring the mirroring port of different switches are slightly different. Taking a certain switch as an example:

[0052] (1) Configure the observation port

[0053] # Configure interface GE0 / 0 / 2 on the Switch as the local observation port.

[0054] <huawei>system-view;

[0055] [HUAWEI]sysname Switch;

[0056] [Switch]observe-port 1 interface gigabitethernet 0 / 0 / 2 / / Configure GE0 / 0 / 2 as the local observation port with the observation port index being 1;

[0057] (2) Configure the mirroring port

[0058] [Switch]interface gigabitethernet 0 / 0 / 1;

[0059] [Switch-GigabitEthernet0 / 0 / 1]port-mirroring to observe-port 1 inbound / / Bind the inbound direction of interface GE0 / 0 / 1 to the observation port with index 1;

[0060] [Switch-GigabitEthernet0 / 0 / 1]return.

[0061] In this implementation, in step S2, the analyzer receives the packets and parses out the device information. Specifically, a packet analyzer is used to parse out the device information, and the specific parsing method refers to the standard network communication protocol, which will not be elaborated here.

[0062] In this implementation, in step S3, the analyzer sends the packets and device information to the database ClickHouse through the kafka system. Specifically, to solve the problem of how to count packets in the case of ultra-large data streams, the packet analyzer sends the packets to the OLAP database ClickHouse through the message queue kafka system. The kafka system processes the real-time data stream and performs deduplication and statistics through the aggregation model of the database ClickHouse.

[0063] In this implementation, in step S4, the database ClickHouse performs deduplication and aggregation processing on the device information and the packets corresponding to the devices, including the steps of: statistically counting the source IP address, source MAC address, source port number, destination IP address, destination MAC address, destination port number, protocol type, and the number of packets in terms of the time dimension to generate a packet table; storing the packet table in the graph database; adjusting the granularity of the time dimension according to the storage load situation. Specifically, the number of packets refers to the number of packets with the same source IP address, source MAC address, source port number, destination IP address, destination MAC address, destination port number, and protocol type within a certain period of time.

[0064] In this embodiment, in step S5, a time range and any node are selected in the graph database to generate a network topology graph. The steps include: using the source IP address, source MAC address, and source port number together to form a starting node, using the destination IP address, destination MAC address, and destination port number together to form an ending node, using the number of packets as the weight, and having a communication relationship from the starting node to the ending node; selecting a time range and any node, and generating a network topology graph within the time range according to the device information of the nodes and the communication relationships between the nodes.

[0065] Specifically, for the graph database, the Neo4j graph database is used; the CREATE command is used to create nodes, the MATCH command is used to create connection relationships, and the Cypher command is used to query any time range and node range to generate a network topology graph. The execution process of the Neo4j graph database is briefly described as follows:

[0066] (1) Create nodes:

[0067] CREATE(d:Device{name:'Router1',type:'Router',ip:'192.168.1.1'});

[0068] CREATE(d:Device{name:'Switch1',type:'Switch',ip:'192.168.1.2'});

[0069] (2) Create connection relationships

[0070] MATCH(a:Device{name:'Router1'}),(b:Device{name:'Switch1'});

[0071] CREATE(a)-[:CONNECTED_TO]->(b);

[0072] (3) Use the Cypher query language to query topology information

[0073] MATCH path=(d:Device)-[:CONNECTED_TO]->(otherDevice);

[0074] RETURN path。

[0075] Specifically, since switches or routers are identified by IP addresses and MAC addresses during the process of generating the network topology graph, devices that cannot be identified can be manually labeled, and data graphics are manually selected to represent the devices.

[0076] In other embodiments, Cypher commands are used to query any time range and node range, generate a graph format supported by the Graphviz graph drawing library, and use the Graphviz graph drawing library to load the graph data to generate a network topology map.

[0077] Specifically, the Neo4j graph database and the Graphviz graph drawing library can be combined to generate a network topology map. In the network topology map, there are not only the communication relationships of the nodes, but also the device graph data. The Neo4j graph database is used to store and manage the logical relationships of the graph data, and the Graphviz graph drawing library is used to draw the network topology map. The specific steps are as follows: Use the Neo4j graph database to execute complex graph queries to obtain specific nodes and relationships; export the query results to a graph format supported by Graphviz, such as the DOT format; use the Graphviz graph drawing library to load the graph data to generate a network topology map, and render the graph data as an image or a visual graph. The Neo4j graph database and the Graphviz graph drawing library have complementary functions in graph data management and visualization, and can be jointly used to process and display complex relational data.

[0078] A method for automatically drawing a network topology map, based on the port mirroring characteristics of routers and switches, sends the packets of network devices to a specified monitoring device through port mirroring, parses the source IP, source port, destination IP, and destination port of the packets through an analyzer, and after processing the packet data through the kafka system and the ClickHouse database, obtains relevant information such as the time, frequency, and quantity of data exchanged by network devices, and finally forms a network topology map.

[0079] Embodiment 2

[0080] An automatic network topology map drawing system includes: a port mirroring module, a packet analysis module, and a network topology map generation module.

[0081] The port mirroring module is used to configure the switch port and copy the packets of the port to a specified observation port.

[0082] The packet analysis module is used to analyze the packets collected by the port mirroring module and determine the communication relationships between nodes; specifically, it includes receiving the packets and parsing out the device information, where the device information includes time, source IP address, source MAC address, source port number, destination IP address, destination MAC address, destination port number, and protocol type; performing deduplication and aggregation processing on the device information and the packets corresponding to the devices, and storing them in the graph database.

[0083] The network topology map generation module selects a time range and any nodes in the graph database to generate a network topology map.

[0084] A network topology diagram display module, which is used to display the network topology diagram and provide a human-computer interaction interface.

[0085] In this embodiment, the device information and the corresponding messages of the devices are de-duplicated and aggregated, including the steps of:

[0086] Statistically analyze the source IP address, source MAC address, source port number, destination IP address, destination MAC address, destination port number, protocol type, and the number of messages in terms of time dimension to generate a message table; store the message table in the graph database; the granularity of the time dimension is adjusted according to the storage load condition;

[0087] In the graph database, select the time range and any node to generate a network topology diagram, including the steps of:

[0088] Use the source IP address, source MAC address, and source port number together to form the starting node, use the destination IP address, destination MAC address, and destination port number together to form the ending node, use the number of messages index as the weight, and there is a communication relationship from the starting node to the ending node;

[0089] Select the time range and any node, and generate the network topology diagram within the time range according to the device information of the nodes and the communication relationships between the nodes.

[0090] The technical effects of an automated network topology diagram drawing system are referred to an automated network topology diagram drawing method, which will not be elaborated here.

[0091] Embodiment 3

[0092] A network device includes a memory and a processor;

[0093] The memory stores computer execution instructions;

[0094] At least one processor executes the computer execution instructions stored in the memory, so that at least one processor executes the automated network topology diagram drawing method of any one of the above.

[0095] Since an automated network topology diagram drawing method has the above technical effects, a network device including an automated network topology diagram drawing method should also have corresponding technical effects, which will not be elaborated here.

[0096] Embodiment 4

[0097] A computer-readable storage medium stores computer execution instructions, and when the computer execution instructions are executed, the automated network topology diagram drawing method of any one of the above is implemented.

[0098] Since an automated method for drawing a network topology diagram has the above technical effects, a computer-readable storage medium including the automated method for drawing a network topology diagram should also have corresponding technical effects, which will not be elaborated here.

[0099] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions according to the embodiments of the present invention are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more integrated available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0100] In the description of the specification, the description with reference to terms such as "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0101] It should also be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variation thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the existence of additional identical elements in the process, method, article or device comprising the element.

[0102] The above content is only an example and explanation of the structure of the present invention. Those skilled in the art of this technology can make various modifications or supplements to the described specific embodiments or use similar ways to replace them. As long as they do not deviate from the structure of the invention or exceed the scope defined by this claim book, they should all fall within the protection scope of the present invention.< / huawei>

Claims

1. A method for automatically drawing a network topology diagram, characterized in that the steps Including: Obtaining the packets of the switch through port mirroring; The analyzer receives the packets and parses out the device information, where the device information includes time, source IP address, source MAC address, source port number, destination IP address, destination MAC address, destination port number, and protocol type; The analyzer sends the packets and device information to the ClickHouse database through the Kafka system; The ClickHouse database performs deduplication and aggregation processing on the device information and the packets corresponding to the devices, and stores them in the graph database; Select a time range and any node in the graph database to generate a network topology graph.

2. The method for automatically drawing a network topology diagram according to claim 1, wherein The ClickHouse database performs deduplication and aggregation processing on the device information and the packets corresponding to the devices, including the steps of: statistically counting the source IP address, source MAC address, source port number, destination IP address, destination MAC address, destination port number, protocol type, and the number of packets in terms of time dimension to generate a packet table; Storing the packet table in the graph database; the granularity of the time dimension is adjusted according to the storage load situation.

3. The method for automatically drawing a network topology diagram according to claim 1, characterized in that, When selecting a time range and any node in the graph database to generate a network topology graph, the steps include: Using the source IP address, source MAC address, and source port number together to form the starting node, using the destination IP address, destination MAC address, and destination port number together to form the ending node, using the number of packets indicator as the weight, and there is a communication relationship from the starting node to the ending node; Select a time range and any node, and generate a network topology graph within the time range according to the device information of the nodes and the communication relationships between the nodes.

4. The method for automatically drawing a network topology diagram according to claim 3, characterized in that, The graph database uses the Neo4j graph database; uses the CREATE command to create nodes, uses the MATCH command to create connection relationships, and uses the Cypher command to query any time range and node range to generate a network topology graph.

5. The method for automatically drawing a network topology diagram according to claim 4, wherein When selecting a time range and any node in the graph database to generate a network topology graph, the steps further include: using the Cypher command to query any time range and node range to generate a graph format supported by the Graphviz graphics drawing library, and using the Graphviz library to load the graph data to generate a network topology graph.

6. An automated network topology diagram drawing system, characterized in that, Including: A port mirroring module, a packet analysis module, and a network topology graph generation module; The port mirroring module is used to configure the switch port and copy the packets of the port to the specified observation port; The packet analysis module is used to analyze the packets collected by the port mirroring module and determine the communication relationships between the nodes; specifically, it includes receiving the packets and parsing out the device information, where the device information includes time, source IP address, source MAC address, source port number, destination IP address, destination MAC address, destination port number, and protocol type; performing deduplication and aggregation processing on the device information and the packets corresponding to the devices, and storing them in the graph database; The network topology graph generation module selects a time range and any node in the graph database to generate a network topology graph.

7. An automated network topology drawing system according to claim 6, wherein Also including: A network topology graph display module, which is used to display the network topology graph and provide a human-computer interaction interface.

8. An automated network topology drawing system according to claim 6, characterized in that, The steps for performing deduplication and aggregation processing on the device information and the packets corresponding to the devices include: Statistically analyze the source IP address, source MAC address, source port number, destination IP address, destination MAC address, destination port number, protocol type, and the number of packets in terms of time dimension to generate a packet table; Store the packet table in a graph database; the granularity of the time dimension is adjusted according to the storage load condition; Select a time range and any node in the graph database to generate a network topology graph, including the steps of: Use the combination of the source IP address, source MAC address, and source port number as the starting node, use the combination of the destination IP address, destination MAC address, and destination port number as the ending node, use the number-of-packets metric as the weight, and there is a communication relationship from the starting node to the ending node; Select a time range and any node, and generate a network topology graph within the time range according to the device information of the nodes and the communication relationships between the nodes.

9. A network device, characterized in that, Comprising a memory and a processor; The memory stores computer-executable instructions; At least one of the processors executes the computer-executable instructions stored in the memory, so that the at least one processor executes the method for automatically drawing a network topology graph according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, Computer-executable instructions are stored in the computer-readable storage medium, and when the computer-executable instructions are executed, the method for automatically drawing a network topology graph according to any one of claims 1 to 8 is implemented.

Images