Network digital map construction method and system based on whole-network operation data collection and routing simulation

Abstract

The application discloses a network digital map construction method and system based on whole-network operation data collection and routing simulation, and belongs to the technical field of network operation and maintenance. The method comprises the following steps: collecting operation protocols, configuration information, state data and other multi-source data of whole-network equipment through multi-protocol cooperation to form a multi-source heterogeneous data pool; performing cleaning and standardization processing on the data to construct a unified model to shield manufacturer differences; recording a complete forwarding path; fusing simulation results and topology data to construct a dynamic network digital map and integrate multiple operation and maintenance functions. The system comprises corresponding function modules and supports distributed deployment and fault self-checking. The application solves the problems of multi-manufacturer equipment heterogeneity, difficult fault positioning and inaccurate path prediction in the prior art, realizes a digital map consistent with a real network, improves network operation and maintenance efficiency, reduces operation and maintenance cost, and guarantees network availability and security.

Description

Technical Field

[0001] This invention belongs to the field of network operation and maintenance technology, specifically relating to a method and system for constructing a network digital map based on the collection of full network operation data and routing simulation. Background Technology

[0002] Current enterprise networks typically include various types of network devices such as switches, routers, firewalls, and load balancers, sourced from multiple mainstream vendors including Cisco, Huawei, H3C, and Juniper. Significant differences exist in the data models, protocol specifications, and configuration methods of devices from different vendors, greatly increasing the difficulty of network operation and maintenance. Network maintenance personnel lack a comprehensive, real-time, and accurate visualization tool for daily troubleshooting, policy changes, and path analysis, making it impossible to fully grasp the network's operational status and packet forwarding paths.

[0003] Existing network topology discovery technologies mostly rely on single protocols such as SNMP and LLDP for data collection, which cannot comprehensively collect all operational data and configuration information of network devices. In particular, they cannot effectively collect complex policy configuration data such as Policy Routing (PBR), Access Control Lists (ACL), and Network Address Translation (NAT), resulting in an inability to fully simulate the forwarding path of data packets in a real network. Furthermore, existing technologies cannot mask data differences between devices from different vendors, making it difficult to achieve unified modeling and simulation across vendors. Simulation results deviate significantly from the real network environment, failing to accurately reflect the impact of policy configuration on forwarding paths. This leads to problems such as prolonged fault location, higher risk of policy changes, and inaccurate path prediction, severely impacting network operation and maintenance efficiency and network availability.

[0004] To address the shortcomings of the existing technologies, this invention proposes a method and system for constructing a network digital map based on full-network operation data collection and routing simulation. By collaboratively collecting data from all network devices through multiple protocols, standardizing processing to mask vendor differences, performing precise path simulation, and constructing a dynamic digital map, a virtual network digital map consistent with the real network environment is achieved. This solves the deficiencies of the existing technologies and provides accurate and efficient visualization support for network operation and maintenance. Summary of the Invention

[0005] In view of this, the present invention provides a method and system for constructing a network digital map based on full network operation data collection and routing simulation, so as to solve or alleviate one of the technical problems existing in the prior art, and at least provide a beneficial option.

[0006] The technical solution of this invention is implemented as follows: a method and system for constructing a network digital map based on full-network operation data collection and routing simulation, comprising the following steps: Step 1: Deployment of the data acquisition layer and multi-source data acquisition. Through the collaborative deployment of five protocols, namely SNMP, SSH, Netconf, gRPC, and RESTful, the system collects the operating protocol data, complete configuration information, real-time status data, full routing tables, complete forwarding tables, and various policy configurations of all communication devices in the network. All collected data is aggregated in real time to form a multi-source heterogeneous data pool. Timestamps and device identifiers are added to each type of data to facilitate subsequent data traceability and correlation analysis.

[0007] Step 2: Multi-source heterogeneous data cleaning and standardization. The data collected from different manufacturers' equipment is denoised and deduplicated to remove invalid, duplicate, and abnormal data. The data models of different manufacturers' equipment are analyzed to extract core fields and data formats. A unified equipment and link abstract model is constructed to map the data from different manufacturers to the unified model, thereby masking manufacturer differences. The standardized data is stored in a unified format and a data standardization report is generated.

[0008] Step 3: Accurate construction of virtual data packets. Based on the actual network operation scenario, generate simulated data packet 5-tuples (source IP address, destination IP address, source port, destination port, protocol type) covering different application scenarios. Add a unique identifier to each virtual data packet, record the data packet construction parameters and generation time, and dynamically adjust the data packet generation frequency according to the network scale as input for path simulation.

[0009] Step 4: Deploy the path simulation engine and perform full-process path simulation. Deploy a distributed path simulation engine to simulate the layered forwarding process of data packets after they enter the network. Perform Layer 2 MIB forwarding table lookup and Layer 3 routing table lookup in sequence. Accurately apply ACL, PBR, security policies, and NAT translation rules. Record forwarding information hop by hop to form a complete forwarding path record. Summarize the simulation results and generate a simulation result report.

[0010] Step 5: Dynamic construction and functional integration of network digital map. Deeply integrate path simulation results with network topology data, establish the correspondence between simulated paths and topology nodes, generate dynamic network digital map, integrate path visualization, strategy impact analysis, fault simulation, change pre-simulation and other functions, and synchronize network operation status and simulation results in real time to ensure the dynamicness and accuracy of the map.

[0011] Furthermore, the operational protocol data mentioned in step 1 includes various routing protocols, link protocols, and management protocol data exchanged between devices; configuration information includes all device-related configuration parameters such as device hardware configuration, software version, port configuration, and VLAN configuration; status data includes real-time operational indicators such as device CPU utilization, memory usage, port traffic, and link connectivity status; routing tables include complete entries for static and dynamic routes and routing priority information; forwarding tables include Layer 2 forwarding entries, Layer 3 forwarding entries, and forwarding priority information; and policy configuration includes complete configuration parameters for ACLs, PBRs, NATs, and various security policies.

[0012] Furthermore, the data denoising process described in step 2 employs a statistical threshold-based denoising algorithm to remove abnormal data that exceeds the normal range, as well as invalid data with format errors or missing fields; data deduplication is achieved through a combination of device identifier, timestamp, and data type; the manufacturer data parsing unit has a built-in mainstream manufacturer data model library that supports real-time updates and uses machine learning algorithms to automatically parse newly added manufacturer device data; the unified device and link abstract model contains core modules and supports flexible expansion.

[0013] Furthermore, the virtual data packet quintuple described in step 3 has source and destination IP addresses covering all network segments within the network and external interactive network segments, source and destination ports covering commonly used service ports and custom ports, and protocol types including common network protocols such as TCP, UDP, and ICMP; the data packet generation frequency is adjusted according to the network size, with a group generated every 10 seconds for small-scale networks and every 5 seconds for medium-to-large-scale networks.

[0014] Furthermore, the path simulation process described in step 4 includes four sub-steps: packet access simulation, layered forwarding simulation, network policy application, hop-by-hop forwarding, and data recording. Among them, the layered forwarding simulation strictly follows the Layer 2 and Layer 3 forwarding rules of the real network, the policy application strictly follows the policy execution order of the real network, and the device identifier, interface, action, policy hit status, and forwarding time of each hop are recorded hop-by-hop.

[0015] Furthermore, the network topology data mentioned in step 5 includes information such as device location, device type, link connection relationship, and link bandwidth; the data fusion adopts a data association algorithm based on deep learning, with an association accuracy of ≥99.5%; the digital map supports two-dimensional and three-dimensional display formats, and the integrated functions can meet the operation and maintenance needs such as fault diagnosis, policy change, and path analysis.

[0016] A network digital map construction system based on full-network operation data collection and routing simulation is provided to implement the above-mentioned method. The system includes a data collection module, a data cleaning and standardization module, a virtual data packet construction module, a path simulation engine module, and a digital map construction module. Each module is connected through a high-speed data bus, has fault self-checking and backup module switching functions, and supports distributed deployment and flexible expansion.

[0017] The data acquisition module is used for the acquisition and preliminary verification of multi-source data. It includes a protocol adaptation unit, a multi-device acquisition unit, a data aggregation unit, and a preliminary data quality verification unit. It supports multi-protocol adaptation and multi-device acquisition to ensure the integrity, continuity, and security of data acquisition.

[0018] The data cleaning and standardization module is used to clean, deduplicate, and standardize the collected data, shielding vendor differences. It includes a data denoising unit, a data deduplication unit, a vendor data parsing unit, a unified model building unit, and a data standardization unit to ensure data accuracy and compatibility.

[0019] The virtual data packet construction module is used to generate simulation data packets, including a scenario configuration unit, a five-tuple generation unit, a data packet identification unit, and a data packet scheduling unit. It supports scenario customization and diversified data packet generation, ensuring the comprehensiveness and efficiency of path simulation.

[0020] The path simulation engine module is used to realize full-process path simulation, including packet access unit, layered forwarding simulation unit, policy application unit, hop-by-hop recording unit and simulation result summary unit, to simulate real network forwarding logic and ensure the authenticity and accuracy of simulation results.

[0021] The digital map building module is used to construct dynamic network digital maps and integrate related functions, including a topology data import unit, a data fusion unit, a map generation unit, and a function integration unit, to achieve real-time map updates and multi-functional support.

[0022] Furthermore, the system also includes a data storage module, a system management module, and a report generation module. The data storage module adopts a distributed storage architecture and supports multiple copy backups and custom data storage periods. The system management module supports user management, permission management, module management, and parameter configuration. The report generation module supports the generation, export, and scheduled push of various types of reports.

[0023] The embodiments of the present invention have the following advantages due to the adoption of the above technical solutions: 1. This invention uses five protocols—SNMP, SSH, Netconf, gRPC, and RESTful—to collect data collaboratively. Compared to the single-protocol collection method in existing technologies, it can comprehensively collect multi-source data such as network device operating protocols, configuration information, status data, routing tables, forwarding tables, and policy configurations, providing more comprehensive data coverage and sufficient data support for subsequent simulation and map construction. At the same time, the continuous and stable data collection is ensured through the deployment of multiple collection nodes and automatic switching in case of protocol failure.

[0024] This invention constructs a unified abstract model of devices and links through data cleaning and standardization, realizing a unified mapping of device data from different manufacturers (Cisco, Huawei, H3C, Juniper, etc.), effectively shielding manufacturer differences, and solving the problems of difficult cross-manufacturer device modeling and data incompatibility in existing technologies. The standardized data ensures the compatibility and accuracy of subsequent simulation and map construction.

[0025] The virtual data packets constructed by this invention cover different application scenarios, contain complete five-tuple information, and are equipped with unique identifiers and timestamps. They can comprehensively simulate data packet communication scenarios in real networks. At the same time, the generation frequency is dynamically adjusted according to the network scale, which takes into account both the comprehensiveness and efficiency of the simulation and provides reliable input for accurate path simulation.

[0026] The path simulation engine of this invention adopts a distributed architecture, strictly follows the routing and forwarding principles of real networks, fully simulates the Layer 2 and Layer 3 forwarding processes and the application of complex policies such as ACL, PBR, and NAT, records forwarding information hop by hop, and the simulation results are highly consistent with the real network environment. It solves the problems of inaccurate path simulation and inability to reflect the impact of policies in the prior art, and can accurately simulate the actual forwarding path of data packets.

[0027] The dynamic network digital map constructed by this invention deeply integrates simulation results with topology data, supports multiple functions such as path visualization, policy impact analysis, fault simulation, and change pre-simulation, and can intuitively display the network operation status and data packet forwarding path. It provides network operation and maintenance personnel with a global, real-time, and accurate visualization tool, which significantly reduces the time cost of fault diagnosis and policy change analysis and improves network operation and maintenance efficiency.

[0028] The system of this invention has comprehensive fault self-diagnosis, backup module switching and distributed deployment functions, which can ensure continuous and stable operation of the system and adapt to networks of different sizes; at the same time, it has data storage, system management and report generation functions, providing comprehensive support for network operation and maintenance, and further improving the convenience and security of network operation and maintenance.

[0029] The above overview is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features of the invention will become readily apparent from the accompanying drawings and the following detailed description. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is a flowchart illustrating the overall method of the present invention; Figure 2 This is a detailed flowchart of the data materials of the present invention; Figure 3 This is a flowchart of the data cleaning and standardization process of the present invention. Detailed Implementation

[0032] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of the invention. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0033] Example 1: A method for constructing a network digital map based on full-network operation data collection and routing simulation This embodiment provides a method for constructing a network digital map based on full-network operation data collection and routing simulation, applied to an enterprise-level local area network. This network includes two Cisco switches, one Huawei router, one H3C firewall, and one Juniper load balancer. The specific steps include: Step 1: Data Acquisition Layer Deployment and Multi-Source Data Acquisition. Deploy the data acquisition module and enable five protocols: SNMP, SSH, Netconf, gRPC, and RESTful. Select the optimal acquisition protocol for different vendor devices: Cisco switches use SSH, Huawei routers use Netconf, H3C firewalls use gRPC, and Juniper load balancers use RESTful. The specific data collected includes: Runtime protocol data: RIP and OSPF routing protocol data, LLDP and CDP link protocol data, SNMP management protocol data; Configuration information: hardware model, software version, port speed, VLAN division, port open status, etc. of each device; Status data: CPU utilization, memory usage, port inbound and outbound traffic, and link connectivity status of each device. Core devices are collected once every 1 second, and ordinary devices are collected once every 5 seconds. The routing table contains 10 static route entries and 25 OSPF dynamic route entries, including information such as route priority and metric. Forwarding table: 30 Layer 2 forwarding entries and 35 Layer 3 forwarding entries, including information such as forwarding port and forwarding priority; Policy configuration: 15 ACL rules, 5 PBR rules, 8 static NAT rules, and 10 security policies; All collected data is aggregated in real time to a multi-source heterogeneous data pool. Each type of data is assigned a device identifier, a collection timestamp, and a collection node identifier to ensure data traceability.

[0034] Step 2: Multi-source heterogeneous data cleaning and standardization. The data cleaning and standardization module processes the collected data: Data denoising: Remove 1 incorrectly formatted port traffic data, 3 duplicate routing table entries, and 2 CPU utilization data that are outside the normal range (greater than 100%). Data deduplication: Five duplicate configuration records are removed by combining device identifier, timestamp, and data type. Manufacturer data analysis: Analyze the data models of Cisco, Huawei, H3C, and Juniper devices, extract core fields, such as mapping "port-speed" of Cisco devices, "port-rate" of Huawei devices, and "port-bandwidth" of H3C devices to the "port bandwidth" field; Unified model construction: Constructing a device abstract model (including basic device information, hardware configuration, software configuration, operating status, routing information, and policy configuration modules) and a link abstract model (including basic link information, link bandwidth, link status, and connected device modules). Data standardization: All parsed data is converted and mapped according to a unified model specification to generate standardized data, verify the accuracy of the standardized data, and generate a data standardization report.

[0035] Step 3: Precise construction of virtual data packets. The virtual data packet construction module is configured with two scenario templates (regular office scenario and file transfer scenario): Typical office scenario: Generate source IP address range 192.168.1.0 / 24, destination IP address range 192.168.2.0 / 24, source port 80, 443, destination port 80, 443, protocol type TCP, generation frequency 1 group every 10 seconds; File transfer scenario: Generate source IP address range 192.168.1.0 / 24, destination IP address range 192.168.3.0 / 24, source port 21, 22, destination port 21, 22, protocol type TCP, UDP, generation frequency 1 group every 10 seconds; Each virtual data packet is assigned a unique identifier, and its construction parameters and generation time are recorded. The data packets are then evenly distributed to the path simulation engine module via the data packet scheduling unit.

[0036] Step 4: Deploy the path simulation engine and perform full-process path simulation. Deploy a distributed path simulation engine, which includes two simulation nodes. The simulation process is as follows: Sub-step 4.1: Data packet access simulation, receive virtual data packets, and record access device and port information, such as data packets in a typical office scenario accessing from the Gi0 / 1 port of Cisco switch 1; Sub-step 4.2: Layered forwarding simulation. After the data packet enters Cisco switch 1, a Layer 2 MIB forwarding table lookup is performed. The forwarding port is matched according to the destination MAC address, and the packet is forwarded to the Huawei router. The Huawei router performs a Layer 3 routing table lookup. The OSPF route is matched according to the destination IP address, and the packet is forwarded to the H3C firewall. Sub-step 4.3: Network policy application. The H3C firewall matches ACL rules to allow TCP ports 80 and 443 to pass through, and matches NAT rules to convert the source IP address to a public IP address. The Huawei router matches PBR rules to redirect data packets in file transfer scenarios to the backup link for forwarding. Sub-step 4.4: Hop-by-hop recording. Record the device identifier, interface, forwarding action, policy hit status, and forwarding time for each hop, such as Cisco switch 1 (Gi0 / 1→Gi0 / 2), Huawei router (GigabitEthernet0 / 0→GigabitEthernet0 / 1), and H3C firewall (Eth0 / 0→Eth0 / 1), forming a complete forwarding path record. The simulation results are summarized and a simulation results report is generated, which includes the forwarding path of all data packets, policy hit status, and simulation accuracy assessment.

[0037] Step 5: Dynamic construction and functional integration of the network digital map. Import network topology data (device location, type, link connection relationship, bandwidth), and associate the simulation results with the topology data through the data fusion unit. The association accuracy is 99.6%, generating a dynamic network digital map. Path visualization: Visually displays the forwarding path of all virtual data packets. Clicking on a device on the path allows you to view the device status, policy hit status, and forwarding time. Policy Impact Analysis: Simulate the deletion of one ACL rule and observe the changes in forwarding paths. It was found that the forwarding paths of three data packets changed. Analyze the scope of the impact of this ACL rule. Fault simulation: Simulate a Huawei router outage, observe changes in the forwarding path, and discover that data packets automatically switch to the backup link to verify the effectiveness of the backup link; Change rehearsal: Add one PBR rule to the rehearsal, check the changes in the forwarding path, and verify the rationality of the rule configuration; The digital map synchronizes the network operation status and simulation results in real time, and updates the map display immediately when the network device status changes.

[0038] Example 2: A Network Digital Map Construction System Based on Full-Network Operation Data Acquisition and Routing Simulation This embodiment provides a network digital map construction system based on full-network operation data collection and routing simulation, used to implement the method described in Embodiment 1. The system includes a data collection module, a data cleaning and standardization module, a virtual data packet construction module, a path simulation engine module, a digital map construction module, a data storage module, a system management module, and a report generation module. Each module is connected through a high-speed data bus, with a data transmission delay of ≤50ms. It has fault self-checking and backup module switching functions and supports distributed deployment.

[0039] The data acquisition module includes a protocol adaptation unit, a multi-device acquisition unit, a data aggregation unit, and a preliminary data quality verification unit. The protocol adaptation unit provides customized adaptation for devices from different vendors: SSH protocol is optimized for Cisco devices, Netconf protocol for Huawei devices, gRPC protocol for H3C devices, and RESTful protocol for Juniper devices. It supports automatic protocol version identification and fault switching. The multi-device acquisition unit deploys two acquisition nodes, covering all network devices. The data aggregation unit forms a multi-source heterogeneous data pool. The preliminary data quality verification unit removes obviously invalid and duplicate data.

[0040] The data cleaning and standardization module includes a data denoising unit, a data deduplication unit, a vendor data parsing unit, a unified model building unit, and a data standardization unit. The vendor data parsing unit has a built-in mainstream vendor data model library, which is updated monthly. The device and link abstract model built by the unified model building unit supports expansion. The data standardization unit ensures that the data format is unified and compatible.

[0041] The virtual data packet construction module includes a scenario configuration unit, a 5-tuple generation unit, a data packet identification unit, and a data packet scheduling unit. The scenario configuration unit has built-in preset templates for general office work and file transfer, and supports custom scenarios. The 5-tuple generation unit generates diverse virtual data packets. The data packet scheduling unit uses a load balancing algorithm to distribute data packets to simulation nodes.

[0042] The path simulation engine module includes a packet access unit, a hierarchical forwarding simulation unit, a policy application unit, a hop-by-hop recording unit, and a simulation result summary unit. The hierarchical forwarding simulation unit simulates real forwarding logic and supports multiple topologies. The policy application unit accurately simulates various network policies. The hop-by-hop recording unit forms a complete forwarding path record. The simulation result summary unit generates a simulation report.

[0043] The digital map construction module includes a topology data import unit, a data fusion unit, a map generation unit, and a function integration unit. The data fusion unit uses a deep learning association algorithm with an association accuracy of ≥99.5%. The map generation unit supports two-dimensional and three-dimensional display. The function integration unit integrates functions such as path visualization and strategy impact analysis.

[0044] Data storage module: It adopts a distributed storage architecture with multiple backups, and the data storage period is set to 3 years. It supports data query, export and deletion.

[0045] System management module: Supports multi-user creation and permission allocation, with different roles having different operation permissions; supports module start, stop, restart, and troubleshooting; supports custom configuration of parameters such as acquisition frequency and simulation speed.

[0046] Report generation module: Generates reports on data collection, data standardization, simulation results, network operation status, etc., supports line charts and bar charts, supports PDF and Excel export, and can be set to automatically generate reports every Monday and push them to the email of maintenance personnel.

[0047] When this system is running, the modules work together to achieve closed-loop management of the entire process, including data collection, cleaning and standardization, virtual data packet construction, path simulation, digital map construction, and operation and maintenance support. The constructed network digital map is highly consistent with the real network environment, effectively solving problems such as heterogeneous equipment from multiple vendors, difficulty in fault location, and inaccurate path prediction, thereby improving network operation and maintenance efficiency and security.

[0048] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope disclosed in the present invention, and these should all be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method and system for constructing a network digital map based on full-network operation data collection and routing simulation, characterized in that, Includes the following steps: Step 1: Data Acquisition Layer Deployment and Multi-Source Data Acquisition. Through the collaborative deployment of five protocols—SNMP, SSH, Netconf, gRPC, and RESTful—comprehensive data acquisition from all communication devices in the network is achieved. The acquisition scope covers network device operational protocol data, complete configuration information, real-time status data, full routing tables, complete forwarding tables, and various policy configurations. Operational protocol data includes various routing protocols, link protocols, and management protocol data exchanged between devices. Configuration information includes all device-related configuration parameters such as hardware configuration, software version, port configuration, and VLAN configuration. Status data includes device CP... Real-time operating metrics include U utilization, memory usage, port traffic, and link connectivity status. The routing table includes complete entries for static and dynamic routes, along with route priority information. The forwarding table includes Layer 2 and Layer 3 forwarding entries, along with forwarding priority information. Policy configurations include complete configuration parameters for Access Control Lists (ACLs), Policy Routing (PBR), Network Address Translation (NAT), and various security policies. All collected data is aggregated in real-time to form a multi-source heterogeneous data pool. The data pool supports multi-threaded concurrent writing to ensure the real-time performance of data collection. Timestamps and device identifiers are added to each type of data for easy data traceability and correlation analysis. Step 2: Multi-source heterogeneous data cleaning and standardization. For the data collected in Step 1 from different vendors' devices (including mainstream vendors such as Cisco, Huawei, H3C, and Juniper), the data is first denoised to remove invalid, duplicate, and abnormal data. Invalid data includes data with incorrect formatting or missing fields. Duplicate data is identified and deduplicated using a combination of device identifier and timestamp. Abnormal data includes operational status data outside the normal range and logically contradictory configuration data. Subsequently, the data models of different vendors' devices are analyzed to extract the core fields and data formats of each vendor's data. A unified device and link abstract model is constructed, mapping the device types, configuration parameters, and protocol data of different vendors to the corresponding fields of the unified model, achieving complete masking of vendor differences. The standardized data is stored in a unified data format to ensure compatibility in subsequent data processing and simulation. A data standardization report is also generated, recording the data cleaning process, standardization mapping rules, and data quality assessment results. Step 3: Precise construction of virtual data packets. Based on actual network operation scenarios, generate simulated data packet quintuples covering different application scenarios. The quintuple specifically includes source IP address, destination IP address, source port, destination port, and protocol type. The source IP address and destination IP address cover all network segments within the network and external interactive network segments. The source port and destination port cover commonly used service ports and custom ports. The protocol type includes common network protocols such as TCP, UDP, and ICMP. At the same time, a unique identifier is added to each virtual data packet to record the data packet construction parameters and generation time. The data packet generation frequency can be dynamically adjusted according to the network scale. A set of data packets is generated every 10 seconds for small-scale networks and every 5 seconds for medium-to-large-scale networks to ensure the comprehensiveness and accuracy of path simulation. The virtual data packet, as the sole input to path simulation, directly determines the realism of the simulation results. Step 4: Deployment of the path simulation engine and full-process path simulation. Deploy an independent path simulation engine. The engine adopts a distributed architecture, supports multi-threaded parallel simulation, and the simulation process strictly follows the routing and forwarding principles of real networks. Specifically, it includes the following sub-steps: Sub-step 4.1: Data packet access simulation. Connect the virtual data packet constructed in step 3 to the simulation engine to simulate the process of the data packet entering the real network from the network entry device and record the device and port information of the data packet access. Sub-step 4.2: Layered forwarding simulation. After the simulated data packet enters the switch / router, it sequentially performs a Layer 2 MIB forwarding table lookup and a Layer 3 routing table lookup. The Layer 2 MIB forwarding table lookup is based on MAC address for port matching and strictly follows the Layer 2 forwarding rules. The Layer 3 routing table lookup is based on destination IP address for route matching, prioritizing the longest prefix route, while also considering route priority and route metric to ensure that the forwarding path is consistent with the real network. Sub-step 4.3: Precise application of network policies. The simulation engine reads the device policy configuration collected in step 1 in real time, identifies and precisely applies the ACL, PBR, security policies and NAT translation rules configured on the device. Among them, ACL rules are matched in the order of configuration to determine whether the data packet is allowed to pass. PBR rules take precedence over ordinary routing rules and adjust the forwarding path of the data packet according to the policy configuration. NAT translation rules translate the IP address or port of the data packet according to the configuration type (static NAT, dynamic NAT, PAT). Security policies filter and control the data packet according to the configured security rules. The policy hit status and execution results are recorded in the entire policy application process. Sub-step 4.4: Hop-by-hop forwarding and data recording. Based on the above forwarding rules and policy application results, simulate the hop-by-hop forwarding process of data packets in the network, record the device identifier, device interface, forwarding action, policy hit status and forwarding time for each hop, and form a complete forwarding path record. The forwarding path record contains all the devices and link information that the data packet passes through from the source IP to the destination IP, ensuring the traceability of the forwarding path. Step 5: Dynamic construction and functional integration of network digital map. The path simulation results from Step 4 are deeply integrated with the network topology data. The topology data includes information such as device location, device type, link connection relationship, and link bandwidth. The integration process uses a data association algorithm to accurately associate the devices and links in the simulation path with the corresponding nodes in the topology data, generating a dynamic network digital map. This digital map supports path visualization, which can intuitively display the complete forwarding path of data packets, the device status of each hop, and the policy hit status. It supports path query, path comparison, and path filtering. It also supports policy impact analysis, which can simulate the changes in forwarding paths after modifying a certain policy and analyze the scope and degree of the policy's impact on network paths. It supports fault simulation function, which can simulate scenarios such as equipment failure and link interruption, view the impact of faults on forwarding paths, and predict fault risks in advance; It supports change simulation function, which can simulate network configuration changes, equipment expansion and other operations to verify the rationality of the changes and avoid network failures caused by the changes; Digital maps synchronize network operation status data and simulation results in real time, ensuring the dynamism and accuracy of the maps.

2. The network digital map construction method and system based on full-network operation data collection and routing simulation according to claim 1, characterized in that: The system includes a data acquisition module, a data cleaning and standardization module, a virtual data packet construction module, a path simulation engine module, and a digital map construction module. The modules are connected through a high-speed data bus to achieve real-time data transmission and collaborative work. Each module has a fault self-diagnosis function. When a module fails, it immediately sends a fault prompt to the system management terminal and records the fault information (including the faulty module, fault time, and fault type). At the same time, the backup module is activated to ensure the continuous and stable operation of the system. The system supports distributed deployment and can flexibly expand the number of module nodes according to the network size. The data acquisition module includes a protocol adaptation unit, a multi-device acquisition unit, a data aggregation unit, and a preliminary data quality verification unit. The protocol adaptation unit supports simultaneous adaptation to five protocols: SNMP, SSH, Netconf, gRPC, and RESTful. It can automatically select the optimal acquisition protocol based on the protocol support of different vendors' devices, ensuring data acquisition compatibility and integrity. The multi-device acquisition unit deploys multiple acquisition nodes, each responsible for data acquisition from devices within a certain range. It supports acquisition from various network devices such as switches, routers, firewalls, and load balancers. The acquisition frequency can be dynamically adjusted according to device type: core devices (such as core routers and firewalls) acquire status data every 1 second, while ordinary devices acquire status data every 5 seconds. The data aggregation unit aggregates the data collected by each acquisition node in real time, classifies and stores it according to data type, forming a multi-source heterogeneous data pool. It also adds device identifiers, acquisition timestamps, and acquisition node identifiers to each data type. The preliminary data quality verification unit performs preliminary verification of the acquired data, identifying obvious invalid and duplicate data, and initially removing them to reduce the workload of subsequent data cleaning. The data cleaning and standardization module includes a data denoising unit, a data deduplication unit, a vendor data parsing unit, a unified model building unit, and a data standardization unit. The data denoising unit uses a statistical threshold-based denoising algorithm to remove abnormal data exceeding the normal range, as well as invalid data with format errors or missing fields, while also repairing fuzzy data. The data deduplication unit identifies and removes duplicate data using a combination of device identifier, timestamp, and data type, ensuring data uniqueness. The vendor data parsing unit performs in-depth analysis of device data models from different vendors such as Cisco, Huawei, H3C, and Juniper, extracting core fields, data formats, and mapping relationships from each vendor's data, and generating a vendor data parsing report. Based on the parsing results, the unified model building unit constructs a unified device and link abstract model, defining unified field specifications, data formats, and encoding rules to achieve unified mapping of data from different vendors. The data standardization unit performs format conversion, field mapping, and encoding standardization on the parsed vendor data according to the unified model specifications, generating standardized data and verifying the accuracy and compatibility of the standardized data to ensure that the standardized data can be directly used in subsequent simulation and map building processes. The virtual data packet construction module includes a scenario configuration unit, a 5-tuple generation unit, a data packet identification unit, and a data packet scheduling unit. The scenario configuration unit allows users to configure the generation scenario of virtual data packets according to the actual network operation scenario, including normal communication scenarios, high-concurrency scenarios, and abnormal communication scenarios. Different scenarios correspond to different data packet generation parameters. The 5-tuple generation unit generates virtual data packet 5-tuples covering different network segments, ports, and protocol types based on the scenario configuration parameters, ensuring the diversity and comprehensiveness of data packets. During the generation process, it strictly follows network address allocation rules to avoid generating invalid IP addresses and ports. The data packet identification unit adds a unique identifier code to each virtual data packet, containing information such as scenario type, generation time, and data packet sequence number, facilitating data packet traceability and result correlation in subsequent simulation processes. The data packet scheduling unit dynamically adjusts the generation frequency and sending order of data packets according to the network scale and simulation requirements, ensuring the efficiency of path simulation and avoiding simulation lag due to excessive data packets. Simultaneously, it sends the generated virtual data packets to the path simulation engine module in sequence. The path simulation engine module includes a packet access unit, a layered forwarding simulation unit, a policy application unit, a hop-by-hop recording unit, and a simulation result aggregation unit. The packet access unit receives virtual packets sent by the virtual packet construction module, records the packet access information, simulates the process of packets entering the network, and simultaneously verifies the validity of the packets. The layered forwarding simulation unit simulates the Layer 2 and Layer 3 forwarding processes of a real network, calls standardized routing tables and forwarding table data, and sequentially performs Layer 2 MIB forwarding table lookups and Layer 3 routing table lookups, strictly adhering to real forwarding rules to ensure the authenticity of the forwarding process. The policy application unit reads standardized device policy configurations. The system identifies ACL, PBR, security policies, and NAT translation rules. Following the policy execution order of the real network, it performs policy matching and execution on data packets, recording the hit status and execution result of each policy. The hop-by-hop recording unit records all information during the hop-by-hop forwarding process of data packets, including the device identifier, device interface, forwarding action, policy hit status, forwarding time, and forwarding status for each hop, forming a complete forwarding path record. The simulation result summarization unit summarizes the forwarding path records and policy execution results of all virtual data packets, generating a simulation result report. The report includes simulation data packet information, forwarding path details, policy application status, and simulation accuracy assessment. The digital map construction module includes a topology data import unit, a data fusion unit, a map generation unit, and a function integration unit. The topology data import unit imports network topology data, including device location, device type, link connection relationships, link bandwidth, and device configuration information, supporting batch import and real-time updates of topology data. The data fusion unit uses a data association algorithm to deeply fuse the simulation results generated by the path simulation engine module with the topology data, establishing a correspondence between simulated paths and topology nodes to ensure consistency between simulation results and topology data. The map generation unit generates a dynamic network digital map based on the fused data. The map uses visualization technology to intuitively display network devices, link connections, and packet forwarding paths, supporting basic operations such as map zooming, panning, and node querying. The function integration unit integrates functions such as path visualization, policy impact analysis, fault simulation, and change pre-simulation, providing users with comprehensive network visualization and operation and maintenance support, while also supporting real-time updates of map data to ensure consistency between the map and the actual network operating status.

3. The network digital map construction method and system based on full-network operation data acquisition and routing simulation according to claim 2, characterized in that: The protocol adaptation unit in the data acquisition module performs dedicated adaptation and optimization for the protocol characteristics of different manufacturers' devices. For Cisco devices, it optimizes the connection stability of the SSH protocol, adopts key authentication to ensure the security of data acquisition, and adapts to Cisco's proprietary CDP protocol to supplement the acquisition of device neighbor information. For Huawei devices, it optimizes the transmission efficiency of the Netconf protocol, supports batch data acquisition, and adapts to Huawei's proprietary LLDP-MED protocol to acquire detailed device status information. For H3C devices, it optimizes the compatibility of the gRPC protocol to ensure the complete acquisition of configuration data. For Juniper devices, it optimizes the response speed of the RESTful protocol to improve data acquisition efficiency. The protocol adaptation unit supports automatic identification and adaptation of protocol versions, is compatible with different versions of protocols, avoids data acquisition failures due to protocol version incompatibility, and has an automatic protocol failure switching function. When the acquisition of a certain protocol fails, it automatically switches to another available protocol to ensure the continuity of data acquisition.

4. The network digital map construction method and system based on full-network operation data acquisition and routing simulation according to claim 2, characterized in that: The vendor data parsing unit within the data cleaning and standardization module has a built-in vendor data model library. This database contains device data models from mainstream vendors such as Cisco, Huawei, H3C, and Juniper, covering configuration parameters, protocol data formats, field definitions, and other information for different models of devices from each vendor. The database supports real-time updates and is updated monthly to supplement data models for newly added device models and protocol versions. The vendor data parsing unit uses machine learning algorithms to automatically parse the data of newly added vendor devices, extracting core fields and mapping relationships without manual intervention, reducing parsing costs. It also generates parsing logs to record anomalies and processing results during the parsing process, facilitating subsequent troubleshooting. The unified model building unit constructs a device abstract model containing core modules such as basic device information, hardware configuration, software configuration, operating status, routing information, and policy configuration. The link abstract model contains core modules such as basic link information, link bandwidth, link status, and connected devices. Both models support expansion, allowing for flexible addition of fields and modules based on the type of newly added device and link.

5. The network digital map construction method and system based on full-network operation data acquisition and routing simulation according to claim 2, characterized in that: The scenario configuration unit in the virtual data packet construction module has a variety of preset scenario templates, including regular office scenarios, video conferencing scenarios, file transfer scenarios, high-concurrency access scenarios, and abnormal attack scenarios. Each scenario template has preset corresponding data packet generation parameters, including the range of five-tuples, generation frequency, and data packet size. Users can directly select preset templates or customize scenario parameters according to actual needs, including custom source IP address range, destination IP address range, port range, protocol type, and generation frequency. The data packet scheduling unit uses a load balancing algorithm to evenly distribute the generated virtual data packets to different simulation nodes of the path simulation engine module, avoiding excessive load on a single node and ensuring the high efficiency and stability of the simulation process. It also supports dynamic expansion of simulation nodes. When the simulation load is too large, the number of simulation nodes is automatically increased, and after the simulation ends, excess nodes are automatically released to save system resources.

6. The network digital map construction method and system based on full-network operation data acquisition and routing simulation according to claim 2, characterized in that: The layered forwarding simulation unit in the path simulation engine module has a built-in real routing forwarding algorithm that completely simulates the actual forwarding logic of switches and routers. The Layer 2 forwarding simulation strictly follows Layer 2 forwarding rules such as MAC address learning, broadcast storm suppression, and VLAN isolation. The Layer 3 forwarding simulation supports the forwarding logic of multiple routing protocols such as static routing, RIP, OSPF, and BGP, and strictly follows the selection rules of route priority and route metric to ensure that the forwarding path is completely consistent with the real network. The hierarchical forwarding simulation unit supports the simulation of complex network topologies, including star, tree, ring, and mesh topologies, and can adapt to networks of different sizes, from small LANs to large WANs, achieving accurate simulation. It also has a simulation speed adjustment function, allowing users to adjust the simulation speed according to their needs. Fast simulation is used for initial problem investigation, while slow simulation is used for precise analysis of path details.

7. The network digital map construction method and system based on full-network operation data acquisition and routing simulation according to claim 2, characterized in that: The policy application unit in the path simulation engine module supports accurate simulation of various network policies such as ACL, PBR, NAT, and security policies. ACL simulation strictly follows the configuration order of ACL rules, matching the source IP, destination IP, source port, destination port, and protocol type of data packets to accurately determine whether a data packet is allowed to pass, while recording the number of ACL rule hits and denials. PBR simulation supports policy routing configuration based on multiple conditions such as source IP, destination IP, protocol type, and port, accurately simulating the adjustment process of forwarding paths by policy routing, and recording the PBR rule hit status and forwarding path change information. NAT simulation supports three translation types: static NAT, dynamic NAT, and PAT, accurately simulating the IP address and port translation process, and recording the IP address and port information before and after the translation. Security policy simulation supports security rules based on multiple dimensions such as application, user, and time, simulating the filtering and control of data packets by security policies, ensuring that the policy application results are consistent with the real network.

8. The network digital map construction method and system based on full-network operation data acquisition and routing simulation according to claim 2, characterized in that: The data fusion unit in the digital map construction module adopts a data association algorithm based on deep learning. By analyzing the device identifiers, link information and corresponding nodes in the topology data in the simulation results, it establishes a precise association relationship with an association accuracy of ≥99.5%, avoiding map display anomalies caused by association errors. The data fusion unit supports real-time fusion. When the network operating status changes, simulation results are updated, or topology data changes, data fusion is performed immediately to update the digital map and ensure the map's real-time performance. It also has a fusion anomaly detection function. When anomalies such as data mismatch or association failure occur during the fusion process, it immediately sends an early warning message and records the abnormal data to facilitate user troubleshooting. The map generation unit supports multiple visualization styles. Users can choose between two-dimensional and three-dimensional maps according to their needs. The two-dimensional map focuses on displaying the distribution of devices, links, and forwarding paths, while the three-dimensional map focuses on displaying the three-dimensional structure of the network and the physical location of devices. The two formats can be switched freely.

9. The network digital map construction method and system based on full-network operation data acquisition and routing simulation according to claim 2, characterized in that: The functional integration unit in the digital map construction module has the following features: Path visualization function supports filtering specific forwarding paths based on data packet identifier, source IP, destination IP, protocol type, etc. It allows for intuitive viewing of the device status, policy hit rate, and forwarding time for each hop along the path. It supports zooming in and out and detailed viewing of the path, and also supports comparative display of multiple paths, facilitating analysis of forwarding differences for different data packets. The policy impact analysis function allows users to simulate modifying a policy (such as adding, deleting, or modifying ACL rules or PBR rules), and view the changes in forwarding paths in real time. It analyzes the policy's impact on network paths, its degree of impact, and potential risks, providing data support for policy adjustments. The fault simulation function supports simulating scenarios such as device failures (e.g., device downtime, port closure) and link failures (e.g., link interruption, reduced link bandwidth), allowing real-time viewing of the impact of faults on forwarding paths, simulating the fault troubleshooting process, predicting fault risks in advance, and developing emergency response plans. The change simulation function supports rehearsing network configuration changes, device expansion, link adjustments, etc., verifying the rationality of the changes, viewing the network's operating status and forwarding path changes after the changes, avoiding network failures caused by changes, and reducing operational risks.

10. The network digital map construction method and system based on full-network operation data acquisition and routing simulation according to claim 2, characterized in that: The system also includes a data storage module, a system management module, and a report generation module. The data storage module adopts a distributed storage architecture, supports multiple copy backups, and ensures the security and integrity of the data. The stored data includes collected raw data, standardized data, virtual data packet data, simulation result data, topology data, digital map data, and system operation logs. The data storage period can be customized (1-5 years), and it supports fast data query, export, and deletion. The system management module supports user management, permission management, module management, and system parameter configuration. User management allows for the creation, modification, and deletion of multiple users. Permission management assigns permissions based on roles, with different roles having different operation permissions to ensure system security. Module management supports starting, stopping, restarting, and troubleshooting various functional modules. System parameter configuration allows for custom settings of parameters such as data acquisition frequency, simulation speed, and map update frequency. The report generation module supports generating various types of reports, including data acquisition reports, data standardization reports, simulation result reports, and network operation status reports. Reports support data statistics and chart display (line charts, bar charts, pie charts) to intuitively display network operation and system working status. Reports can be exported in PDF and Excel formats for easy archiving and analysis. It also supports automatic report generation and scheduled push notifications, and users can set the report generation frequency and push method (email, system message).

Images