A method and test terminal for testing network devices

Abstract

This specification provides a method and a test terminal for testing network devices. The method includes: obtaining a first configuration file of the network device under test; performing semantic parsing and entity relation extraction on the first configuration file to obtain a first service model graph; obtaining a corresponding target test model based on the first service model graph; and using the target test model to test the network device. This method can effectively improve test coverage and increase testing efficiency.

Description

Technical Field

[0001] This specification relates to the field of communication testing technology, and in particular to a method and test terminal for testing network devices. Background Technology

[0002] To meet the needs of cloud-network convergence and digital transformation, modern network operating systems integrate a vast number of features (such as routing, switching, security, VPN, and automation). These features are combined through thousands of lines of configuration commands to define complex business logic (such as an "EVPN VXLAN distributed gateway" service, which involves multiple layers of configuration such as Underlay, Overlay, and host access).

[0003] Currently, there are technical issues such as an explosion in equipment configuration combinations and lagging testing and verification capabilities. Among these, combinatorial explosion refers to the theoretically exponential (M^N) number of configuration combinations and interaction paths for N features. Manual testing can only cover a very small number of these "golden paths."

[0004] Capability Lagging: Existing testing instruments (such as Ixia and Spirent) are essentially high-performance traffic generators and analyzers and protocol simulators. They excel at testing the forwarding performance and protocol consistency of devices "under a given configuration," but they cannot understand the semantics of the configuration itself, let alone proactively and intelligently design destructive and exploratory test scenarios based on existing configurations.

[0005] This results in a large number of potential defects (such as configuration conflicts, resource leaks, state machine deadlocks, etc.) going undetected in the laboratory until they are triggered in the production network by specific user configuration combinations, causing business interruption and significant losses. Summary of the Invention

[0006] To overcome the problems existing in related technologies, this specification provides a method and a test terminal for testing network devices.

[0007] According to a first aspect of the embodiments of this specification, a method for testing a network device is provided, the method comprising: Obtain the first configuration file of the network device under test; Semantic parsing and entity relationship extraction are performed on the first configuration file to obtain the first business model graph; Obtain the corresponding target test model based on the first service model map, and use the target test model to test the network device.

[0008] The step of obtaining the first configuration file of the network device under test includes: Obtain the first configuration file from the network device via the NETCONF or YANG protocol; Clean the text of the first configuration file.

[0009] The step of performing semantic parsing and entity relation extraction on the first configuration file to obtain the first business model graph includes: Parsing data is obtained through syntax / lexical analysis, rule / template analysis, and semantic enhancement analysis. The obtained parsed data is substituted into the preset test business model to complete the entity relationship extraction and obtain the first business model graph.

[0010] The first business model graph includes nodes and edges, wherein nodes represent configuration entities and edges represent business relationships. The first business model graph is stored using a graph database or in-memory graph structure.

[0011] The step of obtaining the corresponding target test model based on the first service model map and testing the network device using the target test model includes: Based on the first business model diagram, stress tests and destructive tests are conducted, and the test results are displayed in the first business model diagram.

[0012] Through the above embodiments, unstructured configuration files can be transformed into a computable and reasonable business model graph through semantic parsing and entity relationship extraction. Based on the business model graph and the corresponding test model, network devices can be tested to ensure test coverage and ultimately achieve a multi-dimensional robustness quantitative evaluation system.

[0013] According to a second aspect of the embodiments of this specification, a test terminal is provided, the test terminal comprising: The acquisition module is used to acquire the first configuration file of the network device under test. The acquisition module is also used to perform semantic parsing and entity relationship extraction on the first configuration file to obtain the first business model graph; The processing module is used to obtain the corresponding target test model based on the first service model map, and to test the network device using the target test model.

[0014] Specifically, the acquisition module is used to acquire the first configuration file from the network device via the NETCONF or YANG protocol, and to perform text cleaning on the first configuration file.

[0015] The acquisition module is further configured to obtain parsed data through syntax / lexical analysis, rule / template analysis, and semantic enhancement analysis. The obtained parsed data is substituted into the preset test business model to complete the entity relationship extraction and obtain the first business model graph.

[0016] The first business model graph includes nodes and edges, wherein nodes represent configuration entities and edges represent business relationships. The first business model graph is stored using a graph database or in-memory graph structure.

[0017] Specifically, the processing module is used to perform stress tests and destructive tests based on the first business model map, and to display the test results in the first business model map.

[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this specification. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this specification and, together with the description, serve to explain the principles of this specification.

[0020] Figure 1 This is a flowchart illustrating a method for testing a network device according to an exemplary embodiment of this specification.

[0021] Figure 2 This is a schematic flowchart illustrating a method for sending messages according to an exemplary embodiment. Detailed Implementation

[0022] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this specification as detailed in the appended claims.

[0023] The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of this specification. The singular forms “a,” “the,” and “the” as used in this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0024] It should be understood that although the terms first, second, third, etc., may be used in this specification to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this specification, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0025] Currently, the mainstream testing technology involves using scripts to perform functional / regression tests on network devices based on human experience. For example, test engineers write static test cases and automated scripts based on product manuals and design documents to verify specific functions on a fixed topology.

[0026] The above methods mainly suffer from the following problems: extremely low coverage: unable to handle massive configuration combinations, resulting in huge test blind spots; fixed scenarios: difficult to simulate complex and dynamic configuration change operations in the live network; inability to perceive business: test cases are disconnected from the actual business logic currently carried by the device, resulting in weak test targeting; low efficiency: high cost of test case writing and maintenance, and inability to quickly adapt to new features and other technical issues.

[0027] Another current testing technique involves stress / performance testing based on traffic flooding and protocol simulation. For example, using specialized testing instruments, line-rate traffic is generated or a large number of protocol sessions (such as BGP neighbors) are simulated to push the device to its limits.

[0028] The above methods have the following main problems: semantic blind testing: the instrument does not understand the business purpose of the device configuration, the test is a "black box" and it is impossible to conduct accurate stress testing on the business critical path; lack of state disturbance: it mainly tests steady-state performance and lacks the ability to actively and intelligently inject and recover from the internal state of the device (routing table, ARP table, session table); and decoupling from configuration changes: it is impossible to systematically test the most important source of failure in daily operation and maintenance, which is "configuration addition, deletion, modification and query".

[0029] To address the aforementioned technical problems, this disclosure provides a method for testing network devices. This method involves establishing a service model graph of the network device's configuration file, allowing the test terminal to recognize the configuration intent. Test cases are then generated using the service model graph and a combination strategy to test the network device. Figure 1 As shown, the method includes: S101 obtains the first configuration file of the network device under test; S102 performs semantic parsing and entity relationship extraction on the first configuration file to obtain the first business model graph; S103 Obtains the corresponding target test model based on the first service model map, and uses the target test model to test the network device.

[0030] In step S101, the first configuration file (such as the full configuration file, display current-configuration) of the network device can be manually entered into the test terminal, or the test terminal can be connected to the network device under test and the first configuration file can be obtained from the network device under test through the NETCONF or YANG protocol.

[0031] In step S102, semantic parsing may include three levels, such as... Figure 2 As shown, they are: syntax / lexical analysis, rule / template analysis, and AI / semantic enhancement analysis, respectively. Syntax / lexical analysis: Based on deterministic rules, a context-free grammar (CFG) is written for a specific device OS, and a parser generator is used to generate a parse tree; Rule / template parsing: Handles variations and combinations, and identifies business relationships between objects. For example: Binding relationship: ACL is bound to the interface.

[0032] Reference relationship: The routing strategy references the prefix list.

[0033] Dependencies: VPN instances depend on routing tables, and VXLAN tunnels depend on Underlay routes for reachability.

[0034] Conflict: The same interface cannot be configured as both Access and Trunk modes simultaneously (as defined by model constraints).

[0035] This is achieved by establishing a configuration template library and a regular expression rule library. For example, it can identify various types of ACL rules and complex route maps.

[0036] Semantic Enhancement Parsing: For example, semantic enhancement parsing can be achieved through AI technology. Specifically, named entity recognition (NER) and relationship classification can be performed by fine-tuning a lightweight LLM. For instance, it can determine whether "vlan 10" is for creating a VLAN or entering a VLAN view. When a configuration references an object that is not explicitly defined in the current configuration (such as referencing a policy in another file), it can be marked as an "external dependency". Handling Unknown Commands: For new commands not covered by the parser library, it can make reasonable generalized guesses and prompt manual calibration.

[0037] In step S102, Entity Relationship Extraction (ERE) can be understood as establishing a correspondence between the configuration data obtained from the first configuration file and the entities (for example, substituting the configuration data into a preset test business model). After completing the entity relationship extraction, the first business model graph is obtained.

[0038] In this embodiment, the first business model graph can be a graph structure containing "configuration entities" as nodes and "business relationships" as edges. This structure can clearly depict "how the various modules inside the device cooperate in order to provide a certain service".

[0039] For example, Table 1 shows a node (entity) classification table: Table 1 Table 2 shows one type of edge (relationship): Table 2 The above structure allows for precise positioning of test targets: directly seeing the "arteries" (critical paths) and "joints" (dependencies) of the business, and realizing visualized business logic: making complex configuration interactions clear at a glance, no longer a "black box".

[0040] In one example, a branch office router needs to establish an IPsec VPN tunnel with the headquarters data center router (HQ-Router) so that the branch intranet (192.168.10.0 / 24) can securely access the headquarters server network segment (10.1.1.0 / 24).

[0041] The key configurations on the network device (router) are as follows: # 1. Interface Configuration interface GigabitEthernet0 / 0 # Connect to the Internet IP address 203.0.113.10 255.255.255.0 nat outbound interface GigabitEthernet0 / 1 # Connect to the branch intranet IP address 192.168.10.1 255.255.255.0 …… The first business model map output through step S102 is: "entities": [ { "id": "e001", "type": "INTERFACE_PHYSICAL", "name": "GigabitEthernet0 / 0", "attributes": {"ip_address": "203.0.113.10 / 24", "status": "up"} }, { ... In this example, a graph database (Neo4j) or an in-memory graph structure (networkx) can be used to store entity attributes for each node and relation attributes and strength for each edge.

[0042] Through the above embodiments, the established first service model graph enables the test terminal to read the first service model graph, and the algorithm can directly analyze, traverse and mutate it; thus, the test terminal can "understand" the purpose or use of the user configuration based on the configuration information of the network device, form an intent model, and subsequently use AI technology to match the corresponding target test model with the intent model, and conduct "stress experiments" and "destructive tests" based on the target test model.

[0043] Before executing step S103, a module combination test database and test cases can be built first, so as to obtain the corresponding target test model and generate the corresponding test cases based on the first business model map.

[0044] Specifically, construct a combined test database, including: 1. General combination rules: such as pairwise and N-wise combination test algorithms.

[0045] 2. Domain Risk Patterns: "Dangerous Combinations" extracted from historical vulnerabilities, such as: "Triggering route oscillation while modifying BGP aggregated routes." 3. Business Weight Table: Identifies core businesses (such as gateway configuration) and their critical paths, and should be given higher weight and more variations during testing.

[0046] Generate dynamic test cases: 1. Path traversal: Select key business paths (such as "the forwarding path of data packets from VLAN10 to VLAN20") on the business model graph.

[0047] 2. Combinatorial Mutation: Applying rules from the database to one or more configuration entities along the path to generate a sequence of change operations. For example: Add: Add a new QoS policy on the interface running OSPF.

[0048] Delete: Delete a static route or key that is being referenced by BGP.

[0049] Modify: Change the action of the ACL rule (change from permit to deny), change the IP address, etc.

[0050] Check: Frequently execute the display command to observe the status.

[0051] Preamble: The order of change operations (A then B vs. B then A).

[0052] Reinforcement learning agents can also be used to explore "which combinations of changes are more likely to trigger anomalies" in a simulated environment, thereby generating more disruptive and efficient use cases.

[0053] Based on the above, the following example demonstrates how to generate key test targets using a visualized business model graph: Use Case 1: Delete the IP address of GE0 / 0 → Expected: Tunnel0 should become invalid (because the source address is unavailable), and VPN service will be interrupted.

[0054] Use Case 2: Modify the pre-shared key of the IKE key chain → Expected outcome: IPsec negotiation fails, VPN service interrupted.

[0055] Verify policy consistency: Use Case 3: Add a deny rule to ACL 3000 → Expected outcome: VPN traffic is blocked, but other services are unaffected.

[0056] Use Case 4: Delete static route 10.1.1.0 / 24 → Expected outcome: Traffic destined for headquarters will not match the route, but the tunnel itself may still exist.

[0057] Test recovery ability: Use Case 5: First delete the IPsec policy, then reconfigure → Monitoring: Tunnel re-establishment time, service interruption duration.

[0058] Use Case 6: During peak VPN traffic periods, restart the GE0 / 0 interface → Monitor: Convergence time for service switching / recovery.

[0059] Decoupling test: Use Case 7: Modify the MTU of GE0 / 1 (not related to VPN) → Verification: VPN services should not be affected.

[0060] Based on the examples above, automated testing and robustness assessments can be performed, such as connecting the test terminal to the network device under test: 1. Test the execution engine: • Issue change operations to the device via standardized interfaces (CLI / Telnet / NETCONF).

[0061] • At the same time, Telemetry is used to collect internal device metrics (CPU / memory / queue depth / protocol state machine) at high speed.

[0062] • Integrates a chaos engineering module to randomly inject minor disturbances (such as packet loss or process restart) during the change process.

[0063] 2. Robustness assessment dimensions: • Functional correctness: After the change, are the basic business functions normal (e.g., ping success)?

[0064] • Business recoverability: After performing a "configuration rollback" or "fault recovery" operation, can the business automatically or manually recover to its original state?

[0065] • System stability: During and after the change, the device did not restart abnormally, there was no core dump, and no critical process crashes.

[0066] • Performance degradation: Whether the forwarding performance (latency, jitter) of critical services degrades within the threshold before and after the change.

[0067] • Resource leakage: After the change operation sequence is executed multiple times, are there any growth trends in resources such as memory and session tables?

[0068] 3. Decoupling Test: For "edge configurations" in the business model graph that are unrelated to the core business path, randomly add or delete them to verify whether they have no impact and ensure that the coupling between modules meets the design expectations.

[0069] As can be seen from the above embodiments, this disclosure transforms unstructured configuration files into a business model graph that can be recognized by the test terminal through semantic parsing and relation extraction. This enables the test terminal to understand the user's "configuration intent" for the network device. Based on the business model graph, combined with combinatorial testing theory and historical defect patterns, a sequence of test cases that can both guarantee coverage and have high destructiveness is dynamically generated, thereby achieving multi-dimensional robustness quantitative evaluation testing.

[0070] Based on the above method embodiments, this disclosure also provides a test terminal, which includes: The acquisition module is used to acquire the first configuration file of the network device under test. The acquisition module is also used to perform semantic parsing and entity relationship extraction on the first configuration file to obtain the first business model graph; The processing module is used to obtain the corresponding target test model based on the first service model map, and to test the network device using the target test model.

[0071] Specifically, the acquisition module is used to acquire the first configuration file from the network device via the NETCONF or YANG protocol, and to perform text cleaning on the first configuration file.

[0072] The acquisition module is further configured to obtain parsed data through syntax / lexical analysis, rule / template analysis, and semantic enhancement analysis. The obtained parsed data is substituted into the preset test business model to complete the entity relationship extraction and obtain the first business model graph.

[0073] The first business model graph includes nodes and edges, wherein nodes represent configuration entities and edges represent business relationships. The first business model graph is stored using a graph database or in-memory graph structure.

[0074] Specifically, the processing module is used to perform stress tests and destructive tests based on the first business model map, and to display the test results in the first business model map.

[0075] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of the solution in this specification according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0076] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0077] Other embodiments of this specification will readily occur to those skilled in the art upon consideration of the specification and practice of the invention claimed herein. This specification is intended to cover any variations, uses, or adaptations that follow the general principles of this specification and include common knowledge or customary techniques in the art not claimed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this specification are indicated by the following claims.

[0078] It should be understood that this specification is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this specification is limited only by the appended claims.

[0079] The above description is merely a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of protection of this specification.

Claims

1. A method for testing network devices, characterized in that, The method includes: Obtain the first configuration file of the network device under test; Semantic parsing and entity relationship extraction are performed on the first configuration file to obtain the first business model graph; Obtain the corresponding target test model based on the first service model map, and use the target test model to test the network device.

2. The method according to claim 1, characterized in that, The process of obtaining the first configuration file of the network device under test includes: Obtain the first configuration file from the network device via the NETCONF or YANG protocol; Perform text cleaning on the first configuration file.

3. The method according to claim 1, characterized in that, The step of performing semantic parsing and entity relation extraction on the first configuration file to obtain the first business model graph includes: Parsing data is obtained through syntax / lexical analysis, rule / template analysis, and semantic enhancement analysis. The obtained parsed data is substituted into the preset test business model to complete the entity relationship extraction and obtain the first business model graph.

4. The method according to claim 1, characterized in that, The first business model graph includes nodes and edges, wherein nodes represent configuration entities and edges represent business relationships; The first business model graph is stored using a graph database or in-memory graph structure.

5. The method according to claim 1, characterized in that, The step of obtaining the corresponding target test model based on the first service model map and testing the network device using the target test model includes: Based on the first business model diagram, stress tests and destructive tests are conducted, and the test results are displayed in the first business model diagram.

6. A testing terminal, characterized in that, The test terminal includes: The acquisition module is used to acquire the first configuration file of the network device under test. The acquisition module is also used to perform semantic parsing and entity relationship extraction on the first configuration file to obtain the first business model graph; The processing module is used to obtain the corresponding target test model based on the first service model map, and to test the network device using the target test model.

7. The test terminal according to claim 6, characterized in that, The acquisition module is specifically used to acquire the first configuration file from the network device via the NETCONF or YANG protocol, and to perform text cleaning on the first configuration file.

8. The test terminal according to claim 6, characterized in that, The acquisition module is also used to obtain parsed data through syntax / lexical analysis, rule / template analysis, and semantic enhancement analysis; The obtained parsed data is substituted into the preset test business model to complete the entity relationship extraction and obtain the first business model graph.

9. The test terminal according to claim 6, characterized in that, The first business model graph includes nodes and edges, wherein nodes represent configuration entities and edges represent business relationships; The first business model graph is stored using a graph database or in-memory graph structure.

10. The test terminal according to claim 6, characterized in that, The processing module is specifically used to perform stress tests and destructive tests based on the first business model map, and to display the test results in the first business model map.

Images