Network device testing system and method

Abstract

A system and method for testing devices for a computer network are disclosed. In some embodiments, test scenarios are executed on one or more virtual machines (VMs) implemented by at least one computing device, such that the one or more VMs generate input data as a result of executing the test scenarios. A network device under test (NDUT) executes the test scenarios in response to the input data from the one or more VMs and generates test result data. Whether the NDUT passed the test scenarios is detected based on the test result data.

Description

[Technical Field]

[0001] A telecommunications network system includes network devices and other types of hardware. Some of this hardware implements software to perform network functions. Occasionally, new hardware and / or software is installed in the network. However, this hardware and / or software is tested before installation into the computer network to ensure that the hardware and / or software operates properly and cannot cause other problems on the computer network. The hardware and / or software is tested in a laboratory with computer network equipment similar to that in the computer network. In this way, the hardware / software is tested to troubleshoot the hardware / software before installation into the computer network.

[0002] Aspects of the present disclosure are best understood from the following detailed description when read in conjunction with the accompanying drawings. In accordance with standard industry practice, various features are not drawn to scale. In fact, the dimensions of various features have been arbitrarily increased or decreased for clarity of illustration. [Brief explanation of the drawings]

[0003] [Figure 1] FIG. 1 is a block diagram of a computing system according to some embodiments.

[0004] [Figure 2] FIG. 2 is a block diagram of an NDUT connected to a VN according to some embodiments.

[0005] [Figure 3] 1 is a flow diagram of a method for performing tests on an NDUT, according to some embodiments.

[0006] [Figure 4] 1 is a table illustrating test scenarios for NDUT, according to some embodiments.

[0007] [Figure 5] 1 is a table illustrating test scenarios for NDUT, according to some embodiments.

[0008] [Figure 6] 1 is test result data according to some embodiments.

[0009] [Figure 7] 1 is test result data according to some embodiments.

[0010] [Figure 8] 1 is test result data according to some embodiments.

[0011] [Figure 9] 1 is a flow diagram of a method for testing devices in a computer network, according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0012] The following disclosure provides many different embodiments or examples for implementing different features of the provided subject matter. To simplify the disclosure, example components, values, operations, materials, arrangements, and the like are described below. Of course, these are examples and are not intended to be limiting. Other components, values, operations, materials, arrangements, and the like are contemplated. For example, a reference to forming a first feature on or above a second feature in the following description includes embodiments in which the first and second features are formed in direct contact with each other, and further includes embodiments in which an additional feature is formed between the first and second features such that the first and second features are not in direct contact with each other. Additionally, the present disclosure repeats reference numerals and / or letters in various examples. This repetition is for the purposes of brevity and clarity and does not dictate a relationship between the various embodiments and / or configurations described.

[0013] Additionally, spatially relative terms such as below, below, lower, above, upper, etc. are used herein for ease of description to describe the relationship of one element or feature, as illustrated in the figures, to one or more other elements or features. Spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation shown in the figures. The device may be otherwise oriented (rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein interpreted accordingly.

[0014] Systems and methods for testing devices for a computer network are disclosed. In some embodiments, the computer network is or includes a cellular network. The devices are referred to as network devices under test (NDUTs). The NDUTs are physical devices to be tested to determine whether the NDUTs are operable in the computer network, whether the physical devices fail to operate properly, or whether the tests result in problems or failures within the computer network in response to being integrated into the computer network. However, in some embodiments, the computer network includes numerous physical computer network devices. Creating replicas of physical computer network devices in a laboratory is tedious and expensive. Therefore, the present disclosure includes embodiments in which the NDUTs are tested by implementing one or more virtual machines (VMs). In some embodiments, the VMs are configured as virtual networks (VNs). The VMs in the VNs are implemented on computer devices (e.g., servers) and emulate the behavior of physical network devices. In some embodiments, the VMs generate the same input data that the NDUTs would receive when integrated into the computer network. By utilizing the VMs in the VNs, the NDUTs are tested without some or all of the physical computer network devices from the computer network. In this way, according to some embodiments, NDUTs are tested in a less cumbersome and inexpensive manner.

[0015] FIG. 1 is a block diagram of a computing system 100 according to some embodiments.

[0016] Computing system 100 includes computer network test device 102, at least one database 104, computer network 105, and user device 107. In Figure 1, computer network 105 includes cellular network 106 and Internet Protocol (IP) network 108. In some embodiments, computer network 105 includes only IP network 108. In some embodiments, computer network 105 includes only cellular network 106. In some embodiments, computer network 105 includes multiple IP networks, such as IP network 108. In some embodiments, computer network 105 includes multiple cellular networks, such as cellular network 106.

[0017] In some embodiments, the computer network test device 102 and the cellular network 106 are connected to each other via an IP network 108. In some embodiments, the IP network 108 includes a wide area network (WAN) (i.e., the Internet), a local area network (LAN), a wide area local area network (WLAN), etc. In some embodiments, the cellular network 106 includes a wireless WAN (WWAN).

[0018] The cellular network 106 includes a Radio Access Network (RAN) 160. The RAN 160 is the wireless element of the cellular network 106. The RAN 160 includes network elements such as base stations, which include one or more radio transceivers. A base station covers a land area called a cell. User equipment, such as a mobile phone, smartphone, or laptop, connects to each of the base stations that cover the cell. The RAN 160 connects to the Core 170 via a backhaul link provided by the Transport 180.

[0019] Core 170 is part of the overall cellular network 106. Core 170 enables mobile subscribers to access services (e.g., international calling, text messaging, local cellular calling). In some embodiments, Core 170 is responsible for functions such as maintaining subscriber profile information, subscriber location, service authentication, and switching functions required for voice and data sessions. Core 170 includes network elements. In some embodiments, the network elements include a Mobility Management Entity (MME), a Serving Gateway, a Multimedia Broadcast Multicast Service (MBMS) Gateway, a Broadcast Multicast Service Center (BM-SC), and a Packet Data Network (PDN) Gateway. In some embodiments, the MME is in communication with a Home Subscriber Server (HSS). The MME is a control node that handles signaling between user equipment and Core 170. Generally, the MME provides bearer and connection management. In some embodiments, Internet Protocol (IP) packets are forwarded through the Serving Gateway, which is connected to IP network 108.

[0020] Transport 180 refers to the transport network connecting the Core 170 and the RAN 160 of the cellular network 106. Transport 180 includes network elements 182, such as backhaul links, connectors, relays, voice-over-IP devices, or other suitable network elements within an embodiment of the present disclosure. In some embodiments, transport 180 includes a fronthaul connecting macrocells to small cells, radio units, digital units, etc.

[0021] The computer network testing device 102 (and in some embodiments the server 102) is a computing device that includes at least one processor 126 and a non-transitory computer-readable medium 128. The non-transitory computer-readable medium 128 stores computer-executable instructions 124. In some embodiments, the non-transitory computer-readable medium 128 includes random access memory (RAM), read-only memory (ROM), electrically erasable programmable ROM (EEPROM), optical disk storage, magnetic disk storage, other magnetic storage, a combination of the aforementioned types of computer-readable medium, or any other medium used to store computer-executable code in the form of instructions or data structures accessed by a computing device. When the processor 126 executes the computer-executable instructions 124, the processor 126 executes the computer network testing software 127.

[0022] In some embodiments, one or more new network devices (e.g., network elements (NEs)) are incorporated into the computer network 105. However, before the network devices are incorporated into the computer network 105, the performance of these new network devices is tested to ensure that the new network devices are properly incorporated into the computer network 105 without causing any issues. Therefore, troubleshooting and system performance are tested before the incorporation of the network devices. In FIG. 1 , a network device being tested before incorporation into the computer network 105 is referred to as a network device under test (NDUT) 130. In some embodiments, the NDUT 130 or a network device such as the NDUT 130 is to be incorporated into the IP network 108. In some embodiments, the NDUT 130 or a network device such as the NDUT 130 is to be incorporated into the RAN 160. In some embodiments, the NDUT 130 or a network device such as the NDUT 130 is to be incorporated into the Core 170. In some embodiments, the NDUT 130 or a network device such as the NDUT 130 is to be incorporated into the Transport 180.

[0023] To test the performance of the NDUT 130, the computer network testing computer software 127 is configured to implement one or more virtual machines (VMs) 132. The VM 132 is computer software that simulates the operation of a physical computer device. In some embodiments, the VM 130 is a software-defined computer running within the server 102 that exists through the implementation of software code. In some embodiments, a virtual version of a computer device, with a dedicated amount of CPU, memory, and storage, is rented from the server 102 (e.g., a server in a cloud provider's data center). In some embodiments, the VM 132 is defined by a computer file (e.g., an image file) that configures the VM 132 to operate like a computer device. In some embodiments, the VM 132 operates as a separate computing environment, distinct from the computing environment of the server 102 on which the VM 132 is implemented. For example, in some embodiments, the VM 132 runs a different operating system than the server 102 on which the VM 132 is implemented. In some embodiments, VM 132 is partitioned from the rest of server 102 so that software implemented within VM 132 does not interfere with the operating system of server 102.

[0024] The computer network testing software 127 is configured to automate change requests to execute test scenarios 148 on VMs 132 implemented by the computer network test computing device 102, such that one or more VMs 132 generate input data 150 as a result of executing the test scenarios 148. In some embodiments, the test scenarios 148 are test scripts stored in the database 104. In some embodiments, different test scenarios 148 are stored in the database 104 to test different types of NDUTs 130 in different situations.

[0025] In some embodiments, the test scenario 148 is selected from a plurality of test scenarios 148 for implementation by the VM 132. In some embodiments, the test scenario 148 to be performed by the VM 132 is selected by the user 130 via the user device 107. The user device 107 includes a non-transitory computer-readable medium 135 and at least one processor 139. The non-transitory computer-readable medium 135 stores computer-executable instructions 137. In response to the processor 139 executing the computer-executable instructions 137, the user device 107 is configured to receive user input and transmit the user input to the computer network testing computer software 127. The computer network testing computer software 127 processes the user input to select an appropriate test scenario 148 and receives associated parameter values ​​for the test scenario from the user device 107.

[0026] In some embodiments, VM 132 is configured to operate as a virtual network (VN) 134. In some embodiments, VN 134 simulates at least a portion of computer network 105. In some embodiments, VN 134 simulates at least a portion of IP network 108. In some embodiments, VN 134 simulates at least a portion of cellular network 106.

[0027] In some embodiments, the input data 150 is the same input data generated by the portion of the computer network 105 that is providing data to the NDUT 132 in response to the NDUT 132 being incorporated into the computer network 105. The input data is received by the NDUT 132 via one or more interfaces. Thus, in some embodiments, the VM 132 in the VN 134 generates the input data 150 in the same format and with the same signaling as the input data generated by the portion of the computer network 105 that is providing data to the NDUT 132 in response to the NDUT 132 being incorporated into the computer network 105. In some embodiments, the input data 150 is provided directly to the NDUT 130 from the VM 132 in the VN 134. In some embodiments, the input data 150 is stored as a log in the database 104. In some embodiments, the NDUT 130 receives the input data 150 after it has been stored in the database 104 by the VM 132.

[0028] The NDUT 130 generates test result data 152 in response to input data 150 from one or more VMs 132. The test result data 152 indicates the performance of the NDUT 130 under the test scenarios 148 implemented by the VMs 132. The computer network testing computer software 127 is configured to detect whether the NDUT 130 passes the implemented test scenarios 148 based on the test result data 152. In some embodiments, the test result data 152 is stored in the database 104. In some embodiments, to determine whether the NDUT 130 passes the implemented test scenarios 148, benchmark data 154 is generated by the computer network testing computer software 127 that defines one or more benchmarks for determining whether the NDUT passes the implemented test scenarios 148. In some embodiments, the benchmarks described by the benchmark data 154 are determined based on user input received from the user device 107. To detect whether the NDUT 130 passes the test scenario 148 based on the test result data 152, the computer network testing computer software 127 is configured to detect whether the test result data 152 meets the benchmarks defined in the benchmark data 154. In response to passing the test scenario 148 (or multiple test scenarios 148), the NDUT 130 or a network device such as the NDUT 130 is integrated into the computer network 105.

[0029] FIG. 2 is a block diagram of an NDUT 200 connected to a VN 202 for testing the NDUT 200, according to some embodiments.

[0030] NDUT 200 is an example of NDUT 130 in FIG. 1 , according to some embodiments. VN 202 is an example of VN 134, according to some embodiments. In some embodiments, VN 202 is implemented by computer network testing computer software 127.

[0031] In FIG. 2 , the NDUT 130 is a centralized unit (CU). More specifically, the CU is for an Open RAN (ORAN) and is an Open CU (OCU) device under test (DUT). Once the NDUT 200 is actually integrated into the computer network 105, system tests are performed to ensure that the NDUT 200 can handle capacity and network traffic by implementing one or more test scenarios using the VN 202. Different types of test scenarios are implemented to determine whether the NDUT 200 performs during handovers, data calls, PWS delivery, F1 procedures, X2C procedures, IP-Sec, or other operational materials for embodiments of the present disclosure that ensure software quality during software overload conditions and hardware performance at peak usage times. Using test result data resulting from the implementation of the test scenarios, network designers develop topology designs that maximize software and hardware resource efficiency.

[0032] The 5G RAN is divided into two physical entities, named CU (Centralized Unit) and DU (Distributed Unit). In relation to the Open Systems Interconnection (OSI) layer description, the CU provides support for the upper layers of the protocol stack, such as Service Data Adaptation Protocol (SDAP, a protocol specified by 3GPP that maps quality of service flows to bearer services), Packet Data Convergence Protocol (PDCP), which provides services to RRC and user plane upper layers, e.g., IP in the UE or relay in the base station, and Radio Resource Control (RRC is a network layer protocol used between the UE and the base station). The DU provides support for the lower layers of the protocol stack, such as Radio Link Control (RLC, a Layer 2 radio link protocol used in UMTS, LTE, and 5G), Media Access Control (MAC, a unique identifier assigned to a network interface controller (NIC) for use as a network address in communications within a network segment), and the physical layer. One CU controls multiple DUs, e.g., over 100 DUs are connected to one CU. Each DU supports one or more cells, such as cell 514, so one CU controls hundreds of cells.

[0033] VMs are denoted by the adjective virtual, and non-virtual machines and physical machines are denoted by the adjective physical throughout this description. In this embodiment, the NDUT 200 is connected to a network device in addition to the VN 202. The network device includes an open distributed unit (O-DU) 204. The O-DU 204 and the NDUT 200 are connected via an F1-C / U interface. The F1 interface connects the gNB CU to the gNB DU. This interface is applicable to a CU-DU split gNB architecture. The F1 (F1-C) control plane enables signaling between the CU and DU, and the F1 (F1-U) user plane enables the transfer of application data. The O-DU 204 is connected to an open radio unit (O-RU) 206. The purpose of the radio unit (RU) is to convert radio signals transmitted to and from the antennas into digital signals transmitted to the DU via the fronthaul. In some embodiments, the O-DU 204 is connected to the O-RU 206 via an open fronthaul (O-FH) interface. The fronthaul interface, which performs communication between the O-DU and O-RU in accordance with the O-RAN fronthaul specifications, consists of multiple hardware (HW) and software (SW) components. The NDUT 200 is connected to the LTE eNB 208 via the X2-C / U interface. Both the O-RU 206 and the LTE eNB 208 are connected to the user equipment (UE) 210. The X2 interface is divided into the X2-C and X2-U interfaces, the former for the control plane and the latter for the user plane. Both were originally designed by 3GPP to transmit information between eNBs in 4G networks or between eNBs and en-gNBs in 5G networks. In O-RAN Alliance documents, the interfaces share the same principles and protocols.

[0034] The NDUT 200 is also coupled to a VN 202. The VN 202 includes multiple VMs. The VMs include an LTE virtual core 212 and a 5G virtual core 214. The LTE virtual core 212 and the 5G virtual core 214 are connected to the Internet 216. In some embodiments, the Internet 216 is an example of the IP network 108 of FIG. 1 . The LTE virtual core 212 is connected to the NDUT 200 via an S1-C / U interface. The LTE virtual core 212 is connected to the LTE eNB 208 via the S1-C / U interface. The 5G virtual core 214 is connected to the NDUT 200 via an NGAP interface. The S1-CP is the control plane, and the S1-U is the user plane external interface (S1-U) defined between the LTE eNodeB and the LTE Serving Gateway (S-GW).

[0035] The VN 202 further includes virtual O-DUs 218. Each of the virtual O-DUs 218 is connected to the NDUT 200 via an F1-C / U interface. Each of the virtual O-DUs 218 is connected to a virtual O-RU 220 via a virtual O-FH interface. Each of the virtual O-RUs 220 is connected to a virtual UE 222. The virtual UE 222 is connected to a virtual LTE eNB 224. The virtual LTE eNB 224 is connected to the NDUT 200 via an X2-C / CU interface.

[0036] Test scenarios are then implemented using the VN 202 to determine the performance of the NDUT 200 under different traffic and network conditions to determine if the NDUT 200 has any performance issues. If not, the NDUT 200 or a network device similar (e.g., the same) as the NDUT 200 is incorporated into the cellular network 106. In some embodiments, the expense and logistics of bringing a network device emulated by the VN 202 into a lab for testing of the NDUT 200 make the benefits of using the VN 202 clear.

[0037] FIG. 3 is a flow diagram 300 of a method for performing tests on an NDUT, according to some embodiments.

[0038] Flowchart 300 includes blocks 302-318. In some embodiments, flow chart 300 is performed by computer network testing computer software 127 shown in Figure 1. In some embodiments, the NDUT is NDUT 200 shown in Figure 2, and the testing is performed using VN 202 shown in Figure 2. The flow begins at block 302.

[0039] In block 302, the computer network testing computer device 102 is prepared to run tests on the NDUT 200, and a pre-check of the computer network testing computer software 127 is performed. In some embodiments, this includes populating the VN 202 with VMs (e.g., VMs 212, 214, 218, 220, 222, and 224) and detecting whether the VMs are operational. In some embodiments, the user 130, using the user device 107, verifies that the VMs are operational and that there are no alarms or flags indicating problems with the operation of the VMs. Flow then proceeds to block 304.

[0040] In block 304, a test scenario is selected from a plurality of test scenarios for execution. In some embodiments, the user 130 selects the test scenario using the user device 107. In some embodiments, the user 130 also generates user input for the test scenario using the user device 107. The user input provides parameter values ​​for the selected test scenario. The test scenarios include a DU configuration scenario (i.e., an F1 configuration procedure), an IP-sec tunnel establishment scenario, a handover profile, and a no handover forwarding scenario. The user input includes several types of VMs to be implemented, a test duration, a description of network traffic, and the amount of network traffic. In some embodiments, the test scenario is a test script. In some embodiments, the test script is loaded in response to the selection of the test script and the user input related to the test script. Flow then proceeds to block 306.

[0041] At block 306, a connection between an interface of the NDUT 200 and a VM is established. Examples of interfaces of the NDUT 200 include F1-C / U, X2-C / U, NGAP (which provides control plane signaling between NG-RAN nodes and the Access and Mobility Management Function (AMF)), etc. Flow then proceeds to block 308.

[0042] In block 308, the computer network test computer software 127 performs the selected test scenario using the VMs in the VN 202 and the NDUT 200. As discussed above with respect to Figure 1, the VMs in the VN 202 generate input data in the same manner as the corresponding network devices under the test scenario. The input data is input to the NDUT 200. Flow then proceeds to block 310.

[0043] In block 310, the NDUT 200 generates test result data in response to the input data resulting from block 308. In some embodiments, the test result data includes reports indicating the performance of the NDUT, system stability parameters, statistical verification of 3GPP specification call flows, and any crashes or anomalous behavior by the NDUT 200.

[0044] At block 312, the test result data is analyzed to determine whether the NDUT 200 complies with the benchmark. In response to determining that the test result data generated by the NDUT 200 complies with the benchmark, the NDUT 200 passes the test scenario at block 314. In some embodiments, the user 130 then proceeds again to block 302 to perform another test scenario. In other embodiments, no further test scenarios are implemented, and the NDUT 200 or a network device such as the NDUT 200 is integrated into the cellular network 106.

[0045] In response to determining that the test result data generated by the NDUT 200 does not conform to the benchmark, the NDUT 200 fails the test scenario in block 316. In response to the NDUT 200 failing the test scenario, the computer network testing computer software 127 generates a report in block 318 regarding any software or hardware issues arising from the test scenario.

[0046] FIG. 4 is a table 400 illustrating various test scenarios for NDUT, according to some embodiments.

[0047] The test scenarios shown in table 400 relate to one embodiment of test scenarios when the NDUT is an O-CU. Table 400 also indicates whether the test scenario relates to LTE, 5G NSA, 5G SA, and test profile technologies.

[0048] Test scenarios 1-5 relate to test scenarios for setting up a portion of a cellular network with NDUT and checking the stability of the portion of the cellular network using NDUT. In some embodiments, test scenarios 1-5 relate to a VN (e.g., VN 202) that is a DU simulation environment running an end-to-end (E2E) O-RAN real-world environment.

[0049] Test scenario 1 is a test scenario that implements the F1 setup scenario to determine the minimum load of the NDUT (ie, O-CU in this example).

[0050] Test scenario 2 is a test scenario to determine the peak load of the NDUT (i.e., the maximum supported DU / cell per O-CU).

[0051] Test scenario 3 is a peak load test scenario for NDUT using IP-Sec.

[0052] Test scenario 4 is a test scenario that verifies the maximum number of associated X2C / X2U interfaces that can be connected to the NDUT.

[0053] Test scenario 5 is a test scenario that verifies the maximum number of associated NGAP interfaces that can be connected to the NDUT.

[0054] Test scenarios 6 to 11 relate to testing the performance of the NDUT for specific scenarios related to the UE being simulated by the VN.

[0055] Test scenario 6 is a test scenario to find the maximum number of supported users connecting to the NDUT.

[0056] Test scenario 7 is a test scenario to find the maximum number of users detaching from the NDUT.

[0057] Test scenario 8 is a test scenario to find the maximum number of users who register the maximum number of user devices in the NDUT.

[0058] Test scenario 9 is a test scenario to detect the number of users who unsubscribe from NDUT.

[0059] Test scenario 10 is a test scenario that tests the maximum supported users providing data upload and data download divided for each cell connected to the NDUT.

[0060] Test scenario 11 is a test scenario that tests the maximum supported users providing undivided data upload and data download for each cell connected to the NDUT.

[0061] FIG. 5 is a table 500 illustrating various test scenarios for the NDUT, according to some embodiments.

[0062] Test scenarios 12-18 relate to testing the performance of the NDUT over an extended period of time by running test scenarios that are simulated by the VN.

[0063] Test scenario 12 is a test scenario to detect the performance of the NDUT when several users use their user devices to browse the web for an extended period of time.

[0064] Test scenario 13 is a test scenario that detects the performance of the NDUT when several users use their user devices to upload and download files over an extended period of time.

[0065] Test scenario 14 is a test scenario to detect the performance of the NDUT when several users use their user devices to perform voice services for a long period of time.

[0066] Test scenario 15 is a test scenario to detect the performance of the NDUT when several users use their user devices to perform video streaming services for a long period of time.

[0067] Test scenario 16 is a test scenario that detects the performance of the NDUT when several users implement a mixture of the services implemented in test scenarios 12 to 15 over an extended period of time using their user devices.

[0068] Test scenario 17 is a test scenario to detect the performance of the NDUT when several users use their user devices to conduct an earthquake or tsunami warning system (ETWS is a type of public warning system (PWS) for notifying UEs in a specific area of ​​an emergency such as an earthquake or tsunami) test over a long period of time.

[0069] Test Scenario 18 is a test scenario designed to detect the performance of NDUT when several users implement network slicing over an extended period of time using their user devices. 5G network slicing is a network architecture that enables the multiplexing of virtualized, independent logical networks on the same physical network infrastructure. Each network slice is an isolated end-to-end network tailored to meet the diverse requirements demanded by a specific application. This technology therefore supports 5G mobile networks designed to efficiently encompass a large number of services with different service level requirements (SLRs). The realization of this service-oriented view of the network leverages the concepts of software-defined networking (SDN) and network function virtualization (NFV), which enable the implementation of flexible and scalable network slices on a common network infrastructure.

[0070] From a business model perspective, each network slice is managed by a Mobile Virtual Network Operator (MVNO). Infrastructure providers (owners of the telecommunications infrastructure) lease physical resources to MVNOs, who share the underlying physical network. Depending on the availability of allocated resources, MVNOs autonomously deploy multiple network slices customized for the various applications offered to users.

[0071] FIG. 6 is test result data according to some embodiments.

[0072] Figure 6 shows test result data for test scenario 8 in Figure 4. The test result data is a data structure 600. Data structure 600 includes du_info structures 602 and 604. Each of du_info structures 602 and 604 includes an identification field for the DU (i.e., the DUID in du_info structures 602 and 604). du_info structure 602 includes carrier information substructures 606, 608, and 610 (i.e., carrier_info in du_info structure 602). du_info structure 604 includes carrier information substructures 612, 614, and 616 (i.e., carrier_info in du_info structure 604).

[0073] Each of the carrier information substructures 606, 608, 610, 612, 614, and 616 includes a cell identity (i.e., cellidentity in the carrier information substructures 606, 608, 610, 612, 614, and 616). Each of the carrier information substructures 606, 608, 610, 612, 614, and 616 includes a XXX (i.e., nrpci in the carrier information substructures 606, 608, 610, 612, 614, and 616). Each 5G new radio (NR) cell corresponds to a physical cell ID (PCI), which is used to distinguish cells on the radio side. The PCI plan for 5G NR is similar to the PCI plan for LTE and the scrambling code plan for 3G UMTS. Each of the carrier information substructures 606, 608, 610, 612, 614, 616 includes a field that identifies the state of the cell (i.e., cell_state in the carrier information substructures 606, 608, 610, 612, 614, 616). Each of the carrier information substructures 606, 608, 610, 612, 614, 616 includes a field that identifies the service state associated with the cell (i.e., service_state in the carrier information substructures 606, 608, 610, 612, 614, 616).

[0074] FIG. 7 is test result data according to some embodiments.

[0075] 7 is test result data for test scenario 1 in FIG. 4. The test result data is a table 700 that identifies different software applications implemented on the NDUT. Table 700 identifies that the NDUT ran the applications for an extended period of 21 hours. In this embodiment, the test result data indicates that the test scenario was successful because the benchmark was whether the applications ran successfully on the NDUT for 21 consecutive hours.

[0076] FIG. 8 is test result data according to some embodiments.

[0077] 8 is test result data for test scenario 15 in FIG. 5. The test result data is a table 800 that identifies whether the NDUT connects to a user device with a particular IP address. Table 800 identifies that the NDUT established a connection with the user device. In this embodiment, the test result data indicates that the test scenario was successful because the benchmark is whether the NDUT succeeded in establishing a connection to each of the user devices identified by the IP address.

[0078] FIG. 9 is a flow diagram 900 of a method for testing devices in a computer network according to some embodiments.

[0079] In some embodiments, the flow diagram 900 is implemented by the computer network testing computer software 127 of the computer network testing computer device 102. The flow diagram 900 includes blocks 902 through 918. The flow begins at block 902.

[0080] At block 902, one or more VMs are implemented. Examples of VMs include VM 132 of Figure 1 and VMs 212, 214, 218, 220, 222, and 224 of Figure 2. In some embodiments, the VMs are configured as VNs, such as VN 134 of Figure 1 and VN 202 of Figure 2. Flow then proceeds to block 904.

[0081] At block 904, a determination is made as to whether one or more VMs are running. Flow then proceeds to block 906.

[0082] At block 906, a connection is established between an interface of the NDUT and one or more VMs. Examples of an NDUT are NDUT 130 of Figure 1 and NDUT 200 of Figure 2. According to some embodiments, an example interface is the interface of NDUT 130 shown in Figure 1. According to some embodiments, other examples of interfaces are the F1-C / U, X2-C / U, and NGAP interfaces shown in Figure 2. Flow then proceeds to block 908.

[0083] At block 908, a test scenario is selected from a plurality of scenarios for implementation of one or more VMs. An example of a test scenario is test scenario 148 of Figure 1. Other examples of test scenarios are test scenarios 1-18 described in Figures 4 and 5. Flow then proceeds to block 910.

[0084] At block 910, benchmark data is generated that defines one or more benchmarks for determining whether the NDUT passes the test scenarios. An example of benchmark data is benchmark data 154 of Figure 1. Flow then proceeds to block 912.

[0085] At block 912, a test scenario is loaded for implementation by one or more VMs. In some embodiments, the test scenario is a test script. Flow then proceeds to block 914.

[0086] At block 914, the test scenarios are executed on one or more VMs implemented by the at least one computing device, such that the one or more VMs generate input data as a result of executing the test scenarios. An example of input data is input data 150 shown in FIG. 1. Flow then proceeds to block 916.

[0087] In block 916, the NDUT is run to generate . An example of test result data is test result data 152 shown in Figure 1. Other examples of test result data are shown in Figures 6, 7, and 8. Flow then proceeds to block 918.

[0088] At block 918, a detection is made as to whether the NDUT passed the test scenario based on the test result data. In some embodiments, whether the NDUT passed the test scenario is based on a benchmark of the benchmark data.

[0089] In some embodiments, a method for testing devices for a computer network includes executing test scenarios on one or more virtual machines (VMs) implemented by at least one computer device to generate input data as a result of the one or more VMs executing the test scenarios, operating a network device under test (NDUT) in response to the input data from the one or more VMs to generate test result data, and detecting, based on the test result data, whether the NDUT passed the test scenarios. In some embodiments, the method further includes incorporating the NDUT or a second network device similar to the NDUT into the computer network in response to the NDUT passing the test scenarios. In some embodiments, the computer network includes a cellular network, and one or more VMs are configured to operate as virtual networks (VNs), where the VN simulates at least a portion of the cellular network. In some embodiments, before executing the test scenarios on the one or more VMs, the method further includes implementing the one or more VMs, detecting whether the one or more VMs are operational, and loading the test scenarios for execution by the one or more VMs. In some embodiments, before loading the test scenario for execution by the one or more VMs, the method further includes selecting a test scenario from the plurality of test scenarios for execution by the one or more VMs. In some embodiments, before executing the test scenario on the one or more VMs, the method further includes establishing a connection between an interface of the NDUT and the one or more VMs. In some embodiments, the method further includes generating benchmark data defining one or more benchmarks to determine whether the NDUT passes the test scenario, and detecting whether the NDUT passes the test scenario based on the test result data includes detecting whether the test result data satisfies the benchmarks defined in the benchmark data.

[0090] In some embodiments, an apparatus for testing devices for a computer network includes a non-transitory computer-readable medium storing computer-executable instructions and one or more processors. In response to the one or more processors executing the computer-executable instructions, the one or more processors are configured to execute test scenarios on one or more virtual machines (VMs), such that the one or more VMs generate input data as a result of executing the test scenarios, operate a network device under test (NDUT) in response to the input data from the one or more VMs to generate test result data, and detect, based on the test result data, whether the NDUT passed the test scenarios. In some embodiments, the one or more VMs are configured to operate as virtual networks (VNs), where the VNs simulate at least a portion of a cellular network. In some embodiments, before executing the test scenarios on the one or more VMs, the one or more processors are further configured to implement the one or more VMs, detect whether the one or more VMs are operational, and load the test scenarios for execution by the one or more VMs. In some embodiments, before loading the test scenarios for execution by the one or more VMs, the one or more processors are further configured to select a test scenario from a plurality of test scenarios for execution by the one or more VMs. In some embodiments, before executing the test scenarios on the one or more VMs, the one or more processors are further configured to establish a connection between an interface of the NDUT and the one or more VMs. In some embodiments, the one or more processors are further configured to generate benchmark data defining one or more benchmarks to determine whether the NDUT passes the test scenarios, and the one or more processors are configured to detect whether the NDUT passes the test scenarios based on the test result data by detecting whether the test result data satisfies the benchmarks defined in the benchmark data.In some embodiments, the test scenario includes a test script.

[0091] In some embodiments, a non-transitory computer-readable medium storing computer-executable instructions is provided, wherein in response to one or more processors executing the computer-executable instructions, the one or more processors are configured to execute test scenarios on one or more virtual machines (VMs), such that the one or more VMs generate input data as a result of executing the test scenarios, operate a network device under test (NDUT) in response to the input data from the one or more VMs to generate test result data, and detect, based on the test result data, whether the NDUT passed the test scenarios. In some embodiments, the one or more VMs are configured to operate as virtual networks (VNs), where the VNs simulate at least a portion of a cellular network. In some embodiments, before executing the test scenarios on the one or more VMs, the one or more processors are further configured to implement the one or more VMs, detect whether the one or more VMs are operational, and load the test scenarios for execution by the one or more VMs. In some embodiments, before loading the test scenarios for execution by the one or more VMs, the one or more processors are further configured to select a test scenario from a plurality of test scenarios for execution by the one or more VMs. In some embodiments, before executing the test scenarios on the one or more VMs, the one or more processors are further configured to establish a connection between an interface of the NDUT and the one or more VMs. In some embodiments, the one or more processors are further configured to generate benchmark data defining one or more benchmarks to determine whether the NDUT passes the test scenarios, and the one or more processors are configured to detect whether the NDUT passes the test scenarios based on the test result data by detecting whether the test result data satisfies the benchmarks defined in the benchmark data.

[0092] The foregoing outlines features of several embodiments so that those skilled in the art may better understand aspects of the present disclosure. Those skilled in the art will readily appreciate that this disclosure may be used as a basis for designing or modifying other processes and structures to carry out the same purposes and / or achieve the same advantages of the embodiments presented herein. Those skilled in the art will also appreciate that such equivalent constructions do not depart from the spirit and scope of the present disclosure, and that various changes, substitutions, and alterations may be made herein without departing from the spirit and scope of the present disclosure.

[0093] Aspects of the present disclosure are best understood from the above detailed description when read in conjunction with the accompanying drawings. In accordance with standard industry practice, various features are not drawn to scale. In fact, the dimensions of various features have been arbitrarily increased or decreased for clarity of illustration.

Claims

1. A method for testing a device for a cellular network, comprising: implementing one or more virtual machines (VMs) implemented by at least one computing device into a virtual network (VN) that simulates at least a portion of the cellular network; executing a user-selected test scenario on the one or more VMs of the VN under a user-selected amount of network traffic to generate input data resulting from the one or more VMs of the VN executing the test scenario; inputting the input data into a network device under test (NDUT); operating the NDUT in response to the input data to generate test result data including performance of the NDUT in response to the amount of network traffic; Detecting whether the NDUT passed the test scenario based on the test result data; Including, The one or more VMs of the VN generate the input data in the same format and with the same signaling as data provided to the NDUT under the assumption that the NDUT is embedded in the cellular network.

2. In response to the NDUT passing the test scenario, incorporating the NDUT into the cellular network. The method of claim 1 further comprising:

3. Before executing the test scenario on the one or more VMs, the method comprises: detecting whether each of the one or more VMs is operational; loading the test scenario for execution by the one or more VMs; The method of claim 1 further comprising:

4. Prior to loading the test scenario for execution by the one or more VMs, the method comprises: allowing a user to select the test scenario from a plurality of test scenarios for execution by the one or more VMs; The method of claim 3 further comprising:

5. Before executing the test scenario on the one or more VMs, the method comprises: Establishing a connection between an interface of the NDUT and the one or more VMs. The method of claim 1 further comprising:

6. generating benchmark data defining one or more benchmarks for determining whether the NDUT passes the test scenarios; further comprising detecting whether the NDUT passes the test scenario based on the test result data; Detecting whether the test result data meets the benchmarks defined in the benchmark data. The method of claim 1 , comprising:

7. An apparatus for testing a device for a cellular network, comprising: implementing one or more virtual machines (VMs) implemented by at least one computing device into a virtual network (VN) that simulates at least a portion of the cellular network; executing a user-selected test scenario on the one or more VMs of the VN under a user-selected amount of network traffic, such that the one or more VMs of the VN generate input data as a result of executing the test scenario; operating a network device under test (NDUT) in response to the input data from the one or more VMs to generate test result data including performance of the NDUT in response to the amount of network traffic; Detecting whether the NDUT passed the test scenario based on the test result data. It is structured as follows: The apparatus, wherein the one or more VMs of the VN generate the input data in the same format and with the same signaling as data provided to the NDUT under the assumption that the NDUT is embedded in the cellular network.

8. prior to executing the test scenario on the one or more VMs; Detecting whether each of the one or more VMs is operational; Loading the test scenario for execution by the one or more VMs. The apparatus of claim 7 further configured to:

9. prior to loading the test scenario for execution by the one or more VMs; allowing a user to select the test scenario from a plurality of test scenarios for execution by the one or more VMs; The apparatus of claim 8 further configured to:

10. prior to executing the test scenario on the one or more VMs; Establishing a connection between an interface of the NDUT and the one or more VMs The apparatus of claim 7 further configured to:

11. generating benchmark data defining one or more benchmarks for determining whether the NDUT passes the test scenarios; It is further structured as follows:

8. The apparatus of claim 7, configured to detect whether the NDUT passes the test scenario based on the test result data by detecting whether the test result data meets the benchmark defined in the benchmark data.

12. The apparatus of claim 7 , wherein the test scenario comprises a test script.

13. On the computer, implementing one or more virtual machines (VMs) implemented by at least one computing device into a virtual network (VN) that simulates at least a portion of the cellular network; causing a user-selected test scenario to be executed on the one or more VMs of the VN under a user-selected amount of network traffic, such that the one or more VMs of the VN generate input data as a result of executing the test scenario; operating a network device under test (NDUT) in response to the input data from the one or more VMs to generate test result data including performance of the NDUT in response to the amount of network traffic; Detecting whether the NDUT passed the test scenario based on the test result data; A computer program, wherein the one or more VMs of the VN generate the input data in the same format and with the same signaling as data provided to the NDUT under the assumption that the NDUT is embedded in the cellular network.

14. Before executing the test scenario on the one or more VMs, the computer detecting whether each of the one or more VMs is operational; Loading the test scenario for execution by the one or more VMs.

14. A computer program according to claim 13.

15. Prior to loading the test scenario for execution by the one or more VMs, the computer: Prompting a user to select the test scenario from a plurality of test scenarios for execution by the one or more VMs.

15. A computer program according to claim 14.

16. Before executing the test scenario on the one or more VMs, the computer Establishing a connection between an interface of the NDUT and the one or more VMs.

14. A computer program according to claim 13.

17. The computer: generating benchmark data defining one or more benchmarks for determining whether the NDUT passes the test scenarios; 14. The computer program of claim 13, further comprising: detecting whether the test result data satisfies the benchmark defined in the benchmark data, thereby detecting whether the NDUT passes the test scenario based on the test result data.

Images