Method and management component for testing a virtualized communication system

Abstract

According to one embodiment, a test management component for a communication system is provided, comprising: The system includes a test input interface configured to receive a request for a test to be performed in the communication system and a specification of the test; a processing unit configured to determine, according to the specification of the test, a plurality of communication network components of the communication system that need to be involved in the test in order to perform the test, and for each communication network component, determine a respective test controller that can manage the communication network component with respect to the test, and for each determined test controller, determine a specification of a sub-test to be performed by the test controller; and a test control interface configured to transmit, for each determined test controller, the determined specification of the sub-test to be performed by the test controller to the test controller.

Description

[Technical Field]

[0001] The present disclosure relates to a test management component, a test controller, a method for managing and organizing testing of a communication system, and a method for processing requests for testing of a communication system. [Background technology]

[0002] A connection between two communicating devices (eg, two physical server systems, or a VM and a virtual function residing on a server) may span multiple administrative domains.

[0003] For example, two systems belonging to different telecommunications operators need to communicate to support roaming operations of subscribers moving between a mobile communications network operated by one operator and a communications network operated by another operator's network.

[0004] These scenarios are particularly challenging with respect to testing: the test controller (i.e., the entity that executes the tests and therefore needs to manage the components involved in the tests) may not actually be able to manage all the components that need to be involved, because these components belong to different operators and the test controller may not be able to manage all these components with respect to the test (e.g., because some components belong to one operator, while components that need to be involved belong to another operator and the first operator does not have access).

[0005] Therefore, an efficient approach to enable testing in multi-domain (eg, multi-operator) communication systems is desirable. Summary of the Invention

[0006] According to one embodiment, a test management component for a communication system is provided, comprising: a test input interface configured to receive a request for a test and a specification of the test to be performed on the communication system; According to the test specifications, Determine the communication network components of the communication system that must be involved in the test to perform the test; for each communication network component, determining a respective test controller capable of managing the communication network component with respect to testing; For each determined test controller, determining a specification of subtests to be performed by the test controller on communication network components of the determined plurality of communication network components that the test controller can manage for testing. a processing unit configured as follows: a test control interface configured to transmit, for each determined test controller, to the test controller a determined specification of the sub-tests to be executed by the test controller; Includes:

[0007] Further, in accordance with the above test management component, a test controller, a method for managing testing of a communication system, and a method for processing requests for testing of a communication system are provided. [Brief explanation of the drawings]

[0008] In the drawings, like reference numbers generally refer to the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the invention. In the following description, various aspects are described with reference to the following drawings: [Figure 1] 1 illustrates a mobile wireless communication system. [Figure 2]1 illustrates an architecture that includes a simplified version of the Network Functions Virtualisation Management and Orchestration (NFV-MANO) architectural framework. [Figure 3] 1 illustrates communication connections in a multi-network and multi-operator (i.e., multi-domain) environment. [Figure 4] 1 illustrates an architecture for testing in a multi-operator and virtualized communication system (i.e., multi-domain in a virtualized environment) according to one embodiment. [Figure 5] 1 illustrates an architecture for testing in a multi-operator and virtualized communication system according to another embodiment. [Figure 6] 10 illustrates an architecture for testing in a multi-operator and virtualized communication system according to a further embodiment. [Figure 7] Illustrates the interaction of Domain Tester with other management systems. [Figure 8] 1 shows an example in which a domain tester manages a first area tester and a second area tester that handle different layers for testing. [Figure 9] 10 shows examples of subtests for the network service test when considering virtualization. [Figure 10] A hierarchical architecture model and a distributed architecture model are shown. [Figure 11] Here is an example architecture where a test broker communicates with five domain testers and the receiver is changing domains. [Figure 12] An example is shown in which a test broker is deployed in an Open Radio Access Network (O-RAN) architecture. [Figure 13] 1 illustrates an exemplary flow for executing a test when a test specification is passed from an operator to one of the domain testers. [Figure 14]1 illustrates an exemplary flow for executing a test when a test specification is passed to a test broker. [Figure 15] A flow diagram showing an example of testing within a single operator Network Function Virtualization Infrastructure Connectivity Point of Presence Network Service (NFVI-Pop NS) is shown. [Figure 16] A flow diagram illustrating an example of a multi-operator NFVI-Pop intra-NS connectivity test is shown. [Figure 17] 1 illustrates test management components of a communications system. [Figure 18] 1 shows a flow diagram illustrating a method for managing testing of a communications system. DETAILED DESCRIPTION OF THE INVENTION

[0009] The following detailed description refers to the accompanying drawings, which show, by way of example, specific details and aspects of the present disclosure in which the present invention may be practiced. Other aspects may be utilized, and structural, logical, and electrical changes may be made, without departing from the scope of the present invention. Various aspects of the present disclosure are not necessarily mutually exclusive, as some aspects of the present disclosure may be combined with one or more other aspects of the present disclosure to form new aspects.

[0010] Various examples corresponding to aspects of the present disclosure are described below.

[0011] Example 1 is the test management component of the above communication system.

[0012] Example 2 is a test management component according to Example 1, wherein each test controller has a respective policy and / or can use a respective (supported) capability and (available) resource for testing, and the test management component is configured to negotiate with each test controller a sub-test to be executed by the test controller in a manner that conforms with the test controller's policy and / or is feasible with the capabilities and resources.

[0013] Example 3 is a test management component according to example 1 or 2, wherein each sub-test includes multiple test actions, and negotiation with each test controller includes determining the test actions to conform to the test controller's policies and the sub-test's specifications.

[0014] Example 4 is the test management component according to any one of Examples 1-3, wherein the test relates to an end-to-end communication test between two components of a communication system, and wherein determining a plurality of communication network components of the communication system that need to be involved in the test includes determining a communication network component that provides the end-to-end communication.

[0015] Example 5 is a test management component according to any one of Examples 1-4, wherein each test controller is capable of managing, with respect to testing, communication network components of a respective domain of a communication system associated with the test controller.

[0016] Example 6 is a test management component according to any one of Examples 1-5, wherein each domain of the communication system includes one or more sub-networks of the communication system and / or one or more layers of a protocol stack.

[0017] Example 7 is the test management component according to any one of Examples 1-6, wherein the domains associated with the different test controllers are domains of different communication network operators.

[0018] Example 8 is a test management component according to any one of Examples 1-7, wherein the test input interface is configured to receive test requests and test specifications from a support system of one of the communications network operators or from one of the test controllers.

[0019] Example 9 is a test management component according to any one of Examples 1-8, wherein the test controllers are test area controllers, each test area controller controlling a respective subset of the communication network components and / or functionality of the communication network components.

[0020] Example 10 is the test management component according to example 9, wherein the test management component is configured to create at least some of the test area controllers according to a specification of the test.

[0021] Example 11 is a test controller for part of a communications system, a test input interface configured to receive a request for a test and a specification of the test to be performed on the communication system; determining whether the test includes one or more communication network components that the test controller does not have control over for the test; If the test includes one or more communication network components that the test controller cannot manage for the test, triggering a test management system function to manage that the test is executed by the test controller and a set of one or more other test controllers that can manage the one or more communication network components that the test controller cannot manage for the test. A processing unit configured as follows: Includes:

[0022] Example 12 is the test controller of example 11, configured to execute a test if the test includes only communication network components that the test controller can manage for the test.

[0023] Example 13 is the test controller of example 11 or 12, wherein the test controller is configured to, when the test includes one or more communication network components that the test controller is unable to manage with respect to the test, execute sub-tests of the test that include only communication network components that the test controller is unable to manage with respect to the test.

[0024] Example 14 is the test controller of example 13, wherein the test input interface is configured to receive specifications of the sub-tests from a test management system that provides test management system functionality.

[0025] Example 15 is the test controller of any one of Examples 11-14, wherein the test controller includes a test management system that provides test management system functionality.

[0026] Example 16 is the test controller of any one of Examples 11-15, wherein the test management system functionality is provided by a test management system external to the test controller, and the test controller includes a test output interface configured to trigger the test management system functionality by transferring a specification of the test to the test management system.

[0027] Example 17 is the test controller of any one of Examples 11-16, wherein the test controller is configured to dynamically determine endpoints assigned to nodes and / or functions involved in a test and to execute the test using the determined endpoints.

[0028] Example 18 is the test controller of any one of Examples 11-17, wherein the test controller is configured to execute the test or sub-tests of the test with multiple test area controllers, each test area controller controlling a respective subset of the communication network components and / or functionality of the communication network components.

[0029] Example 19 is the test controller of example 18, wherein the test controller is configured to dynamically determine endpoints assigned to nodes and / or functions involved in a test or sub-test and control the test area controller to execute the test or sub-test using the determined endpoints.

[0030] Example 20 is the test controller of example 18 or 19, wherein the subset of communication network components is a subset of communication network components for a respective geographic area associated with the test area controller, and / or the subset of communication network component functions is a subset of communication network component functions for a communication layer associated with the test area controller.

[0031] Example 21 is a communication system including the test management component of any one of Examples 1-10 and the test controller of any one of Examples 11-14 or 16-20.

[0032] Example 22 is a method for managing testing of a communications system, the method comprising: receiving a request for a test and a specification of the test to be performed on a communications system; According to the test specifications, Determine the communication network components of the communication system that must be involved in the test to perform the test; for each communication network component, determining a respective test controller capable of managing the communication network component with respect to testing; For each determined test controller, determining a specification of subtests to be performed by the test controller on communication network components of the determined plurality of communication network components that the test controller can manage for testing. Steps and for each determined test controller, sending to the test controller a determined specification of the sub-tests to be executed by the test controller; Includes:

[0033] Example 23 is a method for processing a request for testing a communications system, the method comprising: receiving, by a test controller, a request for a test and a specification of the test to be performed on the communication system; determining whether the test includes one or more communication network components that the test controller is unable to manage for the test; if the test includes one or more communication network components that the test controller cannot manage for the test, triggering a test management system function to manage that the test is executed by the test controller and a set of one or more other test controllers that can manage the one or more communication network components that the test controller cannot manage for the test; Includes:

[0034] It should be noted that one or more features of any of the above examples may be combined with any one of the other examples. In particular, examples described in the context of a device are equally valid for a method.

[0035] According to further embodiments, there is provided a computer program and computer readable medium comprising instructions which, when executed by a computer, cause the computer to perform the method of any one of the above examples.

[0036] Various examples are described in more detail below.

[0037] FIG. 1 illustrates, in simplified form, a mobile wireless communication system 100 configured in accordance with Fifth Generation (5G) standards, for example, as specified by the Third Generation Partnership Project (3GPP).

[0038] The mobile radio communication system 100 includes a mobile radio terminal device 102, such as a user equipment (UE). The mobile radio terminal device 102, also called a subscriber terminal, forms the terminal side, while the other components of the mobile radio communication system 100 described below are part of the mobile communication network side, i.e., part of a mobile communication network (e.g., a public land mobile network, PLMN).

[0039] Furthermore, the mobile wireless communication system 100 includes a Radio Access Network (RAN) 103, which may include multiple radio access network nodes, i.e., base stations configured to provide radio access according to a Fifth Generation (5G) radio access technology (5G New Radio). It should be noted that the mobile wireless communication system 100 may also be configured according to Long Term Evolution (LTE) or another mobile wireless communication standard (e.g., non-3GPP access such as WiFi), although 5G is used herein as an example. Each radio access network node may provide wireless communication with a mobile wireless terminal device 102 over the air interface. It should be noted that the radio access network 103 may include any number of radio access network nodes.

[0040] The mobile radio communication system 100 further comprises a core network (5GC) 119 including an Access and Mobility Management Function (AMF) 101 connected to the RAN 103, a Unified Data Management (UDM) 104 and a Network Slice Selection Function (NSSF) 105, where in the following example the UDM may further comprise an actual UE subscription database, known for example as a Unified Data Repository (UDR). The core network 119 further comprises an Authentication Server Function (AUSF) 114, a Policy Control Function (PCF) 115 and an Application Function (AF) 120.

[0041] The core network 119 may have multiple core network slices 106, 107, and for each core network slice 106, 107, an operator (also called a mobile network operator MNO) may create multiple core network slice instances 108, 109. For example, the core network 119 includes a first core network slice 106 having three core network slice instances 108 for providing enhanced mobile broadband (eMBB) and a second core network slice 107 having three core network slice instances (CNIs) 109 for providing vehicle-to-everything (V2X).

[0042] Typically, when a core network slice is deployed (i.e., created), network functions (NFs) are instantiated or (if already instantiated) referenced to form a core network slice instance, and the network functions belonging to the core network slice instance are configured with a core network slice instance identification information.

[0043] Specifically, in the illustrated example, each instance 108 of the first core network slice 106 includes a first Session Management Function (SMF) 110 and a first User Plane Function (UPF) 111, and each instance 109 of the second core network slice 107 includes a second Session Management Function (SMF) 112 and a second User Plane Function (UPF) 113. The SMFs 110, 112 are for processing Protocol Data Unit (PDU) sessions, i.e., for creating, updating, and removing PDU sessions and managing session contexts with the User Plane Function (UPF).

[0044] Compared to legacy LTE and 5G approaches where base stations were used to provide access to the network, the RAN portion 103 can be decomposed to consider multiple different physical and virtual components. For example, the functionality of a single gNB can be provided by three different entities: a control unit (CU), one or more distributed units (DU), and one or more radio units (RU). Furthermore, these can be deployed as physical network functions (PNF) or virtual network functions (VNF), e.g., virtualized DUs.

[0045] The RAN 103 and the core network 119 form the network side of a mobile radio communication system, or in other words, form a mobile radio communication network. The mobile radio communication network and the mobile terminals accessing the mobile radio communication network together form the mobile radio communication system.

[0046] Similar to the core network 119, the RAN 103 may also be sliced, i.e., may include multiple RAN slices. RAN slices and core network slices 106, 107 may be grouped to form network slices.

[0047] Hereinafter, a "network slice" (or simply a "slice") generally refers to a core network slice, but may also include a RAN slice or even a transport network slice. In other words, each network slice may consist of network slice subnets such as a RAN subnet, a core network subnet, and a transport subnet.

[0048] The Single Network Slice Selection Assistance Information (S-NSSAI) identifies the network slice. The network slice instance (NSI) is identified by the NSI ID.

[0049] The mobile wireless communication system 100 may further include an OAM function (or entity) 116 implemented, for example, by one or more Operation, Administration and Maintenance (OAM) servers connected to the RAN 103 and the core network 119 (connections not shown for simplicity). The OAM 116 may include a Management Data Analytics Service (MDAS). The MDAS may provide, for example, analytical reports on network slice instance load.

[0050] Additionally, the core network 118 includes a Network Repository Function (NRF).

[0051] The core network 119 may further include a Network Data Analytics Function (NWDAF) 117. The NWDAF is responsible for providing network analysis and / or forecasting information upon request from the network functions.

[0052] Various network functions can be implemented on dedicated hardware, i.e., so-called Advanced Telecommunications Computing Architecture (ACTA) devices. This means that the network functions are implemented as physical network functions deployed on special hardware. In that case, the software for implementing the network functions is tightly coupled to the hardware.

[0053] However, it may also be desirable to implement network functions on non-dedicated hardware, i.e., as software applications running on virtual machines (and / or containers) deployed on commercial off-the-shelf (COTS) servers (i.e., on general-purpose or open computing platforms, i.e., devices). This is made possible through the use of virtualization technologies (e.g., hypervisors, operating system (OS) containers, etc.), which can be used to deploy network functions (e.g., SMF, PCF, etc.) as software applications running on virtual machines (VMs) and / or containers. As such, these are referred to as virtual network functions (VNFs).

[0054] Management and orchestration (MANO) is a key element of the European Telecommunications Standards Institute (ETSI) network functions virtualization (NFV) architecture. NFV-MANO is an architectural framework that coordinates network resources for cloud-based applications and the lifecycle management of virtual network functions (VNFs) and network services. Therefore, it is critical to ensure rapid and reliable NFV deployment at scale. NFV-MANO includes several functional blocks and capabilities, such as the NFV orchestrator (NFVO), VNF manager (VNFM), virtual infrastructure manager (VIM), and container infrastructure service management (CISM).

[0055] 2 shows an architecture 200 that includes a simplified version of the Network Functions Virtualization Management and Orchestration (NFV-MANO) architectural framework, which includes a collection of functional blocks and functions, data repositories used by these functional blocks, and reference points and interfaces through which these functional blocks exchange information for the purpose of managing and orchestrating NFVs.

[0056] The architecture 200 includes an operations support system and business support system (OSS / BSS) 201 and an NFV-MANO including an NFVO 202, a VNFM 203, a VIM 204, a container infrastructure service management (CISM) 205, and a CIS cluster management (CCM) 217.

[0057] The NFVO 202 manages network services (NS) 206, which in this example includes the following network functions: AMF 208, SMF 209, and PCF 210, and RAN components 207. A network service (NS) is a composition of physical and virtual network functions and / or services defined by its functional and behavioral specifications.

[0058] The architecture 200 further includes a Network Function Virtualization Infrastructure (NFVI) 211 that includes hardware and software components that create the virtualized environment in which the VNFs are deployed.

[0059] In this example, the AMF 208, SMF 209, and PCF 210 are implemented as VNFs, i.e., NFs that can be deployed on the NFVI 211. The SMF 209 is implemented by a container 212 running in a virtual machine, the PCF 210 on a virtual machine 213, and the AMF 208 in a container 214, all running on a COTS server 215. In contrast, the RAN component 207 is implemented on an ATCA device 216, i.e., is a physical network function.

[0060] The NFVO 202 manages the NS lifecycle and coordinates the overall resource usage of the NS, the VNF lifecycle is supported by the VNFM 203, and the NFVI resource management is supported by the VIM 204, CISM 205 and CCM 217, ensuring optimized allocation of required resources and connections.

[0061] The VNFM 203 is responsible for the lifecycle management of the VNFs, regardless of whether the VNFs are deployed on VMs or containers.

[0062] The VIM 204 is responsible for controlling and managing NFVI compute, storage, and network resources, typically within one operator's infrastructure domain. The CISM 205 is responsible for orchestrating and managing container infrastructure services and the containerized workloads instantiated on them. The CCM 217 is responsible for managing container clusters.

[0063] With regard to network functions, it should be noted that 3GPP is responsible for the management and configuration of network functions (NFs) and the interactions between NFs, but not for the management of virtualized network function aspects. In fact, the 3GPP functions remain the same from an application perspective, whether the network function is virtualized or not. From an ETSI NFV perspective, NFV-MANO handles the orchestration and management of both the virtualized resources and the deployed VNFs, remaining agnostic to the actual VNF application (e.g., if the VNF application is a virtualized PCF or a virtualized SMF).

[0064] Regarding VNF management, not only VNF instantiation but also other life cycle management (LCM) operations such as updates, scaling, etc. are performed by the NFV-MANO (NFVO / VNFM / CISM / VIM), and VNF configuration can be performed by the element manager through interaction with the OSS.

[0065] A Network Point of Presence (N-PoP) is a location where a network function is implemented as either a Physical Network Function (PNF) or a Virtual Network Function (VNF). A Network Function Virtualization Infrastructure Point of Presence (NFVI-PoP) is an N-PoP where a network function is or can be deployed as a Virtual Network Function (VNF).

[0066] In a virtualized environment, test operations can be categorized based on whether they represent VNF ​​application tests (e.g., virtualized SMF performance), VNF connectivity tests with other components, physical and virtual network tests, virtualization management component tests, upgrade component tests, physical resource tests, etc.

[0067] With regard to connectivity testing when virtual network functions are involved in the test, i.e. when the test is performed in a virtualized environment, the test should typically consider several different dimensions, such as: -Connection testing taking into account the underlay network -Connection testing taking overlay networks into account Different sending and receiving entities (i.e., components) such as VNF to VNF, VNFC to VNFC, VNFC to Gateway, Node to Node, etc. Different virtualization technologies (e.g., VM-based, container-based)

[0068] Connectivity tests typically include reachability tests (such as hop-to-hop and end-to-end reachability) and communication quality of service and performance tests.

[0069] The connection test may include tests on one or more layers, such as the PHY layer, the medium access control (MAC) layer, etc. (multi-layer tests may be considered in particular).

[0070] Furthermore, different types of endpoints (associated with the components involved in the test) may be used for the test (e.g., Internet Protocol (IP) address, port, Fully Qualified Domain Name (FQDN), Uniform Resource Locator (URI), etc.). Endpoints can be used to characterize senders and receivers. In virtualized environments, endpoints are not strictly associated with physical nodes and can constantly change (e.g., due to mobility / scaling / failover, etc.). For testing, dynamic identification of endpoints needs to be supported, which can be difficult to manage.

[0071] Furthermore, the communication connection between two such endpoints may extend across multiple communication networks operated by different operators, i.e., testing of the communication connection may need to be performed in a multi-network (and possibly multi-operator) environment.

[0072] FIG. 3 shows communication connections 301, 302, 303, 304 in a multi-network and multi-operator environment.

[0073] For all three communication connections 301, 302, 303, 304, the sender is in a first NFVI-PoP (e.g., a first data center) 305 under the control of operator A. The receiver is In the case of the first communication connection 301, it is in the same NFVI-PoP 305 as the sender. The second communication connection 302 is to an external entity (ie, component) 306 that resides on the public Internet 307 (eg, an external web service). The third communication connection 303 is in a second NFVI-PoP (e.g., a second data center) 308 under the control of the same operator (operator A). This is the case, for example, when the NS spans multiple NFVI-PoPs. The two NFVI-PoPs include respective customer edge (CE) devices 309, 310 for communication over a first wide area network (WAN) 311. The operator of the first wide area network (WAN) 311 may be different from operator A. The fourth communication connection 304 is to a third NFVI-PoP 312 that is under the control of a different operator (Operator B) than the first NFVI-PoP 304 (Operator A). The two NFVI-PoPs include respective CE devices 309, 313 for communication over a second wide area network (WAN) 314.

[0074] At their edges, the WANs 311 and 314 (backbone networks) include provider edge devices (PEs), which are network devices that connect to customer edge (CE) devices 309, 310, and 313. CE devices can be understood as gateway systems. Within the WANs 311 and 314, there are provider devices within each WAN segment, known as P devices.

[0075] A multi-operator communication system such as that shown in Figure 3 comprises multiple (sub)networks, each of which belongs to a different operator, i.e. for each of these networks, the respective operator can manage (e.g. has configuration and control rights) over the network (while other operators cannot manage this network).

[0076] Considering various cases that may arise in testing in a multi-operator and virtualized communication system such as that shown in Figure 3, various embodiments provide means for orchestrating end-to-end multi-operator NS / VNF testing. In particular, various embodiments provide techniques for addressing the following issues that arise in such an environment: In a virtualized environment, endpoints are constantly changing and the endpoints involved in the test need to be defined and communicated across different operator domains (i.e., (sub)networks of communication systems controlled by different operators). To test end-to-end connectivity, the actual communication path that needs to be traversed to reach the destination point from the starting point passes through different network segments (of sub-networks) that may be controlled by different operators (e.g., VNF / VNFC to CE gateway, CE to PE, PE-P, etc.). Besides manual communication between administrators, a means to orchestrate end-to-end testing in such scenarios is desirable. Provides a means to automate multi-operator end-to-end network / service test feasibility testing. A means to automatically and logically segment the network within NFVI-PoP boundaries for more granular testing is desired (typically not possible when each operator domain is considered as one flat network).

[0077] An example of a test use case in a multi-operator communication system is support for testing in roaming use cases where systems and network functions from one operator need to communicate with systems and network functions from another operator.

[0078] FIG. 4 illustrates an architecture for testing in a multi-operator and virtualized communication system (i.e., multi-domain in a virtualized environment) according to one embodiment.

[0079] In this example, a first NFVI-PoP 401 controlled by a first operator (Operator A) and a second NFVI-PoP 402 controlled by a second operator (Operator B) are involved in the test because, for example, the communication connection between a sender 403 in the first NFVI-PoP 401 and a receiver 404 in the second NFVI-PoP 40 is to be tested. The sender 403 and receiver 404 communicate via a (sub)network 405 of the first NFVI-PoP 402, a gateway 406 of the first NFVI-PoP 401, a WAN 407 (comprising P devices and PE devices as described above), a gateway 408 of the second NFVI-PoP 401 and a (sub)network 409 of the second NFVI-PoP 402.

[0080] A first operator operates a first OSS 410 connected to components of a first NFVI-PoP 401, such as one or more element managers 411, a generic OAM function 412, an NFV-MANO 413, and an NFVI 414, such as an NFVI management system. A second operator operates a second OSS 415 connected to components of a second NFVI-PoP 402, such as one or more element managers 416, a generic OAM function 417, an NFV-MANO 418, and an NFVI 419, such as an NFVI management system.

[0081] Each operator has a respective associated domain tester (also called a (domain) test controller) for each NFVI-PoP 401, 402, i.e., Operator A provides and operates a first domain tester 420 for the first NFVI-PoP 401, and Operator B provides and operates a second domain tester 421 for the second NFVI-PoP 402. Each of the first domain tester 420 and second domain tester 421 acts as a test coordinator for the components of the NFVI-PoP 401, 402 that it operates. Additionally, a third domain tester 422 is provided in the WAN 407 (by the operator of the WAN) as a test coordinator for the components of the WAN 407. The third domain tester 422 may be part of a WAN Infrastructure Manager (WIM).

[0082] Each domain tester 420-422 can be used to perform a series of tests at the NS, VNF, VNFC, node and / or network level, for example: Network testing (e.g. VNFC vs. VNFC, VNFC vs. PNF, etc.) VNF Test VNF application testing

[0083] Other function blocks, for example, the NFV-MANO function block, can also be tested (for example, NFVO-to-NFVO connectivity testing that belongs to different domains).

[0084] To support testing in a multi-operator environment (i.e., a communication system having domains (i.e., (sub)networks) controlled by different operators), a test broker (also called a test management component) 423 is provided that communicates with domain testers 420-422.

[0085] In one embodiment, the test broker is part of one of the domain testers, which is shown in FIG.

[0086] FIG. 5 illustrates an architecture for testing in a multi-operator and virtualized communication system according to another embodiment.

[0087] The architecture of Figure 5 corresponds to the architecture of Figure 4, with the difference that one of the domain testers, here domain tester 520 provided by operator A, functions as test broker 423, i.e., has test broker service capabilities (i.e., provides test management system functionality). It is declared as the master (or primary domain tester), while the other domain testers 521, 522 are declared as slaves (or secondary domain testers). Master (test broker) 520 performs the tasks of test broker 423, such as triggering tests, negotiating test specifications, and implementing test configurations.

[0088] Additionally, the domain tester may further segment at least partially its domain (i.e., the domain it controls), as shown in FIG.

[0089] FIG. 6 shows an architecture for testing in a multi-operator and virtualized communication system according to a further embodiment.

[0090] In FIG. 6, the operators and OSS are omitted for simplicity.

[0091] The architecture of Figure 6 corresponds to the architecture of Figure 4, with the difference that one of the domain testers, here the domain tester 620 of Operator A, segments (part of) its domain (specifically the network 405) into test areas (or zones) 626, each with its own area tester (or test area controller) 627. Thus, in each test area 626 in the domain, a respective area tester 627 is used to manage the tests. The area testers 627 are managed by the domain tester 620.

[0092] In the following, the functions of the test broker, domain tester, and area tester are described with reference to the hierarchical architecture model of Figure 4, but all operations and functions equally apply to the distributed architecture model of Figure 5. Similarly, all operations and functions equally described with reference to the architecture of Figure 4 (particularly test broker 423) also apply to the architecture of Figure 6. Furthermore, it should be noted that segmentation of one or more domains as shown in Figure 6 may also occur in the distributed architecture model of Figure 5.

[0093] End-to-end testing involves communication between area testers (if applicable), domain testers (if involved), and test brokers.

[0094] The tests may specifically include any possible aspect related to NS connectivity, for example, a test or sub-test may relate to any possible aspect related to network service communication. Reachability connectivity tests between nodes and / or VNFs, servers, gateway systems, etc. Network and application performance and quality of service (QoS), which refers to the quality of communication. Testing is not necessarily specific to a layer of the protocol stack, rather multi-layer aspects can be considered (PHY layer, MAC layer, L3 testing, etc.). Tests can be next hop, or end-to-end, or for a specific part of the route. Communication tests can be between physical or virtual components (e.g., VNFC to VNFC, VNFC to PNF, VNF to Gateway, PNF to Gateway, etc.). Connectivity tests can refer to underlay or overlay network components (e.g., not just L2, but L2 VPN, L3 VPN, etc.). Tests can be run across different virtualization technologies (e.g., VM-based, container-based).

[0095] Domain Tester The domain testers 420-422 perform dynamic identification of endpoints (e.g., shown as black dots, such as entry point 425 to second WAN 409) to be used for testing. This identification is dynamic because in a virtualized environment, endpoints 425 can constantly change (e.g., due to mobility, scaling, failover, etc.). The domain testers 420-422 further perform translation of Testing Intent Descriptors (TIDs) provided by the operator or test broker 423 into subtests associated with either type of virtual or physical network. Depending on the test, the endpoints 425 may be associated with a digital twin. The domain tester determines whether actual components or digital twins are used for testing, e.g., to avoid interrupting normal traffic. Each domain tester 420-422 automatically determines whether to test actual VNFs or digital twins based on policies, real-time conditions, constraints, etc. In the case of multiple test areas 626, depending on the test intent, each domain tester 420-422 performs automated management of area testers 627 within each domain boundary. Additionally, when multiple test areas 626 are used, each domain tester 420-422 performs tasks such as creating, deleting, updating, and / or configuring area testers 627, identifying area tester boundaries and area entry and exit points, etc.

[0096] For example, a test (eg, initiated by one of the operators) of the connection 424 between the sender 407 and the receiver 404 may be performed as follows. 1) An operator initiating a test passes a Testing Intent Descriptor (TID) to the domain tester 420-422 provided by the operator. The TID describes what the test is about and the high-level goal of the test. It can also provide a detailed specification of the test. 2) If the TID cannot be converted to a local action by the domain tester 420-422, it is communicated to the test broker 423. 3) The test broker 423 automatically identifies appropriate domain testers from other operators (of the domains containing the components involved in the test). In this example, all three domain testers 420-422 are involved (i.e., are from the involved domains, i.e., the domains that have the components involved in the test). 4) The test broker 423 negotiates with the domain testers 420-422 about the test features, tools, configurations, etc. to be used. This includes the domain testers 420-422 of the involved domains providing feedback on test validation and feasibility. Feasibility may include, in particular, for each domain tester, the feasibility with respect to the domain tester's policies (e.g., according to the policies of the domain tester's operator), i.e., whether the test actions required by the subtests are permitted by the policies. 5) Each domain tester executes the (sub)tests according to the results of the negotiation (and, if multiple test areas 626 are used, possibly controls the area testers 627 accordingly).

[0097] FIG. 7 illustrates the functionality of the domain tester 720 in more detail in an example where the domain tester 720 uses an area tester 727 and there are one or more network slice management system components 728, such as one or more NSMFs and / or NSSMFs.

[0098] Domains can be established in various ways, e.g. administrative domains (as in the example above) covering the case of a multi-operator environment, but also network domains covering the case of testing between different network areas within a single administrative domain (operator).

[0099] A domain tester 720 is responsible for its respective domain (i.e., the domain with which it is associated). As noted above, a domain may, for example, include components operated by one operator (i.e., components for which one operator, e.g., OSS / BSS, has configuration rights). A domain may also span multiple NFVI-PoPs of the same operator.

[0100] The domain tester 720 is aware of NFV and software-defined networking (SDN) related information (e.g., obtains topology information, operating protocols, etc.) and can perform connectivity tests on both underlay and overlay networks.

[0101] For testing, the domain tester 720 ·Interact with all appropriate management systems to support test configuration, test execution and result retrieval. An initial analysis of the test intent description (TID, e.g., "Test connectivity between VNF-1 and VNF-2") may be performed. This applies if the operator sends the TID to the domain tester 720 instead of directly to the test broker 423. Dynamically identify the endpoints (e.g., IP addresses, ports, FQDNs, URIs, or other) used for testing. Case 1: All endpoints are inside the NFVI-PoPs of the domain tester's domain. In that case, the domain tester 720 may run the test without other domain testers. This applies especially if the test is a sub-test that the test broker 423 specified for a test (including multiple domains). Case 2: An external endpoint exists. In this case, the domain tester 720 calls the test broker 423 to define an endpoint for the test within its domain (e.g., within a gateway system). This applies when the domain tester 720 receives a test from a component other than the test broker, such as from its operator's OSS (or BSS) 710 (because the test broker sent the subtest). ·Translate received (sub)test specifications into actions (tools / usages / configurations) for end-to-end testing. ·Perform feasibility study analysis. · Providing NFVI-PoP internal segmentation into test areas 626, further area testers 727 can be allocated. ·Manage area testers within the domain. Consider mixed NFVI environments (VM-based, containerized, etc.). Interacting with one or more of the OSS 710, the Test Broker 423, the Area Tester 727 in the Domain Tester NFVI-PoP 701, the EM 711, the Generic OAM Function 712, the NFV-MANO 713, other components (e.g., the NFVI Physical Infrastructure Manager, the NSMF and / or NSSMF in case of network slicing, etc.).

[0102] It should be noted that tests can also be triggered automatically due to lifecycle management actions (e.g., a VNF is created).

[0103] TID The test specification may be in the form of a test intent, specifically a Testing Intent Descriptor (TID). For example, an operator initiating a test passes a Testing Intent Descriptor (TID) to a domain tester or multiple domain testers under the operator's control. The TID provides the test specification in an abstract or very detailed form. The TID describes what the test is about and the high-level goal of the test. An example would be testing L2 connections between all components of network service NS#1.

[0104] The test management component may convert the TID of a test into a specification of sub-tests associated with any type of virtual and physical network.

[0105] Example test intent: Test L2 connectivity for network service #1. The domain tester will translate the test into the following subtests, taking into account all virtualized and physical components that make up the NS: Subtest #1: VNF1-PNF1 or PNF1 Digital Twin depending on dynamics / requirements etc. Subtest #2: VNF1-VNF2 Subtest #3: VNF2-VNF3 up to time t Subtest #4: New endpoints for VNF2-VNF3 after time t o If the endpoints are not passed in the TID, the Domain Tester will identify the endpoints (e.g. IP / Port, FQDN, URI, etc.) to be used for different sub-tests through interaction with NFV-MANO / NFVI / EM, etc. Depending on the constraints / policies / runtime capabilities / configuration / etc., in that case the domain tester may select the corresponding endpoint of the digital twin instead of the VNF / PNF etc.

[0106] Test Area The domain tester 720 may define, create, and manage area testers 727 to segment (at least partially) its domain into test areas. Deductive (before the test intent is known), in which case the domain tester selects the appropriate area tester to use for the test. After the fact, after receiving the test intent (of a test or subtest) by the domain tester, the domain tester can then create a new area tester 627 based on the needs of the experiment.

[0107] The area testers 727 may follow the standardized ETSI GS NFV-TST 011 process. The domain tester 720 keeps an inventory of all area testers 727, manages all area testers 627, and interacts with all area testers 627 to synthesize and control the tests or sub-tests that they should run.

[0108] Test area 726 is a logical abstraction within a domain boundary. · Test area 726 may cover one or more layers of the protocol stack. · Test area 726 may cover a network segment within the domain. Other segmentations of the domain (or parts thereof) into test areas 726, or even combinations of the above, may be considered.

[0109] FIG. 8 shows an example in which a domain tester 801 is configured with a first area tester 802 whose test area is the data link layer, network layer, and transport layer of a communication network 804 within the domain, and a second area tester 803 whose test area is the application layer of the communication network 804.

[0110] Endpoints are used by corresponding domain testers and area testers to identify senders and receivers (which may include intermediate senders and receivers along the end-to-end connection between the endpoint sender and endpoint receiver). In a virtualized environment, endpoints are typically constantly changing (e.g., due to mobility, scaling, failover, etc.). An area tester communicating with a domain tester can define the endpoints to use for each node or function (e.g., intermediate senders and receivers) involved in the test. This information is passed to the domain tester, which can communicate the endpoint information to another operator directly or through a test broker. Endpoints can be IP addresses, network ports, sockets, FQDNs, etc.

[0111] Dynamic identification of endpoints to be used for testing is performed by a domain tester in communication with an area tester. Depending on the test, the endpoints may be associated with a digital twin. The domain tester automatically determines whether to test the actual VNF or the digital twin based on policies, real-time conditions, constraints, etc. For example, to avoid overloading the production network, testing may be performed between the digital twin or the actual NF or VNF.

[0112] Test Broker According to various embodiments, end-to-end network service testing over multiple NFVI-POPs under the control of the same or different operators is provided, including test negotiation and related interfaces. Specifically, a multi-operator / multi-domain test broker is provided that translates end-to-end test intent into sub-domain intent (e.g., network services (NS)) that is passed to domain testers. The test broker negotiates with the domain testers on the test cases, tools, and configurations to be used.

[0113] The test broker 423 identifies administrative boundaries (i.e., boundaries between operator domains) and components (and therefore domain testers 420-422) that need to communicate (interact) for a particular test. Negotiations between domain testers 420-422 occur through the test broker 423. The test broker 423 may also support negotiations between domain testers 420-422 and the OSS (and / or BSS) 410, 415 when end-to-end testing cannot be supported.

[0114] The Test Broker is used to orchestrate, automate, and support multi-operator end-to-end testing. It can perform and support test negotiation, test processing, and test intent processing. More specifically: Use a test inventory with available domain testers 420-422. The inventory can be updated manually or dynamically. · Identify administrative boundaries and the components that need to interact (which domain testers need to interact with). · Execute end-to-end test synthesis based on test intent specifications, constraints, policies, etc. through negotiation with domain testers 420 to 422. Negotiate network demarcation points between different management components (e.g., customer edge (CE) gateways) together with the domain tester. ·Ability to correlate test results from different domain testers after test execution. Performs negotiation and identification of points of interaction between domains. Points of interaction between domains are also considered test endpoints, which may be virtual and not fixed, but are identified by the test broker. · Provides end-to-end test identifiers that map to endpoint identifiers exposed by domain testers.

[0115] As described above with reference to Figures 4 and 5, the test brokers 423, 523 may be arranged according to a hierarchical architecture model or according to a distributed architecture model. The functionality of the test broker 423 may be a public service, or the test broker 423 may be shared by multiple operators or one of the operators.

[0116] Figure 9 shows an example of a test.

[0117] The intent of the test in this example is to test the L2 connectivity between all components of the network service 900.

[0118] One or more domain testers (depending on whether the test spans multiple domains) consider all virtualized and physical components that make up the NS and translate the test into the following subtests: Subtest #1: VNF1-PNF1 or PNF1 Digital Twin depending on dynamics / requirements etc. Subtest #2: VNF1-VNF2 Subtest #3: VNF2-VNF3 up to time t Subtest #4: New endpoints of VNF2-VNF3 after time t

[0119] If the endpoints are not passed to one or more domain testers in the TID, the domain tester identifies, tracks, and updates the endpoints (e.g., IP / port, FQDN, URI, etc.) used for different subtests through interactions with NFV-MANO, NFVI, and / or EM, etc.

[0120] Depending on constraints, policies, runtime capabilities and / or configurations, etc., each domain tester may select the corresponding endpoint of the digital twin instead of the VNF, PNF, etc.

[0121] The test broker 423 analyzes the incomplete test intents that have not yet been completed by the domain testers 420-422 (but have been forwarded by the domain testers 420-422 to the test broker 423 because, for example, they span multiple domains and therefore cannot be completed by the domain testers 420-422). It finds the correct domain tester for the test according to the received test intent and starts negotiations with each found domain tester. It negotiates with each domain tester about the tests, tools, and configurations to be used.

[0122] The end-to-end test may include communication between the area tester 627 and the domain testers 620-622, and between the test broker 423 and the domain testers 420-422.

[0123] FIG. 10 shows a first diagram 1001 illustrating a hierarchical architecture model and a second diagram 1002 illustrating a distributed architecture model, along with corresponding interfaces IF-A and IF-B.

[0124] The interfaces IF-A (hierarchical) and IF-A (distributed) are similar but not identical.

[0125] For example, there may be differences regarding the primary tester (i.e., master) election process and messaging (how the primary tester is advertised to the network) for a distributed architecture. Additionally, in a distributed architecture model, when two secondary testers (i.e., slaves) need to communicate, one is the local primary tester and the other is the local secondary tester.

[0126] Example of operation on IF-A Part of a TID or TID transfer Test negotiations Test processing (e.g., execution) Transfer test results Domain Tester Registration / Unregistration Domain Tester Status IFA-B operation example Part of a TID or TID transfer Test processing (e.g., execution) Transfer test results Test area management

[0127] FIG. 11 shows an example architecture in which a test broker 1101 communicates with five domain testers 1102-1106 to test a communication connection 1107 from a sender 1113 in a first NFVI-PoP 1112 through a first WAN 1109, a second NFVI-PoP 1110, a second WAN 1111 to a receiver 1114 in a third NFVI-PoP 1108.

[0128] As indicated by arrow 1115, the receiver 1114 may have been transitioned from the second NFVI-PoP 1110 to the third NFVI-PoP 1112.

[0129] To address this, according to various embodiments, the test broker 1101 and / or domain testers 1102-1106 may consider that the test endpoints are related to logical entities and may move according to the configuration of the network service or VNF, for example, when a VNF (such as here the receiver 1114) is migrated or logically moved. In that case, a more sophisticated identification of the endpoints used, e.g., the receiver 1114, may be considered.

[0130] 12 shows an example in which a test broker 1201 is deployed in an O-RAN architecture. In this example, the domain of a domain tester 1202 is an O-Cloud 1203 (including, for example, an element manager 1204, a generic OAM function 1205, an NFV-MANO 1206, an NFVI 1207, a communication network 1208, one or more Infrastructure Management Service (IMS) components 1209, one or more Deployment Management Services (DMS) components 1210, and an area tester 1211).

[0131] The IMS component 1209 and the DMS component 1210 are connected to the respective operator's OSS 1212 via a Service Management & Orchestrator (SMO) function 1213, which is also connected to a test broker 1201 and a domain tester 1202. Each domain tester is responsible for a respective O-Cloud, and negotiations take place between the domain testers through the broker.

[0132] FIG. 13 shows an example flow for executing a test when a test specification, i.e., a test intent in the form of a TID, is passed to one of the domain testers, in this example, the first domain tester 420.

[0133] At 1301, Operator A passes the TID to the first domain tester (eg, through the OSS).

[0134] At 1302, the first domain tester analyzes the TID to determine which subtests (particularly the sender and receiver and corresponding endpoints) are involved in the test.

[0135] At 1303, the first domain tester checks whether all the determined endpoints are in its own domain. If so, the domain tester: At 1304, check the feasibility of the test and create and allocate an area tester (if configured or determined to use an area tester). At 1305, fine-tune the test according to the TID (in coordination with the area tester, if used). At 1306, the tests are run (in coordination with the area tester, if used) and the results are collected.

[0136] At 1307, the test results (for all subtests) are reported to the operator(s).

[0137] If the determined endpoint is not in the first domain tester's domain, the domain tester passes the TID or a portion thereof to the test broker, which communicates with one or more other participating domain testers (here, assumed to be only the second domain tester 421) and passes the TID (or a portion thereof relevant to the second domain tester 421, i.e., its assigned subtest, i.e., a subtest (of the overall test specified by the TID) to be executed by the second domain tester 421) to the second domain tester 421. In 1308, each participating domain tester indicates whether it can execute the test according to the TID (or TID portion) passed to it. If not, the participating domain testers (here, the first domain tester 420 and the second domain tester 421) negotiate with and / or through the test broker 423 in 1309 to derive new TIDs for their respective subtests. If the subtests assigned to each domain tester can be executed, the negotiation is completed, the subtests are executed according to steps 1304 to 1306, and the results are reported to the operator or to the operator via the broker in step 1307.

[0138] FIG. 14 shows an exemplary flow for executing a test when a test specification, i.e., a test intent in the form of a TID, is passed to the test broker 423.

[0139] At 1401, Operator A passes the TID directly to the test broker 423.

[0140] At 1402, the test broker analyzes the TID to determine which subtests (specifically, senders and receivers and corresponding endpoints) are involved in the test.

[0141] At 1403, the test broker determines whether all determined endpoints are in the domain of a single domain tester. If so, the test broker sends the TID to that domain tester and the domain tester. At 1404, check the feasibility of the test and create and allocate an area tester (if configured or determined to use an area tester). At 1405, fine-tune the test according to the TID (in coordination with the area tester, if used). At 1406, the tests are run (in coordination with the area tester, if used) and the results are collected.

[0142] At 1407, the domain tester sends the test results to the test broker, which reports the test results to one or more operators at 1408.

[0143] If the determined endpoint is not in the domain of a single domain tester, the test broker communicates with all participating domain testers and passes the TID to each of them (or the portion relevant to each domain tester, i.e., its assigned subtest, i.e., the portion relevant to the subtest (of the overall test specified by the TID) to be executed by the domain tester). In 1409, each participating domain tester indicates whether it can execute the test according to the TID (or TID portion) passed to it. If not, in 1410, the participating domain testers negotiate with and / or through the test broker 423 so that each test broker obtains a new TID for its respective subtest. If each participating test broker can execute the subtest assigned to it, each domain tester executes the subtest according to 1404-1406, and the results (of all subtests) are reported (through the test broker) in 1407 and 1408.

[0144] FIG. 15 shows a flow diagram 1500 illustrating an example of a single operator intra-domain NS connectivity test, ie, assuming that the sender and receiver of the communication connection to be tested are within the domain of a single domain tester.

[0145] An operator 1501, an OSS / BSS 1502, a domain tester 1503, and one or more components 1504 of the domain tester's domain (e.g., NFV-MANO) are involved in the flow.

[0146] At 1505 and 1506, the operator sends a specification of the test intent (eg, TID) to the domain tester 1503 via the OSS / BSS 1506.

[0147] At 1507, the domain tester 1503 translates the test intent into subtests of the component 1504 and requests feedback from the component 1504 at 1508. The component 1504 checks the feasibility state at 1509 and responds to the request of 1508 at 1510.

[0148] In 1511, the domain tester 1503 checks, depending on the response, whether an area tester is required for the test. In this example, it is assumed that an area tester is not required.

[0149] At 1512, the domain tester 1503 begins configuring the components for testing, and at 1513, test configuration and final verification is performed.

[0150] At 1514, the domain tester 1503 notifies the OSS / BSS 1502 that the configuration is complete, and at 1515 the OSS / BSS triggers the test.

[0151] At 1516, the domain tester 1503 and the component 1504 perform the tests, the results of which are collected at 1517 and reported at 1518 and 1519 by the domain tester 1503 to the operator 1501 via the OSS / BSS 1502.

[0152] FIG. 16 shows a flow diagram 1600 illustrating an example of multi-operator intra-domain NS connectivity testing, i.e., assuming that the sender and receiver of the communication connection to be tested are in domains managed by different domain testers (e.g., belonging to different operators).

[0153] The flow involves an operator 1601, an OSS / BSS 1602, a first domain tester 1603, one or more components 1604 (e.g., a first NFVI-PoP management system, resources, etc.) of the first domain tester's domain, a test broker 1605, a second domain tester 1606, and one or more components 1607 (e.g., a second NFVI-PoP management system, resources, etc.) of the second domain tester's domain.

[0154] At 1608 and 1609, the operator sends a specification of the test intent (eg, TID) to the first domain tester 1603 via the OSS / BSS 1602.

[0155] At 1610, the first domain tester 1603 translates the test intent into a sub-test, which is assumed to include a test for a component 1604 in the first domain tester's domain and a test for a component 1607 in the second domain tester's domain.

[0156] Thus, in 1611, the first domain tester 1603 sends a test intent to the test broker 1605, and in 1612, the test broker 1605 identifies which domain testers need to be involved in the test (assumed here to be the second domain tester 1606 in addition to the first domain tester 1603). In 1613, the test broker 1605 sends to the second domain tester 1606 a test intent for the subtests to be executed by the second domain tester 1606.

[0157] At 1614, the domain testers 1603, 1606 and the test broker 1605 negotiate to derive a feasible end-to-end test and fine-tune the test.

[0158] If there are feasible tests at 1615, the components 1604, 1607 of the domain tester's domain are configured accordingly. If there are no feasible tests to be performed, negotiation may also involve the OSS / BSS system.

[0159] At 1616, the first domain tester 1603 triggers the test by signaling the test broker 1605. At 1617, the test broker 1605 notifies the second domain tester 1606 accordingly.

[0160] At 1618, the domain testers 1603, 1606 and components 1604, 1607 perform tests, the results of which are collected at 1619 and reported at 1620 and 1621 by the first domain tester 1603 to the operator 1601 via the OSS / BSS 1602.

[0161] It should be noted that the test trigger can be performed by the OSS / BSS as in the example of FIG. 15 or by the domain tester as in the example of FIG. 16, but may also be triggered by, for example, a life cycle management (LCM) operation (e.g., VNF scaling).

[0162] In summary, according to various embodiments, a test management component is provided as shown in FIG.

[0163] FIG. 17 illustrates test management components 1700 of a communication system.

[0164] The test management component 1700 includes a test input interface 1701 configured to receive requests for and specifications of tests to be performed on a communication system.

[0165] The test management component 1700 executes the test according to the test specifications. Determine the communication network components of the communication system that must be involved in the test to perform the test; for each communication network component, determining a respective test controller capable of managing the communication network component with respect to testing; For each determined test controller, determining a specification of subtests to be performed by the test controller on communication network components of the determined plurality of communication network components that the test controller can manage for testing. The device further includes a processing unit 1702 configured to:

[0166] The test management component 1700 further includes a test control interface 1703 configured to send, for each determined test controller, to the test controller the determined specifications of the sub-tests to be executed by the test controller. The test control interface is also used to orchestrate end-to-end tests (e.g., fine-tune tests and collect test results from multiple domains).

[0167] In other words, according to various embodiments, a central entity is provided that can manage tests across multiple test domains (i.e., domains that are managed by different test controllers with respect to the test, e.g., because the domains belong to different operators) by determining which domain controllers (i.e., test controllers) need to be involved to execute the test and controlling them accordingly (by providing them with sub-tests to be executed so that the test is executed as a whole). A test controller's ability to manage a component with respect to a test may mean that the test controller can manage (e.g., has the rights to) the component with respect to the test. Thus, a test controller's inability to manage a component with respect to a test may mean that the test controller does not have the rights to manage the component or cannot configure the component with respect to the test.

[0168] The test management component corresponds, for example, to the test broker in the above example. In that case, the test controller corresponds to the domain tester. Alternatively, the test management component may correspond to one of the domain testers in the above example. In that case, the test controller corresponds to the area tester (also called the test area controller).

[0169] For example, a method such as that shown in FIG. 18 is performed.

[0170] FIG. 18 shows a flow diagram 1800 illustrating a method for managing testing of a communication system.

[0171] At 1801, a request for a test to be performed on a communication system and a specification of the test are received.

[0172] In 1802, according to the test specifications, A number of communication network components of the communication system that must be involved in the test to perform the test are determined; For each communication network component, a respective test controller capable of managing the communication network component with respect to testing is determined; For each determined test controller, a specification of sub-tests to be performed by the test controller on communication network components of the determined plurality of communication network components that the test controller can manage for testing is determined.

[0173] At 1803, for each determined test controller, after negotiation in case of feasibility issues, a determined specification of the sub-tests to be executed by the test controller is sent to the test controller.

[0174] The test management component corresponds, for example, to the test broker in the above example. In that case, the test controller corresponds to the domain tester. Alternatively, the test management component may correspond to one of the domain testers in the above example. In that case, the test controller corresponds to the area tester (also called the test area controller).

[0175] For example, regarding test management, One of the domains is managed by a domain tester (DT). · Domain testers are managed by operators through OSS / BSS. · A domain tester can manage one or more (test) domains belonging to the same operator. · Domain testers can interact with area testers within their domain. To manage virtualized resources or functions, the domain tester and area tester can interact with the NFV-MANO. To manage physical resources, the domain tester and area tester can interact with the NFVI management system. Considering network slicing, the domain tester can also interact with the network slice management system directly or indirectly through NFV-MANO or OSS / BSS. The Domain Tester interacts with the OSS, Test Broker, Area Tester (in the NFVI-PoP), EM, Generic OAM Function, NFV-MANO, and other entities (e.g., NFVI Physical Infrastructure Manager, NSMF and / or NSSMF in case of network slicing, etc.).

[0176] For example, a domain tester might: · Analyze TID along two dimensions: intra-domain behavior and extra-domain behavior. ○ Within the domain operation, ■ The domain tester can further logically segment the domain into test areas. ■ For each test area in the domain, an area tester is used to manage the tests. ■ Area testers are managed by the domain tester, which can either dynamically create area testers or select area testers from a predefined set. ■ The testing aspects within a domain are organized by a domain tester communicating with an area tester. ■ The domain tester identifies test area boundaries and area entry / exit points, etc., and assigns area testers to the test areas. ○Outside the domain operation, ■ If the TID translation requires communication with an external management entity, the TID or a portion of it is communicated to the test broker. ■ The test broker automatically identifies suitable domain testers from other operators. ■ The test broker negotiates with the domain tester about the test features, capabilities, tools and configurations to be used. ·Provide feedback through test brokers on test validation and feasibility. During negotiation, the test broker can interact with the OSS / BSS if end-to-end testing cannot be supported. In one embodiment, a test broker may be included, i.e., the test broker may be part of one of the domain testers (i.e., correspond to the same entity). o The domain tester that tests the broker service capabilities (i.e., has test management system functionality) is declared as the primary domain tester, and other domain testers are declared as secondary domain testers. o Primary domain tester organizes the testing. · Domain testers and area testers are aware of NFV and SDN related information (e.g., obtain topology information, operating protocols, etc.). Domain and area testers can perform connectivity tests on both underlay and overlay networks. · The Domain Tester and Area Tester interact with NFV-MANO, EM, Generic OAM Functions and NFVI to obtain management information used to support testing. ·Domain and Area Testers will interact with all appropriate management systems to support test configuration, test execution and result retrieval. Test procedures can also be automatically triggered due to lifecycle management actions (e.g., a VNF is created). ·Domain testers and area testers will jointly carry out feasibility study analysis within the domain. · Domain testers and test brokers will jointly perform feasibility study analysis for end-to-end testing.

[0177] As mentioned above, a test area is a logical abstraction. Case 1: An area may cover one or more layers of a protocol stack. · Case 2: The test area may cover a network segment within the domain. Case 3: Other formats or even combinations of the above may be considered.

[0178] An Area Tester can manage testing of one or more test areas within the same domain. Each operator domain is considered a flat network. Area Testers can automatically and logically segment the network within domain boundaries for more granular testing. Area Testers can be defined and created as follows: Case 1: Deductive (before the test intent is known), in which case the domain tester selects the appropriate area tester to use. Case 2: After the fact, after receiving the test intent by the domain tester, the domain tester can create a new area tester based on the needs of the experiment.

[0179] Area testers can follow standardized processes and processes from other inventions. A domain tester, for example, maintains an inventory of all area testers. A domain tester manages all area testers (associated with it, i.e., responsible for testing areas within the domain tester's domain) and interacts with all area testers to synthesize and manage tests.

[0180] The test controller components and the test management components may be implemented, for example, by one or more circuits. A "circuit" may be understood as any kind of logic implementation entity, which may be a dedicated circuit or processor executing software stored in memory, firmware, or any combination thereof. Thus, a "circuit" may be a hard-wired logic circuit or a programmable processor, e.g., a programmable logic circuit such as a microprocessor. A "circuit" may also be a processor executing software, e.g., any kind of computer program. Any other kind of implementation of each of the above functions may also be understood as a "circuit."

[0181] While particular embodiments have been described, it should be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the embodiments of the present disclosure as defined by the appended claims, the scope of which is accordingly indicated by the appended claims, and all changes which come within the meaning and range of equivalents of the claims are therefore intended to be embraced.

[0182] The following items relate to further embodiments.

[0183] Item 1: A test management component of a communication system, comprising: a test input interface configured to receive a request for a test to be performed on the communication system and a specification of the test; According to said specifications of said test, determining a plurality of communication network components of the communication system that need to participate in the test in order to perform the test; determining, for each communication network component, a respective test controller capable of managing said communication network component with respect to testing; For each determined test controller, determining a specification of sub-tests to be performed by the test controller on the communication network components of the determined plurality of communication network components that the test controller can manage for testing. a processing unit configured as follows: a test control interface configured to, for each determined test controller, transmit to the test controller the determined specifications of the sub-tests to be executed by the test controller; Test management components, including:

[0184] Item 2: A test management component as described in Item 1, wherein each test controller has its own policy and / or can use its own capabilities and resources for testing, and the test management component is configured to negotiate with each test controller the sub-tests to be executed by the test controller so as to conform to the policy of the test controller and / or to be feasible using the supported capabilities and available resources.

[0185] Item 3: A test management component described in item 1 or 2, wherein each sub-test includes multiple test actions, and negotiation with each test controller includes determining the test actions to conform to the test controller's policy and the specifications of the sub-test.

[0186] Item 4: A test management component described in any one of items 1 to 3, wherein the test relates to an end-to-end communication test between two components of the communication system, and determining the multiple communication network components of the communication system that need to be involved in the test includes determining communication network components that provide end-to-end communication.

[0187] Item 5: A test management component described in any one of items 1 to 4, wherein each test controller is capable of managing, with respect to testing, communication network components of each domain of the communication system associated with the test controller.

[0188] Item 6: A test management component described in any one of items 1 to 5, wherein each domain of the communication system includes one or more subnetworks of the communication system and / or one or more layers of a protocol stack.

[0189] Item 7: The test management component of any one of items 1 to 6, wherein the domains associated with different test controllers are domains of different communication network operators.

[0190] Item 8: A test management component described in any one of items 1 to 7, wherein the test input interface is configured to receive the request for the test and the specification of the test from a support system of one of the communication network operators or from one of the test controllers.

[0191] Item 9: A test management component described in any one of items 1 to 8, wherein the test controllers are test area controllers, each test area controller controlling a respective subset of communication network components and / or functions of communication network components.

[0192] Item 10: The test management component of item 9, wherein the test management component is configured to create at least some of the test area controllers according to the specifications of the tests.

[0193] Item 11: A test controller for a part of a communication system, a test input interface configured to receive a request for a test to be performed on the communication system and a specification of the test; determining whether the test includes one or more communication network components that the test controller is unable to manage with respect to the test; If the test includes one or more communication network components that the test controller cannot manage for the test, triggering a test management system function to manage that the test is executed by the test controller and a set of one or more other test controllers that can manage the one or more communication network components that the test controller cannot manage for the test. A processing unit configured as follows: A test controller containing:

[0194] Item 12: The test controller of item 11, configured to execute the test if the test includes only communication network components that the test controller can manage for the test.

[0195] Item 13: A communication system including the test management component according to any one of items 1 to 10 and the test controller according to item 11 or 12.

[0196] Item 14: A method for managing testing of a communications system, comprising: receiving a request for a test to be performed on the communications system and a specification of the test; According to said specifications of said test, determining a plurality of communication network components of the communication system that need to participate in the test in order to perform the test; determining, for each communication network component, a respective test controller capable of managing said communication network component with respect to testing; For each determined test controller, determining a specification of sub-tests to be performed by the test controller on the communication network components of the determined plurality of communication network components that the test controller can manage for testing. Steps and for each determined test controller, sending to the test controller the determined specifications of the sub-tests to be executed by the test controller; A method comprising:

[0197] Item 15: A method for processing requests for testing a communications system, comprising: receiving, by a test controller, a request for a test to be performed on the communication system and a specification of the test; determining whether the test includes one or more communication network components that the test controller is unable to manage for the test; if the test includes one or more communication network components that the test controller is not able to manage for the test, triggering a test management system function to manage that the test is executed by the test controller and a set of one or more other test controllers that are able to manage the one or more communication network components that the test controller is not able to manage for the test; A method comprising:

Claims

1. 1. A test management component of a communications system, comprising: a test input interface configured to receive a request for a test to be performed on the communication system and a specification of the test; According to said specifications of said test, determining a plurality of communication network components of the communication system that need to participate in the test in order to perform the test; determining, for each communication network component of the determined plurality of communication network components, a respective test controller capable of managing the communication network component with respect to testing; For each determined test controller, determining a specification of sub-tests to be performed by the test controller on the communication network components that the test controller can manage for testing. a processing unit configured as follows: a test control interface configured to, for each determined test controller, transmit to the test controller the determined specifications of the sub-tests to be executed by the test controller; Including, a test management component, wherein each test controller has its own policy and / or can use its own capabilities and resources for testing, and the test management component is configured to negotiate with each test controller the sub-tests to be executed by the test controller in a manner that conforms to the policy of the test controller and / or is feasible using the supported capabilities and available resources.

2. 2. The test management component of claim 1, wherein each sub-test includes a plurality of test actions, and wherein negotiation with each test controller includes determining the test actions to conform to the policy of the test controller and the specification of the sub-test.

3. 2. The test management component of claim 1, wherein the test relates to an end-to-end communication test between two components of the communication system, and determining the plurality of communication network components of the communication system that need to be involved in the test includes determining communication network components that provide end-to-end communication.

4. The test management component of claim 1 , wherein each test controller is capable of managing, with respect to testing, communication network components of a respective domain of the communication system associated with the test controller.

5. The test management component of claim 4 , wherein each domain of the communication system comprises one or more sub-networks of the communication system and / or one or more layers of a protocol stack.

6. The test management component of claim 5 , wherein the domains associated with different test controllers are domains of different communication network operators.

7. 7. The test management component of claim 6, wherein the test input interface is configured to receive the request for the test and the specification for the test from a support system of one of the telecommunications network operators or from one of the test controllers.

8. 10. The test management component of claim 1, wherein the test controllers are test area controllers, each test area controller controlling a respective subset of communication network components and / or functions of communication network components.

9. The test management component of claim 8 , wherein the test management component is configured to create at least some of the test area controllers according to the specification of the test.

10. 1. A test controller as part of a communication system, comprising: a test input interface configured to receive a request for a test to be performed on the communication system and a specification of the test; determining whether the test includes one or more communication network components that the test controller is unable to manage with respect to the test; If the test includes one or more communication network components that the test controller cannot manage for the test, triggering a test management system function to manage that the test is executed by the test controller and a set of one or more other test controllers that can manage the one or more communication network components that the test controller cannot manage for the test. A processing unit configured as follows: A test controller containing:

11. The test controller of claim 10 , configured to execute the test if the test includes only communication network components that the test controller can manage for the test.

12. A communication system comprising a test management component according to any one of claims 1 to 9 and a test controller according to claim 10 or 11.

13. 1. A method for managing testing of a communications system, comprising: receiving a request for a test to be performed on the communications system and a specification of the test; According to said specifications of said test, determining a plurality of communication network components of the communication system that need to participate in the test in order to perform the test; determining, for each communication network component of the determined plurality of communication network components, a respective test controller capable of managing the communication network component with respect to testing; For each determined test controller, determining a specification of sub-tests to be performed by the test controller on the communication network components that the test controller can manage for testing. Steps and for each determined test controller, sending to the test controller the determined specifications of the sub-tests to be executed by the test controller; Including, Each test controller has its own policy and / or can use its own capabilities and resources for testing, and the method further includes negotiating with each test controller the sub-tests to be executed by the test controller so as to conform to the policy of the test controller and / or to be feasible using the supported capabilities and available resources.

14. 1. A method for processing a request for testing a communications system, comprising: receiving, by a test controller, a request for a test to be performed on the communication system and a specification of the test; determining whether the test includes one or more communication network components that the test controller is unable to manage for the test; if the test includes one or more communication network components that the test controller is not able to manage for the test, triggering a test management system function to manage that the test is executed by the test controller and a set of one or more other test controllers that are able to manage the one or more communication network components that the test controller is not able to manage for the test; A method comprising:

Images