Method and management component for testing a virtualized communication system - Patents.com

Abstract

According to one embodiment, a test management component for a communication system is provided, comprising: The system includes a test input interface configured to receive a request for a test to be performed in the communication system and a specification of the test; a processing unit configured to determine, according to the specification of the test, a plurality of communication network components of the communication system that need to be involved in the test in order to perform the test, and for each communication network component, determine a respective test controller that can manage the communication network component with respect to the test, and for each determined test controller, determine a specification of a sub-test to be performed by the test controller; and a test control interface configured to transmit, for each determined test controller, the determined specification of the sub-test to be performed by the test controller to the test controller.

Description

[Technical field]

[0001] The present disclosure relates to a test management component, a test controller, a method for managing and organizing testing of a communication system, and a method for processing requests for testing of a communication system. [Background technology]

[0002] A connection between two communicating devices (eg, two physical server systems, or a VM and a virtual function residing on a server) may span multiple administrative domains.

[0003] For example, two systems belonging to different telecommunications operators need to communicate to support roaming operations of subscribers moving between a mobile communications network operated by one operator and a communications network operated by another operator's network.

[0004] These scenarios are particularly challenging with respect to testing: the test controller (i.e. the entity that executes the tests and therefore needs to manage the components involved in the tests) may not actually be able to manage all the components that need to be involved, because these components belong to different operators and the test controller may not be able to manage all these components with respect to the tests (e.g. because some components belong to one operator while the components that need to be involved belong to another operator and the first operator does not have access).

[0005] Therefore, an efficient approach to enable testing in multi-domain (eg, multi-operator) communication systems is desirable. Summary of the Invention

[0006] According to one embodiment, a test management component for a communication system is provided, comprising: a test input interface configured to receive a request for a test and a specification of the test to be performed on the communication system; According to the test specifications, Determine a number of communication network components of the communication system that must be involved in the test to perform the test; determining, for each communication network component, a respective test controller capable of managing the communication network component with respect to testing; For each determined test controller, determining a specification of sub-tests to be performed by the test controller on communication network components of the determined plurality of communication network components that the test controller can control with respect to testing. A processing unit configured as described above, a test control interface configured to transmit to the test controller, for each determined test controller, a determined specification of the sub-tests to be executed by the test controller; Includes.

[0007] Further, in accordance with the above test management component, a test controller, a method for managing testing of a communication system, and a method for processing requests for testing of a communication system are provided. [Brief description of the drawings]

[0008] In the drawings, like reference numbers generally refer to the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the invention. In the following description, various aspects are described with reference to the following drawings: [Figure 1] 1 illustrates a mobile wireless communication system. [Diagram 2]1 illustrates an architecture that includes a simplified version of the Network Functions Virtualisation Management and Orchestration (NFV-MANO) architectural framework. [Diagram 3] 1 illustrates communication connections in a multi-network and multi-operator (i.e., multi-domain) environment. [Figure 4] 1 illustrates an architecture for testing in a multi-operator and virtualized communication system (i.e., multi-domain in a virtualized environment) according to one embodiment. [Diagram 5] 1 illustrates an architecture for testing in a multi-operator and virtualized communication system according to another embodiment. [Figure 6] 4 shows an architecture for testing in a multi-operator and virtualized communication system according to a further embodiment. [Figure 7] Illustrates the interaction of Domain Tester with other management systems. [Figure 8] 1 shows an example in which a domain tester manages a first area tester and a second area tester that handle different layers for testing. [Figure 9] 13 shows examples of subtests for a network service test when considering virtualization. [Figure 10] A hierarchical architecture model and a distributed architecture model are shown. [Figure 11] Here is an example architecture where a test broker communicates with five domain testers and the recipients are modifying the domain. [Figure 12] An example is given of a test broker being deployed in an Open Radio Access Network (O-RAN) architecture. [Figure 13] 1 illustrates an exemplary flow for executing a test when a specification of the test is passed from an operator to one of the domain testers. [Figure 14]1 illustrates an example flow for executing a test when a test specification is passed to a test broker. [Figure 15] A flow diagram showing an example of testing within a single operator Network Function Virtualization Infrastructure Connectivity Point of Presence Network Service (NFVI-Pop NS). [Figure 16] A flow diagram illustrating an example of multi-operator NFVI-Pop intra-NS connectivity testing is shown. [Figure 17] 1 illustrates test management components of a communications system. [Figure 18] 1 shows a flow diagram illustrating a method for managing testing of a communications system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] The following detailed description refers to the accompanying drawings, which show by way of example specific details and aspects of the present disclosure in which the present invention may be practiced. Other aspects may be utilized, and structural, logical, and electrical changes may be made, without departing from the scope of the present invention. Various aspects of the present disclosure are not necessarily mutually exclusive, since some aspects of the present disclosure may be combined with one or more other aspects of the present disclosure to form new aspects.

[0010] Various examples corresponding to aspects of the present disclosure are described below.

[0011] Example 1 is a test management component of the above communication system.

[0012] Example 2 is a test management component according to example 1, where each test controller has its own policy and / or can use its own (supported) capabilities and (available) resources for testing, and the test management component is configured to negotiate with each test controller the sub-tests to be executed by the test controller in a manner that conforms with the test controller's policies and / or is feasible with the capabilities and resources.

[0013] Example 3 is a test management component according to example 1 or 2, wherein each sub-test includes a plurality of test actions, and negotiation with each test controller includes determining the test actions to conform to the test controller's policies and the sub-test's specifications.

[0014] Example 4 is a test management component according to any one of Examples 1-3, wherein the testing is for end-to-end communication testing between two components of a communication system, and determining a plurality of communication network components of the communication system that need to be involved in the testing includes determining a communication network component that provides the end-to-end communication.

[0015] Example 5 is a test management component according to any one of Examples 1-4, wherein each test controller is capable of managing, with respect to testing, communication network components of a respective domain of a communication system associated with the test controller.

[0016] Example 6 is a test management component according to any one of Examples 1-5, wherein each domain of the communication system includes one or more sub-networks of the communication system and / or one or more layers of a protocol stack.

[0017] Example 7 is a test management component according to any one of examples 1-6, wherein the domains associated with the different test controllers are domains of different communication network operators.

[0018] Example 8 is a test management component according to any one of Examples 1-7, wherein the test input interface is configured to receive test requests and test specifications from a support system of one of the telecommunications network operators or from one of the test controllers.

[0019] Example 9 is a test management component according to any one of Examples 1-8, wherein the test controllers are test area controllers, each test area controller controlling a respective subset of the communication network components and / or functionality of the communication network components.

[0020] Example 10 is a test management component according to example 9, wherein the test management component is configured to create at least some of the test area controllers according to a specification of the test.

[0021] Example 11 is a test controller for a portion of a communication system, a test input interface configured to receive a request for a test and a specification of the test to be performed on the communication system; determining whether the test includes one or more communication network components over which the test controller has no control; If the test involves one or more communication network components that the test controller cannot manage with respect to the test, triggering a test management system function to manage that the test is executed by the test controller and a set of one or more other test controllers that can manage the one or more communication network components that the test controller cannot manage with respect to the test. A processing unit configured as follows: Includes.

[0022] Example 12 is the test controller of example 11, configured to execute a test if the test includes only communication network components that the test controller can manage for the test.

[0023] Example 13 is the test controller of examples 11 or 12, wherein the test controller is configured to, when a test includes one or more communication network components that the test controller is unable to manage with respect to the test, perform sub-tests of the test that include only those communication network components that the test controller is unable to manage with respect to the test.

[0024] Example 14 is the test controller of example 13, wherein the test input interface is configured to receive a specification of the sub-test from a test management system that provides test management system functionality.

[0025] Example 15 is the test controller of any one of Examples 11-14, wherein the test controller includes a test management system that provides test management system functionality.

[0026] Example 16 is any one of the test controllers of Examples 11-15, wherein the test management system functionality is provided by a test management system external to the test controller, and the test controller includes a test output interface configured to trigger the test management system functionality by transferring a specification of a test to the test management system.

[0027] Example 17 is the test controller of any one of Examples 11-16, wherein the test controller is configured to dynamically determine endpoints assigned to nodes and / or functions involved in a test and to execute the test using the determined endpoints.

[0028] Example 18 is the test controller of any one of Examples 11-17, wherein the test controller is configured to perform tests or sub-tests of a test with multiple test area controllers, each test area controller controlling a respective subset of the communication network components and / or functionality of the communication network components.

[0029] Example 19 is the test controller of example 18, wherein the test controller is configured to dynamically determine endpoints assigned to nodes and / or functions involved in a test or sub-test and control the test area controller to execute the test or sub-test using the determined endpoints.

[0030] Example 20 is the test controller of example 18 or 19, wherein the subset of communication network components is a subset of communication network components for a respective geographic area associated with the test area controller, and / or the subset of communication network component functions is a subset of communication network component functions for a communication layer associated with the test area controller.

[0031] Example 21 is a communication system including the test management component of any one of Examples 1-10 and the test controller of any one of Examples 11-14 or 16-20.

[0032] Example 22 is a method for managing testing of a communications system, the method comprising: receiving a request for and a specification of a test to be performed on a communications system; According to the test specifications, Determine a number of communication network components of the communication system that must be involved in the test to perform the test; determining, for each communication network component, a respective test controller capable of managing the communication network component with respect to testing; For each determined test controller, determining a specification of sub-tests to be performed by the test controller on communication network components of the determined plurality of communication network components that the test controller can control with respect to testing. Steps and for each determined test controller, transmitting to the test controller a determined specification of the sub-tests to be executed by the test controller; Includes.

[0033] Example 23 is a method for processing a request for testing a communications system, the method comprising: receiving, by a test controller, a request for and a specification of a test to be performed on a communications system; determining whether the test involves one or more communication network components that the test controller has no control over for the test; if the test involves one or more communication network components that the test controller is not able to manage for the test, triggering a test management system function to manage that the test is executed by the test controller and a set of one or more other test controllers that are able to manage the one or more communication network components that the test controller is not able to manage for the test; Includes.

[0034] It should be noted that one or more features of any of the above examples may be combined with any one of the other examples. In particular, examples described in the context of a device are equally valid for a method.

[0035] According to further embodiments, there is provided a computer program and computer readable medium comprising instructions which, when executed by a computer, cause the computer to perform the method of any one of the above examples.

[0036] Various examples are described in more detail below.

[0037] FIG. 1 illustrates, in simplified form, a mobile wireless communications system 100 configured in accordance with Fifth Generation (5G) standards, for example, as specified by the Third Generation Partnership Project (3GPP).

[0038] The mobile radio communication system 100 comprises a mobile radio terminal device 102, such as User Equipment (UE), etc. The mobile radio terminal device 102, also called subscriber terminal, forms the terminal side, while the other components of the mobile radio communication system 100 described below are part of the mobile communication network side, i.e. part of a mobile communication network (e.g. public land mobile network PLMN).

[0039] Furthermore, the mobile wireless communication system 100 includes a Radio Access Network (RAN) 103, which may include multiple radio access network nodes, i.e., base stations configured to provide radio access according to a Fifth Generation (5G) radio access technology (5G New Radio). It should be noted that the mobile wireless communication system 100 may also be configured according to Long Term Evolution (LTE) or another mobile wireless communication standard (e.g., non-3GPP access such as WiFi), although 5G is used here as an example. Each radio access network node may provide wireless communication with the mobile wireless terminal device 102 over the air interface. It should be noted that the radio access network 103 may include any number of radio access network nodes.

[0040] The mobile radio communication system 100 further comprises a Core Network (5GC) 119 including an Access and Mobility Management Function (AMF) 101 connected to the RAN 103, a Unified Data Management (UDM) 104 and a Network Slice Selection Function (NSSF) 105, where in the following example the UDM may further comprise an actual UE subscription database, for example known as a Unified Data Repository (UDR). The Core Network 119 further comprises an Authentication Server Function (AUSF) 114, a Policy Control Function (PCF) 115 and an Application Function (AF) 120.

[0041] The core network 119 may have multiple core network slices 106, 107, and for each core network slice 106, 107, an operator (also called a Mobile Network Operator MNO) may create multiple core network slice instances 108, 109. For example, the core network 119 includes a first core network slice 106 with three core network slice instances 108 for providing Enhanced Mobile Broadband (eMBB) and a second core network slice 107 with three core network slice instances (CNI) 109 for providing Vehicle-to-Everything (V2X).

[0042] Typically, when a core network slice is deployed (i.e., created), network functions (NFs) are instantiated or (if already instantiated) referenced to form a core network slice instance, and the network functions belonging to the core network slice instance are configured with a core network slice instance identification information.

[0043] Specifically, in the illustrated example, each instance 108 of the first core network slice 106 includes a first Session Management Function (SMF) 110 and a first User Plane Function (UPF) 111, and each instance 109 of the second core network slice 107 includes a second Session Management Function (SMF) 112 and a second User Plane Function (UPF) 113. The SMFs 110, 112 are for handling Protocol Data Unit (PDU) sessions, i.e., for creating, updating and removing PDU sessions and managing session contexts with the User Plane Function (UPF).

[0044] Compared to legacy LTE and 5G approaches where base stations were used to provide access to the network, the RAN portion 103 can be decomposed considering several different physical and virtual components. For example, the functionality of a single gNB can be provided by three different entities: a Control Unit (CU), one or more Distributed Units (DU), and one or more Radio Units (RU). Furthermore, these can be deployed as physical network functions (PNFs) or virtual network functions (VNFs), e.g., a virtualized DU.

[0045] The RAN 103 and the core network 119 form the network side of a mobile radio communication system, or in other words a mobile radio communication network. The mobile radio communication network and the mobile terminals accessing the mobile radio communication network together form the mobile radio communication system.

[0046] Similar to the core network 119, the RAN 103 may also be sliced, i.e., may include multiple RAN slices. The RAN slices and the core network slices 106, 107 may be grouped to form a network slice.

[0047] In the following, a "network slice" (or simply a "slice") generally refers to a core network slice, but may also include a RAN slice or even a transport network slice. In other words, each network slice may be composed of network slice subnets, such as a RAN subnet, a core network subnet, and a transport subnet.

[0048] Single network slice selection assistance information (S-NSSAI) identifies a network slice. A network slice instance (NSI) is identified by an NSI ID.

[0049] The mobile wireless communication system 100 may further include an OAM function (or entity) 116 implemented, for example, by one or more Operation, Administration and Maintenance (OAM) servers connected to the RAN 103 and the core network 119 (connections not shown for simplicity). The OAM 116 may include a Management Data Analytics Service (MDAS). The MDAS may provide, for example, analytical reports on network slice instance load.

[0050] Additionally, the core network 118 includes a Network Repository Function (NRF).

[0051] The core network 119 may further include a Network Data Analytics Function (NWDAF) 117. The NWDAF is responsible for providing network analysis and / or forecasting information upon request from the network functions.

[0052] Various network functions can be implemented on dedicated hardware, i.e. so-called Advanced Telecommunications Computing Architecture (ACTA) devices. This means that the network functions are implemented as physical network functions deployed on special hardware. The software for implementing the network functions is then tightly coupled to the hardware.

[0053] However, it may be desirable to implement network functions as software applications running on non-dedicated hardware, i.e., on virtual machines (and / or containers) deployed on commercial off-the-shelf (COTS) servers (i.e., on general-purpose or open computing platforms, i.e., devices). This is made possible by using virtualization technologies (e.g., hypervisors, operating system (OS) containers, etc.), which can be used to deploy network functions (e.g., SMF, PCF, etc.) as software applications running on virtual machines (VMs) and / or containers. As such, they are referred to as virtual network functions (VNFs).

[0054] Management and orchestration (MANO) is a key element of the European Telecommunications Standards Institute (ETSI) network functions virtualization (NFV) architecture. NFV-MANO is an architectural framework that orchestrates network resources for cloud-based applications and the lifecycle management of virtual network functions (VNFs) and network services. It is therefore critical to ensure rapid and reliable NFV deployment at scale. NFV-MANO includes several functional blocks and functions such as NFV orchestrator (NFVO), VNF manager (VNFM), virtual infrastructure manager (VIM) and Container Infrastructure Service Management (CISM).

[0055] 2 shows an architecture 200 that includes a simplified version of the Network Functions Virtualisation Management and Orchestration (NFV-MANO) architectural framework, which includes a collection of functional blocks and functions, data repositories used by these functional blocks, and reference points and interfaces through which these functional blocks exchange information for the purpose of managing and orchestrating NFV.

[0056] The architecture 200 includes an operations support system and business support system (OSS / BSS) 201, and an NFV-MANO including an NFVO 202, a VNFM 203, a VIM 204, a container Infrastructure Service Management (CISM) 205, and a CIS Cluster Management (CCM) 217.

[0057] The NFVO 202 manages network services (NS) 206, which in this example includes the following network functions: AMF 208, SMF 209, and PCF 210, and RAN components 207. A network service (NS) is a composition of physical and virtual network functions and / or services defined by its functional and behavioral specifications.

[0058] The architecture 200 further includes a Network Function Virtualization Infrastructure (NFVI) 211 that includes hardware and software components that create the virtualized environment in which the VNFs are deployed.

[0059] In this example, the AMF 208, SMF 209 and PCF 210 are implemented as VNFs, i.e., NFs that can be deployed on an NFVI 211. The SMF 209 is implemented by a container 212 running in a virtual machine, the PCF 210 on a virtual machine 213, and the AMF 208 in a container 214, all running on a COTS server 215. In contrast, the RAN component 207 is implemented on an ATCA device 216, i.e., it is a physical network function.

[0060] The NFVO 202 manages the NS lifecycle and coordinates the overall resource usage of the NS, the VNF lifecycle is supported by the VNFM 203, and the NFVI resource management is supported by the VIM 204, CISM 205 and CCM 217, ensuring optimized allocation of required resources and connections.

[0061] The VNFM 203 is responsible for the lifecycle management of the VNFs, regardless of whether the VNFs are deployed on VMs or containers.

[0062] The VIM 204 is responsible for controlling and managing NFVI compute, storage and network resources, typically within an operator's infrastructure domain. The CISM 205 is responsible for orchestrating and managing container infrastructure services and the containerized workloads instantiated on them. The CCM 217 is responsible for managing container clusters.

[0063] With regard to Network Functions, it should be noted that 3GPP is responsible for the management and configuration of Network Functions (NFs) and the interactions between NFs, but not for the management of virtualized network function aspects. Indeed, the 3GPP functions remain the same from an application perspective, whether a network function is virtualized or not. From an ETSI NFV perspective, NFV-MANO handles the orchestration and management of both the virtualized resources and the deployed VNFs, remaining agnostic to the actual VNF application (e.g., if the VNF application is a virtualized PCF or a virtualized SMF).

[0064] Regarding VNF management, VNF instantiation as well as other life cycle management (LCM) operations such as updates, scaling, etc. are performed by NFV-MANO (NFVO / VNFM / CISM / VIM), and VNF configuration can be performed by element managers through interactions with OSS.

[0065] A Network Point of Presence (N-PoP) is a location where a network function is implemented as either a Physical Network Function (PNF) or a Virtual Network Function (VNF). A Network Function Virtualization Infrastructure Point of Presence (NFVI-PoP) is an N-PoP where a network function is or can be deployed as a Virtual Network Function (VNF).

[0066] In a virtualized environment, test operations can be categorized based on whether they indicate VNF application testing (e.g., performance of a virtualized SMF), VNF connectivity testing with other components, physical and virtual network testing, virtualization management component testing, upgrade component testing, physical resource testing, etc.

[0067] With regard to connectivity testing when virtual network functions are involved in the test, i.e. when the test is performed in a virtualized environment, the test should typically take into account several different dimensions, such as: -Connection testing taking into account the underlay network -Connection testing considering overlay networks Different sending and receiving entities (i.e. components) such as VNF to VNF, VNFC to VNFC, VNFC to Gateway, Node to Node, etc. Different virtualization technologies (e.g. VM-based, container-based)

[0068] Connectivity tests typically include reachability tests (such as hop-to-hop and end-to-end reachability) and communication service quality and performance tests.

[0069] The connectivity tests may include tests on one or more layers, such as the PHY layer, the medium access control (MAC) layer, etc. (multi-layer tests may be considered in particular).

[0070] Furthermore, different types of endpoints (associated with the components involved in the test) may be used for the tests (e.g., Internet Protocol (IP) address, port, Fully Qualified Domain Name (FQDN), Uniform Resource Locator (URI), etc.). Endpoints can be used to characterize senders and receivers. In virtualized environments, endpoints are not strictly associated with physical nodes and can change constantly (e.g., due to mobility / scaling / failover, etc.). For testing, dynamic identification of endpoints needs to be supported, which is difficult to manage.

[0071] Furthermore, a communications connection between two such endpoints may extend across multiple communications networks operated by different operators, i.e., testing of the communications connection may need to be performed in a multi-network (and possibly multi-operator) environment.

[0072] FIG. 3 shows communication connections 301, 302, 303, 304 in a multi-network and multi-operator environment.

[0073] For all three communication connections 301, 302, 303, 304, the sender is in a first NFVI-PoP (e.g., a first data center) 305 under the control of operator A. The receiver is In the case of the first communication connection 301, it is in the same NFVI-PoP 305 as the sender. The second communication connection 302 is to an external entity (ie, component) 306 that resides on the public Internet 307 (eg, an external web service). The third communication connection 303 is in a second NFVI-PoP (e.g., a second data center) 308 under the control of the same operator (operator A). This is the case for example when the NS spans multiple NFVI-POPs. The two NFVI-PoPs include respective customer edge (CE) devices 309, 310 for communication over a first wide area network (WAN) 311. The operator of the first wide area network (WAN) 311 may be different from operator A. The fourth communication connection 304 is in a third NFVI-PoP 312 under the control of a different operator (operator B) than the first NFVI-PoP 304 (operator A). The two NFVI-POPs include respective CE devices 309, 313 for communication over a second wide area network (WAN) 314.

[0074] At their edges, the WANs 311, 314 (backbone networks) include provider edge devices (PE) that are network devices that connect to customer edge (CE) devices 309, 310, 313. A CE device can be understood as a gateway system. Within the WANs 311, 314, there are provider devices (P) in each WAN segment.

[0075] A multi-operator communication system such as that shown in FIG. 3 comprises multiple (sub-)networks, each of which belongs to a different operator, i.e. for each of these networks, the respective operator can manage (i.e. has configuration and control rights) over the network (while other operators cannot manage this network).

[0076] Considering various cases that may arise in testing in a multi-operator and virtualized communication system as shown in Fig. 3, various embodiments provide means for orchestrating end-to-end multi-operator NS / VNF testing. In particular, various embodiments provide approaches for addressing the following problems that arise in such an environment: In a virtualized environment, endpoints are constantly changing and the endpoints involved in the tests need to be defined and communicated across different operator domains (i.e. (sub)networks of communication systems controlled by different operators). To test end-to-end connectivity, the actual communication path that needs to be traversed to reach the destination point from the starting point passes through different network segments (sub-networks) that may be controlled by different operators (e.g. VNF / VNFC to CE Gateway, CE to PE, PE-P, etc.). Besides manual communication between administrators, a means to orchestrate end-to-end testing in such scenarios is desirable. Provide a means to automate multi-operator end-to-end network / service test feasibility testing. A means to automatically and logically segment the network within NFVI-PoP boundaries for more granular testing is desired (typically not possible when each operator domain is considered as one flat network).

[0077] An example of a test use case in a multi-operator communication system is support for testing in roaming use cases where systems and network functions from one operator need to communicate with systems and network functions from another operator.

[0078] FIG. 4 illustrates an architecture for testing in a multi-operator and virtualized communication system (i.e., multi-domain in a virtualized environment) according to one embodiment.

[0079] In this example, a first NFVI-PoP 401 controlled by a first operator (Operator A) and a second NFVI-PoP 402 controlled by a second operator (Operator B) are involved in the test. The reason is that for example a communication connection between a sender 403 in the first NFVI-PoP 401 and a receiver 404 in the second NFVI-PoP 40 should be tested. The sender 403 and the receiver 404 communicate via a (sub)network 405 of the first NFVI-PoP 402, a gateway 406 of the first NFVI-PoP 401, a WAN 407 (comprising P devices and PE devices as above), a gateway 408 of the second NFVI-PoP 401 and a (sub)network 409 of the second NFVI-PoP 402.

[0080] The first operator operates a first OSS 410 connected to components of the first NFVI-PoP 401 such as one or more element managers 411, generic OAM functions 412, an NFV-MANO 413, and an NFVI management system such as an NFVI 414. The second operator operates a second OSS 415 connected to components of the second NFVI-PoP 402 such as one or more element managers 416, generic OAM functions 417, an NFV-MANO 418, and an NFVI management system such as an NFVI 419.

[0081] Each operator has a respective associated domain tester (also called (domain) test controller) for each NFVI-PoP 401, 402, i.e. operator A provides and operates a first domain tester 420 for the first NFVI-PoP 401 and operator B provides and operates a second domain tester 421 for the second NFVI-PoP 402. Each of the first domain tester 420 and the second domain tester 421 acts as a test coordinator for the components of the NFVI-PoP 401, 402 that it operates. Furthermore, a third domain tester 422 is provided in the WAN 407 (by the operator of the WAN) as a test coordinator for the components of the WAN 407. The third domain tester 422 may be part of a WAN Infrastructure Manager (WIM).

[0082] Each domain tester 420-422 can be used to perform a series of tests at the NS, VNF, VNFC, node and / or network level, for example: Network testing (e.g. VNFC vs. VNFC, VNFC vs. PNF, etc.) VNF Testing VNF Application Testing

[0083] Other function blocks, for example, the NFV-MANO function block, may also be subject to testing (for example, NFVO-to-NFVO connectivity testing that belongs to different domains).

[0084] To support testing in a multi-operator environment (i.e., a communication system having domains (i.e., (sub)networks) controlled by different operators), a test broker (also called a test management component) 423 is provided that communicates with the domain testers 420-422.

[0085] In one embodiment, the test broker is part of one of the domain testers, which is shown in FIG.

[0086] FIG. 5 illustrates an architecture for testing in a multi-operator and virtualized communication system according to another embodiment.

[0087] The architecture of Fig. 5 corresponds to that of Fig. 4, with the difference that one of the domain testers, here the domain tester 520 provided by operator A, acts as a test broker 423, i.e. has test broker service capabilities (i.e. provides test management system functionality). It is declared as the master (or primary domain tester), while the other domain testers 521, 522 are declared as slaves (or secondary domain testers). The master (test broker) 520 performs the tasks of the test broker 423, such as triggering tests, negotiating test specifications and implementing test configurations.

[0088] Additionally, the domain tester may at least partially further segment its domain (i.e., the domain it controls), as shown in FIG.

[0089] FIG. 6 shows an architecture for testing in a multi-operator and virtualized communication system according to a further embodiment.

[0090] In FIG. 6, the operators and OSS are omitted for simplicity.

[0091] The architecture of Figure 6 corresponds to that of Figure 4, with the difference that one of the domain testers, here the domain tester 620 of operator A, segments (part of) its domain (specifically the network 405) into test areas (or zones) 626, each with a respective area tester (or test area controller) 627. Thus, in each test area 626 in the domain, a respective area tester 627 is used to manage the tests. The area testers 627 are managed by the domain tester 620.

[0092] In the following, the functionality of the test broker, domain tester, and area tester will be described with reference to the hierarchical architecture model of Figure 4, however, all operations and functionality equally apply to the distributed architecture model of Figure 5. Similarly, all operations and functionality equally described with reference to the architecture of Figure 4 (particularly test broker 423) also apply to the architecture of Figure 6. Furthermore, it should be noted that segmentation of one or more domains as shown in Figure 6 may also be performed in the distributed architecture model of Figure 5.

[0093] End-to-end testing involves communication between area testers (if applicable), domain testers (if involved), and test brokers.

[0094] The tests may specifically involve any possible aspect related to NS connectivity. For example, a test or sub-test may relate to any possible aspect related to network service communication. Reachability connectivity testing between nodes and / or VNFs, servers, gateway systems, etc. Network and application performance and quality of service (QoS), referring to the quality of communication. Testing is not necessarily specific to a layer of the protocol stack, rather multi-layer aspects may be considered (PHY layer, MAC layer, L3 testing, etc.). Tests can be next hop, or end-to-end, or for a specific part of the route. · Communication tests can be between physical or virtual components (e.g. VNFC to VNFC, VNFC to PNF, VNF to Gateway, PNF to Gateway, etc.). Connectivity tests can refer to underlay or overlay network components (e.g. L2, but also L2 VPN, L3 VPN, etc.). Tests can be run across different virtualization technologies (e.g. VM-based, container-based).

[0095] Domain Tester The domain testers 420-422 perform dynamic identification of endpoints (e.g., shown as black dots such as entry points 425 to the second WAN 409) used for testing. This identification is dynamic since in a virtualized environment, the endpoints 425 may constantly change (e.g., due to mobility / scaling / failover, etc.). The domain testers 420-422 further perform translation of Testing Intent Descriptors (TIDs) provided by the operator or test broker 423 into sub-tests related to any type of virtual and physical network. Depending on the test, the endpoints 425 may be related to a digital twin. The domain tester decides whether real components or digital twins are used for testing, e.g., to avoid interrupting normal traffic. Each domain tester 420-422 automatically decides whether to test real VNFs or digital twins based on policies, real-time conditions, constraints, etc. In the case of multiple test areas 626, depending on the test intent, each domain tester 420-422 performs automated management of area testers 627 within each domain boundary. Additionally, when multiple test areas 626 are used, each domain tester 420-422 performs the creation, deletion, updating and / or configuration of area testers 627, identification of area tester boundaries and area entry and exit points, etc.

[0096] For example, a test (eg, initiated by one of the operators) of the connection 424 between the sender 407 and the receiver 404 may be performed as follows. 1) An operator initiating a test passes a Testing Intent Descriptor (TID) to the domain tester 420-422 that the operator provides. The TID describes what the test is about and the high-level goal of the test. It may also provide a detailed specification of the test. 2) If the TID cannot be converted to a local action by the domain testers 420-422, it is communicated to the test broker 423. 3) The test broker 423 automatically identifies suitable domain testers from other operators (of the domains containing the components involved in the test). In this example, all three domain testers 420-422 are involved (i.e., are of the involved domain, i.e., the domains that have the components involved in the test). 4) The test broker 423 negotiates with the domain testers 420-422 about the test features, tools, configurations, etc. to be used. This includes the domain testers 420-422 of the involved domains giving feedback on test validation and feasibility. Feasibility may include, in particular, for each domain tester, the feasibility with respect to the domain tester's policies (e.g. according to the policies of the domain tester's operator), i.e. whether the test actions requested by the subtests are allowed by the policies or not. 5) Each domain tester executes (sub)tests according to the results of the negotiation (and possibly controls the area testers 627 accordingly, if multiple test areas 626 are used).

[0097] FIG. 7 illustrates the functionality of the Domain Tester 720 in more detail in an example where the Domain Tester 720 uses an Area Tester 727 and there are one or more network slice management system components 728, such as one or more NSMFs and / or NSSMFs.

[0098] Domains can be established in different ways, e.g. administrative domains (as in the example above) covering the case of a multi-operator environment, but also network domains, e.g. covering the case of testing between different network areas within a single administrative domain (operator).

[0099] The Domain Tester 720 is responsible for its respective domain (i.e., the domain with which it is associated). As mentioned above, a domain may, for example, include components operated by one operator (i.e., components for which one operator, e.g., OSS / BSS, has configuration rights). A domain may also span multiple NFVI-PoPs of the same operator.

[0100] The domain tester 720 is aware of NFV and Software-Defined Networking (SDN) related information (e.g., obtains topology information, operating protocols, etc.) and can perform connectivity tests on both underlay and overlay networks.

[0101] For testing purposes, the domain tester 720 ·Interact with all appropriate management systems to support test configuration, test execution and result retrieval. · An initial analysis of the test intent description (TID, e.g., "Test connectivity between VNF-1 and VNF-2") may be performed. This applies if the operator sends the TID to the domain tester 720 instead of directly to the test broker 423. Dynamically identify the endpoints (e.g., IP addresses, ports, FQDNs, URIs, or other) used for testing. o Case 1: All endpoints are inside the NFVI-PoP of the domain tester's domain. In that case, the domain tester 720 may run the test without other domain testers. This applies especially if the test is a sub-test that the test broker 423 specified for a test (including multiple domains). o Case 2: An external endpoint exists. In that case, the domain tester 720 calls the test broker 423 to define an endpoint for testing within its domain (e.g., within a gateway system). This applies when the domain tester 720 receives a test from a component other than the test broker (because the test broker sent the subtest), such as from its operator's OSS (or BSS) 710. ·Transforming received (sub)test specifications into actions (tools / usage / configuration) for end-to-end testing. ·Perform feasibility study analysis. · It provides NFVI-PoP internal segmentation into test areas 626 and can allocate further area testers 727. · Manage area testers in the domain. Consider mixed NFVI environments (VM-based, containerized, etc.). · Interacting with one or more of the OSS 710, the Test Broker 423, the Area Tester 727 in the Domain Tester's NFVI-PoP 701, the EM 711, the Generic OAM Function 712, the NFV-MANO 713, other components (e.g., the NFVI Physical Infrastructure Manager, the NSMF and / or NSSMF in case of network slicing, etc.).

[0102] It should be noted that tests can also be triggered automatically due to a lifecycle management action (e.g., a VNF is created).

[0103] TID The specification of a test may be in the form of a test intent, specifically a Testing Intent Descriptor (TID). For example, an operator initiating a test passes a Testing Intent Descriptor (TID) to a domain tester or to multiple domain testers under the operator's control. The TID provides the test specification in an abstract or very detailed form. The TID describes what the test is about and the high level goal of the test. An example is testing L2 connections between all components of the network service NS#1.

[0104] The test management component may convert the TID of a test into a specification of sub-tests associated with any type of virtual and physical network.

[0105] Example test intent: Test L2 connectivity for network service #1. The Domain Tester will convert the test into the following sub-tests, taking into account all virtualized and physical components that make up NS: Subtest #1: VNF1-PNF1 or PNF1 Digital Twin depending on dynamics / requirements etc. Subtest #2: VNF1-VNF2 Subtest #3: VNF2-VNF3 up to time t Subtest #4: New endpoints of VNF2-VNF3 after time t o If the endpoints are not passed in the TID, the Domain Tester, through interaction with NFV-MANO / NFVI / EM etc., identifies the endpoints (e.g. IP / Port, FQDN, URI etc.) to be used for different sub-tests. o Depending on the constraints / policies / runtime capabilities / configuration / etc, in which case the domain tester may select the corresponding endpoint of the digital twin instead of the VNF / PNF etc.

[0106] Test Area The domain tester 720 may define, create and manage area testers 727 to segment (at least partially) its domain into test areas. A priori (before the test intent is known), in which case the domain tester selects the appropriate area tester to use for testing. After-the-fact, after receipt of a test intent (for a test or sub-test) by the domain tester, the domain tester can then create a new area tester 627 based on the needs of the experiment.

[0107] The area testers 727 may follow the standardized ETSI GS NFV-TST 011 process. The domain tester 720 keeps an inventory of all area testers 727, manages all area testers 627, and interacts with all area testers 627 to synthesize and control the tests or sub-tests that it should execute.

[0108] Test area 726 is a logical abstraction within a domain boundary. · A test area 726 may cover one or more layers of a protocol stack. · A test area 726 may cover a network segment within a domain. Other segmentations of the domain (or parts thereof) into test areas 726, or even combinations of the above, may be considered.

[0109] FIG. 8 shows an example in which a domain tester 801 is configured with a first area tester 802 whose test area is the data link layer, network layer, and transport layer of a communication network 804 within the domain, and a second area tester 803 whose test area is the application layer of the communication network 804.

[0110] The endpoints are used by corresponding domain testers and area testers to identify senders and receivers (which may include intermediate senders and receivers along the end-to-end connection between the endpoint sender and the endpoint receiver). In a virtualized environment, the endpoints are typically constantly changing (e.g., due to mobility, scaling, failover, etc.). The area tester in communication with the domain tester can define the endpoints to use for each node or function (e.g., intermediate senders and receivers) involved in the test. This information is passed to the domain tester, which can communicate the endpoint information to another operator directly or through a test broker. The endpoints can be IP addresses, network ports, sockets, FQDNs, etc.

[0111] Dynamic identification of the endpoints used for testing is done by a domain tester in communication with an area tester. Depending on the test, the endpoints may be related to the digital twin. The domain tester automatically decides whether to test the real VNF or the digital twin based on policies, real-time conditions, constraints, etc. For example, to avoid overloading the production network, testing may be done between the digital twin or the real NF or VNF.

[0112] Test Broker According to various embodiments, end-to-end network service testing over multiple NFVI-POPs under the control of the same or different operators is provided, including test negotiation and related interfaces. In particular, a multi-operator / multi-domain test broker is provided that translates end-to-end test intent into sub-domain intent (e.g., network services (NS)) that is passed to a domain tester. The test broker negotiates with the domain tester on the test cases, tools, and configurations to be used.

[0113] The test broker 423 identifies administrative boundaries (i.e., boundaries between operator domains) and components (and therefore domain testers 420-422) that need to communicate (interact) for a particular test. Negotiations between domain testers 420-422 are performed through the test broker 423. The test broker 423 may also support negotiations between domain testers 420-422 and the OSS (and / or BSS) 410, 415 when end-to-end testing cannot be supported.

[0114] The Test Broker is used to orchestrate, automate and support multi-operator end-to-end testing. It can perform and support test negotiation, test processing and test intent processing. More specifically: Use a test inventory with available domain testers 420-422. The inventory can be updated manually or dynamically. · Identify administrative boundaries and the components that need to interact (which domain testers need to interact with). · Perform end-to-end test synthesis based on test intent specifications, constraints, policies, etc. through negotiation with domain testers 420 to 422. · Negotiate network demarcation points between different management components (e.g. customer edge (CE) gateways) together with the Domain Tester. · Ability to correlate test results from different domain testers after test execution. Performs negotiation and identification of points of interaction between domains. Points of interaction between domains are also considered test endpoints, which may be virtual and not fixed, but are identified by the test broker. · Provides end-to-end test identifiers that map to endpoint identifiers exposed by domain testers.

[0115] As described above with reference to Figures 4 and 5, the test brokers 423, 523 may be arranged according to a hierarchical architecture model or according to a distributed architecture model. The functionality of the test broker 423 may be a public service or the test broker 423 may be shared by multiple operators or one of the operators.

[0116] FIG. 9 shows an example of a test.

[0117] The intent of the test in this example is to test the L2 connectivity between all components of network service 900.

[0118] One or more domain testers (depending on whether the test spans multiple domains) translate the test into the following subtests, taking into account all virtualized and physical components that make up the NS: Subtest #1: VNF1-PNF1 or PNF1 Digital Twin depending on dynamics / requirements etc. Subtest #2: VNF1-VNF2 Subtest #3: VNF2-VNF3 up to time t Subtest #4: New endpoints of VNF2-VNF3 after time t

[0119] If the endpoints are not passed to one or more domain testers in the TID, the domain tester identifies, tracks and updates the endpoints (e.g., IP / port, FQDN, URI, etc.) used for different sub-tests through interactions with NFV-MANO, NFVI and / or EM, etc.

[0120] Depending on constraints, policies, runtime capabilities and / or configurations, etc., each domain tester may select a corresponding endpoint of the digital twin instead of a VNF, PNF, etc.

[0121] The test broker 423 analyzes the uncompleted test intents that have not yet been completed by the domain testers 420-422 (but have been transferred from the domain testers 420-422 to the test broker 423 because, for example, they span multiple domains and therefore cannot be completed by the domain testers 420-422). It finds the right domain tester for the test according to the received test intent and starts negotiations with each found domain tester. It negotiates with each domain tester about the tests, tools and configurations to be used.

[0122] The end-to-end test may include communication between the area tester 627 and the domain testers 620-622, and between the test broker 423 and the domain testers 420-422.

[0123] FIG. 10 shows a first diagram 1001 illustrating a hierarchical architecture model and a second diagram 1002 illustrating a distributed architecture model, along with corresponding interfaces IF-A and IF-B.

[0124] The interfaces IF-A(hierarchical) and IF-A(distributed) are similar but not identical.

[0125] For example, there may be differences regarding the primary tester (i.e., master) election process and messaging (how the primary tester is advertised to the network) for a distributed architecture. Furthermore, in a distributed architecture model, when two secondary testers (i.e., slaves) need to communicate, one is the local primary tester and the other is the local secondary tester.

[0126] Example of operation on IF-A Part of a TID or TID transfer Test negotiations Test process (e.g. execution) -Transfer of test results ·Domain Tester Registration / Unregistration ·Domain Tester Status IFA-B operation example Part of a TID or TID transfer Test process (e.g. execution) -Transfer of test results Test area management

[0127] FIG. 11 shows an example architecture in which a test broker 1101 communicates with five domain testers 1102-1106 to test a communication connection 1107 from a sender 1113 in a first NFVI-PoP 1112 through a first WAN 1109, a second NFVI-PoP 1110, and a second WAN 1111 to a receiver 1114 in a third NFVI-PoP 1108.

[0128] As indicated by arrow 1115, the receiver 1114 may be transitioned from the second NFVI-PoP 1110 to the third NFVI-PoP 1112.

[0129] To address this, according to various embodiments, the test broker 1101 and / or the domain testers 1102-1106 may consider that the endpoints of the tests are related to logical entities and may move according to the configuration of the network service or VNF, for example, when a VNF (such as here the receiver 1114) is migrated or logically moved. In that case, a more sophisticated identification of the endpoints used, e.g., the receiver 1114, may be considered.

[0130] 12 shows an example in which a test broker 1201 is deployed in an O-RAN architecture. The domain of the domain tester 1202 in this example is an O-Cloud 1203 (including, for example, an element manager 1204, a generic OAM function 1205, an NFV-MANO 1206, an NFVI 1207, a communication network 1208, one or more Infrastructure Management Service (IMS) components 1209, one or more Deployment Management Services (DMS) components 1210, and an Area Tester 1211).

[0131] The IMS component 1209 and the DMS component 1210 are connected to the respective operator's OSS 1212 via a Service Management & Orchestrator (SMO) function 1213, which is also connected to a test broker 1201 and a domain tester 1202. Each domain tester is responsible for each O-Cloud, and negotiations take place between the domain testers through the broker.

[0132] FIG. 13 shows an example flow for executing a test when a test specification, i.e., a test intent in the form of a TID, is passed to one of the domain testers, in this example, the first domain tester 420.

[0133] At 1301, Operator A passes the TID to a first domain tester (eg, through an OSS).

[0134] At 1302, the first domain tester analyzes the TID to determine which sub-tests (specifically, the senders and receivers and corresponding endpoints) are involved in the test.

[0135] At 1303, the first domain tester checks whether all the determined endpoints are in its own domain. If so, the domain tester: · At 1304, check the feasibility of the test and create and allocate an area tester (if configured or determined to use an area tester). At 1305, fine-tune the test according to the TID (in coordination with the area tester, if used). At 1306, the tests are run (in coordination with the area tester, if used) and the results are collected.

[0136] At 1307, the test results (of all subtests) are reported to an operator (or operators).

[0137] If the determined endpoint is not in the domain of the first domain tester, the domain tester passes the TID or a part thereof to the test broker, which communicates with one or more other participating domain testers (here assumed to be only the second domain tester 421) and passes the TID (or the part relevant to the second domain tester 421, i.e. its assigned sub-tests, i.e. the sub-tests (of the overall test specified by the TID) to be executed by the second domain tester 421) to the second domain tester 421. At 1308, each participating domain tester indicates whether it can execute its test according to the TID (or TID part) passed to it. If not, the participating domain testers (here the first domain tester 420 and the second domain tester 421) negotiate with and / or through the test broker 423 at 1309 to derive new TIDs for their respective sub-tests. If the subtests assigned to each domain tester can be executed, the negotiation is completed, the subtests are executed according to 1304 to 1306, and in 1307 the results are reported to the operator or to the operator via the broker.

[0138] FIG. 14 shows an exemplary flow for executing a test when a test specification, i.e., a test intent in the form of a TID, is passed to the test broker 423.

[0139] At 1401 , Operator A passes the TID directly to the test broker 423 .

[0140] At 1402, the test broker analyzes the TID to determine which sub-tests (specifically, the senders and receivers and corresponding endpoints) are involved in the test.

[0141] At 1403, the test broker determines whether all determined endpoints are in the domain of a single domain tester. If so, the test broker sends the TID to that domain tester and the domain tester. · At 1404, check the feasibility of the test and create and allocate an area tester (if configured or determined to use an area tester). At 1405, fine-tune the test according to the TID (in coordination with the area tester, if used). At 1406, the tests are run (in coordination with the area tester, if used) and the results are collected.

[0142] At 1407, the domain tester sends the test results to a test broker, which reports the test results at 1408 to one or more operators.

[0143] If the determined endpoint is not in the domain of a single domain tester, the test broker communicates with all participating domain testers and passes the TID to each of them (or the part relevant to each domain tester, i.e., its assigned subtest, i.e., the part relevant to the subtest (of the overall test specified by the TID) to be executed by the domain tester). In 1409, each participating domain tester indicates whether it can execute its test according to the TID (or TID part) passed to it. If not, in 1410, the participating domain testers negotiate with and / or through the test broker 423 so that each test broker obtains a new TID for its respective subtest. If each participating test broker can execute the subtest assigned to it, each domain tester executes the subtest according to 1404-1406 and the results (of all subtests) are reported (through the test broker) in 1407 and 1408.

[0144] FIG. 15 shows a flow diagram 1500 illustrating an example of a single operator intra-domain NS connectivity test, ie, assuming that the sender and receiver of the communication connection to be tested are within the domain of a single domain tester.

[0145] An operator 1501, an OSS / BSS 1502, a domain tester 1503, and one or more components 1504 of the domain tester's domain (e.g., NFV-MANO) are involved in the flow.

[0146] At 1505 and 1506 , the operator sends a specification of the test intent (eg, TID) to the domain tester 1503 via the OSS / BSS 1506 .

[0147] At 1507, the domain tester 1503 translates the test intent into subtests of the component 1504 and requests feedback from the component 1504 at 1508. The component 1504 checks the feasibility status at 1509 and responds to the request of 1508 at 1510.

[0148] In 1511, the domain tester 1503 checks, depending on the response, whether an area tester is needed for the test. In this example, it is assumed that an area tester is not needed.

[0149] At 1512, the domain tester 1503 begins configuring the components for testing, and at 1513, the test configuration and final verification is performed.

[0150] At 1514, the domain tester 1503 notifies the OSS / BSS 1502 that the configuration is complete, and at 1515, the OSS / BSS triggers the test.

[0151] At 1516 , the domain tester 1503 and components 1504 perform the tests, the results of which are collected at 1517 and reported at 1518 and 1519 by the domain tester 1503 to the operator 1501 via the OSS / BSS 1502 .

[0152] FIG. 16 shows a flow diagram 1600 illustrating an example of multi-operator intra-domain NS connectivity testing, i.e., assuming that the sender and receiver of the communication connection to be tested are in domains managed by different domain testers (e.g., belonging to different operators).

[0153] Involved in the flow are an operator 1601, an OSS / BSS 1602, a first domain tester 1603, one or more components 1604 (e.g., a first NFVI-PoP management system, resources, etc.) of the first domain tester's domain, a test broker 1605, a second domain tester 1606, and one or more components 1607 (e.g., a second NFVI-PoP management system, resources, etc.) of the second domain tester's domain.

[0154] At 1608 and 1609 , the operator sends a specification of the test intent (eg, TID) to the first domain tester 1603 via the OSS / BSS 1602 .

[0155] At 1610, the first domain tester 1603 translates the test intent into a sub-test, which is assumed to include a test for a component 1604 in the first domain tester's domain and a test for a component 1607 in the second domain tester's domain.

[0156] Thus, at 1611, the first domain tester 1603 sends a test intent to the test broker 1605, which at 1612 identifies which domain testers need to be involved in the test (assumed here to be the second domain tester 1606 in addition to the first domain tester 1603). At 1613, the test broker 1605 sends to the second domain tester 1606 a test intent for the sub-tests to be executed by the second domain tester 1606.

[0157] At 1614, the domain testers 1603, 1606 and the test broker 1605 negotiate to derive a feasible end-to-end test and fine-tune the test.

[0158] If there are feasible tests at 1615, then the components 1604, 1607 of the domain tester's domain are configured accordingly. If there are no feasible tests to be performed, then negotiation may also involve the OSS / BSS system.

[0159] At 1616, the first domain tester 1603 triggers the test by signaling the test broker 1605. At 1617, the test broker 1605 notifies the second domain tester 1606 accordingly.

[0160] At 1618 , the domain testers 1603 , 1606 and components 1604 , 1607 perform tests, the results of which are collected at 1619 and reported at 1620 and 1621 by the first domain tester 1603 to the operator 1601 via the OSS / BSS 1602 .

[0161] It should be noted that the test trigger can be done by the OSS / BSS as in the example of FIG. 15 or by the domain tester as in the example of FIG. 16, but may also be triggered by, for example, a life cycle management (LCM) operation (e.g., VNF scaling).

[0162] In summary, in accordance with various embodiments, a test management component is provided as shown in FIG.

[0163] FIG. 17 illustrates test management components 1700 for a communication system.

[0164] The test management component 1700 includes a test input interface 1701 configured to receive requests for and specifications of tests to be performed on a communication system.

[0165] The test management component 1700 performs the following steps according to the test specifications: Determine a number of communication network components of the communication system that must be involved in the test to perform the test; determining, for each communication network component, a respective test controller capable of managing the communication network component with respect to testing; For each determined test controller, determining a specification of sub-tests to be performed by the test controller on communication network components of the determined plurality of communication network components that the test controller can control with respect to testing. The device further includes a processing unit 1702 configured to:

[0166] The test management component 1700 further includes a test control interface 1703 configured to send, for each determined test controller, to the test controller the determined specifications of the sub-tests to be executed by the test controller. The test control interface is also used to orchestrate end-to-end tests (e.g., fine-tune tests and collect test results from multiple domains).

[0167] According to various embodiments, in other words, a central entity is provided that can manage tests across multiple test domains (i.e., domains that are managed by different test controllers with respect to the tests, e.g., because the domains belong to different operators) by determining which domain controllers (i.e., test controllers) need to be involved to execute the test and controlling them accordingly (by providing them with the sub-tests to be executed so that the test is executed as a whole). A test controller being able to manage a component with respect to a test may mean that the test controller is able (e.g., has the rights) to manage the component with respect to the test. Thus, a test controller not being able to manage a component with respect to a test may mean that the test controller does not have the rights to manage the component or cannot configure the component with respect to the test.

[0168] The test management component corresponds, for example, to the test broker in the above example. In that case, the test controller corresponds to the domain tester. Alternatively, the test management component may correspond to one of the domain testers in the above example. In that case, the test controller corresponds to the area tester (also called the test area controller).

[0169] For example, a method as shown in FIG. 18 may be performed.

[0170] FIG. 18 shows a flow diagram 1800 illustrating a method for managing testing of a communication system.

[0171] At 1801, a request for a test to be performed on a communication system and a specification of the test are received.

[0172] In 1802, according to the test specifications, A number of communication network components of the communication system that need to be involved in the test to perform the test are determined; for each communication network component, a respective test controller capable of managing the communication network component with respect to testing is determined; For each determined test controller, a specification of sub-tests to be performed by the test controller on the communication network components of the determined plurality of communication network components that the test controller is capable of managing for testing is determined.

[0173] At 1803, for each determined test controller, after negotiation in case of feasibility issues, a determined specification of the sub-tests to be executed by the test controller is sent to the test controller.

[0174] The test management component corresponds, for example, to the test broker in the above example. In that case, the test controller corresponds to the domain tester. Alternatively, the test management component may correspond to one of the domain testers in the above example. In that case, the test controller corresponds to the area tester (also called the test area controller).

[0175] For example, regarding test management, One of the domains is managed by a domain tester (DT). · Domain testers are managed by operators through OSS / BSS. · A Domain Tester can manage one or more (test) domains belonging to the same operator. · Domain testers can interact with area testers within their domain. To manage virtualized resources or functions, the domain tester and area tester can interact with NFV-MANO. To manage physical resources, the domain tester and area tester can interact with the NFVI management system. Taking network slicing into account, the domain tester can also interact with the network slice management system directly or indirectly through NFV-MANO or OSS / BSS. The Domain Tester interacts with the OSS, Test Brokers, Area Testers (in NFVI-PoPs), EMs, Generic OAM Functions, NFV-MANO, other entities (e.g. NFVI Physical Infrastructure Manager, NSMF and / or NSSMF in case of network slicing, etc.).

[0176] For example, Domain Tester: · Analyze TID along two dimensions: within-domain behavior and outside-domain behavior. ○ Within the domain operation, ■ The domain tester can further logically segment the domain into test areas. ■ For each test area in the domain, an area tester is used to manage the tests. ■ Area testers are managed by the domain tester, which can either dynamically create area testers or select one from a predefined set. ■ The testing aspects within a domain are organized by a domain tester in communication with an area tester. ■ The domain tester identifies test area boundaries and area entry / exit points, etc., and assigns area testers to the test areas. ○Outside the domain operation, ■ If the TID translation requires communication with an external management entity, the TID or a part of it is communicated to the test broker. ■ The test broker automatically identifies suitable domain testers from other operators. ■ The test broker negotiates with the domain tester about the test features, capabilities, tools and configurations to be used. · Provide feedback through test brokers on test validation and feasibility. During negotiation, the test broker can interact with the OSS / BSS if it cannot support end-to-end testing. In one embodiment, a test broker may be included, i.e., the test broker may be part of one of the domain testers (i.e., correspond to the same entity). o The domain tester that tests the broker service capabilities (i.e. has test management system capabilities) is declared as the primary domain tester and other domain testers are declared as secondary domain testers. o Primary domain tester organizes the testing. · Domain testers and area testers are aware of NFV and SDN related information (e.g., obtain topology information, operating protocols, etc.). · Domain testers and area testers can perform connectivity tests on both underlay and overlay networks. · The Domain Tester and Area Tester interact with NFV-MANO, EM, Generic OAM Functions and NFVI to obtain management information used to support testing. ·Domain Testers and Area Testers interact with all appropriate management systems to support test configuration, test execution and result retrieval. Test procedures can also be triggered automatically due to lifecycle management actions (e.g., a VNF is created). · Domain Testers and Area Testers will jointly carry out feasibility study analysis within the domain. · Domain tester and test broker will jointly perform feasibility study analysis for end-to-end testing.

[0177] As stated above, a test area is a logical abstraction. Case 1: An area may cover one or more layers of a protocol stack. · Case 2: The test area may cover a network segment within the domain. · Case 3: Other formats or even combinations of the above may be considered.

[0178] An Area Tester can manage testing of one or more test areas within the same domain. Each operator domain is considered as a flat network. Area Testers can automatically and logically segment the network within domain boundaries for more granular testing. Area Testers can be defined and created as follows: Case 1: A priori (before the test intent is known), in which case the domain tester selects the appropriate area tester to use. Case 2: Ex post facto after receipt of the test intent by the domain tester. In that case, the domain tester can create a new area tester based on the need for the experiment.

[0179] Area testers can follow standardized processes and processes from other inventions. A domain tester, for example, keeps an inventory of all area testers. A domain tester manages all area testers (associated with it, i.e., responsible for testing areas within the domain of the domain tester) and interacts with all area testers to synthesize and manage tests.

[0180] The test controller components and the test management components may be implemented, for example, by one or more circuits. A "circuit" may be understood as any kind of logic implementation entity, which may be a dedicated circuit or a processor executing software stored in a memory, firmware, or any combination of these. Thus, a "circuit" may be a hardwired logic circuit, or a programmable processor, e.g., a programmable logic circuit such as a microprocessor. A "circuit" may also be a processor executing software, e.g., any kind of computer program. Any other kind of implementation of the respective functions above may also be understood as a "circuit".

[0181] While particular embodiments have been described, it should be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the disclosed embodiments as defined by the appended claims, the scope of which is therefore indicated by the appended claims, and all changes that come within the meaning and range of equivalence of the claims are therefore intended to be embraced.

[0182] The following items relate to further embodiments.

[0183] Item 1: A test management component of a communication system, comprising: a test input interface configured to receive a request for a test to be performed on the communication system and a specification of the test; According to said specifications for said test, determining a number of communication network components of the communication system that need to be involved in the test in order to perform the test; determining, for each communication network component, a respective test controller capable of managing said communication network component with respect to testing; For each determined test controller, determining a specification of sub-tests to be performed by the test controller on the communication network components of the determined plurality of communication network components that the test controller can manage with respect to testing. A processing unit configured as described above, a test control interface configured to transmit, for each determined test controller, to the test controller the determined specification of the sub-test to be executed by the test controller; Test management components including:

[0184] Item 2: A test management component as described in Item 1, wherein each test controller has its own policy and / or can use its own capabilities and resources for testing, and the test management component is configured to negotiate with each test controller the sub-tests to be executed by the test controller in a manner that conforms with the policy of the test controller and / or is feasible using the supported capabilities and available resources.

[0185] Item 3: A test management component as described in item 1 or 2, wherein each sub-test includes a plurality of test actions, and negotiation with each test controller includes determining the test actions to conform to the policy of the test controller and the specifications of the sub-test.

[0186] Item 4: A test management component described in any one of items 1 to 3, wherein the test relates to an end-to-end communication test between two components of the communication system, and determining the multiple communication network components of the communication system that need to be involved in the test includes determining a communication network component that provides end-to-end communication.

[0187] Item 5: A test management component described in any one of items 1 to 4, wherein each test controller is capable of managing, with respect to testing, communication network components of each domain of the communication system associated with the test controller.

[0188] Item 6: A test management component described in any one of items 1 to 5, wherein each domain of the communication system includes one or more subnetworks of the communication system and / or one or more layers of a protocol stack.

[0189] Item 7: A test management component according to any one of items 1 to 6, wherein the domains associated with different test controllers are domains of different communication network operators.

[0190] Item 8: A test management component described in any one of items 1 to 7, wherein the test input interface is configured to receive the request for the test and the specification of the test from a support system of one of the communication network operators or from one of the test controllers.

[0191] Item 9: A test management component described in any one of items 1 to 8, wherein the test controllers are test area controllers, each test area controller controlling a respective subset of communication network components and / or functions of communication network components.

[0192] Item 10: The test management component of item 9, wherein the test management component is configured to create at least some of the test area controllers according to the specifications of the tests.

[0193] Item 11: A test controller for a part of a communication system, a test input interface configured to receive a request for a test to be performed on the communication system and a specification of the test; determining whether the test includes one or more communication network components that the test controller does not have control over for the test; If the test includes one or more communication network components that the test controller cannot manage with respect to the test, triggering a test management system function to manage that the test is executed by the test controller and a set of one or more other test controllers that can manage the one or more communication network components that the test controller cannot manage with respect to the test. A processing unit configured as follows: A test controller containing:

[0194] Item 12: The test controller of item 11, configured to execute the test if the test includes only communication network components that the test controller can manage for the test.

[0195] Item 13: A communication system including a test management component described in any one of items 1 to 10 and a test controller described in item 11 or 12.

[0196] Item 14: A method for managing testing of a communication system, comprising: receiving a request for a test to be performed on the communications system and a specification of the test; According to said specifications for said test, determining a number of communication network components of the communication system that need to be involved in the test in order to perform the test; determining, for each communication network component, a respective test controller capable of managing said communication network component with respect to testing; For each determined test controller, determining a specification of sub-tests to be performed by the test controller on the communication network components of the determined plurality of communication network components that the test controller can manage with respect to testing. Steps and for each determined test controller, sending to said test controller said determined specification of the sub-tests to be executed by said test controller; The method includes:

[0197] Item 15: A method for processing requests for testing a communication system, comprising: receiving, by a test controller, a request for a test to be performed on the communications system and a specification of the test; determining whether the test includes one or more communication network components that the test controller has no control over for the test; if the test involves one or more communication network components that the test controller is not able to manage for the test, triggering a test management system function to manage that the test is executed by the test controller and a set of one or more other test controllers that are able to manage the one or more communication network components that the test controller is not able to manage for the test; The method includes:

Claims

1. 1. A test management component for a communications system, comprising: a test input interface configured to receive a request for a test to be performed on the communication system and a specification of the test; According to said specifications for said test, determining a number of communication network components of the communication system that need to be involved in the test in order to perform the test; determining, for each communication network component of the determined plurality of communication network components, a respective test controller capable of managing the communication network component with respect to testing; For each determined test controller, determine a specification of sub-tests to be performed by the test controller on the communication network components that the test controller can manage for testing. A processing unit configured as described above, a test control interface configured to transmit, for each determined test controller, to the test controller the determined specification of the sub-test to be executed by the test controller; Test management components including:

2. 2. The test management component of claim 1, wherein each test controller has its own policy and / or can use its own capabilities and resources for testing, and the test management component is configured to negotiate with each test controller the sub-tests to be executed by the test controller in a manner that conforms with the policy of the test controller and / or is feasible with the supported capabilities and available resources.

3. 2. The test management component of claim 1, wherein each sub-test includes a plurality of test actions, and negotiation with each test controller includes determining the test actions to conform to a policy of the test controller and the specification of the sub-test.

4. 2. The test management component of claim 1, wherein the test relates to an end-to-end communication test between two components of the communication system, and determining the plurality of communication network components of the communication system that need to be involved in the test comprises determining a communication network component that provides end-to-end communication.

5. 2. The test management component of claim 1, wherein each test controller is capable of managing, with respect to testing, communication network components of a respective domain of the communication system associated with the test controller.

6. 6. The test management component of claim 5, wherein each domain of the communication system includes one or more sub-networks of the communication system and / or one or more layers of a protocol stack.

7. The test management component of claim 6 , wherein the domains associated with different test controllers are domains of different communication network operators.

8. 8. The test management component of claim 7, wherein the test input interface is configured to receive the request for the test and the specification of the test from a support system of one of the telecommunications network operators or from one of the test controllers.

9. 2. The test management component of claim 1, wherein the test controllers are test area controllers, each test area controller controlling a respective subset of communication network components and / or functions of communication network components.

10. The test management component of claim 9 , wherein the test management component is configured to create at least some of the test area controllers according to the specification of the test.

11. 1. A test controller as part of a communication system, comprising: a test input interface configured to receive a request for a test to be performed on the communication system and a specification of the test; determining whether the test includes one or more communication network components that the test controller does not have control over for the test; If the test involves one or more communication network components that the test controller cannot manage with respect to the test, triggering a test management system function to manage that the test is executed by the test controller and a set of one or more other test controllers that can manage the one or more communication network components that the test controller cannot manage with respect to the test. A processing unit configured as follows: A test controller containing:

12. The test controller of claim 11 , configured to execute the test if the test includes only communication network components that the test controller can manage for the test.

13. A communication system comprising a test management component according to any one of claims 1 to 10 and a test controller according to claim 11 or 12.

14. 1. A method for managing testing of a communications system, comprising: receiving a request for a test to be performed on the communications system and a specification of the test; According to said specifications for said test, determining a number of communication network components of the communication system that need to be involved in the test in order to perform the test; determining, for each communication network component of the determined plurality of communication network components, a respective test controller capable of managing the communication network component with respect to testing; For each determined test controller, determine a specification of sub-tests to be performed by the test controller on the communication network components that the test controller can manage for testing. Steps and for each determined test controller, sending to said test controller said determined specification of the sub-tests to be executed by said test controller; The method includes:

15. 1. A method for processing requests for testing a communications system, comprising: receiving, by a test controller, a request for a test to be performed on the communications system and a specification of the test; determining whether the test includes one or more communication network components that the test controller has no control over for the test; if the test involves one or more communication network components that the test controller is not able to manage for the test, triggering a test management system function to manage that the test is executed by the test controller and a set of one or more other test controllers that are able to manage the one or more communication network components that the test controller is not able to manage for the test; The method includes:

Images