Traffic optimization for wireless networks with capacity limits

Abstract

This invention relates to a wireless mesh network for high-capacity backhauling of data traffic such as for telecommunications networks. Reliability is increased through resilience, i.e., data traffic can be transported via multiple paths between customer equipment (e.g., telecommunication cells) and fiber points. Depending on a network status, a node can choose to direct traffic via a different path, thus reducing the chance of packet drops. Once resilience is established, network traffic can be optimized to respond to variable natures of traffic density. By doing this, network reliability can be further increased and throughput optimized while minimizing costs.

Description

[0001] TRAFFIC OPTIMIZATION FOR WIRELESS NETWORKS WITH CAPACITY LIMITS

[0002] FIELD OF THE INVENTION

[0003] The invention relates to the field of communication in wireless networks, such as - but not limited to - telecommunication networks, for use in various different applications.

[0004] BACKGROUND OF THE INVENTION

[0005] Outdoor lighting grids (e.g., street lighting) offer a near-ideal grid to deploy wireless communication infrastructure (such as Wi-Fi, telecommunications 4G / 5G, E-band and V-band backhaul) because they may offer proximity (to people and / or traffic), scale (ubiquitous presence), granularity (distance between light poles matches typical requirements of radio frequency (RF) network design) and elevation (height to mount equipment for signal coverage). On top of that, they may be used to provide electrical power. A challenge to get acceptance from cities (permits) and the public is to provide aesthetic solutions and minimized form factors, as there is a strong desire to hide technology in unseen places.

[0006] From a technology point of view, this may put pressure on the design of overall RF systems which may consist of a base band unit (BBU), a radio unit and an antenna (array), as key building blocks. For lower frequencies (e.g., below about 7 GHz), the BBU, the radio unit and the antenna (array) may be physically separated. However, for higher frequencies, physical separation between the radio unit and the antenna (array) needs to be minimized.

[0007] The ever-growing data consumption (data throughput) requires higher bandwidths, which in turn demands usage of higher frequencies. As an example, radio frequencies used for telecommunication’s 5G standard will increase to 3-6 GHz and later to 26 GHz and beyond. The frequencies of 26 GHz and higher frequencies are typically referred to as mmWave (millimeter wave) because the wave lengths are in the order of (several) millimeter. Moreover, frequencies of wireless backhaul networks (i.e., physical parts of communications networks between central backbones and individual local networks) are typically 60 GHz or 70 GHz and will grow beyond 100 GHz. Signals at these frequencies behave like optical waves in the sense that they do not penetrate walls and objects. Thus, a communication link between a transmitting point or node (transmitter) and a receiving point or node (receiver) requires clear line of sight (LOS), which means that the transmitter and receiver must “see” each other via an uninterrupted, unobstructed, straight line.

[0008] Conventional wireless point-to-multipoint networks that act on layer 2 (i.e., data link layer which is the protocol layer that transfers data between nodes on a network segment across the physical layer) of the Open Systems Interconnection (OSI) model are restricted to tree and hub and spoke topologies. Ring-like structures are not possible in a layer 2 path because this would cause broadcast storms (i.e., an abnormally high number of broadcast packets within a short period of time, which can overload switches and endpoints as they struggle to keep up with processing the flood of packets). Additional protocols like the rapid spanning tree protocol (RSTP) actively enforce trees by blocking ports that could form a ring structure. A disadvantage of this approach is that it might cause traffic delays and data loss due to a slow determination of trees and in the case of failure, rebuilding of a new tree.

[0009] In a more advanced conventional approach, a ring topology may be prevented by splitting the topology into linear and / or tree topologies. Splitting may be done by disabling certain links in the network. The selection of which links are disabled can be arbitrary, as long as the result is that rings are no longer present.

[0010] However, a network may change over time, i.e., there may be nodes added to the network and there may be new customer-premises equipment (CPEs) attached to the network, which may create a new network situation and the network is re-designed and new information needs to be loaded into the network nodes. Note that in this document, we understand the term “network design” to not only include activities performed before the network is installed and commissioned, but also any activities that include change of settings, configurations or parameters during the operational phase of the network. Moreover, in this document, the term “routing” is used as a generic term related to manoeuvring network traffic (packets) between two points in the network, rather than routing in narrow sense of using tables with IP addresses.

[0011] In telecommunications, a customer-premises equipment or customer-provided equipment (CPE) is any terminal and associated equipment located at a subscriber's premises and connected with a carrier's telecommunication circuit at a demarcation point ("demarc"). The demarc is a (physical) point established in a building or complex to separate customer equipment from the equipment located in either the distribution infrastructure or central office of the communications service provider. In this document, the term CPE is intended to refer to any equipment that requires a (broadband) data connection with an external network and that uses the wireless network described in the embodiments as a means to connect to said external network.

[0012] Furthermore, the amount of traffic in the network may not be considered in conventional approaches.

[0013] Fig. 2 shows schematically a linear network architecture and its limitations in case of varying network traffic.

[0014] As can be gathered from Fig. 2, each transmitter (transceiver) A, B can “inject” a maximum data rate (e.g., 1000 Mbps) into the linear network. Regardless of the required capacity that connects to the transmitter, the link created by the transmitter has a maximum capacity of 1000 Mbps. Thus, each node in a network section (e.g., nodes 1 to 9 in Fig. 2) can receive up to 1000 Mbps capacity. If each of the nodes 1 to 5 is a receiver (e.g., has a CPE attached), transmitter A can supply 1000 Mbps in total to a daisy chain of nodes 1 to 5, since the link between nodes 5 and 6 is disabled. Depending on external device traffic requirements, Class of Services (CoS) or other considerations, the 1000 Mbps will be distributed amongst five receivers. Similarly, transmitter B can distribute 1000 Mbps amongst four receivers (nodes 6 to 9). Suppose that under average conditions all CPEs require 150 Mbps capacity. In this average case the network is sufficiently dimensioned to provide the required service.

[0015] However, suppose the CPEs on node 3 and node 5 experience a temporary steep increase in traffic and each requires 600 Mbps, while all other CPEs do not require any traffic. Then, the transmitter A cannot accommodate the combined traffic (1200 Mbps) of the nodes 3 and 5, which results in packet loss and / or reduced download speeds for CPEs connected to nodes 3 and 5. At the same time, transmitter B has excess capacity but cannot provide additional service because of the disabled link between nodes 5 and 6. Hence, the network is not utilized to its full capacity.

[0016] US2022294730A1 is related to a controller. Upon receiving via a user interface a downgrade policy that specifies an objective for path downgrades in the network, the controller identifies traffic of an application conveyed by an edge router in the network via a particular path in the network and using a first type of link. The controller predicts an effect of downgrading the particular path from using the first type of link to using a second type of link to convey the traffic of the application. The controller causes the edge router to convey the traffic of the application via the second type of link, when the effect predicted by the controller satisfies the objective specified by the downgrade policy. US2023145097A1 is related to a method and apparatus for dynamically routing traffic from a first network device to a second network device by optimizing network path routing over network optimization objectives utilizing deep reinforcement learning using the predicted future network statistics.

[0017] SUMMARY OF THE INVENTION

[0018] It is an object of the present invention to provide a fast and dynamic network control functionality that allows for better exploitation of routing options while minimizing network resources and congestions.

[0019] This object is achieved by an apparatus as claimed in claim 1, by a network design system as claimed in claim 9, by a wireless network as claimed in claim 13, by a method as claimed on claim 14, and by a computer program product as claimed in claim 15.

[0020] According to a first aspect, an apparatus is proposed for controlling network design of a wireless network comprising at least one first type of network node, connected to a point of presence, and at least two second type of network nodes, connected to respective customer equipment, wherein the apparatus comprises a traffic orchestrator configured to monitor real-time network performance and to evaluate and / or estimate a potential impact of routing decisions made by network nodes of the wireless network on network performance. Real-time network performance is understood as obtaining performance parameters instantly or with negligible latency or transmission delays. In this context, the term real-time is synonymous with “live”.

[0021] According to a second aspect, a network design system comprising an apparatus of the first aspect is proposed.

[0022] According to a third aspect, a wireless network comprising a network design system of the second aspect and at least one first type of network node, connected to a point of presence, and at least two second type of network nodes, connected to respective customer equipment, is proposed.

[0023] According to a fourth aspect, a method of controlling network design of a wireless network comprising at least one first type of network node, connected to a point of presence, and at least two second type of network nodes, connected to respective customer equipment, is proposed, wherein the method is configured to perform traffic orchestrating by monitoring real-time network performance and evaluating and / or estimating a potential impact of routing decisions made by network nodes of the wireless network on network performance. According to a fifth aspect, a computer program product is provided, which comprises program code for producing the steps of the above method of the fourth aspect when run on a computer device.

[0024] Accordingly, the proposed solution may allow for reactive control of the network design through full use of routing options and fast switching between routes while taking network traffic into account. Individual links can be formed and activated. Traffic routing through the wireless network can be controlled with a logical routing path (tunnel) based on a tunnel identifier in the mesh network. Orchestration of the network can thus be built on and / or made interoperable with the logical routing path. A ring topology may be prevented by defining communication paths (tunnels) such that rings are broken. Even if physical connections are available in a ring topology, logical paths (tunnels) can be used to break the ring. Specifically, a first node in the network may have multiple logical paths defined towards a second node in the network, such that the combination of both paths creates a ring and whereby only one logical path is used (“activated”) at any given time.

[0025] Each node of the network may have a prioritized list of possible logical paths. Prioritization is at least such that one logical path is known as the primary path, which is the logical path that the node would use under normal circumstances. A regular check may be performed on the network, either by the node or by a central agent such as the network design system, to detect exceptions or anomalies. Such checks may be performed multiple times per second, say at least 10 times per second. Exceptions may include loss of continuity in the path (e.g., an interrupted link). When such an exception occurs, the node may temporarily abandon the use of the primary path, and use a logical path with lower priority instead, until the exception has disappeared. Nodes may thereby be capable of switching tunnels multiple times per second. Furthermore, traffic in the network is monitored and the obtained traffic information can be used for designing the network and for modifying the network flow, i.e., modifying the allocation and activation of logical paths (tunnels) on nodes.

[0026] The proposed solution may thus be suitable for scalable networks on which multiple CPEs demand variable traffic load and may allow to minimize network resources both for initial investment in the network as well as for the recurring cost of operating the network. This may be achieved by at least one of reducing data capacity at points of presence (POPs), minimizing the number of nodes in the network and designing the network for traffic that is lower than the peak demand. As a result, the achieved design can balance the capacity more evenly over the POPs, which enables a lower subscription for all POPs. Additionally, the proposed solution may minimize congestion events, both in number of occurrences and duration. Traffic may be switched to a backup path if a primary path is congested. The backup path is to be understood as an alternative path opposed to a certain default path (primary path) that may initially be used. For example, suppose a first node wants to switch to a backup path but the backup path becomes congested because there is already traffic from a second node on this path. Then, the orchestrator of the network design system may switch the second node to another backup path to enable the switch of the first node. Similarly, the traffic routing in a certain state of the network configuration may lead to higher load on one network entry point (e.g., an optical fiber connection) than on other network entry points. By intelligently re-configuring traffic flows (e.g., by reconfiguring logical paths on one or more nodes) traffic can be better balanced between network entry points. Thus, they do not need to be configured for high peak demand.

[0027] Consequently, costs of the network may be minimized by at least one of reducing the number of POPs and / or data capacity that flows to the POPs, increasing the ratio between connected CPEs (e.g., cameras, WiFi access points, telecom units etc.) and node locations, and reducing the overall number of nodes in the network.

[0028] According to a first option of any of the first to fifth aspects, the traffic orchestrator may comprise at least one of a first orchestrator configured to simulate scenarios by varying priorities of tunnels between network nodes of the wireless network and calculating effects on network metrics, a second orchestrator configured to simulate scenarios by adding and / or changing tunnels between network nodes of the wireless network, a third orchestrator configured to simulate scenarios by adding and / or removing links between network nodes of the wireless network, and a fourth orchestrator configured to simulate scenarios by adding and / or moving and / or removing network nodes of the wireless network.

[0029] According to a second option of any of the first to fifth aspects, which may be combined with the first option, the traffic orchestrator may be configured to decide in a three- step process comprising a first step in which a proposed set of virtual tunnel flips (switches, changes) to respective tunnels with different priority is generated based on traffic metrics on link level, a second step in which an effect of the tunnel flips is simulated based on topology and running average or historic network traffic, and a third step in which overall network performance is evaluated based on overall network metrics of a pre-flip and post-flip situation. Next to historical data the simulation could also be done by a mathematical flow model of the network that enables to calculate effect on traffic capacity (later called digital twin simulation) at each of the nodes by any considered action, not only the action of a tunnel flip.

[0030] According to a third option of any of the first to fifth aspects, which may be combined with the first or second option, the traffic orchestrator may be configured to load different tunnel priorities into the network, effectively resulting in a rerouting of traffic through the network, in particular through tunnels, in particular through tunnels, only if a resultant overall network performance metric improves.

[0031] According to a fourth option of any of the first to fifth aspects, which may be combined with any of the first to third options, the traffic orchestrator may be configured to use a simulation of a behavior of a computer-generated model of the wireless network based on actual or historical data obtained from the wireless network, from other networks, or from estimated values, to create a simulated set of overall network metrics.

[0032] According to a fifth option of any of the first to fifth aspects, which may be combined with any of the first to fourth options, at least some of the network nodes of the wireless network may comprise a plurality of physically separated sectors configured to establish a plurality of links to other network nodes.

[0033] According to a sixth option of any of the first to fifth aspects, which may be combined with any of the first to fifth options, the wireless network may form, in terms of a physical or link level of the network nodes, a ring structure that starts at one of the sectors of the first network node and ends at another sector of the first network node, or a linear structure where network traffic is flowing between two first network nodes and two second network nodes, or any combination thereof.

[0034] According to a seventh option of any of the first to fifth aspects, which may be combined with any of the first to sixth options, timing levels of different optimization and deployment procedures used by the traffic orchestrator may range from once per 1-10 minutes through once per 1-10 days and once per 1-10 weeks to once per 1-10 months.

[0035] According to an eighth option of any of the first to fifth aspects, which may be combined with any of the first to seventh options, the wireless network may be controlled based on feedback information received from the wireless network and indicating the performance of the network, to design the wireless network by at least one of transmitting and / or loading control information to the network nodes, setting a tunnel layout, and setting a class of service. According to a ninth option of any of the first to fifth aspects, which may be combined with any of the first to eighth options, the feedback information may comprise network metrics and / or other performance parameters.

[0036] According to a tenth option of any of the first to fifth aspects, which may be combined with any of the first to ninth options, a priority scheme may be provided by defining at least primary and secondary tunnels to allocate an order of priority for use of the tunnels, wherein a primary tunnel is used by the network nodes as default tunnel for data transport unless an exceptional condition occurs.

[0037] It is noted that the above apparatus may be implemented based on discrete hardware circuitries with discrete hardware components, integrated chips, or arrangements of chip modules, or based on signal processing devices or chips controlled by software routines or programs stored in memories, written on a computer readable media, or downloaded from a network, such as the Internet. More specifically, the software may run on a virtual device such as a Docker Container, or virtual machine, which may be implemented on a physical server.

[0038] It shall be understood that the apparatus of claim 1, the network design system of claim 9, the wireless network of claim 13, the method of claim 14, and the computer program product of claim 15 may have similar and / or identical preferred embodiments, in particular, as defined in the dependent claims.

[0039] It shall be understood that a preferred embodiment of the invention can also be any combination of the dependent claims or above embodiments with the respective independent claim.

[0040] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter.

[0041] BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In the following drawings:

[0043] Fig. 1 shows schematically an architecture of a wireless mesh network with dynamic network design system according to various embodiments;

[0044] Fig. 2 shows schematically a linear network architecture and its limitations in case of varying network traffic;

[0045] Fig. 3 shows a diagram indicating network orchestrating options and their timings;

[0046] Fig. 4 shows schematically a network design according to a first embodiment; Fig. 5 shows schematically a network design according to a second embodiment;

[0047] Fig. 6 shows schematically a first orchestrator for the network design system according to various embodiments;

[0048] Fig. 7 shows a flow diagram of an orchestrator decision making process according to an embodiment;

[0049] Fig. 8 shows schematically an interaction structure between the first orchestrator and a second orchestrator for the network design system according to various embodiments;

[0050] Fig. 9 shows schematically an interaction structure between the first and second orchestrators and a third orchestrator for the network design system according to various embodiments; and

[0051] Fig. 10 shows schematically an interaction structure between the third orchestrator and a fourth orchestrator for the network design system according to various embodiments.

[0052] DETAILED DESCRIPTION OF EMBODIMENTS

[0053] Various embodiments of the present invention are now described based on a wireless mesh network, such as a wireless network compliant with IEEE 802.1 lad, IEEE 802.1 lay or similar networks.

[0054] The network can be defined as a network of N network nodes that can establish a connection to other nodes by establishing wireless RF links. The network operates best in Line-Of-Sight (LOS) conditions to maximize connectivity. In its essence, the network may be “wireless fiber” with gigabit speeds, rapid deployment capability, and flexible use case support.

[0055] Furthermore, the network may be a distribution level network designed to augment and expand a fiber optic network. That is, the network may be a high-speed backbone network to which other networks are connected. When combined with fixed access connections or Wi-Fi access points, the network provides a low-cost solution to achieve street-level coverage with high speeds.

[0056] The network may be composed of individual nodes supported by network services operating in a cloud. There may be two types of nodes- distribution nodes (DNs) and client nodes (CNs). DNs are the backbone of the network that distribute internet connectivity from one or more (fiber optic) points of presence (POPs) over multiple hops to CNs. DNs and CNs are connection points for client networks, wireless access points, and other customer premise equipment (CPE) to connect to the internet or private networks (e.g., a network for closed circuit television (CCTV) or the like).

[0057] Each node may use a multi-element, phased array antenna and advanced beamforming techniques to form high-quality communication links between nodes at distances in excess of 100 m, including primarily LOS paths, while also using reflection paths between the nodes. The multiple propagation paths between nodes are called microroutes, and each micro-route is a potential communication link between nodes. Each node may select the strongest communication link from the many possible propagation paths between the nodes. Phased-array antennas may allow nodes to establish links to mitigate cochannel interference that often prevails in dense urban environments.

[0058] According to embodiments, the flow of data traffic in the network is continuously optimized by using an active network design system. Thereby, the fact that networks are dynamic systems where traffic per connected external device varies over time with different timescales and where these variations have very limited predictability can be addressed. Moreover, the wireless nature of such networks adds additional dynamics in that the capacity and number of links between network nodes may vary. In order to continually optimize traffic (minimize congestion) and respond to variations, a layered / nested / hierarchical network design system is described in the below embodiments.

[0059] In a typical architecture of such networks, a network node (or “node” throughout the present disclosure) may have a central network processor that handles network traffic (or “traffic” throughout the present disclosure) from and to a plurality of sectors. A sector comprises a modem, an RF transceiver and a segmented antenna (e.g., phased-array antenna) that together use beamforming to establish links with sectors of other nodes. As an example, one node may comprise four sectors to enable a 360-degree coverage horizontally (azimuthally).

[0060] It is further assumed that each node can establish up to four network connections, paths or channels to other nodes, which may be accomplished by four physically separated antennas, called sectors. It is however noted that this assumption is only intended to simplify the description of embodiments. It is not a necessary restriction. Each node may also have more (or fewer) than four network connections, paths or channels to other nodes, especially when several CPE's are connected to a node. Each sector might indeed be used for no, a single or multiple connections to other nodes are CPE's. It is still further assumed that the network is operated on a symmetric time division duplex (TDD) or time-division multiple access (TDMA) scheme, where nodes alternate between transmission and reception modes and where equal time periods may be allocated for each mode (symmetric mode). However, the below embodiments are also valid for TDD / TDMA schemes that are asymmetric.

[0061] The network may be built on odd / even polarity, i.e., while odd nodes are transmitting, even nodes are receiving and vice versa, at least on networks using a fixed symmetric TDD scheme. This enables infinite scalability of the network and reduces interference. When a node is transmitting / receiving, all antennas on the node are transmitting / receiving synchronously.

[0062] Fig. 1 shows schematically an architecture of a wireless mesh network with a dynamic network design or management system (network design system in the following) according to various embodiments.

[0063] It is noted that - throughout the present disclosure - the structure and / or function of blocks with identical reference numbers that have been described before are not described again, unless an additional specific functionality is involved. Moreover, only those structural elements and functions are shown, which are useful to understand the embodiments. Other structural elements and functions are omitted for brevity reasons.

[0064] The wireless mesh network 100 provides a data transport network consisting of N nodes 110, indicated as a black circular dot in Fig. 1, wherein each node 110 is a distribution point (DP).

[0065] Furthermore, one or more of the DP nodes 110 is coupled to an external network, considered an entry point (EP). The entry point may be an optical fiber network, or gateway or point of presence (POP) 20 (A, B and C), indicated as a rhombus in Fig. 1. Such a network node is called a “DP -EP” node 120. The POP 20 may be an artificial demarcation point or network interface point between communicating entities. In examples, the POP 20 may be a fiber cable sticking out of the ground. The cable may be connected to a network switch within the fiber provider's fiber network. The fiber provider may sell capacity and connections like the POP 20 as a service to data-service companies such as telecommunication operators, internet service providers, companies who operate a CCTV network, etc. In terms of networking, the POP 20 could be an edge data center.

[0066] Additionally, one or more of the DP nodes 110 is coupled to a customerpremise-device or CPE 30 (D and E), indicated in Fig. 1 as a triangle. Such a node 140 is called a “DP-XP”. The network 100 is designed to transport data between a DP -EP 120 and a DP- XP 140 over the wireless connections between the nodes 110. Note that continuous lines between network devices (i.e., nodes, POPs and CEs) of Fig. 1 indicate wired connections while dotted lines between network devices of Fig. 1 indicate wireless connections.

[0067] The network 100 is made resilient in the sense that more than one logical path or channel can be established or selected for transporting data (between DP -EP 120 and DP- XP 140). These logical paths or channels or connections are called “tunnels” throughout the present disclosure. More specifically, a tunnel is to be understood as an available logical path, progressing along a sequence of wireless (physical) links, between two network nodes, whereby one of the network nodes is connected to an external network such as a fiber optics network. In other words, a series of links between a node and a DP -EP node. Thus, a tunnel is a logical path, which implies that multiple tunnels can make use of (a number of) the same physical links, whereby all tunnels are logically separated.

[0068] Furthermore, a network design system (NDS) 10 is provided, that controls the network 100 to design the network 100 e.g. by transmitting / loading control information (e.g., network design information (NDI)) to the nodes (DP -EP 120, DP-XP 140, DP 110) and / or setting a tunnel layout and / or setting a potential class of service (COS).

[0069] In an example, the network design system 10 may be configured to provide a priority scheme by defining primary, secondary and tertiary (and so on) tunnels to allocate an order of priority for the use of the tunnels. As an example, the nodes (DP -EP 120, DP-XP 140, DP 110) may by default use a primary tunnel for data transport unless an exceptional condition occurs.

[0070] Logical paths within the network 100 may be proprietary tunnels, virtual local area network (VLAN) connections, or 802. IQ tunneling connections (i.e., aka Q-in-Q, a technique often used by Metro Ethernet providers as a layer 2 VPN for customers, wherein the provider puts an 802. IQ tag on all the frames that it receives from a customer with a unique VLAN tag). 802. IQ is the original base standard that defines the usage of a single tag VLAN in a L2 network. Q-in-Q is a VLAN implementation (introduced by IEEE std 802. lad) with a dual VLAN tag, e.g., to create separate logical networks per provider and also per customer or service. By using a different VLAN tag for each customer, the traffic can be separated from different customers and also transparently transferred throughout a service provider network.

[0071] A VLAN is a logical overlay network that groups together a subset of devices that share a physical LAN, isolating the traffic for each group, where the path is coded into the data stream. VLANs may thus define a tunnel from the transmitter to the POP without the need for physically disabling the link, which is a time-consuming process.

[0072] In an embodiment, the network 100 may be transparent (ignorant) of routing protocols that are used outside of the network. Logical paths outside the network 100 may be VLAN and / or Q-in-Q connections.

[0073] However, tunneling may be based on other systems such as Multi Protocol Label Switching (MPLS), Virtual Private LAN Service (VPLS) tag (label) based data etc.

[0074] Data packets that are routed between a given DP -EP 120 and a given DP-XP 140 may have a tunnel identifier that makes sure that the data is transported along a predefined path (tunnel).

[0075] The network design system 10 may control the network 100 based on feedback information (e.g., network performance information (NPI) such as network metrics and / or other performance parameters) received from the network 100 and indicating the performance of the network. The network design system 10 may also control the network 100 based on simulated (expected) performance metrics, based on any combination of the known topology of the network combined with the data usage by the CPEs and capacity of POPs.

[0076] In an example, the network design system 10 may include a storage and analysis system and a design system that can adjust and / or control the network design based on the performance data (e.g., NPI) received back from the network 100. Furthermore, a system to receive contextual information may be provided. Optionally, a further simulation or emulation system may be provided for checking performance parameters of network design adjustments.

[0077] The network design system 10 may be configured to apply the following procedures to adjust / control the network design. To achieve this, the network design system 10 may comprise one or more processors controlled by instructions of software routines (stored in or loaded into a memory of the network design system 10) to perform steps or operations of the following procedures.

[0078] In the example of Fig. 1, a data transport network 100 with N nodes is controlled by the network design system 10. The network may be intended to transport data to and from a location where there is an external broadband connection, i.e., the POPs 20 (A to C). These may be fiber POPs, wherein the DP node 110 closest to the POP 20 is the DP- EP node 120. Furthermore, in Fig. 1, there are three DP -EP nodes 120 (including node 1 in the upper left corner of the network 100) which are connected to the POPs A, B and C 20. The data is transported via a number of DP nodes 110 to a location where there is a need for connectivity by a device (e.g., CPE 30 (D, E)) that is external to the network, but at which location no wired connectivity is available. An example of such an external device can be a camera, a Wi-Fi access point or a small cell radio. If the location of the external device coincides with the location of the respective DP-XP node 140, it will be connected by means of a wired connection (in Fig. 1 this is a connection to CPE D). The wired connection can be Ethernet, Power over Ethernet, fiber, etc. If the locations are not coinciding, the connection to the nearest DP-XP node 140 can be achieved by means of a terminal unit (in Fig. 1 this is a connection to CPE E). Such terminal unit (not shown) can make a wireless link to any node that is in its range in the same way as links between nodes are set up. The terminal unit may also have a connector to make a wired connection to the device external to the network (e.g., CPE E). In both cases (direct connection and the connection via a terminal unit) the (nearest) connected node is a DP-XP node 140. All nodes in the network that are neither DP -EP nodes 120 nor DP-XP nodes 140 are called DP nodes 110.

[0079] DP nodes 110 can become DP -EP nodes 120 when they are connected to a fiber POP 20, and they can become DP-XP nodes when an external device (e.g., CPE 30) is connected. It should also be noted that at the location of the fiber POP 20 also external devices can be connected, but they are not considered a part of the wireless network to be designed.

[0080] In the example of Fig. 1, the network 100 comprises of a collection of N nodes, with a number L of wireless communication links between the nodes. Each node can connect to multiple other nodes, thus creating an interlinked network, or mesh network. The links have a finite (limited) throughput capacity for data transport. The physical ports of the nodes also have finite throughput capacity. For simplicity it is assumed that the throughput capacity of a physical port that connects to an external device (e.g., a CPE 30) always exceeds the throughput requirements of the external device, i.e., the physical port on a DP- XP node 140 is suitable to deliver at least the peak data consumption of the CPE 30.

[0081] It is further assumed that the physical port to the external network on the DP- EP node 120 has finite capacity. This capacity can be limited by the physical properties of the port, or it can be limited by a data subscription that is purchased.

[0082] In an example, the physical port may comprise a small form-factor pluggable (SFP+) module which is a compact, hot-pluggable network interface module format used for both telecommunication and data communications applications.

[0083] For example, a 10 Gbps SFP+ port has a physical limitation of the port’s capabilities to 10 Gbps. The optical fiber connected to this port may have a data transport capacity that is a multiple of the port capacity. However, the capacity of the connection to the external data infrastructure, e.g., a data center, may be limited to only 5 Gbps (or any other value). This limitation may be due to a physical limitation somewhere in the external network, or an externally imposed limitation based on an agreed data subscription.

[0084] In the below embodiments, it will be assumed that data traffic on the CPEs 30 is asymmetric (i.e., the download data requirements (resulting in data transfer from the DP- EP node 120 to the DP-XP node 140) are higher than upload requirements (resulting in data traffic from the DP-XP node 140 to the DP -EP node 120). The asymmetry may be significant, such as 80-90% of data traffic is download while only 10-20% of data traffic is upload, which is a common ratio for internet services or telecommunication services. Because of the asymmetry, the download traffic is most likely the largest and can cause congestion in the network sooner than upload traffic.

[0085] Therefore, in the following embodiments, download traffic will be discussed only. Arrows indicate the direction of the traffic. Congestion occurs when the data transport demand is higher than the transport demand at any current tunnel in the network 100. It should be understood that there may also be situations where the upload is higher than the download need, but for simplicity all below embodiments describe the case that download need is highest.

[0086] In order to transport the data between a DP -EP node 120 and a DP-XP node 140, the resultant traffic may need to be routed through one or more DP nodes 110 and potentially other DP-XP nodes 140. Such routing may be based on an OSI Layer 2 implementation, in which traffic is routed along predetermined paths to route traffic from the DP -EP node 120 to the DP-XP node 140.

[0087] Fig. 3 shows a diagram indicating network design orchestration options (e.g., optimization (O) and deployment (D) procedures) and their timings, which may be applied by the network design system 10.

[0088] In various embodiments, the network design system 10 may be capable of applying various procedures to adjust the network design. Note that individual embodiments will be described later.

[0089] The procedures may include a procedure to determine a momentary quality of wireless network links of the network 100. Quality of a link may be determined based on data extracted from the network nodes and possibly processed or derived data e.g. by calculating ratios. Quality parameters may include at least one of availability, (excess) capacity of the links, latency, packet delay, signal-to-noise ratio, modular coding scheme (MCS) levels and the like. Applicability and / or usefulness of various parameters may depend on preference and technical feasibility. Extracted information from the nodes may include buffer status in the network node, traffic per link, link budget, MCS level and the like. The link quality may be recorded periodically (e.g., node reports to a central network management software (NMS) of the network design system 10 or the NMS polls such data) or can be reported by network nodes whenever certain thresholds (set points) are exceeded. The reporting of data may be initiated by the nodes themselves (push model) or may be initiated by the network design system 10 (poll model).

[0090] Furthermore, the procedures may include a procedure to store recorded quality data over an extended period of time.

[0091] Additionally, the procedures may include a procedure to determine historic information on quality of wireless network links, e.g. running average of quality parameters, longer term averages and standard deviations, maximum / minimum values during a certain period such as the last 24 hours.

[0092] Moreover, the procedures may include a procedure that combines quality parameters beyond a single node or link, to create a set of overall network metrics, such as of the entire network, based on performance information (e.g., NPI) received from the network 100. An example of such metrics may be the sum of links that experience congestion, or a summation over all links of a total time period that congestion was experienced in the past 10 seconds (e.g., if 5 links experienced congestion during 0.5 seconds during the past 10 seconds, this metric would amount to 5*0.5=2.5). There may be numerous other examples including variations and min / max / average values during certain time periods.

[0093] Further, the procedures may include a procedure that uses simulation of the behavior of a computer-generated model of the network (i.e., a digital twin simulation) based on historical data obtained from the real network (on link quality and traffic) to create a simulated set of overall network metrics. A digital twin simulation operates by digitally replicating the network 100 or a part or path thereof in a virtual environment, including its functionality, features, and behavior. A real-time digital representation of the asset may be created based on performance data (e.g., NPI) collected from the nodes. The digital twin simulations may be based on information that includes at least the network topology, data consumption of the CPEs 30 and data capacity to the external networks connected to the DP- EP nodes 120. Information on such data consumption and capacity may be based on actual or historic data from the real network, from other (similar) networks, or from estimated values. In the diagram of Fig. 3, the left-hand axis indicates timing (TMG) levels for scheduling orchestration procedures which may be available at the network design system 10. The timing levels range from once per second (1 / s), once per minute (1 / m), once per hour ( 1 / h), once per day (1 / D) through once er week (1 / W) and once per month (1 / M) to once per three months or quarter (1 / 3M). The blocks in Fig. 3 indicate the available orchestration procedures, wherein the vertical placing of the blocks indicates the allocated timing level of the procedures.

[0094] More specifically, a first optimization procedure 01 of a first type of orchestration may use digital twin simulation(s) of the network 100 or parts or paths thereof to optimize allocation of primary tunnel(s) (i.e., logical path(s) or channels with highest priority) on nodes (typically daily or multiple times per day).

[0095] For example, in a configuration A, each DP-XP node 140 may have a list of logical paths, and one of these logical paths (channels) may be allocated as primary tunnel (the primary tunnel may be used for data traffic). In another configuration B, each node may have the same list of logical paths, but an alternative allocation of primary tunnels. If simulated congestion in configuration B is less than in configuration A, configuration B may be selected as preferred. By iterating multiple configurations, an optimization can be performed. Furthermore, a first deployment procedure DI of the first type of orchestrator may be provided to deploy (e.g., change) the new primary tunnel allocations (CHG PTA) to the network nodes, e.g. by modifying the priority of one or more logical paths on one or more nodes. The procedures 01 and DI may be performed at least once per minute (1 / m).

[0096] On another timing level, a second optimization procedure 02 of a second type of orchestration may be performed that uses digital-twin-simulations to optimize a definition of tunnels on / to nodes.

[0097] For example, in configuration A, each node may have a list of tunnels, and one of these tunnels is allocated as primary tunnel. In another configuration B, one or more nodes may have a different primary tunnel, which is not defined in any of the tunnels in configuration A. If simulated congestion in configuration B is less than in configuration A, configuration B may be selected as preferred. By iterating multiple configurations, optimization can be performed. Note that this procedure can be used to add tunnel(s) to certain nodes. It may be so that adding a new tunnel to one node can lead to different results of optimization procedure 01. Thus, procedure 01 can be performed afterwards to determine optimal allocation of primary tunnels. Furthermore, the second optimization procedure 02 may be performed at a lower frequency than the first optimization procedure 01. For example, 02 may be performed on a daily or weekly basis (1 / D or 1 / W) or be initiated by an operator.

[0098] A related second deployment procedure D2 of the second type of orchestration may be performed to assign the new (updated) tunnels to the network nodes (ASS NT). Procedure D2 may include, or may be followed by, procedure DI which assigns appropriate priorities (CHG PTA) to one or more tunnels.

[0099] Additionally, a third optimization procedure 03 of a third type or orchestration may be configured to use digital-twin-simulations to optimize a link topology without changing (e.g., adding / relocating) the location or quantity of nodes. This may be achieved by using input data that include at least one of LOS options between nodes and expected link quality and capacity of each LOS option. The third optimization procedure 03 may be similar to the second optimization procedure 02 in that it uses an existing configuration of the network and enhances this configuration. Instead of only adding tunnels, the third optimization procedure 03 may add links between nodes of the network, thus effectively modifying the network topology. Besides this, the third optimization procedure 03 may add new tunnels to nodes that are connected to the new links. By trying out multiple options of new links and comparing simulated overall network quality parameters, optimization can be performed.

[0100] Adding a new link to nodes may lead to different results of the second and first optimization procedures 02 and 01. Thus, procedures 02 and 01 may be performed afterwards to determine optimal allocation of primary tunnels.

[0101] A related third deployment procedure D3 of the third type of orchestration, in which the result of the third optimization procedure 03 is implemented in the network, may be performed at lower frequency than optimization procedures 02 or 01. For example, O3 / D3 may be performed on a monthly basis or be initiated by an operator.

[0102] The third deployment procedure D3 may be performed to deploy the new link topology on the network nodes (e.g., update the link topology between nodes). Deployment of link topologies may include adding and / or removing RF links (A / R RF-LK) between existing nodes in the network. Procedure D3 may include, or may be followed by, procedures D2 and DI to assign appropriate tunnels and priorities to one or more nodes.

[0103] Moreover, a fourth optimization procedure 04 of a fourth type of orchestration may be configured to use digital-twin-simulations to optimize topology by adding nodes to the network and / or relocating nodes in the network. Such a procedure may use input data that may include at least one of potential node locations, LOS options between nodes, and expected link quality and capacity of each LOS option. Addition or relocation of a node may involve adding and / or altering links in the network.

[0104] Relocating a node may only modify the length of links, e.g., if nodes X, Y and Z are in a linear configuration, moving node Y closer to node X would shorten link X-Y and lengthen link Y-Z, thus changing capacity of these two links.

[0105] Addition of links may occur if an action opens possibilities of creating new links, e.g., if nodes X, Y and Z are in a linear configuration, moving node Y out of the line may create a triangle configuration that enables link X-Z to be created.

[0106] Alteration of links may also occur, e.g., in the reverse of the previous example (e.g., nodes X, Y and Z are in a triangle configuration and node Y is moved in between nodes X and Z such that link X-Z is no longer possible).

[0107] Besides links, the procedure 04 may be configured to add or alter tunnels to nodes that are connected to new links. By trying out multiple options of new links and comparing simulated overall network quality parameters, optimization can be performed.

[0108] A related reporting procedure D4 of the fourth type of orchestration may be configured to report outcomes of the optimization procedure 04, e.g., as an advised network upgrade. Note that procedure 04 requires a physical modification of the network by adding / removing nodes (A / RND).

[0109] The related reporting procedure D4 and the fourth optimization procedure 04 may be performed at lower frequency than the other optimization procedures 03, 02 or 01. For example, 04 and D4 may be performed on a 3-monthly (quarterly) basis or be initiated by an operator.

[0110] In an example, the above optimization procedures 01, 02, 03 and 04 may be combined and performed together at same or respective different timing levels.

[0111] A still further fifth optimization procedure 05 of a fifth type of orchestration may be configured to use digital twin simulations to predict the effect of contextual data (CD) to network performance on an instant (INST) timing level. 05 may use one or more of the optimization procedures 01, 02, 03 and 04 to minimize the effect of anticipated external influences. Examples of such contextual data may include emergencies, weather conditions, weather forecasts, maintenance planning. Depending on the type of contextual data, different optimization methods and frequencies may be preferred.

[0112] For example, if maintenance on a node is planned, traffic may be redirected by changing primary tunnels (first optimization procedure 01). If such maintenance isolates a certain area of the network, some additional links may be created (fourth optimization procedure 04) to ensure that this network area retains some connectivity. Even if such additional links are weaker (e.g., lower throughput) than in the original topology, such weaker connections may be preferred to retain at least some connection to the area.

[0113] The fifth optimization procedure 05 may use a related fifth deployment procedure D5 and / or any of the previously listed deployment procedures (DI, D2, D3), or reporting procedure D4 (e.g., to generate a report with suggested temporary addition of nodes to ensure high-capacity connection to a network area that is undergoing maintenance).

[0114] In an example, at least some of the above multiple optimization procedures 01 to 04 may be implemented as software routine(s), e.g., by automatically and periodically running one or more optimizations and / or automizing decision and deployment of the results of one or more optimizing procedures (optimizers) in the network.

[0115] Furthermore, in an example, a software-based reporting of the results of the optimizing procedures 01, 02, 03 and / or 04 may be provided for use and decision making by an authorized user. Such reports may at least include overall network quality parameters of the existing situation and one improved situation as determined by the optimizing procedure(s) (optimizer).

[0116] Note that the allocated timing levels described above and shown in Fig. 3 are indicative only and serve the purpose of creating a hierarchy between the procedures. There may be various reasons to apply faster or slower timing schemes to the procedures. Applied frequencies of the procedures may also vary, e.g., the first procedures 01 / DI may be applied multiple times per hour during certain expected high-traffic times of the network, while only once daily during expected low-traffic times.

[0117] Fig. 4 shows schematically a network design according to a first embodiment, which is controlled (orchestrated) by the network design system 10 of Fig. 1, e.g., by directly overriding tunnel switching or by changing tunnel priorities. The network design of the first embodiment contains one DP -EP node (A) 120 and nine other nodes 1 to 9, of which at least two nodes (nodes 2 and 3) are DP-XP nodes 140 and the remaining nodes (nodes 1 and 4 to 9) are DP nodes 110. Each node includes four sectors that are indicated as rectangular bars at each side of the rectangular node symbol. Arrows with black content indicate a primary tunnel, while arrows with white content indicate a secondary tunnel. The direction of the arrows indicates download traffic.

[0118] In terms of the physical or link level the nodes form a ring structure that starts at one of the sectors of the DP -EP node 120 and ends at another sector of the same DP -EP node 120. This network, when implemented in a layer 2 mode, can become instable in itself because of broadcast storms. For that reason, logical routing paths are implemented that prevent these broadcast storms. The primary tunnels of both DP-XP nodes 140 are routed towards the same sector of the DP -EP node 120, while the secondary tunnels are routed to the other sector of the DP -EP node 120. The DP-XP nodes 140 (nodes 2 and 3) may have implemented a first procedure to validate the availability / connectivity of the primary tunnel. Alternatively, the DP -EP node 120 or the network design system may have implemented such a first procedure. In addition, a second procedure may be implemented to determine the traffic in the two primary tunnels, that is flowing between the DP -EP node 120 and the two DP-XP nodes 140. The traffic determination may be implemented in the DP -EP node 120, in the DP-XP nodes 140 or in a traffic manager (e.g., at the network design system 10 of Fig. 1) that oversees traffic in the whole network. The procedure to determine the traffic may involve a measurement of the transmit and / or receive buffer (e.g., fill percentage or overflow) used in the nodes of the network. If there is a sudden interruption in the availability of the link between the node 2 and 3 (as output of the first procedure), the network design system may decide to use the secondary tunnel of node 3 to the DP -EP node 120, while leaving the primary tunnel of node 2 available (intact). It may also occur that the link availability remains available (intact), and there is a sudden increase in traffic on the DP-XP nodes 2 and 3, which is larger than the link capacity between nodes 2 and 1 or between node 1 and the DP -EP node 120. In that case, the second procedure may detect that and decide that the secondary tunnel of either node 2 or node 3 is activated and the traffic is divided to flow to two of the sectors of the same DP -EP 120. There may be reasons related to a difference in quality of service of the traffic to nodes 2 and 3 to determine which node should use the secondary tunnel. When the summed traffic of nodes 2 and 3 has dropped below the link capacity, the network design system may decide to fall back to the two primary tunnels.

[0119] Fig. 5 shows schematically a network design according to a second embodiment, which is controlled (orchestrated) by the network design system 10 of Fig. 1, e.g., by directly overriding tunnel switching or by changing tunnel priorities.

[0120] It should be noted that the second embodiment is very similar to the first embodiment in terms of network layout, except for the fact that the traffic is now flowing between two DP-EPs nodes 120 (nodes A and B) and two DP-XP nodes 140 (nodes 2 and 3).

[0121] Also note that this embodiment may solve the problem of under-utilizing network capacity initially described with reference to Fig. 2. Through the above first optimization procedure 01, availability / connectivity of the primary tunnels can be validated. In addition, the above second procedure may determine the traffic in the primary paths. If there is a sudden increase in traffic which is larger than the link capacity, the above second optimization procedure 02 may detect that and decide that a secondary tunnel is activated, and the traffic is divided to flow to two of the sectors of the same node. Thus, if the DP -EP node A cannot accommodate a combined traffic, which results in packet loss and / or reduced download speeds for connected CPEs, the secondary tunnel can be activated to allow provision of additional services by the second DP- EP node B so that the network can be utilized to its full capacity.

[0122] The following embodiments may be based on an exemplary VLAN implementation of tunnels, wherein various orchestration functions (orchestrators) for controlling network design are described based on the procedures of the optimization / deployment diagram of Fig. 3, which may be implemented based on software routine(s) executed at the network design system 10 of Fig. 1.

[0123] Fig. 6 shows schematically a first orchestrator (ORC1) 31 for implementing the above first procedures 01 / DI of the network design system according to various embodiments.

[0124] The first orchestrator 31 may be configured to receive at least three types of input information including first reference data (REFI), first settings (SI) and a first state estimation output (ESTI) from a first state estimator (SEI) 310. The first orchestrator 31 may be configured to simulate scenarios by varying tunnel priorities and calculating resulting effects on network metrics.

[0125] The first reference data may include a list of all network nodes and connectivity between these nodes, i.e., the network topology, and a list of all VLAN tunnel options per node, and priority of each tunnel on every node. Typically, one VLAN tunnel may have the highest priority, which signifies the default tunnel. The other tunnels may have lower priority. Each tunnel may have different priorities.

[0126] The first settings may include parameters that may be set by an authorized user or administrator, such as priority (or CoS) for CPEs or data streams, and a frequency setting that determines the frequency at which the first orchestrator 31 operates, which may typically be in the range of multiple seconds or minutes. And there may be a duration parameter for decisions (duration period), that determines how long the effect of a decision will last. Such duration will typically (but not necessarily) be the same or lower than the frequency parameter, a preferred duration may be 5-10 times the period of the operating frequency. For networks with relatively stable traffic requirements, the above frequencies may be lower, such as an operating frequency in the order of hours.

[0127] Other input parameters of the first orchestrator 31 may be disturbances (DIS), such as network fluctuations (e.g., MCS levels on the links between nodes), and demand variations (DEM VAR) such as CPE traffic variations.

[0128] Output information (OUT) of the first orchestrator 31 may be new priorities (primary, secondary etc.) for tunnels. The first orchestrator 31 acts on a shorter timescale compared to the other orchestrators and is designed to react to variations in link quality and demand variations, or repeated continuity failures of tunnels. When it detects (potential) congestion or loss of efficiency in the network, it will send a notification to a selection of DP- XP nodes to change tunnel priorities. Such new primary tunnel allocation (CHG PTA) may have a temporary duration, i.e., after the abovementioned duration period, the node switches back to its default tunnel. In other words, the first orchestrator 31 hands out a “CHG PTA ticket” with a validity of a predetermined time period (e.g., X seconds). A change of the highest priority will thus result in a change of traffic to a different tunnel. The time period would generally be longer than the frequency at which orchestrator 31 acts (by evaluating input from state estimator 310).

[0129] In examples, the first orchestrator 310 may be configured to modify the priorities "permanently" or temporary (using a duration period) to prevent propagation of changes through the network, based on network knowledge including at least topology and / or evaluation of the effect of tunnel changes on traffic in other tunnels, and to decide not to change a tunnel if an assessed post-situation is worse than the pre-situation.

[0130] The first state estimator 310 supports the first orchestrator 310 to take decisions based on link quality information (e.g., buffer sizes per node, traffic per node, link budget etc.). The first state estimation of the first state estimator 310 is at least based on momentary information but may also include historic information and / or running average data.

[0131] The first orchestrator 310 may use the first state estimation and / or other input information to simulate effect(s) of tunnel changes on traffic in other network areas (e.g., tunnels on other nodes).

[0132] In another embodiment, momentary information may be obtained on diagnostic parameters on wireless links, such as link budget, link traffic, buffer usage. Combinations of such parameters and derived parameters are also possible (a derived parameter may be for example “available link capacity” which could be link capacity minus data transport on that link. Momentary link diagnostic values may be obtained from the nodes via a poll or push mechanism.

[0133] Since CPE traffic can vary and fluctuate, it may be advantageous to filter out high-frequency fluctuations, for example by keeping a running average of mentioned diagnostic parameters. Such running average would preferably span a period longer than the frequency setting, preferably at least 5 times longer than the period of the frequency setting. Furthermore, it may be advantageous to keep historic data on the state estimation over longer periods of time, such as hours or days. Such historic data can be used to find patterns, e.g., based on artificial intelligence algorithms. One such pattern may be typical duration of traffic peaks, or typical time-of-day of traffic peaks. Such information may be used to optimize decisions, e.g., at certain times of the day, tunnel priorities on certain nodes will be set such that primary tunnel is tunnel B, whereas at other times of the day these nodes will be configured so that tunnel C is the primary tunnel.

[0134] All tunnel changes may be controlled on a temporary basis, after which the tunnel is switched back to the default state. The reason for this is that this reduces risks of “propagation”, which we will explain below.

[0135] In an example, a network may comprise many nodes and POPs (or DP -EP nodes). In a network section close to two POPs, two nodes may get notified by the first orchestrator 31 to change tunnel priorities (CHG PT A). As a result, traffic will flow via a different tunnel, i.e. the tunnel that has now become primary tunnel. Now, another link may get congested due to the traffic change. Another tunnel priority change (CHG PTA) is triggered, which induces another congestion, and so on. Such propagation may eventually end up at the same network section (link) that had a congestion problem in the first place.

[0136] In order to reduce the chances of such propagation, a period of the decision duration is defined. When this duration expires, the node switches its traffic back to the default tunnel. If congestion still occurs, the first orchestrator 31 may simply issue another tunnel reprioritization. The value of the decision duration may be adjustable, to fine-tune behavior of the first orchestrator 31.

[0137] The above example of propagation illustrates another challenge. Re-routing traffic to a different tunnel (e.g., VLAN tunnel) will affect traffic on other links. As the first orchestrator 31 can decide on tunnel changes for multiple nodes at the same time (or in quick succession), there is a significant chance for creating new congestion issues, even making overall network performance worse. The first orchestrator 31 may therefore need network awareness. Such awareness may consist at least of the following elements: topology, traffic per link and capacity per link. Historic and / or running average traffic information per link may be beneficial as well. One solution may be to create a set of rules and parameters for tunnel reprioritization, which dictate when tunnels can be reprioritized or not.

[0138] Fig. 7 shows a flow diagram of an orchestrator decision making process according to an embodiment.

[0139] The first orchestrator 31 of Fig. 6 may be configured to decide in three-step process. In a first step S 101 , a proposed set of new tunnel priorities (PR) is generated based on traffic metrics on link level. In a second step S102, the effect of the tunnel priorities is simulated (SI) based on topology and running average (or historic) network traffic. Finally, in a third step SI 03, overall network performance is evaluated (EV) based on overall network metrics, of the pre-PR and post-PR situation. The first orchestrator 31 may be configured to execute tunnel priority changes only if the resultant overall network metric improves.

[0140] In order to take the decision, the first orchestrator 31 may evaluate options in a few ways. Firstly, if a (set of) nodes experience(s) frequent tunnel flips, e.g. due to continuity problems, and the frequency of these tunnel flips exceeds a certain threshold, the tunnel priorities may be modified accordingly. For example, if a labelled node has a first tunnel as default and is switched to a second tunnel for more than 30% of the time, the second tunnel becomes the default tunnel (primary tunnel with highest priority). The first tunnel may then become secondary tunnel with second highest priority and the remaining third tunnel may become tertiary tunnel with third highest priority, and so on. Alternatively, the first orchestrator 31 may assign different primary tunnels on one or more different nodes, typically nodes that are adjacent to the labelled node, if that seems a more effective method of reducing the tunnel switching on the labelled node.

[0141] Secondly, if orchestrator 31 uses a finite time duration for the tunnel reprioritization, and if such reprioritization is done at a frequency that exceeds a certain threshold, the orchestrator may decide to increase the duration period.

[0142] Thirdly, the first orchestrator 31 may periodically run a traffic simulator to determine the impact on overall network metrics and to change tunnels or tunnel priorities if network metrics improve.

[0143] Overall network metrics may include parameters such as at least one of number of congested links in the network, number of nearly congested links in the network, duration of the congestion, total sum of excess link capacity in the network, number of CPEs that have restricted throughput, etc. All or at least some of the parameters may be weighted based on CoS or other priority schemes. By simulating the impact of VLAN priority changes on affected links in the network, the first orchestrator 31 can adapt to network changes that are made by other orchestrators.

[0144] Fig. 8 shows schematically an interaction structure between the first orchestrator 31 and a second orchestrator (ORC2) 32 for implementing the second procedures O2 / D2 (shown in Fig. 3) of the network design system according to various embodiments.

[0145] The second orchestrator 32 operates on slower time scales than the first orchestrator 32. Time scales may be typically in order of hours, or even days. The second orchestrator 32 takes at least three types of input information, which are second reference data (REF2), second settings (S2) and a second state estimation (EST2) obtained from a second state estimator 320.

[0146] The second reference data may include a list of all network nodes and connectivity between these nodes, i.e. the network topology, a list of all VLAN tunnel options per node, and priority of each tunnel on every node. Typically, one VLAN tunnel may have primary (highest) priority, which signifies the default tunnel. The other tunnels will have lower priority. Each tunnel may have different priorities.

[0147] The second settings may include parameters that may be set by an authorized user and / or administrator, such as priority (or CoS) for CPEs or data streams, and a frequency setting that determines the frequency at which the orchestrator operates, which may typically be in the range of once per day(s). The settings may also include a list of parameters that limit the number of possible tunnels on the network, such as maximum path length (e.g., number of hops) for a tunnel, maximum number of tunnels per node.

[0148] The second orchestrator 32 may be configured to periodically analyze the topology and determine if there are options for new VLAN definitions. In theory, a network node can have multiple tunnels to a particular DP -EP node. The number of possible tunnels depends on the level of resilience built into the network, locations of POP nodes (or DP -EP nodes) and the location of the node itself.

[0149] In an example, three possible tunnels may have been established to connect a DP-XP node to DP -EP nodes. Two further VLAN options may have been rejected by the second orchestrator 31 due to the parameter settings that a VLAN should contain a maximum of seven hops and a VLAN should contain a maximum of one DP -EP node. Now, the second orchestrator 32 can decide to add / remove tunnel options for each node and subsequently change the priority of the tunnels on nodes (for example, it may change the default VLAN on a labelled node from a first tunnel to a newly added tunnel). In order to take the decision, the second orchestrator 32 may evaluate options in two ways. The second orchestrator 32 may assign different primary tunnels on one or more different nodes, typically nodes that are adjacent to the labelled node, if that seems a more effective method of reducing the tunnel switching on the labelled node. As a second alternative, the second orchestrator 32 may define one or more new tunnel options the labelled node or other nodes, and assign at least one of these new tunnels as a primary tunnel, in order to reduce the amount of tunnel switches on the labelled node.

[0150] In other cases, the second orchestrator 32 may periodically run a traffic simulator (same or similar as the first orchestrator 31) to determine the impact on overall network metrics and to change tunnels or tunnel priorities if network metrics improve.

[0151] Thus, the decisions (output) made by the second orchestrator 32 may modify the input of the first references of the first orchestrator 31.

[0152] Fig. 9 shows schematically an interaction structure between the first and second orchestrators 31, 32 and a third orchestrator (ORC3) 33 for implementing the above procedures O3 / D3 (shown in Fig. 3) of the network design system according to various embodiments.

[0153] The third orchestrator 3 may operate on slower time scales than the second orchestrator 32. Time scales may be typically in order of days / daily or weeks / weekly. The third orchestrator 33 takes at least three types of input, which are third reference data (REF3), third settings (S3) and a third state estimation (EST3) provided by a third state estimator (SE3) 330.

[0154] The third reference data may include a list of all nodes and their locations, as well as connectivity between these nodes, i.e., the network topology, and a list of lines of sight between all nodes, their distances and directions. Such a list of lines of sights may have been determined by “external” means, i.e., by software or other means not known to the third orchestrator 33. Secondly, the list of lines of sight may be updated by a state estimation routine of the third state estimator 330.

[0155] The third settings may include parameters that may be set by an authorized user and / or administrator, such as a frequency setting that determines the frequency at which the orchestrator operates, and “autonomy settings” that indicate to what extent the third orchestrator 33 may modify the network topology by adding / removing links. The third orchestrator 33 may be configured to generate options only and report these options to an authorized user. The authorized user may then decide which topology changes shall be implemented. The third state estimation obtained by the third state estimator 330 may include assessment of the link quality for all links in the topology, e.g., by obtaining buffer levels, link budget, modulation coding scheme (MCS) levels. Moreover, the third state estimator 330 may be configured to check if there are other links that may be available and that may be activated if desired. In order to check for new links, network nodes could scan for, and try to establish a communication link with, nearby nodes. Radio nodes may typically have embedded functionality to perform such operations. As such scanning may require the node to reboot or otherwise affect operation of existing links, such scanning needs to be carefully scheduled. For automatic operation, certain settings and / or thresholds may need to be set. For example, scanning for new links may only be done between 2:00-4:00 am and only if traffic on a node is below a certain threshold value. A more robust (and preferred) procedure may be that the third orchestrator 33 triggers temporary VLAN changes (e.g., via a command to the first orchestrator 31) to re-route traffic around a particular node. Subsequently, this node may scan for all possible links to neighbors and assesses the quality of those links. Once these scans are done, the third state estimation may be updated and VLANs may be changed back to their default values.

[0156] The third state estimator 330 may use a database with LOS information, e.g. to expedite the link scanning.

[0157] The third orchestrator 33 may be configured to use (historic) traffic information to determine benefits of adding links to the network topology. To achieve this, it may run / trigger the VLAN optimization procedure of the second orchestrator 32 to evaluate the resulting effect on the network metrics. For example, a node may have two links to other nodes, and each link may have a stable capacity of 1 Gbps and may have never been congested. The node has a possibility to add a third link, but the capacity of this link is only 400 Mbps. At first sight, this third link doesn’t add any benefits to the node under consideration. However, adding this third link may improve overall network metrics, for example by opening new VLAN options on another node. Because of this, traffic to the other node may be organized differently, which may reduce congestion in an area of the network.

[0158] Fig. 10 shows schematically an interaction structure between the third orchestrator 33 and a fourth orchestrator (ORC4) 34 for implementing the above fourth procedures O4 / D4 (shown in Fig. 3) of the network design system according to various embodiments. The fourth orchestrator 34 takes at least three types of input, which are fourth reference data (REF4), fourth settings (S4) and a fourth state estimation (EST4) provided by a fourth state estimator (SE4) 340.

[0159] The fourth reference data may include a list of potential node locations and environmental data.

[0160] The fourth settings may include frequency setting that determines the frequency at which the orchestrator operates, and reporting procedure(s) defining reporting options.

[0161] The fourth state estimation obtained by the fourth state estimator 340 may include LOS information for all potential node locations to all other potential node locations.

[0162] The fourth orchestrator 34 may be configured to check for modification options of the third refence input parameters of the third orchestrator 33. Since the third orchestrator 33 operates on link level, the fourth orchestrator 34 may be configured to operate on node level. Addition or relocation of nodes may result in better network performance, evaluated on previously mentioned network metrics. For example, adding a few nodes may increase network resilience and / or may improve traffic flows to POPs, and thus reduce the amount of congestion. Moreover, the overall number of nodes may be reduced without reducing overall network metrics. Such a modification may reduce network operations cost without affecting traffic or quality of service.

[0163] In order to evaluate options for node locations, the fourth orchestrator 34 may need input on all potential (available) node locations. A limited set (population) of such potential node locations may be all locations of streetlight poles, or other city furniture. The state estimation for such a population may entail an LOS check between all members of the population, possibly trunked by some threshold values. For example, the LOS check may be limited to population members that are less than 200m distant from one another. Such LOS check may be software based, using environmental data such as locations and shapes of trees, buildings and other objects.

[0164] Adding or (re)moving nodes in a network may require human intervention, by literally (de)installing hardware on site. Therefore, the fourth orchestrator 34 may not be able to take autonomous decisions. It may evaluate LOS states, run / trigger simulators of the first to third orchestrators 31 to 33 in sequence and determine if topologies can be optimized. If so, the fourth orchestrator 34 may report options and metrics, e.g. by automatically generated reports by a reporting function (REP) 342 as dictated by its fourth settings. Then, after physical changes have been made on site, the third reference input to the third orchestrator 33 may be modified.

[0165] All of the above first to fourth orchestrators 31 to 34 may be implemented based on software routines and / or tools that can run automatically, on a scheduled basis, or through some level of user interaction. They may be based on artificial intelligence, deterministic algorithms or any other software-based routines.

[0166] To summarize, a wireless mesh network for high-capacity backhauling of data traffic such as for telecommunications networks has been described. Reliability is increased through resilience, i.e., data traffic can be transported via multiple paths between customer equipment (e.g., telecommunication cells) and fiber points (or other connections to an external network). Depending on a network status, a node or network design system can choose to direct traffic via a different path, thus reducing the chance of packet drops. Once resilience is established, network traffic can be optimized to respond to variable nature of traffic density. By doing this, network reliability can be further increased and throughput optimized while minimizing costs.

[0167] While the invention has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive. The invention is not limited to the disclosed mesh-type communication network with sectorized nodes. It may be applied to all kinds of wireless networks that make use of wireless LOS connections (e.g., 60 GHz connections) between network nodes. Moreover, the invention is not limited to routing control via tunnel priorities. The orchestration procedures of Fig. 3 and the orchestrators of Figs. 6 to 10 may be implemented in connection with alternative routing control options such as increasing link bandwidth through channel bonding, channel aggregation, orthogonality etc.). The orchestrator decision making process of Fig. 7 may be applied in all orchestrator types of Figs. 6 and 8 to 10. Other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure and the appended claims. In the claims, the word “comprising” does not exclude other elements or steps, and the indefinite article “a” or “an” does not exclude a plurality. A single processor or other unit may fulfil the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. The foregoing description details certain embodiments of the invention. It will be appreciated, however, that no matter how detailed the foregoing appears in the text, the invention may be practiced in many ways, and is therefore not limited to the embodiments disclosed. It should be noted that the use of particular terminology when describing certain features or aspects of the invention should not be taken to imply that the terminology is being re-defined herein to be restricted to include any specific characteristics of the features or aspects of the invention with which that terminology is associated.

[0168] A single unit or device may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0169] The described operations like those indicated in Figs. 3 and 6 to 10 can be implemented as program code means of a computer program and / or as dedicated hardware of the transmitter devices, receiver devices or transceiver devices, respectively. The computer program may be stored and / or distributed on a suitable medium, such as an optical storage medium or a solid-state medium, supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems.

Claims

CLAIMS:

1. An apparatus for controlling network design of a wireless network (100) comprising at least one first type of network node (120), connected to a point of presence (20), and at least two second type of network nodes (140), connected to respective customer equipment (30), wherein the apparatus comprises a traffic orchestrator (31 to 34) configured to monitor real-time network performance and to evaluate and / or estimate a potential impact of routing decisions made by network nodes (110, 120, 140) of the wireless network (100) on network performance, wherein the traffic orchestrator is configured to evaluate and / or estimate the potential impact on network performance in a three-step process comprising a first step (S 101) in which a proposed set of virtual tunnel flips to respective tunnels with different priority is generated based on traffic metrics on link level, a second step (SI 02) in which an effect of the tunnel flips is simulated based on topology and running average or historic network traffic, and a third step (SI 03) in which overall network performance is evaluated based on overall network metrics of a pre-flip and post-flip situation.

2. The apparatus of claim 1, wherein the traffic orchestrator comprises at least one of a first orchestrator (31) configured to simulate scenarios by varying priorities of tunnels between network nodes (110, 120, 140) of the wireless network (100) and calculating effects on network metrics, a second orchestrator (32) configured to simulate scenarios by adding and / or changing tunnels between network nodes (110, 120, 140) of the wireless network (100), a third orchestrator (33) configured to simulate scenarios by adding and / or removing links between network nodes (110, 120, 140) of the wireless network (100), and a fourth orchestrator (34) configured to simulate scenarios by adding and / or moving and / or removing network nodes (110, 120, 140) of the wireless network (100).

3. The apparatus of claim 1, wherein the traffic orchestrator is configured to load different tunnel priorities into the network, effectively resulting in a rerouting of traffic through the network, in particular through tunnels, only if a resultant overall network performance metric improves.

4. The apparatus of any one of the preceding claims, wherein the traffic orchestrator (31 to 34) is configured to use a simulation of a behavior of a computergenerated model of the wireless network (100) based on actual or historical data obtained from the wireless network (100), from other networks, or from estimated values, to create a simulated set of overall network metrics.

5. The apparatus of any of the preceding claims, wherein at least some of the network nodes (110, 120, 140) of the wireless network (100) comprise a plurality of physically separated sectors configured to establish a plurality of links to other network nodes (110, 120, 140).

6. The apparatus of claim 5, wherein the wireless network (100) forms, in terms of a physical or link level of the network nodes (110, 120, 140), a ring structure that starts at one of the sectors of the first network node (120) and ends at another sector of the first network node (120), or a linear structure where network traffic is flowing between two first network nodes (120) and two second network nodes (140), or any combinations thereof.

7. The apparatus of any of the preceding claims, wherein timing levels of different optimization and deployment procedures used by the traffic orchestrator (31 to 34) range from once per 1-10 minutes through once per 1-10 days and once per 1-10 weeks to once per 1-10 months.

8. A network design system (10) comprising an apparatus according to claim 1.

9. The system of claim 8, wherein the system is configured to control the wireless network (100) based on feedback information received from the wireless network (100) and indicating the performance of the network, to design the wireless network (100) by at least one of transmitting and / or loading control information to the network nodes (100, 120, 140), setting a tunnel layout, and setting a class of service.

10. The system of claim 9, wherein the feedback information comprises network metrics and / or other performance parameters.

11. The system of any one of claims 8 to 10, wherein the network design system (10) is configured to provide a priority scheme by defining at least primary and secondary tunnels to allocate an order of priority for use of the tunnels, wherein a primary tunnel is used by the network nodes (110, 120, 140) as default tunnel for data transport unless an exceptional condition occurs.

12. A wireless network comprising a network design system according to claim 8 and at least one first network node (120), connected to a point of presence (20), and at least two second network nodes (140), connected to respective customer equipment (30).

13. A method of controlling network design of a wireless network (100) comprising at least one first type of network node (120), connected to respective points of presence (20), and at least two second type of network nodes (140), connected to respective customer equipment (30), wherein the method is configured to perform traffic orchestrating (31 to 34) by monitoring real-time network performance and evaluating and / or estimating a potential impact of routing decisions made by network nodes (110, 120, 140) of the wireless network (100) on network performance; wherein the method further comprising evaluating and / or estimating the potential impact on network performance in a three-step process comprising a first step (S 101) in which a proposed set of virtual tunnel flips to respective tunnels with different priority is generated based on traffic metrics on link level, a second step (SI 02) in which an effect of the tunnel flips is simulated based on topology and running average or historic network traffic, and a third step (SI 03) in which overall network performance is evaluated based on overall network metrics of a pre-flip and post-flip situation.

14. A computer program product comprising program code for producing the steps of claim 13 when run on a computer device.

Images