MPLS-EVPN cloud networking overlays with optimized load-balancing and frame size

Abstract

A method to establish an optimized frame format used by a first compute node (21) acting as ingress PE and a second compute node (22) acting as egress PE in the transport of overlay ethernet traffic containing an IP payload from the source overlay endpoint attached to or hosted in the first compute node (21) to the destination overlay endpoint attached to or hosted in the second compute node (22) through a Multi Protocol Label Switching, MPLS, underlay network (2), where the frame format is a conventional MPLS tagged ethernet frame format, wherein an MPLS label stack is inserted between the ethernet header and the IP header of the overlay ethernet frame, where the single ethernet header carries the destination and source overlay Media Access Control, MAC, addresses when the frame was originated from the source overlay endpoint, where the MPLS header of the frame comprises a top label with the Label Switch Path, LSP, label towards the destination Provider Edge, PE, router, ie. egress PE, and a bottom label, e.g. an optimization MPLS label, holding information enabling the forwarding mechanism in the egress PE router to restore the original destination and source overlay MAC addresses.

Description

[0001] MPLS-EVPN CLOUD NETWORKING OVERLAYS WITH OPTIMIZED LOAD¬

[0002] BALANCING AND FRAME SIZE

[0003] TECHNICAL FIELD

[0004] Embodiments herein relate to a first compute node, hosting or running overlay endpoints virtual machines and / or containers the first compute node being adapted to function as ingress PE, and a second compute node, hosting or running overlay endpoints virtual machines and / or containers, the second compute node being adapted to function as egress PE, and methods therein. Some aspects relate to a method to establish an optimized frame format used by a first and second compute node with ingress and egress PE functions in the transport of overlay ethernet traffic containing an IP payload from the source overlay endpoint or host to the destination overlay endpoint or host through a Multi Protocol Label Switching, MPLS, underlay network, where the frame format is a conventional MPLS tagged ethernet frame format.

[0005] Embodiments herein also relate to a first computer program comprising instructions, which when executed by a processor belonging to a first compute node will cause the first compute node to perform the actions of an ingress PE.

[0006] Embodiments herein also relate to a second computer program comprising instructions, which when executed by a processor belonging to a second compute node will cause the second compute node to perform the actions of an egress PE.

[0007] BACKGROUND

[0008] In a typical wireless communication network, wireless devices, also known as wireless communication devices, mobile stations, stations (STA) and / or User Equipment (UE), communicate via a Wide Area Network or a Local Area Network such as a Wi-Fi network or a cellular network comprising a Radio Access Network (RAN) part and a Core Network (CN) part. The RAN covers a geographical area which is divided into service areas or cell areas, which may also be referred to as a beam or a beam group, with each service area or cell area being served by a radio network node such as a radio access node e.g., a Wi-Fi access point, a Base Station (BS) or a radio base station (RBS), which in some networks may also be denoted, for example, a Base Station (BS), a NodeB, eNodeB (eNB), or gNodeB (gNB) as denoted in Fifth Generation (5G) telecommunications. A service area or cell area is a geographical area where radio coverage is provided by the radio network node. The radio network node communicates over an air interface operating on a radio frequency with the wireless devices within the range of the radio network node.

[0009] 3rd Generation Partnership Project (3GPP) is the standardization body for specifying the standards for the cellular system evolution, e.g., including 3G, 4G, 5G and the future evolutions. Specifications for Evolved Universal Terrestrial Radio Access (E- UTRA) and Evolved Packet System (EPS) have been completed within the 3GPP. In 4G also called a Fourth Generation (4G) network, EPS is core network and E-UTRA is radio access network. In 5G, 5G Core (5GC) is core network, NR is radio access network. As a continued network evolution, the new release of 3GPP specifies a 5G network also referred to as 5G New Radio (NR) and 5GC.

[0010] Frequency bands for 5G NR are being separated into two different frequency ranges, Frequency Range 1 (FR1) and Frequency Range 2 (FR2). FR1 comprises sub-6 GHz frequency bands. Some of these bands are bands traditionally used by legacy standards but have been extended to cover potential new spectrum offerings from 410 MHz to 7125 MHz. FR2 comprises frequency bands from 24.25 GHz to 52.6 GHz. Bands in this millimeter wave range have shorter range but higher available bandwidth than bands in the FR1.

[0011] Multi-antenna techniques may significantly increase the data rates and reliability of a wireless communication system. For a wireless connection between a single user, such as UE, and a base station (BS), the performance is in particular improved if both the transmitter and the receiver are equipped with multiple antennas, which results in a Multiple-Input Multiple-Output (MIMO) communication channel. This may be referred to as Single-User (SU)-MIMO. In the scenario where MIMO techniques is used for the wireless connection between multiple users and the base station, MIMO enables the users to communicate with the base station simultaneously using the same time-frequency resources by spatially separating the users, which increases further the cell capacity. This may be referred to as Multi-User (MU)-MIMO. Note that MU-MIMO may benefit when each UE only has one antenna. The cell capacity can be increased linearly with respect to the number of antennas at the BS side. Due to that, more and more antennas are employed in BS. Such systems and / or related techniques are commonly referred to as massive MIMO.

[0012] Cloud networking uses overlay networking technologies using VXLAN or Multi Protocol Label Switching (MPLS) to transport overlay traffic through an IP underlay network. The standard RFC7432 Border Gateway Protocol (BGP) MPLS-Based Ethernet Virtual Private Network (EssVPN) describes overlay networking using the MPLS encapsulation of overlay ethernet traffic to transport it through the MPLS capable underlay network. However, the RFC7432 BGP MPLS-Based Ethernet VPN EVPN solution has some well-problems related to traffic load-balancing and frame reordering in the MPLS underlay network due to which it is not widely used for cloud overlay networking in data centers. These problems are generally due to the inability of the underlay MPLS routers to detect and look beyond the inner ethernet header of MPLS EVPN frame for load balancing purposes. The improper load balancing of the MPLS EVPN frame in the underlay MPLS routers and the resulting frame reordering problems are explained in detail in the MPLS EVPN RFC7432 section 18 and RFC4448 section 4.6.

[0013] As per that description, the Equal Cost Multi Path (ECMP) implementations on underlay MPLS routers wrongly interpret and examine the first nibble after the MPLS header as IP header version number whereas in fact those bits are part of the inner ethernet header MAC addresses. If the nibble value is 0x4 or 0x6, it could be mistaken for an IPv4 or IPv6 header and the load balancing happens using subsequent bits in the inner ethernet header being wrongly interpreted as an IP header. This results in not only the improper load balancing but frame reordering issues in the underlay network for the MPLS EVPN traffic.

[0014] The existing solutions to this load balancing problems are complex and mandate the use of additional control-plane protocol signaling and frame enhancements such as employment of a control word or entropy MPLS labels etc. One standard solution to the frame reordering problems of MPLS L2 VPN is the employment of the Ethernet control word in the frame as discussed in RFC4448 section 4.6, RFC7432 section 18 and RFC8469. However, the employment of the Ethernet control word requires its manual enabling configuration or additional protocol signaling outside EVPN RFC7432 for its usage negotiation and also requires such frame awareness and processing capability at the ingress and egress Provider Edge (PE) routers. The underlay MPLS routers need to have the minimum control word awareness to check its presence and if so further the ability to load balance based on the bottom most MPLS label as described in RFC4448.

[0015] Another ECMP load balancing technique for MPLS EVPN frame is based adding additional entropy labels as per RFC6790. The Use of Entropy Labels in MPLS Forwarding does not only require additional protocol signaling outside RFC7432 for negotiation of its use but also increases the frame size by 8 bytes (entropy indication and entropy labels). Further the ingress and egress PE routers need to have the ability to compute and insert / delete these entropy labels at Label Switched Path (LSP) ingress / egress and the underlay MPLS routers should have the ability to load balance the EVPN frame based on these MPLS entropy labels.

[0016] These mechanisms make the control-plane and data-plane extremely complex for employing MPLS based cloud overlay networking implementations. It is highly desirable to have a ECMP load balancing solution for MPLS EVPN traffic that avoids these additional mechanisms and their associated complexities.

[0017] SUMMARY

[0018] As part of developing embodiments herein, some problems have been identified that first will be described.

[0019] RFC7432 based MPLS EVPN solution for layer 2 overlay cloud networking suffers from the ECMP load balancing and frame reordering problems in the MPLS underlay network due to the general inability of underlay MPLS routers to looks beyond the inner ethernet header and load balance based on the IP and transport header that is present after the inner ethernet header of the MPLS encapsulated ethernet frame that is the EVPN frame. These problems are described in RFC7432 section 18, RFC4448 section 4.6 and RFC8469 and the existing solutions to these problems are complex involving employment of additional control-plane protocol signaling, configuration and frame enhancements to carry additional load balancing information and further requires the hardware support at the ingress and egress PE routers as well as in the core routers of the underlay to process such traffic. Details of these problems are described below:

[0020] RFC7432 BGP MPLS EVPN frame uses the MPLS encapsulation frame format of RFC4448 in which the overlay ethernet frame is embedded in an MPLS frame. I.e., the entire frame consists of first the outer or underlay ethernet frame header and next the MPLS header and then the inner ethernet frame header containing the overlay source and destination MAC addresses and finally the IP or any other payload. The regular IP and transport headers based ECMP load balancing of such frames in the MPLS underlay network when the frame contains unicast IP payload does not work due to the known general problem of the underlay MPLS routers’ inability to detect and look beyond the inner ethernet header of the MPLS I2VPN frame for ECMP load balancing purposes.

[0021] The improper ECMP load balancing of the MPLS EVPN traffic in the underlay MPLS routers and the resulting frame reordering problems are explained in detail in the MPLS EVPN RFC7432 section 18 and RFC4448 section 4.6. As per that description, the ECMP implementations on underlay MPLS routers wrongly interpret and examine the first nibble after the MPLS header as IP header version number whereas in fact those bits are part of the inner ethernet header MAC addresses. If the nibble value is 0x4 or 0x6, it could be mistaken for an IPv4 or IPv6 header and the load balancing happens using subsequent bits in the inner ethernet header being wrongly interpreted as an IP header. This results in improper ECMP load balancing and frame reordering issues in the underlay network for the MPLS EVPN traffic.

[0022] The standard solution to the frame reordering problems of MPLS EVPN is the employment of the Ethernet control word in the frame as discussed in RFC4448 section 4.6, RFC7432 section 18 and RFC8469. The employment of the Ethernet control word requires its manual enabling configuration or additional protocol signaling outside the RFC7432 scope for its usage negotiation and also requires such frame awareness and hardware processing capability at the ingress and egress PE routers. The underlay MPLS routers need to have the minimum control word awareness to check its presence and if so further the ability to load balance based on the bottom most MPLS label as described in RFC4448 if ECMP is needed. The Ethernet control word also gives rise to additional 4 bytes in frame size. For all these reasons the Ethernet control word-based solution is very complex and requires hardware support by routers.

[0023] Another ECMP load balancing technique for MPLS EVPN frame is based adding additional entropy labels as per RFC6790 which not only requires additional protocol signaling outside the RFC7432 scope for negotiation of its use but also increases the frame size by 8 bytes (entropy indication and entropy labels). Further the ingress and egress PE routers need to have the ability to compute and insert / delete these entropy labels at LSP ingress / egress and the underlay MPLS routers should have the ability to load balance the EVPN frame based on these MPLS entropy labels.

[0024] The MTU in the underlay should be configured to be more by 24 bytes (that of layer 2 overlay header + control word size) for the MPLS EVPN frame when compared to a normal MPLS tagged ethernet frame containing an IP packet.

[0025] Embodiments herein proposes a solution to the ECMP load balancing and frame reordering problems of RFC 7432 BGP MPLS-Based Ethernet VPN.

[0026] According to an aspect of embodiments herein, the object is achieved by a method to establish an optimized frame format used by a first compute node acting as ingress PE and a second compute node acting as egress PE in the transport of overlay ethernet traffic containing an IP payload from the source overlay endpoint attached to or hosted in the first compute node to the destination overlay endpoint attached to or hosted in the second compute node through a Multi Protocol Label Switching, MPLS, underlay network. The frame format is a conventional MPLS tagged ethernet frame format,

[0027] It is proposed that an MPLS label stack is inserted between the ethernet header and the IP header of the overlay ethernet frame. The single ethernet header carries the destination and source overlay Media Access Control, MAC, addresses when the frame was originated from the source overlay endpoint. It is proposed that the MPLS header of the frame comprises:

[0028] - a top label with the Label Switch Path, LSP, label towards the destination Provider Edge, PE, router, i.e. egress PE, and

[0029] - a bottom label, e.g. an optimization MPLS label, holding information enabling the forwarding mechanism in the egress PE router to restore the original destination and source overlay MAC addresses.

[0030] According to an aspect of embodiments herein, the object is achieved by a method performed by an ingress PE router to establish a route between the ingress PE router and an egress PE router, where the frame format proposed above is used in the transport of overlay ethernet traffic. The method comprises:

[0031] - receiving, from an egress PE router, in a type 2 MAC / IP route advertisement message through BGP protocol, an Ethernet Virtual Private Network, EVPN, Network Layer Reachability Information, NLRI, and an optimization MPLS label where an overlay destination endpoint is attached; and

[0032] - creating forwarding rules to send overlay layer 2 traffic with unicast IP payload destined to the overlay destination endpoint based on the received EVPN NLRI and an optimization MPLS label.

[0033] According to another aspect of embodiments herein, the object is achieved by a method performed by an egress PE router to establish a route between the egress PE router and an ingress PE router, where the frame format proposed above is used in the transport of overlay ethernet traffic. The method comprises:

[0034] - advertising an optimization MPLS label in a type 2 MAC / IP route advertisement message through BGP protocol for an attached overlay endpoint; and

[0035] - creating the forwarding rules to receive and process the overlay traffic sent to the mentioned overlay endpoint by other remote overlay endpoints in the optimized frame format.

[0036] According to another aspect of embodiments herein, the object is achieved by a method performed by an egress PE router for the transport of overlay ethernet traffic through a route as established between an ingress PE router and the egress PE router as proposed above. A frame is routed through the MPLS undelay network towards the egress PE router based the top label. The method comprises:

[0037] - restoring, by the forwarding mechanism in the egress PE router, the original overlay ethernet header MAC addresses in the ethernet header of the frame using the bottom label, being the optimization label, and

[0038] - delivering the frame to the corresponding attached destination overlay endpoint.

[0039] According to another aspect of embodiments herein, the object is achieved by a first compute node, hosting or running overlay endpoints virtual machines and / or containers and adapted to use the frame format proposed above in a cloud overlay networking system based on the RFC7432 BGP MPLS EVPN, where the first compute node is adapted to function as ingress PE. The first compute node comprises:

[0040] - a virtual switch and router component adapted to function as an ingress PE router forwarding plane function as per RFC 7432;

[0041] - a Border Gateway Protocol, BGP, function adapted to function as an ingress PE control plane function as per RFC 7432; and

[0042] - a virtual switch and router function adapted to function as an egress PE router.

[0043] According to another aspect of embodiments herein, the object is achieved by a second compute node, hosting or running overlay endpoints virtual machines and / or containers and adapted to use the frame format proposed above in a cloud overlay networking system based on the RFC7432 BGP MPLS EVPN, where the second compute node is adapted to function as egress PE. The second compute node comprises:

[0044] - a virtual switch and router component adapted to function as an egress PE router forwarding plane function as per RFC 7432;

[0045] - a BGP function adapted to function as an egress PE control plane function as per RFC 7432; and

[0046] - a virtual switch and router function adapted to function as an ingress PE router.

[0047] According to another aspect of embodiments herein, the object is achieved by a first computer program comprising instructions, which when executed by a processor belonging to a first compute node, causes the processor to perform actions of a first compute node as proposed above.

[0048] According to another aspect of embodiments herein, the object is achieved by a second computer program comprising instructions, which when executed by a processor belonging to a second compute node, causes the processor to perform actions of a second compute node as proposed above. According to another aspect of embodiments herein, the object is achieved by a carrier comprising the first and / or second computer program as proposed above, wherein the carrier is one of an electronic signal, an optical signal, an electromagnetic signal, a magnetic signal, an electric signal, a radio signal, a microwave signal, or a computer- readable storage medium.

[0049] Thanks to these embodiments a simple solution to the load balancing and frame reordering problems of MPLS EVPN is provided.

[0050] Embodiments herein may provide one or more of the following advantages over the conventional RFC7432 EVPN.

[0051] The ECMP load balancing in the MPLS underlay network of the overlay traffic based on overlay IP and transport headers is realized, which was not possible using the MPLS L2VPN encapsulation of ethernet frames format as per RFC7432, RFC4448.

[0052] The frame reordering problems of the MPLS L2VPN encapsulated frames in the underlay MPLS routers described in RFC7432 section 18 and RFC4448 section 4.6 are solved as they were caused by the presence of the inner ethernet header in the EVPN frame and the optimized frame format of the proposed solution does not employ the inner ethernet header.

[0053] The standard Ethernet control word based complex solution to be employed as described in RFC4448 section 4.6, RFC7432 section 18 and RFC8469 for the MPLS L2VPN encapsulated frame reordering problem is not required and avoided.

[0054] The use of complex RFC6790 entropy MPLS labels based ECMP load balancing solution of MPLS traffic is not required and avoided.

[0055] Saving in the frame size by 24 bytes, meaning the inner ethernet header + control word size, when compared to the conventional MPLS L2VPN encapsulation frame size for the same payload size. Further, saving of additional 8 bytes which would have been required had the entropy MPLS labels based ECMP load balancing solution been employed.

[0056] Avoiding the MTU extra 24 bytes configuration needed with regards to overlay MTU in the underlay network otherwise when the conventional MPLS L2VPN encapsulation frame format was used. Otherwise, the MTU in the underlay should have been configured to be more by 24 bytes, that of layer 2 overlay header + control word size, when compared to a normal MPLS tagged ethernet frame containing an IP packet. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Examples of embodiments herein are described in more detail with reference to attached drawings in which:

[0058] Figure 1 is a schematic block diagram illustrating embodiments of a communications network.

[0059] Figure 2 is a schematic block diagram illustrating a cloud overlay networking system, Figures 3a to 3d are illustrations of an Overlay Ethernet frame and an optimized frame according to embodiments herein.

[0060] Figure 4 is a simplified flowchart illustrating the process of establishing and using a frame format.

[0061] Figure 5 is a generalized block diagram of embodiments of a first compute node.

[0062] Figure 6 is a generalized block diagram of embodiments of a second compute node.

[0063] Figure 7 is a generalized block diagram of embodiments of a communication system.

[0064] Figure 8 is a generalized block diagram of embodiments of a user equipment.

[0065] Figure 9 is a generalized block diagram of embodiments of a network node.

[0066] Figure 10 is a generalized block diagram of embodiments of a virtualization environment.

[0067] DETAILED DESCRIPTION

[0068] Embodiments proposed herein suggests a solution to the ECMP load balancing and frame reordering problems of RFC 7432 BGP MPLS-Based Ethernet VPN using a method and system to transport the overlay ethernet traffic containing the IP payload from a source overlay endpoint or host to a destination overlay endpoint or host through the MPLS underlay network in the RFC7432 implementation context. This new method and system avoid using the MPLS encapsulation based EVPN frame format and associated inner ethernet header for the overlay ethernet traffic when it contains the IP payload. Instead, the standard MPLS tagged ethernet frame format is used and with new MPLS label tagging called the optimization MPLS label which is different from the convention VPN label, and which had been signaled previously between the ingress and egress PE routers to which the source and destination overlay endpoints or hosts respectively are attached. Unlike the MPLS EVPN frame which contains two ethernet headers namely the outer and inner, the MPLS tagged overlay frame contain only one ethernet header and in which the MPLS header is inserted between the ethernet header and the IP header of the overlay ethernet frame. This proposal calls this frame format the optimized frame format for the EVPN. The MPLS header of this optimized frame is proposed to contain two MPLS labels, the top being the MPLS LSP label towards the destination or the egress PE router and the bottom label will be the optimization MPLS label which is unique to the destination MAC address and the broadcast domain of the destination overlay endpoint or host. This optimized frame is routed through the MPLS undelay network towards the egress PE router based the top LSP label. At the egress PE router the optimization MPLS label is used to restore the original overlay ethernet header MAC addresses in the ethernet header of the frame and then deliver the frame to the corresponding attached destination overlay endpoint. The mentioned restoration of overlay MAC addresses in the ethernet header of the frame are needed because they have been changed at every routing hop in the underlay network due to next hop MAC rewriting during the underlay router forwarding process. The purpose of the optimization MPLS label is to distinguish and process the EVPN traffic in the proposed format and also to facilitate the restoration of overlay ethernet header destination MAC address that had been overwritten in the underlay due to the next hop rewriting process. The optimization MPLS label is allocated at the egress PE router unique to and for each attached overlay endpoint or host. The optimization MPLS label value allocated for an attached overlay endpoint must be different from the conventional VPN label allocated for the same endpoint by the conventional RFC7432 EVPN implementation. The optimization MPLS label of an overlay endpoint is signaled through BGP by the egress PE router where it is attached to other remote PE routers.

[0069] As per this proposal only the overlay ethernet traffic with IPv4 or IPv6 payload is forwarded and processed using the proposed optimized frame format by the ingress and egress PE routers. Rest of the overlay ethernet traffic, i.e., those containing non IP payload such as ARP, DHCP payload etc. are forwarded processed as per the conventional RFC7432 EVPN frame format and implementation.

[0070] Since the proposed frame format does not contain an inner ethernet header after the MPLS header and instead contains the overlay IP header, all the reported ECMP load balancing and frame reordering problems in the underlay network are not encountered and hence solved and further the ECMP load balancing of the overlay traffic now happens in the underlay network based on the overlay IP and transport headers as desired and without the necessity to use the additional and complex solutions based on employment of the ‘Ethernet control word’ described in RFC4448 section 4.6, RFC7432 section 18 nor the MPLS entropy labels solution as described in RFC6790.

[0071] Figure 1 is a schematic overview depicting a wireless communications network 100 wherein embodiments herein may be implemented. The wireless communications network 100 comprises one or more RANs, one or more CNs and a conversation AR network 105. The communications network 100 may use 5G NR but may further use a number of other different technologies, such as, 6G, Wi-Fi, Long Term Evolution (LTE), LTE-Advanced, Wideband Code Division Multiple Access (WCDMA), Global System for Mobile communications / enhanced Data rate for GSM Evolution (GSM / EDGE), Worldwide Interoperability for Microwave Access (WiMax), or Ultra Mobile Broadband (UMB), just to mention a few possible implementations.

[0072] Base stations, such as a first base station 111 and a second base station 112, operate in the RAN the communications network 100. The base stations 111 , 112, may each be a transmission and reception point e.g. a radio access network node such as a base station, e.g. a radio base station such as a NodeB, an evolved Node B (eNB, eNode B), an NR Node B (gNB), a base transceiver station, a radio remote unit, an Access Point Base Station, a base station router, a transmission arrangement of a radio base station, a stand-alone access point, a Wireless Local Area Network (WLAN) access point or an Access Point Station (AP STA), an access controller, or any other network unit capable of communicating with UEs, such as a UE 121, within a cell, served by the respective base station 111, 112. The respective base station 111, 112 may be referred to as a serving radio network node and may communicate with the UE 121 with Downlink (DL) transmissions to the UE 121 and Uplink (UL) transmissions from the UE 121.

[0073] Embodiments proposed herein solves the ECMP load balancing and frame reordering problems of RFC7432 based EVPN for cloud overlay networking and also avoids other complex solutions to these problems such as the employment of ethernet control word and the MPLS entropy labels etc. These problems are solved by avoiding the use of the standard MPLS encapsulation based EVPN frame format, as illustrated in Figure 3b, for transporting the overlay IP traffic through the MPLS underlay network as the EVPN frame format caused these problems due to having two, i.e.. , the outer and inner ethernet headers and the inability of the underlay MPLS routers to detect the inner ethernet header and then ECMP load balance using the subsequent overlay IP header.

[0074] The core essence of proposed embodiments is to employ an optimized frame format illustrated in Figures 3c and 3d, that is the standard MPLS tagged ethernet frame and that has only a single ethernet header and the MPLS tagging is done using a proposed optimization MPLS label. Since the optimized frame contains a single ethernet header followed by the IP header containing the source and destination overlay IP addresses the ECMP load balancing happens in the conventional manner based the IP and transport header fields. This solution proposes the following ways to signal and use the mentioned optimization MPLS label per the EVPN Network Layer Reachability Information (NLRI) of an overlay endpoint, i.e., the overlay MAC address, IP address and broadcast domain tuple and then based on this transport the overlay ethernet traffic with IP payload in the MPLS underlay network in the proposed optimized frame format and then deliver to the destination overlay end point workload.

[0075] The RFC7432 EVPN frame format is the MPLS encapsulation of ethernet frames and is described in RFC4448 and illustrated in Figure 3b. This frame format is used by RFC7432 based EVPN to encapsulate and transport the overlay layer 2 ethernet frames through the underlay network between the ingress and egress PE routers where respectively the source and destination overlay endpoints or hosts are attached. The RFC7432 EVPN frame format contains two ethernet headers namely the outer ethernet header carrying the underlay next hop destination and source MAC addresses and the inner ethernet header carrying the destination and source overlay MAC addresses. The EVPN frame when it contains the unicast IP payload causes the ECMP load balancing and frame reordering problems in the underlay network as had been discussed earlier and these problems needed complex ethernet control word and MPLS entropy labels based standard solutions which also were mentioned earlier. This proposal solves these problems using a different and simple approach without the need for the mentioned complex standard solutions.

[0076] Instead of using the RFC7432 EVPN frame format, this proposal suggests to use a frame format called the optimized frame format to transport the overlay layer 2 ethernet frames with unicast IP payload between the ingress PE router 41 and the egress PE router 42 through the MPLS underlay network. This suggested frame format is illustrated in Figure 3c, showing the conventional MPLS tagged ethernet frame format in which the MPLS label stack is inserted between the ethernet header and the IP header of the overlay ethernet frame. Thus, the proposed optimized frame format contains only one ethernet header and that carries the destination and source overlay MAC addresses when the frame was originated from the source overlay endpoint. The MPLS header of the optimized frame is proposed to contain the two MPLS labels of which the top label is the LSP label towards the destination PE router (egress PE) and the bottom label is a new proposed optimization MPLS label which is different from the conventional VPN label of RFC7432 but was previously signaled along with the latter. Such a frame undergoes rewriting of the MAC address in the ethernet header during the MPLS routing in the underlay network and hence before the frame is delivered to the destination overlay endpoint the original destination and source overlay MAC addressed are restored by the forwarding mechanism in the egress PE router as described further in this proposal. The employment of the proposed optimization MPLS label helps to restore the mentioned overlay MAC addresses restoration and also to deliver the frame to the corresponding destination overlay endpoint at the egress PE router as discussed further. Since the optimized frame format contains only one ethernet header and after that it contains the overlay IP header the mentioned ECMP load balancing and frame reordering problems associated with the conventional EVPN frame are not encountered and further the frame gets ECMP load balanced based on the overlay IP and transport headers in the frame as desired.

[0077] This proposal suggests that the BGP control-plane signaling and forwarding-plane usage of the mentioned optimization MPLS label to facilitate the identification and forwarding of the overlay layer 2 traffic with unicast IP payload using the proposed optimized frame format. For each attached overlay endpoint EVPN NLRI, i.e., the overlay MAC address, overlay IP address and the broadcast domain, the BGP control plane in an egress PE signals an additional MPLS label called optimization MPLS label in the otherwise unused MPLS Iabel2 field of the RFC7432 specified type 2 MAC / IP route advertisement. This new optimization MPLS label field value should be different from the conventional VPN label that also is included in the MPLS labell field value in the mentioned type 2 MAC IP route update. The optimization MPLS label thus advertised signifies the MPLS label to be used to send the overlay layer 2 ethernet traffic with IP payload destined to the mentioned overlay endpoint using the proposed optimized frame format by any other remote ingress PEs from their attached overlay endpoints.

[0078] As illustrated in Figure 4, this proposal suggests that in the forwarding plane, each ingress PE 41 creates 403 the forwarding rules to send 405 the overlay layer 2 traffic with unicast IP payload destined to an overlay destination endpoint based on the type 2 MAC / IP route advertisement information namely the EVPN NLRI and the optimization MPLS label they received 401 from an egress PE where the mentioned overlay destination is attached as was mentioned above.

[0079] These rules employ the optimized frame format in which the overlay layer 2 frame containing unicast IP payload is MPLS tagged by inserting an MPLS label stack between the ethernet header and IP header, as illustrated in Figure 3cr. The MPLS label stack inserted thus should contain the underlay LSP label towards the egress PE that advertised the mentioned type 2 MAC / IP route advertisement as the top label and the received optimization MPLS label as the bottom label. The mentioned forwarding rules take precedence over the conventional RFC7432 VPN label based forwarding rules. Due to this higher precedence of the optimized frame rules, the overlay layer frames with unicast IP traffic will be forwarded using the optimized frame format and the rest of the overlay traffic types, i.e. broadcast, DHCP, ARP etc., will be forwarded using the conventional RFC7432 MPLS encapsulation of layer 2 frame format by the ingress PE.

[0080] This proposal suggests that in the forwarding plane, each egress PE 42 advertises 402 an optimization MPLS label in a type 2 MAC / IP route for an attached overlay endpoint as mentioned above and creates 404 the forwarding rules to receive and process the overlay traffic sent to the mentioned overlay endpoint by other remote overlay endpoints in the optimized frame format and then to deliver 406 the traffic to the mentioned overlay destination endpoint. These forwarding rules pop the labels in the MPLS label stack of the received frame and match and verify that the top label is the LSP label towards this PE and the bottom label is the mentioned optimization MPLS label this PE advertised for the mentioned overlay endpoint. If they match then these rules reconstruct the overlay layer 2 frame from the received frame in the optimized format by first stripping the MPLS header and then restoring 405 the original destination and source MAC address, as they were rewritten and lost due to the next hop rewriting in the undelay network routing, to the actual values as was present when the frame was originally generated and sent by the remote source overlay source endpoint. This restoring 405 of the destination MAC address in the frame to the overlay MAC address corresponding the optimization MPLS label and the source MAC address to the overlay MAC corresponding to the source IP address in the frame is based on the corresponding EVPN NLRI MAC - IP - optimization label mappings which were sent or received via the type 2 MAC / IP advertisements 402 earlier. After this the forwarding rules deliver 406 the frame that is now a proper overlay layer 2 frame to the mentioned attached overlay endpoint on the mentioned egress PE 42.

[0081] As per this proposal, the end-to-end forwarding of the overlay layer 2 frames with unicast IP payload happens as follows. A source overlay endpoint sends an overlay ethernet frame containing unicast IP payload to its attached ingress PE router. This frame is a normal ethernet frame and its ethernet header contains the MAC address of the source overlay source endpoint and the MAC address of the destination overlay endpoint respectively as source and destination MAC addresses. Further, its IP header contains the source IP address of the overlay source endpoint and the destination IP address that of the overlay destination endpoint. The forwarding rules mentioned above in the mentioned ingress PE router inserts the MPLS label stack to this overlay frame between the ethernet and IP headers. This MPLS stack consists of the top label being the MPLS LSP label towards the egress PE router attached to the destination overlay endpoint and bottom label being the optimization MPLS label corresponding to the mentioned destination overlay endpoint. Next the mentioned forwarding rules routes this frame to the MPLS underlay network. The mentioned frame is now in the optimized frame format. This frame gets routed along the LSP corresponding to the top MPLS LSP label and finally gets delivered by the MPLS underlay network to the mentioned egress PE router. During the routing of the frame in the underlay, the source and destination MAC addresses of the ethernet header are changed at every routing hop due to the source MAC and next hop MAC rewriting in the underlay network. When the frame is received at the mentioned egress PE router the forwarding rules mentioned above pop both the MPLS labels and the original destination and source overlay MAC addresses in the ethernet header of the frame are reconstructed based on the optimization MPLS label and the source IP address in the frame respectively and the frame is delivered to the attached overlay workload as previously explained.

[0082] A number of embodiments will now be described, some of which may be seen as alternatives, while some may be used in combination.

[0083] A method according to embodiments will first be described with regards to the establishment of an optimized frame format used by a first compute node acting as ingress PE 21 and a second compute node acting as egress PE 22 in the transport of overlay ethernet traffic containing an IP payload from the source overlay endpoint attached to or hosted in the first compute node to the destination overlay endpoint attached to or hosted in the second compute node through a Multi Protocol Label Switching, MPLS, underlay network.

[0084] The proposed frame format, exemplified in Figure 3c, is a conventional MPLS tagged ethernet frame format, wherein an MPLS label stack is inserted between the ethernet header and the IP header of the overlay ethernet frame, where the single ethernet header carries the destination and source overlay Media Access Control, MAC, addresses when the frame was originated from the source overlay endpoint, and where the MPLS header of the frame comprises:

[0085] - a top label with the Label Switch Path, LSP, label towards the destination Provider Edge, PE, router, i.e. egress PE, and

[0086] - a bottom label, e.g. an optimization MPLS label, holding information enabling the forwarding mechanism in the egress PE router to restore the original destination and source overlay MAC addresses, exemplified by Lo1 in Figure 3c.

[0087] A method according to embodiments will now be described as performed by an ingress PE router 41 to establish a route between the ingress PE 41 router and an egress PE router 42, wherein the frame format as established above is used in the transport of overlay ethernet traffic.

[0088] It is proposed that this method comprises:

[0089] - receiving 401, from an egress PE router, in a type 2 MAC / IP route advertisement message through BGP protocol, an Ethernet Virtual Private Network, EVPN, Network Layer Reachability Information, NLRI, and an optimization MPLS label where an overlay destination endpoint is attached; and

[0090] - creating 403 forwarding rules to send overlay layer 2 traffic with unicast IP payload destined to the overlay destination endpoint based on the received EVPN NLRI and an optimization MPLS label.

[0091] A method according to embodiments will now be described as performed by an egress PE router 42 to establish a route between the egress PE router 42 and an ingress PE router 41, wherein the frame format as established above is used in the transport of overlay ethernet traffic.

[0092] It is proposed that this method comprises:

[0093] - advertising 402 an optimization MPLS label in a type 2 MAC / IP route advertisement message through BGP protocol for an attached overlay endpoint; and

[0094] - creating 404 the forwarding rules to receive and process the overlay traffic sent to the mentioned overlay endpoint by other remote overlay endpoints in the optimized frame format.

[0095] A method according to embodiments will now be described as performed by an egress PE router 42 for the transport of overlay ethernet traffic through a route according established as described above. It is proposed that a frame is routed 405 through the MPLS undelay network towards the egress PE router 42 based the top label. It is proposed that the method comprises:

[0096] - restoring 406, by the forwarding mechanism in the egress PE router 42, the original overlay ethernet header MAC addresses in the ethernet header of the frame using the bottom label, being the optimization label, and

[0097] - delivering 407 the frame to the corresponding attached destination overlay endpoint.

[0098] A first compute node 21 according to proposed embodiments will now be described, where the first compute node is adapted to host or run overlay endpoints virtual machines and / or containers. The first compute node 21 is adapted to use a frame format as proposed above in a cloud overlay networking system based on the RFC7432 BGP MPLS EVPN, and to function as ingress PE 41.

[0099] It is proposed that the first compute node 21 comprises:

[0100] - a virtual switch and router component 211 adapted to function as an ingress PE router forwarding plane function as per RFC 7432;

[0101] - a Border Gateway Protocol, BGP, function 212 adapted to function as an ingress PE control plane function as per RFC 7432; and

[0102] - a virtual switch and router function adapted to function as an egress PE router.

[0103] A second compute node 22 according to proposed embodiments will now be described, where the second compute node 22 is adapted to host or run overlay endpoints virtual machines and / or containers.

[0104] The second compute node 22 is adapted to use a frame format as proposed above in a cloud overlay networking system based on the RFC7432 BGP MPLS EVPN, and to function as egress PE 42.

[0105] It is proposed that the second compute node 22 comprises:

[0106] - a virtual switch and router component 221 adapted to function as an egress PE router forwarding plane function as per RFC 7432;

[0107] - a BGP function 222 adapted to function as an egress PE control plane function as per RFC 7432; and

[0108] - a virtual switch and router function adapted to function as an ingress PE router.

[0109] The working of the method and system of this proposal in the context of an RFC7432 BGP MPLS EVPN implementation will now be described.

[0110] A typical cloud overlay networking system based on RFC7432 BGP MPLS EVPN consists of a set of compute nodes hosting the overlay endpoint workloads which are virtual machines (VM) and / or Containers belonging to various applications. These compute nodes are connected to a layer 3 MPLS underlay network 2 as illustrated in Figure 2 schematically showing a first compute node 21 and a second compute node 22.

[0111] The underlay network 2 may consist of a set of interconnected routers providing the conventional MPLS label switch path (LSP) based IP forwarding. These compute nodes 21, 22 are also running a virtual switch and router function software component 211, 221. This virtual switch and router component 211, 221 may be realized through the open source Open vswitch or an equivalent hardware function such as a smartNIC or Data Processing Unit (DPU) to which the overlay endpoint workloads are connected using the virtual links created in the compute node’s 21, 22 operating system. The virtual switch and router function 211, 221 in a compute node 21, 22 may be capable of doing the standard openflow rules, also called flow entries, based packet forwarding path programming. The virtual switch and router function 211, 221 may also act as the Provider Edge router (PE) for all the local overlay endpoint workloads attached to it.

[0112] The virtual switch and router function 211, 221 on a compute node 21 , 22 may be configured with a distinct underlay IP address IPu. An MPLS LSP label Lu may be established for IPu in the underlay network. Lu represents the LSP to reach IPu from any other remote PE router through the MPLS underlay network. The LSP label Lu for IPu might be established via LDP signaling in the underlay network 2 or it could use the MPLS segment routing label (Segment Identifyer or SID) of IPu if the underlay is employing MPLS segment routing.

[0113] Further Lu may be more than one label representing the LSP towards IPu. In this way the virtual switch and router functions in each of the compute nodes may have a unique underlay IP address IPu and the corresponding LSP label Lu.

[0114] Generally, an administrator creates an overlay network N and creates and bring up the overlay endpoint workloads i.e., either a VM or a container on the compute nodes 21 , 22 in the overlay network N. Each overlay endpoint workload is assigned generally via DHCP an overlay IP address IPo from the overlay IP subnet prefix range corresponding to the overlay network N the endpoint workload belongs to.

[0115] According to this proposal the overlay endpoint workload IP address IPo and MAC address MACo are locally mapped to a locally allocated unique MPLS label Lo called the optimization MPLS label by the control component on the compute and advertised by the BGP function 212, 222 on the compute node 21, 22 using the route type 2 MAC / IP route advertisement as per the RFC7432 in the otherwise unused MPLS Iabel2 field of the type 2 MAC / IP route advertisement. It should be understood that the normal RFC7432 based EVPN forwarding control-plane and data-plane construction and behavior is not described in this proposal and is assumed to be understood and functional. Only the optimization MPLS label signaling and the optimized forwarding plane construction and operation according to proposed embodiments are described in this proposal.

[0116] The block diagram in Figure 2 shows two such compute nodes the first compute node 21 and the second compute node 22, in a cloud overlay networking system based on the RFC7432 BGP MPLS EVPN.

[0117] It is proposed that the first compute node 21 may contain the first virtual switch and router component 211 that act as the ingress PE router, and the second compute node 22 contain the second virtual switch and router component 212 that act as the egress PE router, forwarding plane function as per RFC 7432. The first BGP function 212 on the first compute node 21 , and second BGP function 222 on the second compute node 22 act as the ingress PE and egress PE control plane function as per RFC 7432. The proposed optimized forwarding control plane and forwarding plane functionalities are explained with the first compute nodes 21 virtual switch and router function 211 as the egress PE router and the second compute nodes 22 virtual switch and router 221 as the ingress PE router.

[0118] The first compute nodes 21 virtual switch and router function 211 may have the underlay network IP address I Pu1 and MAC address MACul. The MPLS LSP label established by the standard means in the underlay for I Pu1 is Lu1. N1 is an administratively created overlay network whose broadcast domain identifier allocated or configured as per RFC7432 is ETAG1. The first compute node 21 may have an administratively created overlay endpoint workload first endpoint which is a first virtual machine VM1 whose overlay IP and MAC addresses are IPo1 and MACol respectively in an overlay network N1.

[0119] The first overlay endpoint endpoint1 / VM1 vnic is Vnid and is attached using a virtual link to openflow port Po1 of the first virtual switch router function 211. The overlay network NTs Ethernet Tag Id allocated as per RFC 7432 is ETAG1 which represents the layer 2 broadcast domain for the overlay network N1. The virtual switch and router function 211 on the first compute node 21 is attached to the underlay network through the network interface card Pnid and Pnid has the openflow port Pethl on the virtual switch and router function 211.

[0120] Similarly, the second virtual switch and router function 221 of the second compute node 22 may have the underlay network IP address IPu2 and MAC address MACu2. The MPLS LSP label established by the standard means in the underlay for IPu2 is Lu2. The second compute node 22 may have an administratively created second overlay endpoint which is a virtual machine VM2 whose IP and MAC addresses are IPo2 and MACo2 respectively in the same overlay network N1. The second overlay endpoint endpoint2 / VM2 vnic is Vnic2 and is attached through a virtual link to the second virtual switch router function’s 221 openflow port Po2. The second virtual switch and router function 221 on the second compute node 22 is attached to the underlay network through the network interface card Pnic2 and Pnic2 has the openflow port Peth2 on the second virtual switch and router function 221. Figure 3a shows an overlay ethernet frame generated from VM1 and destined to VM2. Figure 3b shows the RFC7432 MPLS-EVPN encapsulation the frame in Figure 3a. It has two ethernet headers, the outer and inner ones.

[0121] Figure 3c shows the proposed optimized frame for the overlay frame of Figure 3a. It has only one ethernet header and it is MPLS tagged using the proposed optimization MPLS label instead of the VPN label. These figures will be used in the following proposed forwarding plane behaviors.

[0122] In the following the term ingress PE refers to a first compute node 21 and the virtual switch and router function 211 on it receiving traffic from an overlay endpoint attached to it and forwards it to the underlay 2 to a remote PE 22. The egress PE refers to the second compute node 22 and the virtual switch and router function 221 on it receiving traffic from the underlay network 2 and then delivers it to an overlay endpoint attached to it. These terminologies are in line with the RFC 7432.

[0123] The specific control plane signaling operational behaviors of this proposal will now be explained in the context of RFC7432 control plane behavior.

[0124] When the administratively created overlay endpoint workload VM1 boots and become operational on the first compute node 21, a control component 213 on the first compute node 21 creates an openflow rule, i.e. flow entry, in the PE router of the first compute nodes 21 , i.e. in the flow table 0 of the openflow table of the virtual switch and router function 211 , to classify the overlay layer 2 traffic received from VM1 through attached openflow port Po1 into the Ethernet Tag ID / broadcast domain ETAG1 of the network N1 to which VM1 belongs.

[0125] This openflow rule is the local frame classifying rule and is denoted as rule A1 in this proposal. The Rule A1 is shown in table 1 below.

[0126] Table 1. Openflow rules, i.e. flow entries, created on the first virtual switch and router component 211 belonging to the first compute node 21. Rule A1 matches the in port openflow field of the locally received layer 2 frame with openflow port Po1 and then sets the tun-id openflow field of the frame to the Ethernet Tag Id / broadcast domain identifier value ETAG1. This frame classifying rule is created as per RFC7432 implementation and is not anything specific to this proposal. This proposal depends on this rule for getting the local overlay layer 2 ethernet traffic classified into the configured overlay network’s broadcast domain of the overlay endpoint workload.

[0127] Similarly, on the second compute node 22 when the overlay endpoint VM2 boots and become operational, a Control component 223 on the second compute node 22 creates the openflow rule, i.e. flow entry Rule A2, a local frame classifying rule, in the flow table 0 of the openflow table of the second compute nodes 22 PE router, i.e. in the virtual switch and router function 221 as shown in table 2.

[0128] Table 2. Openflow rules, i.e. flow entries, created on the second virtual switch and router component 221 belonging to the second compute node 22. These local classification rules classified ethernet traffic from VM1 and VM2 into the same broadcast domain ETAG1 that correspond to the configured overlay network N1 for these VMs.

[0129] It is proposed that when the overlay endpoint workload VM1 boots and become operational on the first compute node 21 in the overlay network N1 as previously mentioned, the first BGP function 212 on the first compute node 21 may advertise the endpoint’s EVPN NLRI which consists of the IP address IPo1 , MAC address MACol and Ethernet Tag ID / broadcast domain ETAG1 with the next hop as I Pu1 using the MP-BGP route type 2 MAC / IP advertisement as per RFC 7432.

[0130] This advertisement also contains the MPLS VPN label Lv1 of VM1 in the MPLS labell field of the advertisement as per RFC 7432.

[0131] This proposal requires that a new MPLS label called the optimization MPLS label

[0132] Lo1 is to be allocated locally by the first control function 213 on the first compute node 21 and locally unique to the NLRI MACol, IPo1 and ETAG1 of VM1. The same optimization MPLS label is to be advertised in the same type2 MAC / IP advertisement mentioned above in the unused MPLS Iabel2 route attribute field of the advertisement payload. This optimization MPLS label should be different from the VPN label Lv1 of VM1.

[0133] This advertisement payload that includes the NLRI and the optimization MPLS label Lo1 is shown in the block diagram 2A in Figure 2.

[0134] Alternatively, instead of using the MPLS Iabel2 field, it is proposed that this optimization label Lo1 may be advertised as a custom BGP Extended Community attribute as part of the mentioned route type 2 MAC / IP advertisement. Further, this proposal requires that the Control function on the Computes create the send and receive traffic processing and forwarding rules on the respective PE routers, corresponding to this enhanced route advertisement as will be described further below.

[0135] Similarly, on the second compute node 22 when the overlay endpoint VM2 boots and become operational, it is proposed that the second BGP function 222 on the second compute node 22 may advertise the endpoint’s EVPN NLRI which consists of the IP address IPo2, MAC address MACo2 and Ethernet Tag ID / broadcast domain ETAG1 with the next hop IPu2 and VPN label Lv2 and the optimization MPLS label Lo2 using the MP- BGP route type 2 MAC / IP advertisement.

[0136] As per this proposal, each compute nodes PE router, i.e., the virtual switch and router component, may create the routing rules to send overlay traffic tagged with the optimization MPLS label as per the received type 2 MAC / IP advertisements. Accordingly, when the type 2 MAC / IP route advertisement from the second compute node mentioned above is received with the NLRI as MACo2, IPo2 and ETAG1 and the route attributes next hop as IPu2 and MPLS Iabel2 as Lo2, the BGP function on the first compute node 21 instructs the first Control function 213 to create the openflow rule, i.e. flow entry Rule B1 , called the Optimization and LSP label push rule on the virtual switch and router function’s openflow table in the flow table 1 as shown in table 1. Alternatively, Lo2 might be received through the custom BGP enhanced community attribute of the previously mentioned type 2 MAC / IP route advertisement if it was sent thus by the send side as mentioned above.

[0137] The purpose this openflow rule is to route any overlay IP unicast traffic originating from any local overlay endpoint workloads on the first compute nide 21 belonging to network N1 and destined to the received NLRI from the second compute node 22, through the underlay network using the LSP label towards the destination PE router on the second compute node 22. Rule B1 matches the ethernet type of the overlay frame with IP4, destination MAC of the frame with MACo2 and the tunjd field of the frame with ETAG1 and if matched then first inserts the optimization MPLS label Lo2 followed by the LSP label Lu2 corresponding to the next hop IPu2 to the overlay frame between its ethernet and IP headers (normal MPLS tagging) and then sets the destination MAC address to the first hop MAC address MACf2 of the LSP label Lu2, ie, the next hop MAC address for Lu2 on the first compute node 21.

[0138] Finally, Rule B1 outputs the frame to the underlay network through the corresponding openflow port Pethl.

[0139] It can be noted that at the first compute node 21 , MACf2 is the underlay first hop router’s MAC for the MPLS LSP label Lu2 and was learned by the first Control function 213 belonging to the first compute node 21 through the ARP protocol and the LSP establishment process through LDP or segment routing etc., all of which are part of the regular underlay MPLS route and LSP establishment process and nothing specific to this proposal.

[0140] Similarly, the second BGP function 222, belonging to the second compute node 22, receives the type 2 MAC / IP route advertisement from the first BGP 212 for its overlay endpoint VM1 with NLRI MACol , IPo1 and ETAG1 and with the next hop as I Pu 1 with route attributes containing the optimization label Lo1. Based on this the second Control function 223, belonging to the second compute node 22, creates the openflow rule, i.e. flow entry Rule B2, i.e. the Optimization and LSP label push rule, in the flow table 1 of the in the openflow table of the second compute node 22 as shown in table 2.

[0141] As per this proposal each compute node sending a type 2 MAC / IP advertisement with an optimization MPLS label may create the corresponding receive processing routing rules for each of the advertised endpoints EVPN NLRIs. Accordingly, when the overlay endpoint workload VM1 boots and become operational on the first compute nide 21 and the type 2 route advertisement for its EVPN NLRI MACol, IPo1 and ETAG1 is sent as mentioned above, the first Control function 213 on the first compute node 21 creates the following openflow rules, i.e. flow entries C1, D1, F1, in the openflow table of the first virtual switch and router function 211 to receive and process the received traffic in the optimized frame format from other remote endpoints via the underlay network, sent towards the advertised NLRI. These routing rules strip the MPLS labels from the received frame and restore the destination and source overlay MAC addresses in the frame based on the NLRI and optimization label mappings that had been established trough the routing advertisements mentioned earlier and then deliver the frames to the right local destination endpoint workload corresponding to the optimization MPLS label in the received frame.

[0142] Accordingly, Rule C1 is created in the openflow flow table 0 and is called the remote frame accept rule and it matches the in_port of a received frame from the underlay network with openflow port id Pethl and the top MPLS label of the frame with LSP label Lu1 and then accepts the frame for further processing and sends it to openflow flow table 10 after popping the top LSP label Lu1.

[0143] Again, accordingly, Rule D1 is created in the openflow flow table 10 and is called the remote frame classification rule, and it matches the now top MPLS label field of the frame with the optimization MPLS label Lo1 and classifies the frame into the corresponding broadcast domain ETAG1 by setting the tun-id openflow field in the frame and then sends the frame to openflow flow table 11 without popping the MPLS label.

[0144] Rule E1 is created in the openflow flow table 11 for the remote frame overlay source MAC restoration as will be described below and it matches the source IP address of the frame and rewrites the source MAC address with corresponding overlay source MAC address.

[0145] Again, accordingly, Rule F1 is created in the openflow flow table 12 and is called the remote frame destination MAC restoration rule, matches the MPLS label field of the frame with Lo1 and then restores its overlay destination MAC address to the corresponding MAC MACol and set the TTL field of the frame to 255 and finally outputs the frame to the openflow port Po1 towards the attached endpoint VM1 that correspond to the optimization MPLS label Lo1. The frame thus delivered is a proper overlay ethernet frame.

[0146] As per this proposal, when the type 2 MAC / IP route advertisement from the second compute node 22 is received with the NLRI as MACo2, IPo2 and ETAG1, the first BGP function 212 instructs the first Control function 213 on the first compute node 21 to create the openflow rule, i.e. flow entry Rule E1 mentioned earlier, in the openflow flow table 11, and is called the remote frame source MAC restoration rule. The purpose of this rule is to restore the source MAC address of a remote fame to MACo2 based on the corresponding source IP address in the frame IPo2. This rule is created for each remote NLRI received thru a type 2 MAC / IP route advertisement.

[0147] Similarly, on the second compute node 22, the openflow rules, i.e. flow entries Rule C2, D2, E2 and F2 are created for processing and delivering the received remote originated overlay traffic to the second compute node 22 hosted endpoint VM2.

[0148] As per this proposal, in this fashion as described above, for each received type 2 MAC / IP route advertisement for a remote overlay endpoint workload, a local PE router on a compute creates the send routing rule A1 that sends the overlay ethernet traffic with IP payload to that destination in the optimized frame format.

[0149] Further, for each type 2 MAC / IP advertisement sent by the local PE router on a compute node for its locally attached overlay endpoint workloads, the local PE router creates the receive routing rules C1, D1 and F1. For each received type 2 MAC / IP route advertisement for a remote overlay endpoint workload, a local PE router on a compute node creates the receive rule E1 to restore the source overlay MAC address of the overlay ethernet frame from that corresponding overlay source endpoint. These routing rules then form the complete optimized forwarding plane on the PE routers for sending and receiving via the underlay network the overlay layer 2 network traffic with unicast IP payload in the optimized frame format.

[0150] The forwarding of overlay layer 2 traffic with unicast IP payload by the above created openflow rules, i.e. flow entries on both compute nodes virtual switch and router functions, is explained for the overlay traffic generated by the overlay endpoint workload VM1 on the first compute node 21 and destined to the overlay endpoint workload VM2 on the second compute node 22. Note that both overlay endpoint workloads VM1 and VM2 were created to belong to network N1 whose corresponding broadcast domain is ETAG1 as had been mentioned.

[0151] On the first compute node 21, when a layer 2 overlay ethernet frame is received by the ingress PE router, i.e. the virtual switch and router function on the port Po1 from the attached overlay endpoint VM1 and the frame is destined to the remote destination endpoint workload mac address MACo2 and IP address IPo2, then the frame processing happens and starts at openflow flow table 0 in the first virtual switch and router function 211 of the first compute nodes 21. This frame is shown in Figure 3a. In openflow flow table 0 the frame classification rule A1 matches the source port of the frame with the VMTs attached port Po1 and the rule action sets the tunjd openflow field of the frame to ETAG1 thus classifying the frame into the broadcast domain ETAG1 and the frame is directed to openflow flow table 1 for further processing. In openflow flow table 1 the optimization label push rule B1 matches and the tunjd in the frame with ETAG1 and the eth_type of the frame with IPv4 and the destination MAC in the frame with MACo2 and if they match the actions of the rule are triggered. The actions of the rule B1 pushes the MPLS label stack with bottom label as the optimization MPLS label Lo2 and the top label as the LSP label Lu2 between the ethernet and IP headers of the frame. The frame at this point is shown in Figure 3c. Further actions in the rule B1 sets the destination MAC in the frame to the first hop MAC MACf2 of the LSP label Lu2 and send the frame to the underlay network through the openflow port Pethl . The frame thus sent out is in the optimized frame format of this proposal and is shown in Figure 3d.

[0152] Embodiments propose that if the overlay frame mentioned above contains the unicast destination MAC address and IP payload then the Rule B1 conditions will match, and the frame forwarding may happen as mentioned above with the optimized frame format of this proposal. On the other hand, if the overlay frame contains a non IP payload then it will not match with the eth type=IPv4 condition of Rule B1 and hence rule B1 will not be applied. Instead the frame will be processed by the subsequent rules in openflow flow table 0 or further openflow tables which are supposed to contain the conventional RFC7432 frame forwarding rules and the traffic will be accordingly sent out with the conventional RFC7432 MPLS EVPN encapsulation frame format.

[0153] Assuming that the overlay ethernet frame contained the unicast IP payload and hence got processed using Rule B1 of this proposal, based on the top LSP label the frame is MPLS forwarded by the MPLS underlay network and delivered to the second virtual switch and router function 221 belonging to the second compute node 22 on port Peth2. During the forwarding of the frame in the underlay network its destination and source MAC addresses are rewritten due to the next hop rewriting in the underlay network forwarding. Hence the original overlay destination and source MAC addresses MACo2, MACol respectively are no more present in the frame. They will be restored at the receive side egress PE router forwarding rule processing as described further.

[0154] When the mentioned remote frame is received from the underlay network by the second virtual switch and router function 221 on the second compute node 22 through the port Peth2 the rule C2 in openflow flow table 0 matches the frame’s in_port with Peth2 and MPLS top label with Lu2 and if both conditions matched then pops the top LSP label Lu2 and sends the frame to openflow flow table 10 for further processing. In openflow table 10 the remote frame classification rule D2 matches the current top MPLS label of the frame with the optimization MPLS label Lo2 and based on this the frame is classified by the action of the rule D2 into ETAG1 broadcast domain by setting tunjd to ETAG1 and the next action of the rule sends the frame to openflow flow table 11 for further processing. In openflow flow table 11 the remote frame source MAC restoration rule E2 matches the frames tunjd and source IP address fields with ETAG1 and IPo1 respectively and restores the source MAC of the frame to the original overlay source MAC address MACol and sends the frame to openflow flow table 12 for further processing. In openflow flow table 12 the remote frame destination MAC restoration and local delivery rule matches the top MPLS label of the fame with Lo2 and restores the destination MAC address of the frame to the original overlay destination MAC MACo2 and sets the IP header TTL of the frame to 255 and delivers the frame to the destination overlay workload VM2 on its attached port Po2. The frame thus delivered is the normal Iayer2 overlay ethernet frame with the unicast IP payload as was generated and sent by the remote source overlay endpoint workload VM1.

[0155] If the remote frame received was a conventional RFC7432 MPLS EVPN frame and not an optimized frame of this proposal, then the Rule D2 will not match the bottom label in frame and hence the frame will be processed by the conventional RFC4432 forwarding rules, i.e. flow entries, that are supposed to exist in openflow flow tables.

[0156] The openflow rules, i.e. flow entries, of this proposal are mentioned to be created in openflow flow tables 1, 10, 11 and 12. However, they can be created in openflow flow tables of different numbers and in that case the goto action of these flow entries should specify the corresponding flow table numbers such that the frame processing order as specified by these rules through the goto action are preserved.

[0157] The above description of the control-plane and forwarding-plane processing for the traffic between overlay endpoint workload pairs VM1 and VM2 is exemplary and sufficiently cover the overlay IP unicast traffic processing for any pair of source, destination overlay workloads in any broadcast domain.

[0158] The overlay non-IP and broadcast traffic handling is not done as per this proposal and instead they are expected to be handled by the conventional RFC7432 implementation as had been mentioned.

[0159] Embodiments herein may be implemented through a respective processor or one or more processors, such as the respective processor 510 of a processing circuitry in the first compute node 21 depicted in Figure 5, and processor 610 of a processing circuitry in the second compute node 22 depicted in Figure 6 together with respective computer program code for performing the functions and actions of the embodiments herein. The program code mentioned above may also be provided as a computer program product, for instance in the form of a data carrier carrying computer program code for performing the embodiments herein when being loaded into the respective first compute node 21 and second compute node 22. One such carrier may be in the form of a CD ROM disc. It is however feasible with other data carriers such as a memory stick. The computer program code may furthermore be provided as pure program code on a server and downloaded to the respective first compute node 21 and second compute node 22. The first compute node 21 and second compute node 22 may further comprise a respective memory 520 and memory 620 comprising one or more memory units. The respective memory 520 and memory 620 comprises instructions executable by the processor in the respective first compute node 21 and second compute node 22. The respective memory 520 and memory 620 are arranged to be used to store e.g., media functions, indications, tags, information, data, configurations, communication data, and applications to perform the methods herein when being executed in the respective first compute node 21 and second compute node 22.

[0160] In some embodiments, a respective computer program 530 and computer program 630 comprises instructions, which when executed by the respective at least one processor 510 and processor 610, cause the at least one processor of respective first compute node 21 and second node 22 to perform the actions above.

[0161] In some embodiments, a respective carrier 540 and carrier 640 comprises the respective computer program 530 and computer program 630, wherein the respective carrier 540 and carrier 640 is one of an electronic signal, an optical signal, an electromagnetic signal, a magnetic signal, an electric signal, a radio signal, a microwave signal, or a computer-readable storage medium.

[0162] Those skilled in the art will appreciate that units in the respective first compute node 21 and second compute node 22 described above may refer to a combination of analog and digital circuits, and / or one or more processors configured with software and / or firmware, e.g. stored in the respective first compute node 21 and second compute node 22, that when executed by the respective one or more processors such as the processors described above. One or more of these processors, as well as the other digital hardware, may be included in a single Application-Specific Integrated Circuitry ASIC, or several processors and various digital hardware may be distributed among several separate components, whether individually packaged or assembled into a System-on-a-Chip (SoC).

[0163] ADDITIONAL EXPLANATION

[0164] Some of the embodiments contemplated herein will now be described more fully with reference to the accompanying drawings. Embodiments are provided by way of example to convey the scope of the subject matter to those skilled in the art.

[0165] Figure 7 shows an example of a communication system QQ100 in accordance with some embodiments.

[0166] In the example, the communication system QQ100 includes a telecommunication network QQ102 that includes an access network QQ104, such as a radio access network (RAN), and a core network QQ106, which includes one or more core network nodes QQ108. The access network QQ104 includes one or more access network nodes, such as network nodes QQ110a and QQ110b (one or more of which may be generally referred to as network nodes QQ110), or any other similar 3rdGeneration Partnership Project (3GPP) access nodes or non-3GPP access points. Moreover, as will be appreciated by those of skill in the art, a network node is not necessarily limited to an implementation in which a radio portion and a baseband portion are supplied and integrated by a single vendor. Thus, it will be understood that network nodes include disaggregated implementations or portions thereof. For example, in some embodiments, the telecommunication network QQ102 includes one or more Open-RAN (ORAN) network nodes. An ORAN network node is a node in the telecommunication network QQ102 that supports an ORAN specification (e.g., a specification published by the O-RAN Alliance, or any similar organization) and may operate alone or together with other nodes to implement one or more functionalities of any node in the telecommunication network QQ102, including one or more network nodes QQ110 and / or core network nodes QQ108.

[0167] Examples of an ORAN network node include an open radio unit (0-Rll), an open distributed unit (0-Dll), an open central unit (O-CU), including an O-CU control plane (O- CLI-CP) or an O-CU user plane (O-CU-UP), a RAN intelligent controller (near-real time or non-real time) hosting software or software plug-ins, such as a near-real time control application (e.g., xApp) or a non-real time control application (e.g., rApp), or any combination thereof (the adjective “open” designating support of an ORAN specification). The network node may support a specification by, for example, supporting an interface defined by the ORAN specification, such as an A1, F1, W1, E1 , E2, X2, Xn interface, an open fronthaul user plane interface, or an open fronthaul management plane interface. Moreover, an ORAN access node may be a logical node in a physical node. Furthermore, an ORAN network node may be implemented in a virtualization environment (described further below) in which one or more network functions are virtualized. For example, the virtualization environment may include an O-Cloud computing platform orchestrated by a Service Management and Orchestration Framework via an 0-2 interface defined by the O-RAN Alliance or comparable technologies. The network nodes QQ110 facilitate direct or indirect connection of user equipment (UE), such as by connecting UEs QQ112a, QQ112b, QQ112c, and QQ112d (one or more of which may be generally referred to as UEs QQ112) to the core network QQ106 over one or more wireless connections.

[0168] Example wireless communications over a wireless connection include transmitting and / or receiving wireless signals using electromagnetic waves, radio waves, infrared waves, and / or other types of signals suitable for conveying information without the use of wires, cables, or other material conductors. Moreover, in different embodiments, the communication system QQ100 may include any number of wired or wireless networks, network nodes, UEs, and / or any other components or systems that may facilitate or participate in the communication of data and / or signals whether via wired or wireless connections. The communication system QQ100 may include and / or interface with any type of communication, telecommunication, data, cellular, radio network, and / or other similar type of system.

[0169] The UEs QQ112 may be any of a wide variety of communication devices, including wireless devices arranged, configured, and / or operable to communicate wirelessly with the network nodes QQ110 and other communication devices. Similarly, the network nodes QQ110 are arranged, capable, configured, and / or operable to communicate directly or indirectly with the UEs QQ112 and / or with other network nodes or equipment in the telecommunication network QQ102 to enable and / or provide network access, such as wireless network access, and / or to perform other functions, such as administration in the telecommunication network QQ102.

[0170] In the depicted example, the core network QQ106 connects the network nodes QQ110 to one or more host computing systems, such as host QQ116. These connections may be direct or indirect via one or more intermediary networks or devices. In other examples, network nodes may be directly coupled to hosts. The core network QQ106 includes one more core network nodes (e.g., core network node QQ108) that are structured with hardware and software components. Features of these components may be substantially similar to those described with respect to the UEs, network nodes, and / or hosts, such that the descriptions thereof are generally applicable to the corresponding components of the core network node QQ108. Example core network nodes include functions of one or more of a Mobile Switching Center (MSC), Mobility Management Entity (MME), Home Subscriber Server (HSS), Access and Mobility Management Function (AMF), Session Management Function (SMF), Authentication Server Function (AUSF), Subscription Identifier De-concealing function (SIDF), Unified Data Management (UDM), Security Edge Protection Proxy (SEPP), Network Exposure Function (NEF), and / or a User Plane Function (UPF).

[0171] The host QQ116 may be under the ownership or control of a service provider other than an operator or provider of the access network QQ104 and / or the telecommunication network QQ102. The host QQ116 may host a variety of applications to provide one or more service. Examples of such applications include live and pre-recorded audio / video content, data collection services such as retrieving and compiling data on various ambient conditions detected by a plurality of UEs, analytics functionality, social media, functions for controlling or otherwise interacting with remote devices, functions for an alarm and surveillance center, or any other such function performed by a server.

[0172] As a whole, the communication system QQ100 of Figure 7 enables connectivity between the UEs, network nodes, and hosts. In that sense, the communication system may be configured to operate according to predefined rules or procedures, such as specific standards that include, but are not limited to: Global System for Mobile Communications (GSM); Universal Mobile Telecommunications System (UMTS); Long Term Evolution (LTE), and / or other suitable 2G, 3G, 4G, 5G standards, or any applicable future generation standard (e.g., 6G); wireless local area network (WLAN) standards, such as the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standards (WiFi); and / or any other appropriate wireless communication standard, such as the Worldwide Interoperability for Microwave Access (WiMax), Bluetooth, Z-Wave, Near Field Communication (NFC) ZigBee, LiFi, and / or any low-power wide-area network (LPWAN) standards such as LoRa and Sigfox.

[0173] In some examples, the telecommunication network QQ102 is a cellular network that implements 3GPP standardized features. Accordingly, the telecommunications network QQ102 may support network slicing to provide different logical networks to different devices that are connected to the telecommunication network QQ102. For example, the telecommunications network QQ102 may provide Ultra Reliable Low Latency Communication (URLLC) services to some UEs, while providing Enhanced Mobile Broadband (eMBB) services to other UEs, and / or Massive Machine Type Communication (mMTC) / Massive loT services to yet further UEs.

[0174] In some examples, the UEs QQ112 are configured to transmit and / or receive information without direct human interaction. For instance, a UE may be designed to transmit information to the access network QQ104 on a predetermined schedule, when triggered by an internal or external event, or in response to requests from the access network QQ104. Additionally, a UE may be configured for operating in single- or multi- RAT or multi-standard mode. For example, a UE may operate with any one or combination of Wi-Fi, NR (New Radio) and LTE, i.e. being configured for multi-radio dual connectivity (MR-DC), such as E-UTRAN (Evolved-UMTS Terrestrial Radio Access Network) New Radio - Dual Connectivity (EN-DC).

[0175] In the example, the hub QQ114 communicates with the access network QQ104 to facilitate indirect communication between one or more UEs (e.g., UE QQ112c and / or QQ112d) and network nodes (e.g., network node QQ110b). In some examples, the hub QQ114 may be a controller, router, content source and analytics, or any of the other communication devices described herein regarding UEs. For example, the hub QQ114 may be a broadband router enabling access to the core network QQ106 for the UEs. As another example, the hub QQ114 may be a controller that sends commands or instructions to one or more actuators in the UEs. Commands or instructions may be received from the UEs, network nodes QQ110, or by executable code, script, process, or other instructions in the hub QQ114. As another example, the hub QQ114 may be a data collector that acts as temporary storage for UE data and, in some embodiments, may perform analysis or other processing of the data. As another example, the hub QQ114 may be a content source. For example, for a UE that is a VR device, display, loudspeaker, or other media delivery device, the hub QQ114 may retrieve VR assets, video, audio, or other media or data related to sensory information via a network node, which the hub QQ114 then provides to the UE either directly, after performing local processing, and / or after adding additional local content. In still another example, the hub QQ114 acts as a proxy server or orchestrator for the UEs, in particular if one or more of the UEs are low energy loT devices.

[0176] The hub QQ114 may have a constant / persistent or intermittent connection to the network node QQ110b. The hub QQ114 may also allow for a different communication scheme and / or schedule between the hub QQ114 and UEs (e.g., UE QQ112c and / or QQ112d), and between the hub QQ114 and the core network QQ106. In other examples, the hub QQ114 is connected to the core network QQ106 and / or one or more UEs via a wired connection. Moreover, the hub QQ114 may be configured to connect to an M2M service provider over the access network QQ104 and / or to another UE over a direct connection. In some scenarios, UEs may establish a wireless connection with the network nodes QQ110 while still connected via the hub QQ114 via a wired or wireless connection. In some embodiments, the hub QQ114 may be a dedicated hub - that is, a hub whose primary function is to route communications to / from the UEs from / to the network node QQ110b. In other embodiments, the hub QQ114 may be a non-dedicated hub - that is, a device which is capable of operating to route communications between the UEs and network node QQ110b, but which is additionally capable of operating as a communication start and / or end point for certain data channels.

[0177] Figure 8 shows a UE QQ200 in accordance with some embodiments. The UE QQ200 presents additional details of some embodiments of the UE QQ112 of Figure 1. As used herein, a UE refers to a device capable, configured, arranged and / or operable to communicate wirelessly with network nodes and / or other UEs. Examples of a UE include, but are not limited to, a smart phone, mobile phone, cell phone, voice over IP (VoIP) phone, wireless local loop phone, desktop computer, personal digital assistant (PDA), wireless cameras, gaming console or device, music storage / playback device, wearable terminal device, wireless endpoint, mobile station, tablet, laptop, laptop-embedded equipment (LEE), laptop-mounted equipment (LME), an Augmented Reality (AR) or Virtual Reality (VR) device, wireless customer-premise equipment (CPE), vehicle, vehiclemounted or vehicle embedded / integrated wireless device, etc. Other examples include any UE identified by the 3rd Generation Partnership Project (3GPP), including a narrow band internet of things (NB-loT) UE, a machine type communication (MTC) UE, and / or an enhanced MTC (eMTC) UE.

[0178] A UE may support device-to-device (D2D) communication, for example by implementing a 3GPP standard for sidelink communication, Dedicated Short-Range Communication (DSRC), vehicle-to-vehicle (V2V), vehicle-to-infrastructure (V2I), or vehicle-to-everything (V2X). In other examples, a UE may not necessarily have a user in the sense of a human user who owns and / or operates the relevant device. Instead, a UE may represent a device that is intended for sale to, or operation by, a human user but which may not, or which may not initially, be associated with a specific human user (e.g., a smart sprinkler controller). Alternatively, a UE may represent a device that is not intended for sale to, or operation by, an end user but which may be associated with or operated for the benefit of a user (e.g., a smart power meter).

[0179] The UE QQ200 includes processing circuitry QQ202 that is operatively coupled via a bus QQ204 to an input / output interface QQ206, a power source QQ208, a memory QQ210, a communication interface QQ212, and / or any other component, or any combination thereof. Certain UEs may utilize all or a subset of the components shown in Figure QQ2. The level of integration between the components may vary from one UE to another UE. Further, certain UEs may contain multiple instances of a component, such as multiple processors, memories, transceivers, transmitters, receivers, etc.

[0180] The processing circuitry QQ202 is configured to process instructions and data and may be configured to implement any sequential state machine operative to execute instructions stored as machine-readable computer programs in the memory QQ210. The processing circuitry QQ202 may be implemented as one or more hardware-implemented state machines (e.g., in discrete logic, field-programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), etc.); programmable logic together with appropriate firmware; one or more stored computer programs, general-purpose processors, such as a microprocessor or digital signal processor (DSP), together with appropriate software; or any combination of the above. For example, the processing circuitry QQ202 may include multiple central processing units (CPUs).

[0181] In the example, the input / output interface QQ206 may be configured to provide an interface or interfaces to an input device, output device, or one or more input and / or output devices. Examples of an output device include a speaker, a sound card, a video card, a display, a monitor, a printer, an actuator, an emitter, a smartcard, another output device, or any combination thereof. An input device may allow a user to capture information into the UE QQ200. Examples of an input device include a touch-sensitive or presence-sensitive display, a camera (e.g., a digital camera, a digital video camera, a web camera, etc.), a microphone, a sensor, a mouse, a trackball, a directional pad, a trackpad, a scroll wheel, a smartcard, and the like. The presence-sensitive display may include a capacitive or resistive touch sensor to sense input from a user. A sensor may be, for instance, an accelerometer, a gyroscope, a tilt sensor, a force sensor, a magnetometer, an optical sensor, a proximity sensor, a biometric sensor, etc., or any combination thereof. An output device may use the same type of interface port as an input device. For example, a Universal Serial Bus (USB) port may be used to provide an input device and an output device.

[0182] In some embodiments, the power source QQ208 is structured as a battery or battery pack. Other types of power sources, such as an external power source (e.g., an electricity outlet), photovoltaic device, or power cell, may be used. The power source QQ208 may further include power circuitry for delivering power from the power source QQ208 itself, and / or an external power source, to the various parts of the UE QQ200 via input circuitry or an interface such as an electrical power cable. Delivering power may be, for example, for charging of the power source QQ208. Power circuitry may perform any formatting, converting, or other modification to the power from the power source QQ208 to make the power suitable for the respective components of the UE QQ200 to which power is supplied.

[0183] The memory QQ210 may be or be configured to include memory such as random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic disks, optical disks, hard disks, removable cartridges, flash drives, and so forth. In one example, the memory QQ210 includes one or more application programs QQ214, such as an operating system, web browser application, a widget, gadget engine, or other application, and corresponding data QQ216. The memory QQ210 may store, for use by the UE QQ200, any of a variety of various operating systems or combinations of operating systems.

[0184] The memory QQ210 may be configured to include a number of physical drive units, such as redundant array of independent disks (RAID), flash memory, USB flash drive, external hard disk drive, thumb drive, pen drive, key drive, high-density digital versatile disc (HD-DVD) optical disc drive, internal hard disk drive, Blu-Ray optical disc drive, holographic digital data storage (HDDS) optical disc drive, external mini-dual in-line memory module (DIMM), synchronous dynamic random access memory (SDRAM), external micro-DIMM SDRAM, smartcard memory such as tamper resistant module in the form of a universal integrated circuit card (UICC) including one or more subscriber identity modules (SIMs), such as a USIM and / or ISIM, other memory, or any combination thereof. The UICC may for example be an embedded UICC (eUlCC), integrated UICC (iUICC) or a removable UICC commonly known as ‘SIM card.’ The memory QQ210 may allow the UE QQ200 to access instructions, application programs and the like, stored on transitory or non-transitory memory media, to off-load data, or to upload data. An article of manufacture, such as one utilizing a communication system may be tangibly embodied as or in the memory QQ210, which may be or comprise a device-readable storage medium.

[0185] The processing circuitry QQ202 may be configured to communicate with an access network or other network using the communication interface QQ212. The communication interface QQ212 may comprise one or more communication subsystems and may include or be communicatively coupled to an antenna QQ222. The communication interface QQ212 may include one or more transceivers used to communicate, such as by communicating with one or more remote transceivers of another device capable of wireless communication (e.g., another UE or a network node in an access network). Each transceiver may include a transmitter QQ218 and / or a receiver QQ220 appropriate to provide network communications (e.g., optical, electrical, frequency allocations, and so forth). Moreover, the transmitter QQ218 and receiver QQ220 may be coupled to one or more antennas (e.g., antenna QQ222) and may share circuit components, software or firmware, or alternatively be implemented separately.

[0186] In the illustrated embodiment, communication functions of the communication interface QQ212 may include cellular communication, Wi-Fi communication, LPWAN communication, data communication, voice communication, multimedia communication, short-range communications such as Bluetooth, near-field communication, location-based communication such as the use of the global positioning system (GPS) to determine a location, another like communication function, or any combination thereof. Communications may be implemented in according to one or more communication protocols and / or standards, such as IEEE 802.11, Code Division Multiplexing Access (CDMA), Wideband Code Division Multiple Access (WCDMA), GSM, LTE, New Radio (NR), UMTS, WiMax, Ethernet, transmission control protocol / internet protocol (TCP / IP), synchronous optical networking (SONET), Asynchronous Transfer Mode (ATM), QUIC, Hypertext Transfer Protocol (HTTP), and so forth.

[0187] Regardless of the type of sensor, a UE may provide an output of data captured by its sensors, through its communication interface QQ212, via a wireless connection to a network node. Data captured by sensors of a UE can be communicated through a wireless connection to a network node via another UE. The output may be periodic (e.g., once every 15 minutes if it reports the sensed temperature), random (e.g., to even out the load from reporting from several sensors), in response to a triggering event (e.g., when moisture is detected an alert is sent), in response to a request (e.g., a user initiated request), or a continuous stream (e.g., a live video feed of a patient).

[0188] As another example, a UE comprises an actuator, a motor, or a switch, related to a communication interface configured to receive wireless input from a network node via a wireless connection. In response to the received wireless input the states of the actuator, the motor, or the switch may change. For example, the UE may comprise a motor that adjusts the control surfaces or rotors of a drone in flight according to the received input or to a robotic arm performing a medical procedure according to the received input.

[0189] A UE, when in the form of an Internet of Things (loT) device, may be a device for use in one or more application domains, these domains comprising, but not limited to, city wearable technology, extended industrial application and healthcare. Non-limiting examples of such an loT device are a device which is or which is embedded in: a connected refrigerator or freezer, a TV, a connected lighting device, an electricity meter, a robot vacuum cleaner, a voice controlled smart speaker, a home security camera, a motion detector, a thermostat, a smoke detector, a door / window sensor, a flood / moisture sensor, an electrical door lock, a connected doorbell, an air conditioning system like a heat pump, an autonomous vehicle, a surveillance system, a weather monitoring device, a vehicle parking monitoring device, an electric vehicle charging station, a smart watch, a fitness tracker, a wearable for tactile augmentation or sensory enhancement, a water sprinkler, an animal- or item-tracking device, a sensor for monitoring a plant or animal, an industrial robot, an Unmanned Aerial Vehicle (UAV), and any kind of medical device, like a heart rate monitor or a remote controlled surgical robot. A UE in the form of an loT device comprises circuitry and / or software in dependence of the intended application of the loT device in addition to other components as described in relation to the UE QQ200 shown in Figure QQ2.

[0190] As yet another specific example, in an loT scenario, a UE may represent a machine or other device that performs monitoring and / or measurements, and transmits the results of such monitoring and / or measurements to another UE and / or a network node. The UE may in this case be an M2M device, which may in a 3GPP context be referred to as an MTC device. As one particular example, the UE may implement the 3GPP NB-loT standard. In other scenarios, a UE may represent a vehicle, such as a car, a bus, a truck, a ship and an airplane, or other equipment that is capable of monitoring and / or reporting on its operational status or other functions associated with its operation.

[0191] In practice, any number of UEs may be used together with respect to a single use case. For example, a first UE might be or be integrated in a drone and provide the drone’s speed information (obtained through a speed sensor) to a second UE that is a remote controller operating the drone. When the user makes changes from the remote controller, the first UE may adjust the throttle on the drone (e.g. by controlling an actuator) to increase or decrease the drone’s speed. The first and / or the second UE can also include more than one of the functionalities described above. For example, a UE might comprise the sensor and the actuator, and handle communication of data for both the speed sensor and the actuators.

[0192] Figure 9 shows a network node QQ300 in accordance with some embodiments. As used herein, network node refers to equipment capable, configured, arranged and / or operable to communicate directly or indirectly with a UE and / or with other network nodes or equipment, in a telecommunication network. Examples of network nodes include, but are not limited to, access points (APs) (e.g., radio access points), base stations (BSs) (e.g., radio base stations, Node Bs, evolved Node Bs (eNBs) and NR NodeBs (gNBs)), O- RAN nodes or components of an O-RAN node (e.g., O-RU, O-DU, O-CU).

[0193] Base stations may be categorized based on the amount of coverage they provide (or, stated differently, their transmit power level) and so, depending on the provided amount of coverage, may be referred to as femto base stations, pico base stations, micro base stations, or macro base stations. A base station may be a relay node or a relay donor node controlling a relay. A network node may also include one or more (or all) parts of a distributed radio base station such as centralized digital units, distributed units (e.g., in an O-RAN access node) and / or remote radio units (RRUs), sometimes referred to as Remote Radio Heads (RRHs). Such remote radio units may or may not be integrated with an antenna as an antenna integrated radio. Parts of a distributed radio base station may also be referred to as nodes in a distributed antenna system (DAS).

[0194] Other examples of network nodes include multiple transmission point (multi-TRP) 5G access nodes, multi-standard radio (MSR) equipment such as MSR BSs, network controllers such as radio network controllers (RNCs) or base station controllers (BSCs), base transceiver stations (BTSs), transmission points, transmission nodes, multi- cell / multicast coordination entities (MCEs), Operation and Maintenance (O&M) nodes, Operations Support System (OSS) nodes, Self-Organizing Network (SON) nodes, positioning nodes (e.g., Evolved Serving Mobile Location Centers (E-SMLCs)), and / or Minimization of Drive Tests (MDTs).

[0195] The network node QQ300 includes a processing circuitry QQ302, a memory QQ304, a communication interface QQ306, and a power source QQ308. The network node QQ300 may be composed of multiple physically separate components (e.g., a NodeB component and a RNC component, or a BTS component and a BSC component, etc.), which may each have their own respective components. In certain scenarios in which the network node QQ300 comprises multiple separate components (e.g., BTS and BSC components), one or more of the separate components may be shared among several network nodes. For example, a single RNC may control multiple NodeBs. In such a scenario, each unique NodeB and RNC pair, may in some instances be considered a single separate network node. In some embodiments, the network node QQ300 may be configured to support multiple radio access technologies (RATs). In such embodiments, some components may be duplicated (e.g., separate memory QQ304 for different RATs) and some components may be reused (e.g., a same antenna QQ310 may be shared by different RATs). The network node QQ300 may also include multiple sets of the various illustrated components for different wireless technologies integrated into network node QQ300, for example GSM, WCDMA, LTE, NR, WiFi, Zigbee, Z-wave, LoRaWAN, Radio Frequency Identification (RFID) or Bluetooth wireless technologies. These wireless technologies may be integrated into the same or different chip or set of chips and other components within network node QQ300.

[0196] The processing circuitry QQ302 may comprise a combination of one or more of a microprocessor, controller, microcontroller, central processing unit, digital signal processor, application-specific integrated circuit, field programmable gate array, or any other suitable computing device, resource, or combination of hardware, software and / or encoded logic operable to provide, either alone or in conjunction with other network node QQ300 components, such as the memory QQ304, to provide network node QQ300 functionality.

[0197] In some embodiments, the processing circuitry QQ302 includes a system on a chip (SOC). In some embodiments, the processing circuitry QQ302 includes one or more of radio frequency (RF) transceiver circuitry QQ312 and baseband processing circuitry QQ314. In some embodiments, the radio frequency (RF) transceiver circuitry QQ312 and the baseband processing circuitry QQ314 may be on separate chips (or sets of chips), boards, or units, such as radio units and digital units. In alternative embodiments, part or all of RF transceiver circuitry QQ312 and baseband processing circuitry QQ314 may be on the same chip or set of chips, boards, or units.

[0198] The memory QQ304 may comprise any form of volatile or non-volatile computer- readable memory including, without limitation, persistent storage, solid-state memory, remotely mounted memory, magnetic media, optical media, random access memory (RAM), read-only memory (ROM), mass storage media (for example, a hard disk), removable storage media (for example, a flash drive, a Compact Disk (CD) or a Digital Video Disk (DVD)), and / or any other volatile or non-volatile, non-transitory device- readable and / or computer-executable memory devices that store information, data, and / or instructions that may be used by the processing circuitry QQ302. The memory QQ304 may store any suitable instructions, data, or information, including a computer program, software, an application including one or more of logic, rules, code, tables, and / or other instructions capable of being executed by the processing circuitry QQ302 and utilized by the network node QQ300. The memory QQ304 may be used to store any calculations made by the processing circuitry QQ302 and / or any data received via the communication interface QQ306. In some embodiments, the processing circuitry QQ302 and memory QQ304 is integrated.

[0199] The communication interface QQ306 is used in wired or wireless communication of signaling and / or data between a network node, access network, and / or UE. As illustrated, the communication interface QQ306 comprises port(s) / terminal(s) QQ316 to send and receive data, for example to and from a network over a wired connection. The communication interface QQ306 also includes radio front-end circuitry QQ318 that may be coupled to, or in certain embodiments a part of, the antenna QQ310. Radio front-end circuitry QQ318 comprises filters QQ320 and amplifiers QQ322. The radio front-end circuitry QQ318 may be connected to an antenna QQ310 and processing circuitry QQ302. The radio front-end circuitry may be configured to condition signals communicated between antenna QQ310 and processing circuitry QQ302. The radio front-end circuitry QQ318 may receive digital data that is to be sent out to other network nodes or UEs via a wireless connection. The radio front-end circuitry QQ318 may convert the digital data into a radio signal having the appropriate channel and bandwidth parameters using a combination of filters QQ320 and / or amplifiers QQ322. The radio signal may then be transmitted via the antenna QQ310. Similarly, when receiving data, the antenna QQ310 may collect radio signals which are then converted into digital data by the radio front-end circuitry QQ318. The digital data may be passed to the processing circuitry QQ302. In other embodiments, the communication interface may comprise different components and / or different combinations of components.

[0200] In certain alternative embodiments, the network node QQ300 does not include separate radio front-end circuitry QQ318, instead, the processing circuitry QQ302 includes radio front-end circuitry and is connected to the antenna QQ310. Similarly, in some embodiments, all or some of the RF transceiver circuitry QQ312 is part of the communication interface QQ306. In still other embodiments, the communication interface QQ306 includes one or more ports or terminals QQ316, the radio front-end circuitry QQ318, and the RF transceiver circuitry QQ312, as part of a radio unit (not shown), and the communication interface QQ306 communicates with the baseband processing circuitry QQ314, which is part of a digital unit (not shown).

[0201] The antenna QQ310 may include one or more antennas, or antenna arrays, configured to send and / or receive wireless signals. The antenna QQ310 may be coupled to the radio front-end circuitry QQ318 and may be any type of antenna capable of transmitting and receiving data and / or signals wirelessly. In certain embodiments, the antenna QQ310 is separate from the network node QQ300 and connectable to the network node QQ300 through an interface or port.

[0202] The antenna QQ310, communication interface QQ306, and / or the processing circuitry QQ302 may be configured to perform any receiving operations and / or certain obtaining operations described herein as being performed by the network node. Any information, data and / or signals may be received from a UE, another network node and / or any other network equipment. Similarly, the antenna QQ310, the communication interface QQ306, and / or the processing circuitry QQ302 may be configured to perform any transmitting operations described herein as being performed by the network node. Any information, data and / or signals may be transmitted to a UE, another network node and / or any other network equipment.

[0203] The power source QQ308 provides power to the various components of network node QQ300 in a form suitable for the respective components (e.g., at a voltage and current level needed for each respective component). The power source QQ308 may further comprise, or be coupled to, power management circuitry to supply the components of the network node QQ300 with power for performing the functionality described herein. For example, the network node QQ300 may be connectable to an external power source (e.g., the power grid, an electricity outlet) via an input circuitry or interface such as an electrical cable, whereby the external power source supplies power to power circuitry of the power source QQ308. As a further example, the power source QQ308 may comprise a source of power in the form of a battery or battery pack which is connected to, or integrated in, power circuitry. The battery may provide backup power should the external power source fail.

[0204] Embodiments of the network node QQ300 may include additional components beyond those shown in Figure 9 for providing certain aspects of the network node’s functionality, including any of the functionality described herein and / or any functionality necessary to support the subject matter described herein. For example, the network node QQ300 may include user interface equipment to allow input of information into the network node QQ300 and to allow output of information from the network node QQ300. This may allow a user to perform diagnostic, maintenance, repair, and other administrative functions for the network node QQ300. In some embodiments providing a core network node, such as core network node 108 of FIG. QQ1, some components, such as the radio front-end circuitry QQ318 and the RF transceiver circuitry QQ312 may be omitted.

[0205] Figure 10 is a block diagram illustrating a virtualization environment QQ400 in which functions implemented by some embodiments may be virtualized. In the present context, virtualizing means creating virtual versions of apparatuses or devices which may include virtualizing hardware platforms, storage devices and networking resources. As used herein, virtualization can be applied to any device described herein, or components thereof, and relates to an implementation in which at least a portion of the functionality is implemented as one or more virtual components. Some or all of the functions described herein may be implemented as virtual components executed by one or more virtual machines (VMs) implemented in one or more virtual environments QQ400 hosted by one or more of hardware nodes, such as a hardware computing device that operates as a network node, UE, core network node, or host. Further, in embodiments in which the virtual node does not require radio connectivity (e.g., a core network node or host), then the node may be entirely virtualized. In some embodiments, the virtualization environment QQ400 includes components defined by the O-RAN Alliance, such as an O-Cloud environment orchestrated by a Service Management and Orchestration Framework via an 0-2 interface. Virtualization may facilitate distributed implementations of a network node, UE, core network node, or host.

[0206] Applications QQ402 (which may alternatively be called software instances, virtual appliances, network functions, virtual nodes, virtual network functions, etc.) are run in the virtualization environment Q400 to implement some of the features, functions, and / or benefits of some of the embodiments disclosed herein.

[0207] Hardware QQ404 includes processing circuitry, memory that stores software and / or instructions executable by hardware processing circuitry, and / or other hardware devices as described herein, such as a network interface, input / output interface, and so forth. Software may be executed by the processing circuitry to instantiate one or more virtualization layers QQ406 (also referred to as hypervisors or virtual machine monitors (VMMs)), provide VMs QQ408a and QQ408b (one or more of which may be generally referred to as VMs QQ408), and / or perform any of the functions, features and / or benefits described in relation with some embodiments described herein. The virtualization layer QQ406 may present a virtual operating platform that appears like networking hardware to the VMs QQ408.

[0208] The VMs QQ408 comprise virtual processing, virtual memory, virtual networking or interface and virtual storage, and may be run by a corresponding virtualization layer QQ406. Different embodiments of the instance of a virtual appliance QQ402 may be implemented on one or more of VMs QQ408, and the implementations may be made in different ways. Virtualization of the hardware is in some contexts referred to as network function virtualization (NFV). NFV may be used to consolidate many network equipment types onto industry standard high volume server hardware, physical switches, and physical storage, which can be located in data centers, and customer premise equipment.

[0209] In the context of NFV, a VM QQ408 may be a software implementation of a physical machine that runs programs as if they were executing on a physical, non-virtualized machine. Each of the VMs QQ408, and that part of hardware QQ404 that executes that VM, be it hardware dedicated to that VM and / or hardware shared by that VM with others of the VMs, forms separate virtual network elements. Still in the context of NFV, a virtual network function is responsible for handling specific network functions that run in one or more VMs QQ408 on top of the hardware QQ404 and corresponds to the application QQ402.

[0210] Hardware QQ404 may be implemented in a standalone network node with generic or specific components. Hardware QQ404 may implement some functions via virtualization. Alternatively, hardware QQ404 may be part of a larger cluster of hardware (e.g. such as in a data center or CPE) where many hardware nodes work together and are managed via management and orchestration QQ410, which, among others, oversees lifecycle management of applications QQ402. In some embodiments, hardware QQ404 is coupled to one or more radio units that each include one or more transmitters and one or more receivers that may be coupled to one or more antennas. Radio units may communicate directly with other hardware nodes via one or more appropriate network interfaces and may be used in combination with the virtual components to provide a virtual node with radio capabilities, such as a radio access node or a base station. In some embodiments, some signaling can be provided with the use of a control system QQ412 which may alternatively be used for communication between hardware nodes and radio units.

[0211] Although the computing devices described herein (e.g., UEs, network nodes) may include the illustrated combination of hardware components, other embodiments may comprise computing devices with different combinations of components. It is to be understood that these computing devices may comprise any suitable combination of hardware and / or software needed to perform the tasks, features, functions and methods disclosed herein. Determining, calculating, obtaining or similar operations described herein may be performed by processing circuitry, which may process information by, for example, converting the obtained information into other information, comparing the obtained information or converted information to information stored in the network node, and / or performing one or more operations based on the obtained information or converted information, and as a result of said processing making a determination. Moreover, while components are depicted as single boxes located within a larger box, or nested within multiple boxes, in practice, computing devices may comprise multiple different physical components that make up a single illustrated component, and functionality may be partitioned between separate components. For example, a communication interface may be configured to include any of the components described herein, and / or the functionality of the components may be partitioned between the processing circuitry and the communication interface. In another example, non-computationally intensive functions of any of such components may be implemented in software or firmware and computationally intensive functions may be implemented in hardware.

[0212] In certain embodiments, some or all of the functionality described herein may be provided by processing circuitry executing instructions stored on in memory, which in certain embodiments may be a computer program product in the form of a non-transitory computer-readable storage medium. In alternative embodiments, some or all of the functionality may be provided by the processing circuitry without executing instructions stored on a separate or discrete device-readable storage medium, such as in a hard-wired manner. In any of those particular embodiments, whether executing instructions stored on a non-transitory computer-readable storage medium or not, the processing circuitry can be configured to perform the described functionality. The benefits provided by such functionality are not limited to the processing circuitry alone or to other components of the computing device but are enjoyed by the computing device as a whole, and / or by end users and a wireless network generally.

[0213] When using the word "comprise" or “comprising” it shall be interpreted as nonlimiting, i.e. meaning "consist at least of".

[0214] The embodiments herein are not limited to the preferred embodiments described above. Various alternatives, modifications and equivalents may be used.

[0215] Abbreviations

[0216] PS Packet Switch Domain

[0217] CS GSM / WCDMA Circuit Switched (CS)

[0218] LTE Long Term Evolution

[0219] 5G NR 5G Ny Radio

[0220] ASR Abort Session Request

[0221] ASA Abort Session Answer

[0222] MTAS Multimedia Telephony Application Server

[0223] EPS FB Evolved Packet System Fallback

[0224] IMS IP Multimedia Subsystem (IMS) telephony service engine

[0225] ECMP Equal Cost Multi Path

[0226] MPLS Multi Protocol Label Switching

[0227] VPN Virtual Private Network

[0228] EVPN Ethernet Virtual Private Network

[0229] LSP Label Switched Path

[0230] NLRI Network Layer Reachability Information

[0231] PE Provider Edge router

[0232] SID Segment Identifier

[0233] BGP Border Gateway Protocol

Claims

CLAIMS1. A method to establish an optimized frame format used by a first compute node acting as ingress PE and a second compute node acting as egress PE in the transport of overlay ethernet traffic containing an IP payload from the source overlay endpoint attached to or hosted in the first compute node to the destination overlay endpoint attached to or hosted in the second compute node through a Multi Protocol Label Switching, MPLS, underlay network, where the frame format is a conventional MPLS tagged ethernet frame format, wherein an MPLS label stack is inserted between the ethernet header and the IP header of the overlay ethernet frame, where the single ethernet header carries the destination and source overlay Media Access Control, MAC, addresses when the frame was originated from the source overlay endpoint, where the MPLS header of the frame comprises: a top label with the Label Switch Path, LSP, label towards the destination Provider Edge, PE, router, i.e. egress PE, and a bottom label, e.g. an optimization MPLS label, holding information enabling the forwarding mechanism in the egress PE router to restore the original destination and source overlay MAC addresses.

2. A method performed by an ingress PE router to establish a route between the ingress PE router and an egress PE router, wherein the frame format according to claim 1 is used in the transport of overlay ethernet traffic, the method comprising: receiving, from an egress PE router, in a type 2 MAC / IP route advertisement message through BGP protocol, an Ethernet Virtual Private Network, EVPN, Network Layer Reachability Information, NLRI, and an optimization MPLS label where an overlay destination endpoint is attached; and creating forwarding rules to send overlay layer 2 traffic with unicast IP payload destined to the overlay destination endpoint based on the received EVPN NLRI and an optimization MPLS label.

3. A method performed by an egress PE router to establish a route between the egress PE router and an ingress PE router, wherein the frame format according to claim 1 is used in the transport of overlay ethernet traffic, the method comprising:- advertising an optimization MPLS label in a type 2 MAC / IP route advertisement message through BGP protocol for an attached overlay endpoint; andcreating the forwarding rules to receive and process the overlay traffic sent to the mentioned overlay endpoint by other remote overlay endpoints in the optimized frame format.

4. A method performed by an egress PE router for the transport of overlay ethernet traffic through a route according to claim 2 and 3, where a frame is routed through the MPLS undelay network towards the egress PE router based the top label, the method comprising:- restoring, by the forwarding mechanism in the egress PE router, the original overlay ethernet header MAC addresses in the ethernet header of the frame using the bottom label, being the optimization label, and- delivering the frame to the corresponding attached destination overlay endpoint.

5. A first compute node, hosting or running overlay endpoints virtual machines and / or containers and adapted to use a frame format according to claim 1 in a cloud overlay networking system, the first compute node being adapted to function as ingress PE, and the first compute node comprising:- a virtual switch and router component adapted to function as an ingress PE router forwarding plane function;- a Border Gateway Protocol, BGP, function adapted to function as an ingress PE control plane function; and- a virtual switch and router function adapted to function as an egress PE router.

6. A second compute node, hosting or running overlay endpoints virtual machines and / or containers and adapted to use a frame format according to claim 1 or 2 in a cloud overlay networking system, the second compute node being adapted to function as egress PE, and the second compute node comprising:- a virtual switch and router component adapted to function as an egress PE router forwarding plane function;- a BGP function adapted to function as an egress PE control plane function; and- a virtual switch and router function adapted to function as an ingress PE router.

7. A first computer program comprising instructions, which when executed by a processor (12) belonging to a first compute node, causes the processor (12) to perform actions according to claim 1 or 2.

8. A second computer program comprising instructions, which when executed by a processor (12) belonging to a second compute node, causes the processor (12) to perform actions according to any of claims 1, 3 or 4.

9. A carrier comprising the computer program of claim 7 or 8, wherein the carrier is one of an electronic signal, an optical signal, an electromagnetic signal, a magnetic signal, an electric signal, a radio signal, a microwave signal, or a computer-readable storage medium.

Images