Distributed user plane functions for wireless networks - Patents.com

The distributed implementation of UPF components within the RAN and cloud-native microservices architecture in 5G networks addresses latency and failure issues, enhancing bandwidth and availability for advanced applications.

JP7584681B2Active Publication Date: 2024-11-15AMAZON TECH INC
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
JP2023572884
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2021-05-26
Filing Date
2022-05-10
Publication Date
2024-11-15
Estimated Expiration
2042-05-10

AI Technical Summary

Technical Problem

Existing 5G wireless networks face issues with latency, bandwidth overhead, and single points of failure due to the encapsulation of network traffic using GPRS Tunneling Protocol (GTP), which increases packet size and causes fragmentation, and the centralized implementation of User Plane Functions (UPF) leading to potential communication disruptions if a single UPF fails.

Method used

Implementing a distributed architecture where the user plane components of UPF are placed within the Radio Access Network (RAN) while maintaining the control plane components in the core network, avoiding GTP encapsulation, and utilizing a cloud-native, microservices-based approach for network functions to enhance flexibility, reduce latency, and ensure high availability.

Benefits of technology

This solution reduces latency, improves bandwidth efficiency, and eliminates single points of failure by distributing UPF components, enabling seamless communication and supporting advanced applications like factory IoT, augmented reality, and autonomous navigation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007584681000001
    Figure 0007584681000001
  • Figure 0007584681000002
    Figure 0007584681000002
  • Figure 0007584681000003
    Figure 0007584681000003
Patent Text Reader

Abstract

Various embodiments are disclosed for providing a distributed user plane function in a wireless network. In one embodiment, data is received from a client device connected to a radio access network of the wireless network. The data is forwarded to a user plane component of the user plane function. The user plane component is implemented in a computing device of the radio access network. The data is processed by the user plane component. The processed data is then routed to a data network.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to and the benefit of co-pending U.S. patent application Ser. No. 17 / 331,327, filed May 26, 2021, entitled "DISTRIBUTED USER PLANE FUNCTIONS FOR RADIO-BASED NETWORKS," which is incorporated by reference in its entirety as if set forth herein. [Background technology]

[0002] 5G is the fifth generation technology standard for broadband cellular networks and is planned to eventually replace the fourth generation (4G) standard of Long-Term Evolution (LTE). 5G technology will provide significantly increased bandwidth, thereby expanding the cellular market beyond smartphones to provide last-mile connectivity to desktops, set-top boxes, laptops, Internet of Things (IoT) devices, and more. Some 5G cells may use a frequency spectrum similar to 4G, while other 5G cells may use the millimeter wave frequency spectrum. Millimeter wave cells will have a smaller coverage area but will provide much higher throughput than 4G. Summary of the Invention

[0003] Many aspects of the present disclosure can be better understood with reference to the following drawings. The components in the drawings are not necessarily to scale, emphasis instead being placed upon clearly illustrating the principles of the present disclosure. Moreover, in the drawings, like reference numerals designate corresponding parts throughout the several views. [Brief description of the drawings]

[0004] [Figure 1A] 1 is a drawing of an example communications network that may be deployed and managed in accordance with various embodiments of the present disclosure. [Figure 1B] 1 is a depiction of an example radio access network protocol stack for use in various embodiments of the present disclosure. [Figure 1C] 1 is a diagram illustrating an example data flow in a wireless network, according to various embodiments of the present disclosure. [Figure 2A] 1A illustrates an example of a network environment including a cloud provider network and further including various provider substrate extensions of the cloud provider network that may be used at various locations within the communications network of FIG. 1A, in accordance with some embodiments of the present disclosure. [Figure 2B] 1B illustrates an example of cellularization and geographic distribution of the communication network of FIG. 1A. [Figure 3A] 2B illustrates an example of the network environment of FIG. 2A including geographically distributed provider substrate extensions in accordance with some embodiments of the present disclosure. [Figure 3B] 1 illustrates an example networked environment for a distributed User Plane Function (UPF), in accordance with various embodiments. [Figure 4] 2B is a schematic block diagram of the network environment of FIG. 2A in accordance with various embodiments of the present disclosure. [Diagram 5] 2C is a flowchart illustrating an example of functionality implemented as part of a distribution unit (DU) / centralized unit (CU) executing in computing hardware from the network environment of FIG. 2B according to various embodiments of the present disclosure. [Figure 6] 5 is a flowchart illustrating an example of functionality implemented as part of a UPF controller executing in a computing environment within the network environment of FIG. 4 in accordance with various embodiments of the present disclosure. [Figure 7] 5 is a schematic block diagram providing an illustration of an example computing environment for use in the network environment of FIG. 4, in accordance with various embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0005] The present disclosure relates to implementing a distributed User Plane Function (UPF) in wireless networks, such as 4G and 5G radio access networks, or parts of such wireless networks, and associated core networks using cloud provider network infrastructure. The UPF is the interconnection point between mobile infrastructure and data networks, and the UPF acts as a protocol data unit session anchor point to provide mobility within and between different radio access networks. The UPF runs to apply policies to customer network traffic, and then forwards the network traffic via the user data plane as required. The UPF is similar to a router in that network traffic passes through it, rather than being terminated there. The UPF also performs application discovery, implements network slicing and quality of service requirements, and monitors traffic for billing purposes.

[0006] Internet Protocol (IP) network traffic between the Radio Access Network (RAN) and the UPF of the core network is typically encapsulated using the GPRS Tunneling Protocol (GTP). GTP adds an outer IP header (20 bytes in IPv4), a UDP header (8 bytes in IPv4), and a GPRS header (16 bytes in IPv4). This increases the packet size by almost 3 percent when the maximum transmission unit (MTU) is 1500 bytes. Using GTP increases latency not only from the encapsulation and decapsulation functions, but also from the potential for fragmentation to occur because the GTP headers drive the packet size beyond the MTU limit.

[0007] The UPF also addresses potential traffic bottlenecks and single points of failure in radio-based networks. In some embodiments, the UPF must scale to handle up to millions of IP addresses connected to the Internet, which may be contiguous sub-networks routed to the same destination. In some embodiments, all network traffic for a particular end user device must pass through a single UPF, and thousands of user devices may be mapped to individual UPFs. If the UPF of the core network fails, user devices in the RAN will not be able to communicate, even if the data link between the RAN and the core network is still functional.

[0008] Various embodiments of the present disclosure introduce a distributed implementation of the UPF that places the user plane component of the UPF inside the RAN, as opposed to conventional implementations where the UPF is implemented entirely in the core network. By placing the user plane component of the UPF in the RAN, encapsulation of network traffic using GTP can be avoided. In some embodiments, the control plane component of the UPF remains in the core network and functions in parallel with the user plane component of the UPF in the RAN. Furthermore, the user plane component of the UPF can continue to run in the core network to maintain backward compatibility and interoperability with RANs that do not support the distributed architecture. The user plane component of the UPF in the core network can also be used for scenarios including, for example, handovers, resource-constrained RANs, etc.

[0009] As one skilled in the art will appreciate in light of this disclosure, certain embodiments may achieve certain advantages, including some or all of the following: (1) improving the availability of a communications network by distributing the UPF so that it is not a potential single point of failure; (2) improving the flexibility of a communications network by allowing user plane components of the UPF to be implemented inside small cells or other RAN side equipment; (3) improving the operation of a communications network by reducing the latency and bandwidth overhead associated with GTP encapsulation and decapsulation and the resulting fragmentation of packets; (4) enabling communication between devices in the RAN without the need to route network traffic to a UPF in the core network, thereby reducing latency and improving bandwidth, and the like.

[0010] Among the advantages of the present disclosure is the ability of a cellularized control plane, which controls the operation of a wireless network that operates at least in part using a cloud provider infrastructure, to deploy and chain network functions across different physical sites, as well as manage failover between the physical sites to provide a distributed UPF. Cellularizing the control plane refers to having multiple independently operable copies of the control plane (called "cells") so that failure of an individual control plane does not affect all deployments. According to the present disclosure, network functions organized into microservices work together to provide end-to-end connectivity (sometimes called a "network function stack"). One set of network functions is part of the wireless network and operates within base stations, performing radio signal to IP conversion. Other network functions run in large data centers that perform subscriber-related business logic and route IP traffic to and from the Internet. For applications to use the new features of 5G, such as low latency communications and reserved bandwidth, both of these types of network functions need to work together to properly schedule and reserve the radio spectrum, and perform real-time computation and data processing.

[0011] The techniques disclosed herein provide edge location hardware (described further below) integrated with network functions that run across the entire network, from cell sites to internet breakouts, and orchestrate the network functions across these sites to provide high availability. Specifically, within each control plane infrastructure cell, multiple redundant network function stacks can be provisioned, and the control plane cell can shift traffic to the secondary stack as needed to provide the required availability. These redundant network function stacks can be provisioned across different or combinations of edge locations, customer data centers, and cloud provider availability zones. This enables an entirely new set of applications with stringent quality of service (QoS) requirements that were previously not possible to run on mobile networks, ranging from factory-based IoT to augmented reality (AR), virtual reality (VR), game streaming, and autonomous navigation support for connected vehicles.

[0012] The described "Elastic 5G" service provides and manages all the hardware, software, and network functions required to build the network and can orchestrate the network functions across various physical sites as described herein. In some embodiments, the network functions can be developed and managed by a cloud service provider, but the described control plane can manage the network functions across various providers so that customers can use a single set of APIs to invoke and manage selected network functions on the cloud infrastructure. The Elastic 5G service has the advantage of automating the creation of an end-to-end 5G network, from hardware to network functions, resulting in reduced deployment time and operational costs for network operations. By providing APIs that expose the network functions, the disclosed Elastic 5G service allows applications to specify the desired QoS simply as constraints, and then deploys and chains the network functions to deliver an end-to-end network slice that reflects the network characteristics required by the software application, while managing failover to provide the availability level required by the software application.

[0013] This disclosure describes embodiments related to creating and managing cloud-native 5G core and / or cloud-native 5G RAN and associated control plane components. Cloud-native refers to an approach to building and running applications that leverage the benefits of the cloud computing delivery model, such as dynamic scalability, distributed computing, and high availability (including geographic distribution, redundancy, and failover). Cloud-native refers to the way these applications are created and deployed to make them suitable for deployment in the public cloud. Cloud-native applications can (and often do) run in the public cloud, but can also run in on-premise data centers. Some cloud-native applications can be containerized, e.g., different parts, functions, or subunits of an application are packaged in their own containers, which can be dynamically orchestrated such that each part is actively scheduled and managed to optimize resource utilization. These containerized applications can be built using a microservices architecture to improve the overall agility and maintainability of the application. In a microservices architecture, an application is arranged as a collection of smaller subunits ("microservices") that can be deployed and scaled independently of each other and communicate with each other over a network. These microservices usually have a particular technical and functional granularity, and often implement lightweight communication protocols, making them fine-grained. An application's microservices can perform different functions, may be independently deployable, and may use different programming languages, databases, and hardware / software environments.Decomposing an application into smaller services beneficially increases the modularity of the application, allows individual microservices to be replaced as needed, and parallelizes development by allowing teams to develop, deploy, and maintain the microservices independently of one another. Microservices may, in some examples, be deployed using virtual machines, containers, or serverless functions. The disclosed core and RAN software may follow a microservices architecture such that the described wireless network is composed of independent sub-units that can be deployed and scaled on demand.

[0014] Referring now to FIG. 1A, an example of a communications network 100 deployed and managed in accordance with various embodiments of the present disclosure is shown. The communications network 100 includes a wireless network 103 that may correspond to a cellular network, such as a fourth generation (4G) Long-Term Evolution (LTE) network, a fifth generation (5G) network, a 4G-5G hybrid core with both 4G and 5G RAN, or another network that provides wireless network access. The wireless network 103 may be operated by a public telecommunications provider or a cloud service provider of an enterprise or other organization. Various deployments of the wireless network 103 may include one or more of a core network and a RAN network, as well as a control plane for running the core network and / or the RAN network on a cloud provider infrastructure. As described above, these components may be developed in a cloud-native manner, such as using a microservices architecture, such that traffic and transactions can be efficiently scaled using centralized control and distributed processing. These components may be based on 3GPP specifications by following an application architecture with separated control and user plane processing (CUPS architecture).

[0015] The wireless network 103 provides wireless network access to a number of wireless devices 106, which may be mobile devices or fixed location devices. In various examples, the wireless devices 106 may include devices such as smartphones, connected vehicles, IoT devices, sensors, machines (such as in a manufacturing facility), hotspots, etc. The wireless devices 106 may also be referred to as user equipment (UE) or customer premises equipment (CPE).

[0016] The wireless network 103 may include a RAN that provides wireless network access to multiple wireless devices 106 through multiple cells 109. Each of the cells 109 may be equipped with one or more antennas and one or more radio units (RUs) that transmit and receive wireless data signals to and from the wireless devices 106. The antennas may be configured for one or more frequency bands, and the radio units may also be frequency agile or frequency tunable. To focus the signal in a particular direction or azimuth range, the antennas may be associated with a particular gain or beamwidth, which may allow for frequency reuse in different directions. Additionally, the antennas may be horizontally, vertically, or circularly polarized. In some examples, the radio units may utilize multiple-input multiple-output (MIMO) technology to transmit and receive signals. Thus, the RAN implements radio access technologies that enable wireless connections with the wireless devices 106 and provides connectivity to the core network of the wireless network. Components of the RAN include base stations and antennas that cover a given physical area, as well as the necessary core network items to manage connections to the RAN.

[0017] Data traffic is routed to the core network through a fiber transport network, often consisting of multiple hops of Layer 3 routers (e.g., at aggregation sites). The core network is typically housed in one or more data centers. Typically, the core network aggregates data traffic from end devices, authenticates subscribers and devices, applies personalized policies, manages device mobility, and then routes traffic to operator services or the Internet. For example, the 5G core can be decomposed into several microservice elements, separating the control plane and the user plane. The 5G core can include virtualized software-based (e.g., deployed as microservices) network functions rather than physical network elements, and thus can be instantiated within a multi-access edge computing (MEC) cloud infrastructure. The network functions of the core network can include a user plane function (UPF), an access and mobility management function (AMF), and a session management function (SMF), which are described in more detail below. For data traffic destined for locations external to the communications network 100, the network functions typically include firewalls to external networks, such as the Internet or cloud provider networks, through which traffic can enter or exit the communications network 100. Note that in some embodiments, the communications network 100 can include facilities that allow traffic to enter or exit sites further downstream from the core network (e.g., aggregation sites or wireless network 103).

[0018] The UPF provides an interconnection point between the mobile infrastructure and the data network (DN), i.e., General Packet Radio Service (GPRS) Tunneling Protocol for User Plane (GTP-U) encapsulation and decapsulation. The UPF may also provide a session anchor point to provide mobility within the RAN, such as sending one or more end marker packets to the RAN base station. The UPF may also handle packet routing and forwarding, such as steering flows to specific data networks based on traffic match filters. Another function of the UPF includes per-flow or per-application QoS handling, such as uplink (UL) and downlink (DL) transport-level packet marking and rate limiting. The UPF can be implemented as a cloud-native network function using modern microservices techniques, for example, deployed within a serverless framework (which abstracts the underlying infrastructure on which the code runs via managed services).

[0019] The AMF may receive connection and session information from the wireless device 106 or the RAN and may handle connection and mobility management tasks. For example, the AMF may manage handovers between base stations within the RAN. In some examples, the AMF may be considered an access point to the 5G core by terminating traffic for a particular RAN control plane and the wireless device 106. The AMF may also implement encryption and integrity protection algorithms.

[0020] The SMF may handle session establishment or modification, for example by creating, updating, and deleting Protocol Data Unit (PDU) sessions and managing session context within the UPF. The SMF may also implement Dynamic Host Configuration Protocol (DHCP) and IP Address Management (IPAM). The SMF may be implemented as a cloud-native network function using modern microservices approaches.

[0021] Various network functions for implementing the wireless network 103 may be deployed in a distributed computing device 112, which may correspond to a general-purpose computing device configured to perform network functions. For example, the distributed computing device 112 may execute one or more virtual machine instances that are in turn configured to execute one or more services that implement the network functions. In one embodiment, the distributed computing device 112 is a ruggedized machine deployed at each cell site. For example, the distributed computing device may execute a distributed unit (DU) function of a RAN.

[0022] In contrast, one or more centralized computing devices 115 may perform various network functions at a central site operated by a customer. For example, the centralized computing devices 115 may be centrally located at a customer's premises in a coordinated server room. The centralized computing devices 115 may run one or more virtual machine instances that are in turn configured to run one or more services that implement the network functions. For example, the centralized computing devices 115 may perform a centralized unit (CU) function for a RAN.

[0023] In one or more embodiments, network traffic from wireless network 103 is backhauled to one or more computing devices of core network 118, which may be located in one or more data centers located remotely from customer sites. Core network 118 may also perform various network functions, including routing network traffic to and from network 121, which may correspond to the Internet and / or other external public or private networks. The core network may perform functions related to management of communications network 100 (e.g., billing, mobility management, etc.) and transport functions for relaying traffic between communications network 100 and other networks.

[0024] 1B, an example of a RAN protocol stack 120 for use in various embodiments of the present disclosure is shown. The RAN protocol stack 120 is divided into a user plane 122 and a control plane 124. The RAN consists of workloads corresponding to a central unit (CU) and distributed units (DU). The DU is connected to the CU as well as radio units (RUs) and antennas.

[0025] L1 126 corresponds to Layer 1, which represents an upper physical layer (PHY) that performs channel coding or digital signal processing (DSP) operations. Such DSP operations may include Fast Fourier Transform (FFT), forward error correction, radio channel estimation, precoding, modulation, etc., as required by the wireless network. The portion of the DU that performs L1 operations may be connected to the RU via an Ethernet network.

[0026] L2 128, or Layer 2, is primarily a scheduling layer, a medium access control (MAC) layer that schedules uplink and downlink transmissions based on the radio conditions of individual client devices, which may be updated periodically, such as every millisecond or less. L2 128 determines how to best utilize the available radio capacity among existing client devices to meet capacity, latency, and traffic requirements. In the case of cellular networks, network traffic to and from client devices is explicitly scheduled by a scheduler. The more efficient the scheduler is in terms of using optimal coding for the uplink and downlink, the more client devices can function with the same hardware, thus providing a cheaper and more scalable solution. In contrast, Wi-Fi networks use a carrier sense multiple access with collision avoidance (CSMA-CA) protocol in which client devices try to sense network conditions to avoid collisions, and as soon as they detect a collision, the client device backs off and then retries. For CSMA-CA, there is no central scheduling of network traffic.

[0027] For example, L2 128 may implement a hybrid automatic repeat request (HARQ) technique, which is a combination of high rate forward error correction (FEC) and automatic repeat request (ARQ) error control at the transport channel level. L2 128 may also implement ARQ error control at the logical channel level. For example, Radio Link Control (RLC), a radio link protocol, may be used to implement the logical channels. The L1 126 and L2 128 layers may be implemented at cell sites or at smaller aggregation sites relatively close to the RUs due to strict latency constraints.

[0028] L3 130 or Layer 3 includes the control plane-like layer of the RAN that is responsible for the configuration of the other layers and makes decisions around the mobility of client devices across cell sites or RANs. These are CU workloads that can run in the cloud provider network because they can tolerate higher latency. From a physical topology perspective, there may be multiple cells and base stations (DU / RU sets) per geographical area depending on the radio frequency used. The higher the radio frequency, the smaller the area that a single cell can cover. A client device is exclusively connected to one cell at a time and can be handed off to another if the client device changes location or if another cell can better serve the client device.

[0029] At L3 130, the Packet Data Convergence Protocol (PDCP) may be used to implement services including user plane data transfer, control plane data transfer, header compression, encryption, integrity protection, etc. Additionally, the Service Data Adaptation Protocol (SDAP) may be used for Quality of Service (QoS) management. For example, the SDAP may map flows of a particular quality of service within a session to corresponding data radio bearers established with an appropriate level of QoS. The SDAP may mark transmitted packets with the correct QoS flow identifier.

[0030] Radio Resource Control (RRC) is also in L3 130, but on the control plane 124 side. RRC provides information transfer services to the Non-Access Stratum (NAS). RRC is also responsible for controlling the configuration of the air interface, facilitating the establishment and management of radio bearers, mobility management and security. NAS is used to manage the establishment of communication sessions and to maintain continuous communication with a client device as it moves. The control plane 124 also includes a Subscriber Identity (SI) channel and a paging channel.

[0031] 1C is a diagram illustrating an example data flow 140 in wireless network 103 (FIG. 1A). Data flow 140 illustrates the transfer of data between client device 106, wireless access network 143, UPF 146, and data network 149. Client device 106 communicates with wireless access network 143 over wireless interface 152. Wireless access network 143 forwards network traffic from client device 106 to UPF 146 via network interface 154. In one example, wireless access network 143 is operated by a cloud service provider, while in another example, wireless access network 143 is operated by a communications service provider.

[0032] UPF 146 forwards the return network traffic to wireless access network 143, also via network interface 154. After UPF 146 processes the network traffic from wireless access network 143, it forwards the processed network traffic to data network 149 via network interface 156. Similarly, UPF 146 receives return network traffic from data network 149 via network interface 156, processes it, and then forwards the processed network traffic to wireless access network 143 via network interface 154.

[0033] Network traffic on the network interface 154 is encapsulated using GTP, with each tunnel uniquely identified by a tuple that includes a tunnel identifier and the network address of the client device 106. This encapsulation adds header data overhead to each data packet. The additional data can cause a data packet to exceed the MTU value, resulting in fragmentation of the packet. The processes of encapsulation and decapsulation, along with potential fragmentation, add latency to the communication.

[0034] FIG. 2A illustrates an example of a network environment 200 including a cloud provider network 203 and further including various provider substrate extensions of the cloud provider network 203 that may be used at various locations in the communications network 100 of FIG. 1A, according to some embodiments. The cloud provider network 203 (sometimes simply referred to as a "cloud") refers to a network-accessible pool of computing resources (e.g., compute, storage, and networking resources, applications, and services), which may be virtualized or bare metal. A cloud may provide convenient on-demand network access to a shared pool of configurable computing resources that may be programmatically provisioned and released in response to customer commands. These resources may be dynamically provisioned and reconfigured to adjust to changing loads. Thus, cloud computing may be viewed as both applications delivered as a service over a publicly accessible network (e.g., the Internet, a cellular communications network) and the hardware and software in the cloud provider's datacenter that provides those services.

[0035] The cloud provider network 203 may provide users with an on-demand, scalable computing platform over the network, for example, allowing users to have scalable "virtual computing devices" at their disposal through the use of compute servers (which provide compute instances through the use of central processing units (CPUs) and / or graphics processing units (GPUs), optionally with local storage) and block store servers (which provide virtualized persistent block storage for designated compute instances). These virtual computing devices have the attributes of a personal computing device, including hardware (various types of processors, local memory, random access memory (RAM), hard disk, and / or solid state drive (SSD) storage), a selected operating system, network capabilities, and preloaded application software. Each virtual computing device may also virtualize its console input / output (e.g., keyboard, display, and mouse). This virtualization allows users to configure and use their virtual computing devices as if they were personal computing devices by connecting to them using computer applications, such as browsers, application programming interfaces (APIs), software development kits (SDKs), and the like. Unlike a personal computing device, where a user has a fixed amount of hardware resources available to them, the hardware associated with a virtual computing device can be scaled up or down depending on the resources needed by the user.

[0036] As mentioned above, users can use various interfaces 206 (e.g., APIs) to connect to virtualized computing devices and other cloud provider network 203 resources and services via intermediate network(s) 212 to configure and manage telecommunications networks, such as a 5G network. An API refers to an interface and / or communication protocol between a client device 215 and a server such that if the client makes a request in a predefined format, the client should receive a response in a specific format or initiate a defined action. In the context of a cloud provider network, an API provides a gateway for customers to access the cloud infrastructure by allowing customers to retrieve data from the cloud provider network 203 or perform actions within the cloud provider network 203, enabling the development of applications that interact with resources and services hosted in the cloud provider network 203. An API can also enable various services in the cloud provider network 203 to exchange data with each other. Users can choose to deploy their own virtual computing systems to provide network-based services for their own use and / or for use by their customers or clients.

[0037] Cloud provider network 203 may include a physical network (e.g., sheet metal boxes, cables, rack hardware) called a substrate. The substrate may be considered a network fabric that includes the physical hardware that runs the provider network's services. The substrate may be isolated from the rest of cloud provider network 203, e.g., it may not be possible to route from a substrate network address to an address in the production network that runs the cloud provider's services or to a customer network that hosts customer resources.

[0038] Cloud provider network 203 may also include an overlay network of virtualized computing resources running on a substrate. In at least some embodiments, a hypervisor or other device or process on the network substrate may use encapsulation protocol techniques to encapsulate and route network packets (e.g., client IP packets) on the network substrate between client resource instances on different hosts in the provider network. Encapsulation protocol techniques may be used on the network substrate to route encapsulated packets (also called network substrate packets) between endpoints on the network substrate via overlay network paths or routes. The encapsulation protocol techniques may be viewed as providing a virtual network topology overlaid on the network substrate. As such, network packets may be routed along the substrate network according to constructs in the overlay network (e.g., virtual networks that may be called virtual private clouds (VPCs), port / protocol firewall configurations that may be called security groups). A mapping service (not shown) may coordinate the routing of these network packets. The mapping service may be a regionally distributed lookup service that maps overlay Internet Protocol (IP) and network identifier combinations to substrate IPs, allowing distributed substrate computing devices to look up packet destinations.

[0039] To illustrate, each physical host device (e.g., compute server, block store server, object store server, control server) may have an IP address in the substrate network. Hardware virtualization techniques may enable multiple operating systems to run simultaneously on a host computer, e.g., as virtual machines (VMs) on a compute server. A hypervisor, or virtual machine monitor (VMM), on the host allocates the host's hardware resources to the various VMs on the host and monitors the execution of the VMs. Each VM may be given one or more IP addresses in the overlay network, and the VMM on the host may know the IP addresses of the VMs on the host. The VMM (and / or other devices or processes on the network substrate) may use encapsulation protocol techniques to encapsulate and route network packets (e.g., client IP packets) over the network substrate between virtualized resources on different hosts in the cloud provider network 203. Encapsulation protocol techniques may be used on the network substrate to route encapsulated packets between endpoints on the network substrate via overlay network paths or routes. The encapsulation protocol technology may be viewed as providing a virtual network topology overlaid on a network substrate. The encapsulation protocol technology may include a mapping service that maintains a mapping directory that maps IP overlay addresses (e.g., IP addresses visible to customers) to substrate IP addresses (IP addresses invisible to customers), which may be accessed by various processes on the cloud provider network 203 to route packets between endpoints.

[0040] As shown, the traffic and operations of the cloud provider network substrate may be roughly subdivided into two categories, in various embodiments: control plane traffic, carried on a logical control plane 218, and data plane operations, carried on a logical data plane 221. The data plane 221 represents the movement of user data through the distributed computing system, and the control plane 218 represents the movement of control signals through the distributed computing system. The control plane 218 generally includes one or more control plane components or services distributed across and implemented by one or more control servers. The control plane traffic generally includes management operations such as establishing isolated virtual networks for various customers, monitoring resource usage and health, identifying specific hosts or servers on which requested compute instances will be launched, and provisioning additional hardware as needed. The data plane 221 includes customer resources (e.g., compute instances, containers, block storage volumes, databases, file storage) that are implemented on the cloud provider network. The data plane traffic generally includes non-management operations such as transferring data to and from customer resources.

[0041] Control plane components are typically implemented on a separate set of servers from the data plane servers, and control plane traffic and data plane traffic may be sent over separate and distinct networks. In some embodiments, control plane traffic and data plane traffic may be supported by different protocols. In some embodiments, messages (e.g., packets) sent over cloud provider network 203 include a flag indicating whether the traffic is control plane traffic or data plane traffic. In some embodiments, the payload of the traffic may be inspected to determine its type (e.g., control plane or data plane). Other approaches to distinguish between traffic types are possible.

[0042] As shown, data plane 221 may include one or more compute servers, which may be bare metal (e.g., single tenant) or may be virtualized by a hypervisor to run multiple VMs (sometimes referred to as “instances”) or micro-VMs for one or more customers. These compute servers may support virtualized computing services (or “hardware virtualization services”) of cloud provider network 203. The virtualized computing services may be part of control plane 218 and allow customers to issue commands via interfaces 206 (e.g., APIs) to launch and manage compute instances (e.g., VMs, containers) for applications. The virtualized computing services may provide virtual computing instances with various computational and / or memory resources. In one embodiment, each of the virtual computing instances may correspond to one of several instance types. An instance type may be characterized by its hardware type, computational resources (e.g., the number, type, and configuration of CPUs or CPU cores), memory resources (e.g., the capacity, type, and configuration of local memory), storage resources (e.g., the capacity, type, and configuration of locally accessible storage), network resources (e.g., the characteristics of its network interfaces and / or network capabilities), and / or other suitable descriptive characteristics. The instance type selection function may be used to select an instance type for a customer, e.g., based (at least in part) on input from the customer. For example, a customer may select an instance type from a set of predefined instance types. As another example, a customer may specify the desired resources of the instance type and / or the requirements of the workload the instance will run, and the instance type selection function may select an instance type based on such specifications.

[0043] The data plane 221 may also include one or more block store servers, which may include persistent storage for storing volumes of customer data, and software for managing these volumes. Such block store servers may support the managed block storage services of the cloud provider network. The managed block storage services are part of the control plane 218 and allow customers to issue commands via interfaces 206 (e.g., APIs) to create and manage volumes for applications running on the compute instances. The block store servers include one or more servers where data is stored as blocks. A block is a sequence of bytes or bits, and typically contains an integer number of records with a maximum length of the block size. Blocked data is typically stored in a data buffer, and an entire block is read and written at a time. In general, a volume may correspond to a logical collection of data, such as a set of data maintained on behalf of a user. A user volume, which may be treated as an individual hard drive, for example, ranging in size from 1 GB to 1 terabyte (TB) or more, is composed of one or more blocks stored in a block store server. Although treated as individual hard drives, it will be understood that volumes may be stored as one or more virtualized devices implemented on one or more underlying physical host devices. A volume may be partitioned a small number of times (e.g., up to 16 times) with each partition hosted by a different host. A volume's data may be replicated across multiple devices within a cloud provider network to provide multiple replicas of the volume (such replicas may collectively represent a volume on a computing system).Replicas of volumes in a distributed computing system can beneficially provide automatic failover and recovery, for example, by allowing a user to access either a primary replica of the volume or a secondary replica of the volume synchronized with the primary replica at the block level, such that failure of either the primary or secondary replica does not prevent access to the information of the volume. The role of the primary replica may be to facilitate reads and writes (sometimes called "input / output operations" or simply "I / O operations") on the volume and to reflect any writes to the secondary (possibly using asynchronous replication, but preferably synchronously in the I / O path). Secondary replicas are updated synchronously with the primary replica and can provide a seamless transition during a failover operation, whereby the secondary replica takes over the role of the primary replica and the previous primary is designated as the secondary, or a new replacement secondary replica is provisioned. While specific examples herein describe primary and secondary replicas, it will be understood that a logical volume can include multiple secondary replicas. A compute instance may virtualize its I / O to the volume via a client. A client corresponds to instructions that enable a compute instance to connect to a remote data volume (e.g., a data volume stored on a physically separate computing device accessed over a network) and perform I / O operations on the remote data volume. A client may be implemented on an offload card of a server that includes a processing unit (e.g., a CPU or GPU) of the compute instance.

[0044] The data plane 221 may also include one or more object store servers, which represent another type of storage in the cloud provider network 203. The object storage servers may include one or more servers where data is stored as objects in resources called buckets, and may be used to support the managed object storage services of the cloud provider network 203. Each object typically includes the data stored, a variable amount of metadata that enables various functions of the object storage server regarding analysis of the stored objects, and a globally unique identifier or key that can be used to retrieve the object. Each bucket is associated with a given user account. A customer may store as many objects as desired in his bucket, may write, read, and delete objects in his bucket, and may control access to his bucket and the objects contained therein. Furthermore, in an embodiment where several different object storage servers are distributed in different ones of the above-mentioned regions, a user may choose the region (or regions) in which a bucket is stored, e.g., to optimize latency. A customer may use a bucket to store various types of objects, such as machine images that can be used to boot a VM, or snapshots that represent a point-in-time view of a volume's data.

[0045] Provider substrate extension 224 ("PSE") provides resources and services of cloud provider network 203 within a separate network, such as a telecommunications network, thereby extending the capabilities of cloud provider network 203 to new locations (e.g., for reasons related to latency of communications with customer devices, regulatory compliance, security, etc.). In some embodiments, PSE 224 may be configured to provide capacity for cloud-based workloads executing within the telecommunications network. In some embodiments, PSE 224 may be configured to provide core and / or RAN functions of the telecommunications network and may be configured with additional hardware (e.g., radio access hardware). Some embodiments may be configured to enable both, for example, by allowing unused capacity by the core and / or RAN functions to be used for executing cloud-based workloads.

[0046] As shown, such provider substrate extensions 224 may include a cloud provider network managed provider substrate extension 227 (e.g., formed by servers located in cloud provider managed facilities separate from those associated with the cloud provider network 203), a communications service provider substrate extension 230 (e.g., formed by servers associated with communications service provider facilities), a customer managed provider substrate extension 233 (e.g., formed by servers located on-premise at a customer or partner facility), among other possible types of substrate extensions.

[0047] As shown in the exemplary provider substrate extension 224, the provider substrate extension 224 may similarly include a logical separation between a control plane 236 and a data plane 239, which respectively extend the control plane 218 and the data plane 221 of the cloud provider network 203. The provider substrate extension 224 may be pre-configured, for example by a cloud provider network operator, with an appropriate combination of hardware with software and / or firmware elements to support various types of computing-related resources and to reflect the experience of using the cloud provider network 203 to do so. For example, one or more provider substrate extension location servers may be provisioned by a cloud provider for deployment within the provider substrate extension 224. As noted above, the cloud provider network 203 may offer a set of pre-defined instance types, each with various types and amounts of underlying hardware resources. Each instance type may also be offered in various sizes. The servers may be heterogeneous servers to allow customers to continue to use the same instance types and sizes in the provider substrate extension 224 as they use in the region. A heterogeneous server can simultaneously support multiple instance sizes of the same type and can also be reconfigured to host any instance type supported by its underlying hardware resources. Reconfiguration of a heterogeneous server can be performed on the fly using the server's available capacity, i.e., while other VMs are still running and consuming other capacity of the provider substrate extension location server.This can improve utilization of computing resources in edge locations by enabling better packing of running instances on servers, and can also provide a seamless experience in using instances across cloud provider networks 203 and cloud provider network managed provider substrate extensions 227.

[0048] A provider substrate extension server may host one or more compute instances. A compute instance may be a VM, or a container, that packages code and all its dependencies, allowing an application to run quickly and reliably across computing environments (including, for example, VMs and micro-VMs). Additionally, depending on the customer's desires, the server may host one or more data volumes. In a region of the cloud provider network 203, such volumes may be hosted on dedicated block store servers. However, because the capacity of the provider substrate extension 224 may be significantly smaller than in a region, a suboptimal usage experience may be provided if the provider substrate extension 224 includes such dedicated block store servers. Thus, the block storage service may be virtualized within the provider substrate extension 224 such that one of the VMs runs the block store software to store the data of the volume. Similar to the operation of the block storage service in a region of the cloud provider network 203, the volumes within the provider substrate extension 224 may be replicated for durability and availability. Volumes may be provisioned within their own isolated virtual network within the provider substrate extension 224. The compute instances and any volumes collectively constitute a data plane 239 extension of the provider network data plane 221 within the provider substrate extension 224 .

[0049] Servers within the provider substrate extension 224 may, in some embodiments, host certain local control plane components, such as components that allow the provider substrate extension 224 to continue functioning if the connection back to the cloud provider network 203 is broken. Examples of these components include a migration manager that can move compute instances between provider substrate extension servers as needed to maintain availability, and a key-value data store that indicates the location of volume replicas. In general, however, the control plane 236 functions of the provider substrate extension 224 remain in the cloud provider network 203 to allow customers to use as much of the provider substrate extension 224's resource capacity as possible.

[0050] The migration manager may have a centralized coordination component running within the region as well as a local controller running on the PSE server (and servers in the cloud provider's data center). The centralized coordination component may identify the target edge location and / or target host when migration is triggered, and the local controller may coordinate the data transfer between the source and target hosts. The described movement of resources between hosts in different locations may take one of several migration forms. Migration refers to moving virtual machine instances (and / or other resources) between hosts in a cloud computing network, or between hosts outside the cloud computing network and hosts in the cloud. There are various types of migrations, such as live migration and reboot migration. During a reboot migration, the customer experiences a shutdown and effective power cycle of the virtual machine instance. For example, the control plane service may coordinate a reboot migration workflow that includes destroying the current domain on the original host and then creating a new domain for the virtual machine instance on the new host. The instance is restarted by shutting it down on the original host and starting it up again on the new host.

[0051] Live migration refers to the process of moving a running virtual machine or application between different physical machines without significant disruption to the availability of the virtual machine (e.g., virtual machine downtime is not noticeable to the end user). When the control plane executes a live migration workflow, it can create a new "inactive" domain associated with the instance, while the original domain of the instance continues to run as an "active" domain. The virtual machine's memory (including the in-memory state of the running applications), storage, and network connections are transferred from the original host, which is an active domain, to the destination host, which is an inactive domain. The virtual machine may be briefly paused to prevent state changes while transferring memory contents to the destination host. The control plane may transition the inactive domain to be the active domain, demote the original active domain to be the inactive domain (also called "flipping"), and then destroy the inactive domain.

[0052] Various types of migration techniques include managing critical phases (times when a virtual machine instance is unavailable to customers), which should be as short as possible. In the currently disclosed migration techniques, this can be particularly challenging since resources are moved between hosts in geographically distant locations that may be connected through one or more intermediate networks. In the case of live migration, the disclosed techniques can dynamically determine how much memory state data to copy up front (e.g., while the instance is still running on the source host) and how much memory state data to copy after (e.g., after the instance starts running on the destination host) based on, for example, the latency between the locations, the network bandwidth / usage patterns, and / or which memory pages are most frequently used by the instance. Furthermore, the specific time at which the memory state data is transferred can be dynamically determined based on the network conditions between the locations. This analysis may be performed by a migration management component in the region or by a migration management component running locally at the source edge location. If the instance has access to virtualized storage, both the source and target domains may be attached to the storage simultaneously to allow uninterrupted access to that data during migration and in the event that a rollback to the source domain is required.

[0053] Server software running on the provider substrate extension 224 may be designed by the cloud provider to run on the cloud provider substrate network, and this software may be enabled to run unmodified in the provider substrate extension 224 by creating a private replica of the substrate network ("shadow substrate") in the edge location using local network manager(s) 242. The local network manager(s) 242 operate on the provider substrate extension 224 servers and can bridge the shadow substrate with the provider substrate extension 224 network, for example, by acting as one or more virtual private network (VPN) endpoints between the provider substrate extension 224 and proxies 245, 248 in the cloud provider network 203, as well as by implementing a mapping service (for encapsulation and decapsulation of traffic) that associates data plane traffic (from the data plane proxy 248) and control plane traffic (from the control plane proxy 245) with the appropriate server(s). By implementing a local version of the provider network's substrate overlay mapping service, the local network manager(s) 242 enable resources in the provider substrate extension 224 to seamlessly communicate with resources in the cloud provider network 203. In some implementations, a single local network manager 242 may perform these actions for all servers hosting compute instances in the provider substrate extension 224. In other implementations, each of the servers hosting compute instances may have a dedicated local network manager 242.In multi-rack edge locations, inter-rack communications can be routed through the local network managers 242, with the local network managers maintaining open tunnels between each other.

[0054] The provider substrate extension locations may utilize secure network tunnels through the provider substrate extension 224 network to the cloud provider network 203, for example, to maintain the security of customer data as it traverses the provider substrate extension 224 network and other intermediate networks (potentially including the public Internet). Within the cloud provider network 203, these tunnels are comprised of virtual infrastructure components including isolated virtual networks (e.g., within an overlay network), control plane proxies 245, data plane proxies 248, and substrate network interfaces. Such proxies 245, 248 may be implemented as containers running on the compute instances. In some embodiments, each server at the provider substrate extension 224 locations that hosts a compute instance may utilize at least two tunnels: one for control plane traffic (e.g., Constrained Application Protocol (CoAP) traffic) and one for encapsulated data plane traffic. A connection manager (not shown) within the cloud provider network 203 manages the lifecycle of the cloud provider network side of these tunnels and their components, for example, by automatically provisioning them as needed and maintaining them in a healthy operational state. In some embodiments, direct connections between provider substrate extension 224 locations and cloud provider network 203 may be used for control plane and data plane communications. Compared to VPNs over other networks, direct connections may provide constant bandwidth and more consistent network performance because their network paths are relatively fixed and stable.

[0055] A control plane (CP) proxy 245 may be provisioned in the cloud provider network 203 to represent a particular host(s) at an edge location. The CP proxy 245 is an intermediary between the control plane 218 in the cloud provider network 203 and the control plane targets in the control plane 236 of the provider substrate extension 224. That is, the CP proxy 245 provides an infrastructure for tunneling management API traffic destined for a provider substrate extension server from the regional substrate to the provider substrate extension 224. For example, a virtualized computing service in the cloud provider network 203 may issue a command to a VMM of a server in the provider substrate extension 224 to launch a compute instance. The CP proxy 245 maintains a tunnel (e.g., a VPN) to the local network manager 242 of the provider substrate extension. Software implemented in the CP proxy 245 ensures that only eligible API traffic leaves the substrate and returns to the substrate. The CP proxy 245 provides a mechanism to expose remote servers on the cloud provider substrate while still protecting the substrate's security material (e.g., encryption keys, security tokens) from leaving the cloud provider network 203. The one-way control plane traffic tunnel imposed by the CP proxy 245 also prevents any (potentially compromised) device from calling back into the substrate. The CP proxy 245 may be instantiated one-to-one with a server in the provider substrate extension 224, or may be able to manage the control plane traffic for multiple servers within the same provider substrate extension 224.

[0056] Data plane (DP) proxies 248 may also be provisioned within the cloud provider network 203 to represent a particular server(s) within the provider substrate extension 224. The DP proxies 248 may be used by services within the cloud provider network 203 to act as a shadow or anchor for the server(s) and monitor the host's health (including availability, used / free compute and capacity, used / free storage and capacity, and network bandwidth usage / availability). The DP proxies 248 also act as proxies for the server(s) within the cloud provider network 203, thereby enabling isolated virtual networks to span the provider substrate extension 224 and the cloud provider network 203. Each DP proxy 248 may be implemented as a packet forwarding compute instance or container. As shown, each DP proxy 248 may maintain a VPN tunnel with a local network manager 242 that manages traffic to the server(s) that the DP proxy 248 represents. This tunnel may be used to transmit data plane traffic between the provider substrate extension server(s) and the cloud provider network 203. Data plane traffic flowing between the provider substrate extension 224 and the cloud provider network 203 may pass through the DP proxy 248 associated with that provider substrate extension 224. For data plane traffic flowing from the provider substrate extension 224 to the cloud provider network 203, the DP proxy 248 may receive the encapsulated data plane traffic, verify its accuracy, and allow it to enter the cloud provider network 203. The DP proxy 248 may forward the encapsulated traffic from the cloud provider network 203 directly to the provider substrate extension 224.

[0057] The local network manager(s) 242 may provide secure network connectivity with proxies 245, 248 established in the cloud provider network 203. After connectivity between the local network manager 242 and the proxies 245, 248 is established, the customer may issue commands to instantiate compute instances (and / or perform other operations using compute instances) via the interface 206 using the provider substrate extension resources in the same manner that commands would be issued for compute instances hosted in the cloud provider network 203. From the customer's perspective, the customer can now seamlessly use local resources in the provider substrate extension (and resources in the cloud provider network 203, if desired). Computing instances set up on servers in the provider substrate extension 224 may communicate with both electronic devices in the same network and other resources set up in the cloud provider network 203, as desired. A local gateway 251 may be implemented to provide network connectivity between the provider substrate extension 224 and a network coupled to the extension (eg, a communications service provider network in the example of provider substrate extension 230).

[0058] In some circumstances, data transfer between the object storage service and the provider substrate extension (PSE) 224 may be required. For example, the object storage service may store machine images used to boot VMs, and snapshots representing point-in-time backups of volumes. The object gateway may provide on the PSE server or specialized storage device, and provide customers with a configurable per-bucket cache of the contents of object storage buckets in the PSE 224 to minimize the impact of PSE region latency on customer workloads. The object gateway may also temporarily store snapshot data from snapshots of volumes in the PSE 224, and synchronize with object servers in the region when possible. The object gateway may also store machine images designated by the customer for use in the PSE 224 or on the customer's premises. In some implementations, data in the PSE 224 may be encrypted with a unique key, and the cloud provider may restrict the key from being shared from the region to the PSE 224 for security reasons. Thus, data exchanged between the object store server and the object gateway may utilize encryption, decryption, and / or re-encryption to maintain security boundaries regarding encryption keys or other sensitive data. A means of transformation can perform these operations, and a PSE bucket can be created (on the object store server) to store the snapshot data and the machine image data using the PSE encryption key.

[0059] As described above, the PSE 224 forms an edge location in that it provides resources and services of the cloud provider network 203 outside of a traditional cloud provider datacenter and closer to customer devices. The edge location referred to herein may be structured in several ways. In some implementations, an edge location may be an extension of the cloud provider's network infrastructure that includes a limited amount of capacity provided outside of an availability zone (e.g., in a small datacenter or other facility of the cloud provider that is located closer to customer workloads and may be far away from any availability zone). Such edge locations may be referred to as "far zones" (because they are far from other availability zones) or "near zones" (because they are closer to customer workloads). Near zones may be connected to publicly accessible networks such as the Internet in various ways, e.g., directly, through another network, or through a private connection to a region. Typically, near zones have limited capacity compared to regions, but in some cases, near zones may have significant capacity, such as several thousand racks or more.

[0060] In some implementations, an edge location may be an extension of a cloud provider network substrate formed by one or more servers located on-premise at a customer or partner facility, such server(s) communicating over a network (e.g., a publicly accessible network such as the Internet) with a nearby availability zone or region of the cloud provider network. This type of substrate extension located outside of a cloud provider network datacenter may be referred to as an "outpost" of the cloud provider network. Some outposts may be integrated into a telecommunications network, for example, as a multi-access edge computing (MEC) site with a physical infrastructure that spans across telecommunications datacenters, telecommunications aggregation sites, and / or telecommunications base stations in the telecommunications network. In an on-premise example, the limited capacity of the outpost may be available only to the customer that owns the facility (and any other accounts authorized by the customer). In a telecommunications example, the limited capacity of the outpost may be shared among multiple applications (e.g., gaming, virtual reality applications, healthcare applications) that transmit data to users of the telecommunications network.

[0061] An edge location may include data plane capacity that is at least partially controlled by a control plane of a nearby availability zone in the provider network. Thus, an availability zone group may include a "parent" availability zone and any "child" edge locations that are homed to the parent availability zone (e.g., controlled at least in part by its control plane). Certain limited control plane functions (e.g., functions that require low latency communication with customer resources and / or functions that allow the edge location to continue functioning when disconnected from the parent availability zone) may also be present in some edge locations. Thus, in the above example, an edge location refers to at least an extension of data plane capacity located at the edge of the cloud provider network, close to customer devices and / or workloads.

[0062] In the example of FIG. 1A, the distributed computing device 112 (FIG. 1A), the centralized computing device 115 (FIG. 1A), and the core network 118 (FIG. 1A) may be implemented as a provider substrate extension 224 of the cloud provider network 203. The placement or arrangement of the provider substrate extension 224 within the communication network 100 may vary depending on the particular network topology or architecture of the communication network 100. The provider substrate extension 224 may generally connect anywhere the communication network 100 may generate packet-based traffic (e.g., IP-based traffic). Furthermore, communications between a given provider substrate extension 224 and the cloud provider network 203 typically traverse at least a portion of the communication network 100 securely (e.g., via a secure tunnel, a virtual private network, a direct connection, etc.).

[0063] In 5G wireless network development efforts, edge locations may be considered as a possible implementation of multi-access edge computing (MEC). Such edge locations may be connected to various points in the 5G network that provide data traffic breakout as part of the user plane function (UPF). Even older wireless networks may incorporate edge locations. For example, in a 3G wireless network, the edge locations may be connected to a packet-switched network portion of the communications network 100, such as a Serving General Packet Radio Service Support Node (SGSN) or a Gateway General Packet Radio Service Support Node (GGSN). In a 4G wireless network, the edge locations may be connected to a Serving Gateway (SGW) or a Packet Data Network Gateway (PGW) as part of the core network or evolved packet core (EPC). In some embodiments, traffic between the provider substrate extension 224 and the cloud provider network 203 may be decoupled from the communications network 100 without being routed through the core network.

[0064] In some embodiments, the provider substrate extension 224 may connect to multiple communication networks associated with each customer. For example, if two communication networks of each customer share or route traffic through a common point, the provider substrate extension 224 may connect to both networks. For example, each customer may assign a portion of its network address space to the provider substrate extension, and the provider substrate extension 224 may include a router or gateway capable of distinguishing traffic exchanged with each of the communication networks 100. For example, traffic destined for the provider substrate extension 224 from one network may have a different destination IP address, source IP address, and / or virtual local area network (VLAN) tag than traffic received from another network. Traffic from the provider substrate extension destined for a destination in one of the networks may likewise be encapsulated to have the appropriate VLAN tag, source IP address (e.g., from a pool allocated to the provider substrate extension from the destination network address space), and destination IP address.

[0065] FIG. 2B illustrates an example 253 of cellularization and geographic distribution of the communication network 100 (FIG. 1A). In FIG. 2B, a user device 254 communicates with a request router 255 to route requests to one of a number of control plane cells 257a and 257b. Each control plane cell 257 may include a network service API gateway 260, a network slice configuration 262, a function for network service monitoring 264, site planning data 266 (including layouts describing customer site requirements, device types, device quantities, etc.), a network service / function catalog 268, a function for orchestration 270, and / or other components. To reduce the possibility of a large-scale error affecting a wide range of customers, a large control plane may be divided into cells by providing one or more cells operating independently, for example, per customer, per network, or per region.

[0066] The network service / function catalog 268 is also referred to as the NF repository function (NRF). In a service-based architecture (SBA) 5G network, the control plane functions and common data repository may be delivered through a set of interconnected network functions built using a microservices architecture. The NRF may maintain a record of available NF instances and their supported services, allowing other NF instances to subscribe and be notified of registrations from NF instances of a given type. Thus, the NRF may support service discovery by receiving discovery requests from NF instances and specify which NF instances support a particular service. The network function orchestrator 270 may perform NF lifecycle management, including instantiation, scale out / in, performance measurement, event correlation, and termination. The network function orchestrator 270 may onboard new NFs, manage migration of existing NFs to new or updated versions, identify a suitable set of NFs for a particular network slice or larger network, and orchestrate NFs across the various computing devices and sites that make up the wireless network 103.

[0067] The control plane cell 257 may communicate with one or more cell sites 272, one or more customer local data centers 274, one or more local zones 276, and one or more regional zones 278. The cell site 272 includes computing hardware 280 that runs one or more distributed unit (DU) network functions 282. The customer local data center 274 includes computing hardware 283 that runs one or more DU or central unit (CU) network functions 284, a network controller 285, a UPF 286, one or more edge applications 287 that support the customer workloads, and / or other components.

[0068] The local zone 276 may be in a data center operated by a cloud service provider and may run one or more core network functions 288, such as the AMF, the SMF, a network publishing function (NEF) that securely publishes services and capabilities of other network functions, and a unified data management (UDM) function that manages subscriber data for authorization, registration, and mobility management. The local zone 276 may also run the UPF 286, a service 289 for metric processing, and one or more edge applications 287.

[0069] The regional zone 278 may be located within a data center operated by the cloud service provider and may run one or more core network functions 288; a UPF 286; an operational support system (OSS) 290 supporting network management systems, service delivery, service fulfillment, service assurance, and customer care; an Internet Protocol Multimedia Subsystem (IMS) 291; a business support system (BSS) 292 supporting product management, customer management, revenue management, and / or order management; one or more portal applications 293, and / or other components.

[0070] In this example, the communication network 100 employs a cellular architecture to reduce the blast radius of individual components. At the highest level, the control plane resides within multiple control plane cells 257 to prevent failure of an individual control plane from affecting the entire deployment.

[0071] Within each control plane cell 257, multiple redundant stacks may be provided with a control plane that shifts traffic to a secondary stack as needed. For example, cell site 272 may be configured to utilize a nearby local zone 276 as its default core network. If the local zone 276 goes down, the control plane can redirect cell site 272 to use a backup stack in regional zone 278. Traffic that would normally be routed from the Internet to local zone 276 may be shifted to an endpoint in regional zone 278. Each control plane cell 257 may implement a "stateless" architecture that shares a common session database across multiple sites (e.g., across availability zones or edge sites).

[0072] FIG. 3A illustrates an exemplary cloud provider network 203 including geographically distributed provider substrate extensions 224 (FIG. 2A) (or "edge locations 303") according to some embodiments. As illustrated, the cloud provider network 203 may be formed as multiple regions 306, where a region 306 is a distinct geographic area in which the cloud provider has one or more data centers 309. Each region 306 may include two or more availability zones (AZs) connected to each other via a private high-speed network, such as a fiber communication connection. An availability zone refers to an isolated failure domain that includes one or more data center facilities with separate power sources, separate networks, and separate cooling relative to other availability zones. Cloud providers may strive to place availability zones far enough apart from each other within a region 306 to prevent multiple availability zones from being taken offline at the same time due to a natural disaster, major power outage, or other unforeseen event. Customers can connect to resources within an availability zone of the cloud provider network 203 via a publicly accessible network (e.g., the Internet, a cellular communication network, a communication service provider network). Transit Centers (TCs) are the main backbone locations that link customers to the cloud provider network 203 and may be co-located with other network provider facilities (e.g., Internet service providers, telecommunications providers). Each region 306 may operate two or more TCs for redundancy. The regions 306 are connected to a global network that includes a private network infrastructure (e.g., fiber connections controlled by the cloud service provider) that connects each region 306 to at least one other region. The cloud provider network 203 may deliver content from points of presence ("PoPs") that are outside of, but networked with, these regions 306 via edge locations 303 and regional edge cache servers.This partitioning and geographic distribution of computing hardware enables the cloud provider network 203 to offer customers global, low-latency resource access with a high degree of fault tolerance and stability.

[0073] The number of edge locations 303 may be much larger compared to the number of regional data centers or availability zones. Such widespread deployment of edge locations 303 may provide low latency connectivity to the cloud for a much larger group of end user devices (compared to end user devices that happen to be very close to the regional data center 309). In some embodiments, each edge location 303 may peer with a portion of the cloud provider network 203 (e.g., a parent availability zone or regional data center). Such peering allows various components operating within the cloud provider network 203 to manage the computing resources of the edge locations 303. In some cases, multiple edge locations 303 may be located or installed in the same facility (e.g., separate racks of computer systems) and managed by different zones or data centers 309 to provide additional redundancy. It should be noted that although the edge location 303 is generally depicted herein as being within the communications service provider network or wireless network 103 (FIG. 1A), in some cases, such as when a cloud provider network facility is relatively close to the communications service provider facility, the edge location 303 may remain within the physical premises of the cloud provider network 203 while being connected to the communications service provider network via fiber or other network links.

[0074] Edge locations 303 may be structured in several ways. In some implementations, edge locations 303 may be an extension of the cloud provider's network substrate, including a limited amount of capacity provided outside of availability zones (e.g., in the cloud provider's small data center or other facilities located close to customer workloads and that may be far from any availability zones). Such edge locations 303 may be referred to as local zones (because they are closer to a local or group of users than traditional availability zones). Local zones may be connected in various ways to a publicly accessible network, such as the Internet, for example, directly, through another network, or through a private connection to a region 306. Typically, local zones have limited capacity compared to a region 306, but in some cases, local zones may have significant capacity, such as several thousand racks or more. Some local zones may use infrastructure similar to a typical cloud provider data center instead of the edge location 303 infrastructure described herein.

[0075] As shown herein, the cloud provider network 203 can be formed as a number of regions 306, each of which represents a geographic area in which the cloud provider clusters data centers 309. Each region 306 may further include multiple (e.g., two or more) availability zones (AZs) connected to each other via a private high-speed network, e.g., fiber communication connections. An AZ may provide an isolated failure domain, including one or more data center facilities with separate power sources, separate networks, and separate cooling from another AZ. The AZs within a region 306 are preferably located in locations far enough apart from each other so that the same natural disaster (or other failure-causing event) does not affect or take multiple AZs offline at the same time. Customers can connect to the AZs of the cloud provider network via a publicly accessible network (e.g., the Internet, a cellular communication network, etc.).

[0076] Parenting of a given edge location 303 to an AZ or region 306 of a cloud provider network 203 may be based on several factors. One such parenting factor is data sovereignty. For example, an edge location 303 deployed within a country's telecommunications network may be parented to an AZ or region 306 within the country to keep data originating from that country within the country. Another factor is service availability. For example, some edge locations 303 may have different hardware configurations depending on the presence or absence of components such as local non-volatile storage for customer data (e.g., solid state drives), graphics accelerators, etc. Some AZs or regions 306 may lack services to leverage those additional resources, and therefore the edge location may be parented to an AZ or region 306 that supports the use of those resources. Another factor is the latency between the AZ or region 306 and the edge location 303. Although deploying edge locations 303 within a communications network provides benefits in terms of latency, those benefits can be negated by parenting the edge location 303 to a distant AZ or region 306, which introduces significant latency to the edge location 303's regional traffic. Thus, edge locations 303 are often parented to nearby (in terms of network latency) AZs or regions 306.

[0077] Additionally, the disclosed service may provide a private zone for running local applications within the cloud provider network. This private zone may be connected to and effectively become a part of the broader regional zone, and customers may manage the private zone using the same APIs and tools used in the cloud provider network. Similar to availability zones, private zones may be assigned virtual private network subnets. Using APIs, subnets may be created and assigned to all zones that customers wish to use, including private zones and other existing zones. A management console may provide a simplified process for creating private zones. Virtual machine instances and containers may be launched in private zones as well as regional zones. Customers may configure network gateways to define routes, assign IP addresses, set up network address translation (NAT), and so on. Autoscaling allows the private zone to scale virtual machine instance or container capacity as needed. The same management and authentication APIs as the cloud provider network may be available within the private zone. In some cases, cloud services available in the regional zone may be accessed remotely from the private zone over a secure connection, allowing access to these cloud services without the need to upgrade or modify local deployments.

[0078] Turning now to FIG. 3B, an example networked environment 315 of a distributed UPF is illustrated in accordance with various embodiments. Although FIG. 3B specifically references a distributed UPF, other distributed data processing network functions may be substituted in other implementations. In the networked environment 315, wireless devices 106 are connected to a radio access network 143. Some of the wireless devices 106 communicate with respective DU / CUs 284a, 284b...284N, which in turn communicate with multiple UPF-Us. r320a, 320b, . . . 320N. In contrast to the central network 118, the UPF-U r 320 operates in RAN 143. In some cases, DU / CU 284 and UPF-U r 320 may be implemented in a single small cell device 321 that may include a radio unit. Such computing hardware may be relatively resource constrained. Alternatively, the UPF-U r 320 may be implemented in computing hardware at the same location (e.g., the same rack) or at a location near the DU / CU 284. r The UPF-U 320 corresponds to the user plane component of the UPF 286 (Figure 2B). r The UPF-U 320 processes user plane network traffic originating from or destined for each wireless device 106. r 320 then routes or forwards the processed network traffic to the network 121 or to other wireless devices 106 in the RAN 143 .

[0079] The network environment 315 in this example also includes a UPF-U r The legacy DU / CU 322 does not include the UPF-U of the core network 318. The legacy DU / CU 322 may be operated by a different communication service provider in some cases. The legacy DU / CU 322 routes its user plane network traffic to the UPF-U of the core network 318. c Transfer to 323. UPF-U c H.323 also corresponds to the user plane component of UPF286, but c 323 may operate in the core network 118 rather than in the RAN 143. The core network 118 may correspond to the cloud provider network 203 (FIG. 2A) and / or the provider substrate extension 224 (FIG. 2A) of the cloud provider network 203. c323 handles user plane network traffic originating or destined for each wireless device 106 that communicates with the legacy DU / CU 322. c 323 routes or forwards the processed network traffic to other wireless devices 106 in the network 121 or the RAN 143. c 323 is a separate RAN 143 that does not operate user plane components, or a UPF-U r 320 can handle network traffic.

[0080] UPF-C, the control plane component of UPF286 c 328 operates in the core network 118. The DU / CU 284 and the legacy DU / CU 322 can exchange control plane messages via the AMF 330 of the core network 118. The AMF 330 can exchange control plane messages with the SMF 332 of the core network 118. c The UPF-C 328 can exchange control plane messages with the SMF 332. Upon learning about any control plane changes from the SMF 332, the UPF-C c 328 will use this information as UPF-U r Propagate to 320 nodes.

[0081] DU / CU284 and UPF-U r Network traffic exchanged between the legacy DU / CU 322 and the UPF-U does not need to be encapsulated or decapsulated using GTP. c Network traffic exchanged between the UPF-U and the 323 encapsulation and decapsulation is done using GTP. By refraining from encapsulating / decapsulating network traffic, latency is reduced. r 320 is run on a device that is capacity or resource limited and uses UPF-Ur The processing by 320 may be based on QoS parameters, the network slice, the wireless device 106, and / or other factors. r 320 and forward the packet to UPF-U for processing. C 323.

[0082] 4, a network environment 400 is shown in accordance with various embodiments. The network environment 400 includes a computing environment 403, one or more client devices 406, one or more wireless access networks 143 that may be operated by one or more third-party communications service providers, and one or more wireless networks 103 in data communication with each other via a network 412. The network 412 may include, for example, the Internet, an intranet, an extranet, a wide area network (WAN), a local area network (LAN), a wired network, a wireless network, a cable network, a satellite network, or other suitable network, or any combination of two or more of these such networks.

[0083] The computing environment 403 may include, for example, a server computer or any other system that provides computing capacity. Alternatively, the computing environment 403 may employ multiple computing devices that may be arranged, for example, in one or more server or computer banks or other arrangements. Such computing devices may be located in a single facility or distributed among many different geographic locations. For example, the computing environment 403 may include multiple computing devices that together may comprise host computing resources, grid computing resources, and / or any other distributed computing arrangement. In some cases, the computing environment 403 may correspond to elastic computational resources in which the allocated capacity of processing, network, storage, or other computing-related resources may change over time. For example, the computing environment 403 may correspond to a cloud provider network 203 (FIG. 2A), where customers are charged according to the use of computing resources based on a utility computing model.

[0084] In some embodiments, computing environment 403 may correspond to a virtualized private network within a physical network that includes virtual machine instances running on physical computing hardware, for example by a hypervisor. The virtual machine instances and containers running on those instances may be provided network connectivity through virtualized network components made available by physical network components such as routers and switches.

[0085] Various applications and / or other functions may be executed in the computing environment 403 according to various embodiments. Additionally, various data is stored in a data store 415 that is accessible to the computing environment 403. The data store 415 may be representative of multiple data stores 415, as can be appreciated. The data stored in the data store 415 may be associated with the operation of the various applications and / or functional entities described below, for example.

[0086] The computing environment 403 as part of a cloud provider network offering utility computing services includes computing devices 418 and other types of computing devices 418. The computing devices 418 may correspond to different types of computing devices 418 and may have different computing architectures. The computing architectures may differ by utilizing processors with different architectures, such as x86, x86_64, ARM, Scalable Processor Architecture (SPARC), PowerPC, etc. For example, one computing device 418 may have an x86 processor, while another computing device 418 may have an ARM processor. The computing devices 418 may also differ in available hardware resources, such as local storage, graphics processing units (GPUs), machine learning extensions, and other characteristics.

[0087] Computing devices 418 may have various forms of allocated computing capacity 421, which may include virtual machine (VM) instances, containers, serverless functions, etc. A VM instance may be instantiated from a VM image. To this end, a customer may specify that a virtual machine instance should be launched within a particular type of computing device 418 and not other types of computing devices 418. In various examples, one VM instance may run alone on a particular computing device 418, or multiple VM instances may run on a particular computing device 418. Also, a particular computing device 418 may run different types of VM instances, which may provide different amounts of resources available via the computing device 418. For example, one type of VM instance may provide more memory and processing power than other types of VM instances.

[0088] Components executing in the computing environment 403 may include, for example, one or more UPF-C c 328, one or more UPF-U c 323, a UPF controller 424, and other applications, services, processes, systems, engines, or functions not described in detail herein. Each of these components may be executed as allocated computing capacity 421 on a computing device 418 that may be located at a cell site, a customer site, a local zone data center, a regional data center, etc.

[0089] The UPF controller 424 is a UPF-C of the wireless network 103. c 328, UPF-U c 323 and UPF-U r In particular, the UPF controller 424 is implemented to manage the operation of the distributed UPF architecture, including monitoring the UPF-C 320. c 328, UPF-Uc 323 and UPF-U r 320, the UPF controller 424 receives status information, which may include performance indicators, heartbeat signals, etc. from the UPF-C 320. c 328, UPF-U c 323 and UPF-U r The UPF controller 424 may also coordinate the allocation of network traffic flows to the UPF-C 320. This may include coordinating handovers of wireless devices 106 (FIG. 1A) moving from one cell 109 (FIG. 1A) to another. The UPF controller 424 may also coordinate the allocation of network traffic flows to the UPF-C 320 by starting up additional instances, reallocating other computing capacity to make room for the additional instances, etc. c 328, UPF-U c 323 and UPF-U r 320 can be expanded.

[0090] Data stored in data store 415 may include, for example, one or more network plans 439, one or more cellular topologies 442, one or more spectrum allocations 445, device data 448, one or more RBN metrics 451, customer billing data 454, radio unit configuration data 457, antenna configuration data 460, network function configuration data 463, one or more network function workloads 466, one or more customer workloads 469, UPF status information 472, UPF controller configuration 475, and potentially other data.

[0091] The network plan 439 is a specification of the wireless network 103 to be deployed for a customer. For example, the network plan 439 may include the location or geographic area of ​​the premises to be covered, the number of cells, device identification and permissions, the desired maximum network latency, the desired bandwidth or network throughput for one or more classes of devices, one or more quality of service parameters of an application or service, and / or other parameters that can be used to create the wireless network 103. The customer may manually specify one or more of these parameters via a user interface. One or more of the parameters may be pre-set as default parameters. In some cases, the network plan 439 may be generated for a customer based at least in part on an automated site survey using an unmanned aerial vehicle. The values ​​of the parameters that define the network plan 439 may be used as a basis for the cloud service provider to bill the customer under a utility computing model. For example, if the service level agreement (SLA) has a lower latency target and / or a higher bandwidth target, the customer may be charged a higher amount, and the customer may be billed per device, per cell, on a geographic area served basis, on a spectrum availability basis, etc.

[0092] The cellular topology 442 includes an arrangement of multiple cells for a customer, taking into account the location of the cells and, where possible, the reuse of frequency spectrum. The cellular topology 442 can be generated automatically by performing a site survey. In some cases, the number of cells in the cellular topology 442 can be automatically determined based on the desired geographic area to be covered, the availability of backhaul connections at various sites, signal propagation, the available frequency spectrum, and / or other parameters.

[0093] Spectrum allocations 445 include frequency spectrum available for allocation to wireless network 103 and frequency spectrum currently allocated to wireless network 103. Frequency spectrum may include spectrum that is publicly accessible without restrictions, spectrum that is privately owned or leased by a customer, spectrum that is owned or leased by a provider, spectrum that is free to use but requires reservation, etc.

[0094] The device data 448 corresponds to data describing the wireless device 106 that is authorized to connect to the wireless network 103. This device data 448 may include the corresponding user, account information, billing information, data plan, permitted applications or uses, an indication of whether the wireless device 106 is mobile or fixed, location, current cell, network address, device identifier (e.g., International Mobile Equipment Identity (IMEI) number, Equipment Serial Number (ESN), Media Access Control (MAC) address, Subscriber Identity Module (SIM) number, etc.).

[0095] The RBN metrics 451 include various metrics or statistics indicative of the performance or health of the wireless network 103. Such RBN metrics 451 may include bandwidth metrics, dropped packet metrics, signal strength metrics, latency metrics, etc. The RBN metrics 451 may be aggregated per device, per cell, per customer, etc.

[0096] The customer billing data 454 specifies the charges to be incurred by the customer for the provider's operation of the wireless network 103 for the customer. The charges may include a fixed fee based on the equipment deployed to the customer and / or a usage-based fee. In some cases, the customer may have purchased the equipment up front and may only be charged for the bandwidth or back-end network costs. In other cases, the customer may not incur any upfront costs and may be charged purely on a usage basis. Because the equipment is provided to the customer based on a utility computing model, the cloud service provider can select the optimal configuration of equipment to meet the customer's target performance metrics while avoiding unnecessary hardware over-provisioning.

[0097] The wireless unit configuration data 457 may correspond to configuration settings for wireless units deployed in the wireless network 103. Such settings may include the frequency to use, the protocol to use, modulation parameters, bandwidth, network routing and / or backhaul configurations, etc.

[0098] The antenna configuration data 460 may correspond to the configuration settings of the antenna, including the frequency to be used, the azimuth angle, the vertical or horizontal orientation, the beam tilt, and / or other parameters that may be controlled automatically (e.g., by networked motors and controls on the antenna) or manually by instructing a user to mount the antenna in a particular manner or make physical changes to the antenna.

[0099] Network function configuration data 463 corresponds to configuration settings that configure the operation of various network functions of wireless network 103. In various embodiments, the network functions may be deployed on VM instances located on computing devices 418 at cell sites, customer aggregation sites, or data centers located remotely from the customers. Non-limiting examples of network functions may include access and mobility management functions, session management functions, user plane functions, policy control functions, authentication server functions, unified data management functions, application functions, network publishing functions, network function repositories, network slice selection functions, and / or others. Network function workloads 466 correspond to machine images, containers, or functions that are to be launched within allocated computing capacity 421 to perform one or more network functions.

[0100] Customer workloads 469 correspond to customer machine images, containers, or functions that may be executed in parallel with, or instead of, network function workloads 466 within allocated computing capacity 421. For example, customer workloads 469 may provide or support a customer application or service.

[0101] UPF status information 472 may include information collected by or reported to UPF controller 424. Such information may include heartbeat signals and performance indicators such as processor utilization, memory utilization, latency metrics, bandwidth utilization, etc. UPF controller configuration 475 includes data that configures the operation of UPF controller 424. Such data may include rules that control the scaling up or down of various components of the distributed UPF and rules that control the allocation of network traffic flows to particular UPF user plane components in either the radio access network 143 or the core network 118 (FIG. 1A).

[0102] Client device 406 represents a plurality of client devices 406 that may be coupled to network 412. Client device 406 may include, for example, a processor-based system such as a computer system. Such a computer system may be embodied in the form of a desktop computer, a laptop computer, a personal digital assistant, a mobile phone, a smartphone, a set-top box, a music player, a web pad, a tablet computer system, a gaming console, an e-reader, a smart watch, a head-mounted display, a voice interface device, or other device. Client device 406 may include a display comprising one or more devices such as, for example, a liquid crystal display (LCD), a gas plasma-based flat panel display, an organic light-emitting diode (OLED) display, an electronic ink (E-ink) display, an LCD projector, or other types of display devices.

[0103] The client device 406 may be configured to execute various applications, such as a client application 436 and / or other applications. The client application 436 may execute within the client device 406 to render a user interface on a display, for example, by accessing network content provided by the computing environment 403 and / or other servers. To this end, the client application 436 may comprise, for example, a browser, a dedicated application, etc., and the user interface may comprise a network page, an application screen, etc. In addition to the client application 436, the client device 406 may be configured to execute applications, such as, for example, an email application, a social networking application, a word processor, a spreadsheet, and / or other applications.

[0104] 5, a flowchart presenting an example of the operation of a portion of the DU / CU 284 according to various embodiments is shown. It is understood that the flowchart of FIG. 5 provides merely an example of many different types of functional arrangements that may be used to implement the operation of the portion of the DU / CU 284 described herein. Alternatively, the flowchart of FIG. 5 may be considered to illustrate an example of elements of a method implemented in the computing hardware 280 or 283 (FIG. 2B) according to one or more embodiments.

[0105] Beginning at box 503, the DU / CU 284 receives data from a client device, such as the wireless device 106 (FIG. 1A). The client device is connected to the radio access network 143 (FIG. 3B) of the DU / CU 284, and the data is received over a radio channel. The radio access network 143 is associated with the core network 118 (FIG. 3B) of the wireless network 103 (FIG. 4).

[0106] In box 506, the DU / CU 284 then identifies an instance of a user plane component of the UPF 286 (FIG. 2B) that handles the processing of the data. An example of a user plane component is the UPF-U of the radio access network 143. r 320 (FIG. 3B), or the UPF-U of the core network 118 c 323 (FIG. 3B). In some cases, UPF-U r 320 may be located within the same device as the DU / CU 284, or may be within the same device as the radio unit. r 320 may be within the same rack or physical location as the DU / CU 284 and / or radio units (e.g., computing hardware 280). r 320 may be implemented in computing hardware 283 within the customer's local data center 274 (FIG. 2B).

[0107] In identifying an instance of a user plane component, the DU / CU 284 operates in conjunction with the UPF controller 424 (FIG. 4) and / or the UPF-C c 328 (FIG. 3B). In some cases, the default scenario is for the DU / CU 284 to route network traffic to the local UPF-U. r 320, but UPF-U r The hardware of the UPF-U may be constrained. In some cases, the computing capacity of the hardware may need to be shared with higher priority customer workloads. r 320 can be completely preempted, which means that the network traffic of the DU / CU 284 can be transferred to a different cell 109 (FIG. 1) or UPF-U c Another UPF-U associated with cell 323 r This means that the request must be processed by 320.

[0108] The decision as to which user plane component is used may be based on a variety of factors. For example, a flow of network traffic with higher QoS parameters or requirements may be routed to minimize latency and improve throughput, including routing the network traffic flow to a collocated UPF-U in the same cell 109 as the client device. r In contrast, network traffic flows with lower QoS parameters or requirements may include being directed to the UPF-U c323. QoS parameters may be assigned based on application, destination, source (i.e., client device), cost, time of day, day of the week, and / or other factors. Network traffic for a particular network slice may be routed to a particular user plane component based on the network slice. Also, if a client device is moving from one cell 109 to another cell 109, the network traffic of the client device may be routed to a different UPF-U corresponding to the cell 109 serving the client device. r 320 and / or UPF-U c 323 to prompt for handover.

[0109] Once the instance is allocated, the DU / CU 284 proceeds to box 509 and determines whether the allocated instance is in the core network 118. That is, the DU / CU 284 determines whether the user plane component is in the UPF-U c 323. Determine whether the assigned user plane component is UPF-U c If so, the DU / CU 284 continues to box 512 and prepares the data for transfer to the core network 118 (FIG. 1A). In some implementations, this may involve transmitting the data to the UPF-U through the network interface 154 (FIG. 1C). c The process may include encapsulating the data using GTP so that it can be forwarded to H.323. The DU / CU 284 then proceeds to box 515. If there is no allocated instance in the core network 118, the DU / CU 284 proceeds from box 509 to box 515 without preparing the data for forwarding to the core network 118 or encapsulating the data using GTP.

[0110] In box 515, the DU / CU 284 forwards the data to the identified instance of the user plane component for processing. Depending on the destination, the DU / CU 284 may set a custom MTU value or use a different transport protocol, such as Transmission Control Protocol (TCP). Similarly, the DU / CU 284 receives data from the instance of the user plane component, decapsulates the data if necessary, and returns the data to the client device. The operation of the DU / CU 284 then ends.

[0111] Continuing with reference to Figure 6, a flowchart is shown presenting an example of an operation of a portion of the UFP controller 424 in accordance with various embodiments. It will be understood that the flowchart of Figure 6 provides merely an example of many different types of functional arrangements that may be used to implement the operation of a portion of the UFP controller 424 described herein. Alternatively, the flowchart of Figure 6 may be considered to illustrate an example of elements of a method implemented within the computing environment 403 (Figure 4) in accordance with one or more embodiments.

[0112] Starting at box 603, the UPF controller 424, via the control plane, c 328 and UPF-U r 320 and UPF-U c The UPF receives UPF status information 472 (FIG. 4) from distributed UPF components, including instances of UPF 323. The UPF status information 472 may include key performance indicators, a heartbeat signal, and / or other information.

[0113] In box 606, the UPF controller 424 determines whether to scale the distributed UPF components. Such components can be scaled up or down as appropriate to meet demand and QoS requirements. For example, additional computing capacity of the cell hardware can be achieved by transferring customer workloads 469 (FIG. 4) or lower priority other network function workloads 466 (FIG. 4) to another hardware or cloud provider network 203 (FIG. 2A). c Additional instances of 323, or instances with more computational power, may be launched.

[0114] If the UPF controller 424 determines in box 609 to scale the component, then the UPF controller 424 performs the scaling in box 612. The UPF controller 424 then proceeds to box 615. If the UPF controller 424 does not scale the component in box 609, then the UPF controller 424 moves to box 615.

[0115] In box 615, the UPF controller 424 distributes network traffic from multiple client devices to instances of the UPF user plane components. r 320 and / or UPF-U c323. The UPF controller 424 can determine to distribute network traffic based at least in part on the UPF control configuration 475 (FIG. 4), which can include a rule set for allocating network traffic to types of user plane components. Such a rule set can allocate network traffic based on client device, QoS parameters, network slice, application, cost, time, and / or other factors. In allocating network traffic, the UPF controller 424 can move network traffic from one instance of a user plane component to another instance to reduce or balance load. The UPF controller 424 can also reallocate network traffic to another instance to provide failover upon detecting a lack of a heartbeat signal from an instance.

[0116] In box 618, the UPF controller 424 can configure the DU / CU 284 (FIG. 3B) to forward network traffic to a particular instance of a user plane component using encapsulation with GTP or another protocol, no encapsulation, a custom MTU value, or another transport protocol. For example, network traffic to a user plane component of the core network 118 may be encapsulated using GTP, while network traffic to a user plane component of the radio access network 143 may not be encapsulated. Operation on the part of the UPF controller 424 then ends.

[0117] 7, a schematic block diagram of a computing environment 403 is shown, according to an embodiment of the present disclosure. The computing environment 403 includes one or more computing devices 700. Each computing device 700 includes at least one processor circuit having, for example, a processor 703 and a memory 706, both of which are coupled to a local interface 709. To this end, each computing device 700 may include, for example, at least one server computer or similar device. The local interface 709 may include, for example, a data bus with an associated address / control bus, or other bus structure as may be understood.

[0118] Stored in the memory 706 are both data and several components executable by the processor 703. In particular, stored in the memory 706 and executable by the processor 703 are UPF-C c 328, UPF-U c 323, UPF controller 424, and potentially other applications. Also stored in memory 706 may be a data store 415 and other data. Additionally, an operating system may be stored in memory 706 and executable by processor 703.

[0119] As can be appreciated, it is understood that there may be other applications stored in memory 706 and executable by processor 703. Any of the components described herein, if implemented in the form of software, may employ any one of a number of programming languages, such as, for example, C, C++, C#, Objective C, Java, JavaScript, Perl, PHP, Visual Basic, Python, Ruby, Flash, or other programming languages.

[0120] Some software components are stored in memory 706 and executable by processor 703. In this regard, the term "executable" refers to a program file in a form that is ultimately executable by processor 703. Examples of executable programs may include, for example, a compiled program that can be loaded into a random access portion of memory 706 and converted into machine code in a format that is executable by processor 703, source code that may be expressed in a suitable format such as object code that can be loaded into a random access portion of memory 706 and executed by processor 703, or source code that can be interpreted by another executable program that generates instructions in the random access portion of memory 706 to be executed by processor 703, etc. Executable programs may be stored in any portion or component of memory 706, including, for example, random access memory (RAM), read only memory (ROM), hard drive, solid state drive, USB flash drive, memory card, optical disk such as a compact disk (CD) or digital versatile disk (DVD), floppy disk, magnetic tape, or other memory components.

[0121] Memory 706 is defined herein to include both volatile and non-volatile memory and data storage components. A volatile component is one that does not retain a data value upon loss of power. A non-volatile component is one that retains data upon loss of power. Thus, memory 706 may include, for example, random access memory (RAM), read only memory (ROM), a hard disk drive, a solid state drive, a USB flash drive, a memory card accessed via a memory card reader, a floppy disk accessed via an associated floppy disk drive, an optical disk accessed via an optical disk drive, a magnetic tape accessed via a suitable tape drive, and / or other memory components, or a combination of any two or more of these memory components. Additionally, RAM may include, for example, static random access memory (SRAM), dynamic random access memory (DRAM), or magnetic random access memory (MRAM), and other such devices. ROM may include, for example, a programmable read only memory (PROM), an erasable programmable read only memory (EPROM), an electrically erasable programmable read only memory (EEPROM), or other similar memory devices.

[0122] Also, processor 703 may represent multiple processors 703 and / or multiple processor cores, and memory 706 may represent multiple memories 706 each operating in parallel processing circuits. In such a case, local interface 709 may be a suitable network facilitating communication between any two of the multiple processors 703, between any processor 703 and any of the memories 706, or between any two of the memories 706, etc. Local interface 709 may include additional systems designed to regulate this communication, including, for example, implementing load balancing. Processor 703 may be an electrical structure or some other available structure.

[0123] UPF-Cc 328, UPF-U c 323, UPF controller 424, and various other systems described herein may be embodied in software or code executed by general purpose hardware, as discussed above, although, alternatively, the same may be embodied in dedicated hardware or a combination of software / general purpose hardware and dedicated hardware. If embodied in dedicated hardware, each may be implemented as a circuit or state machine using any one or combination of a number of technologies. These technologies may include, but are not limited to, discrete logic circuits having logic gates to implement various logic functions upon application of one or more data signals, application specific integrated circuits (ASICs) having appropriate logic gates, field programmable gate arrays (FPGAs), or other components, and the like. Such technologies are generally well known to those skilled in the art and will not be described in detail herein.

[0124] The flowcharts of Figures 5 and 6 show the functionality and operation of the implementation of the DU / CU 284 and the UPF controller 424. When embodied in software, each block may represent a module, segment, or portion of code including program instructions to implement a particular logical function(s). The program instructions may be embodied in the form of source code including human-readable statements written in a programming language, or machine code including numerical instructions recognizable by a suitable execution system, such as the processor 703 in a computer system or other system. The machine code may be translated from the source code, or the like. When embodied in hardware, each block may represent a circuit or a number of interconnected circuits to implement a particular logical function(s).

[0125] Although the flow charts of Figures 5 and 6 show a specific order of execution, it is understood that the order of execution may differ from that shown. For example, the order of execution of two or more blocks may be swapped out of the order shown. Also, two or more blocks shown in succession in Figures 5 and 6 may be executed concurrently or with partial concurrence. Furthermore, in some embodiments, one or more of the blocks shown in Figures 5 and 6 may be omitted or omitted. In addition, any number of counters, state variables, alert semaphores, or messages may be added to the logic flows described herein for purposes such as improving usability, explaining, measuring performance, or providing clues for solving problems. It is understood that all such variations are within the scope of the present disclosure.

[0126] Also, UPF-C c 328, UPF-U c Any logic or application described herein, including, for example, the processor 323, and the UPF controller 424, may be embodied in any non-transitory computer-readable medium for use by or in association with an instruction execution system, such as, for example, the processor 703 in a computer system or other system. In this sense, logic may include, for example, statements, including instructions and declarations, retrievable from a computer-readable medium and executable by an instruction execution system. In the context of this disclosure, a "computer-readable medium" may be any medium that can contain, store, or retain logic or applications described herein for use by or in association with an instruction execution system.

[0127] The computer-readable medium may include any one of a number of physical media, such as, for example, magnetic, optical, or semiconductor media. More specific examples of suitable computer-readable media may include, but are not limited to, magnetic tape, magnetic floppy disk, magnetic hard drive, memory card, solid-state drive, USB flash drive, or optical disk. The computer-readable medium may also be, for example, random access memory (RAM), including static random access memory (SRAM) and dynamic random access memory (DRAM), or magnetic random access memory (MRAM). In addition, the computer-readable medium may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or other types of memory devices.

[0128] In addition, UPF-C c 328, UPF-U c Any logic or application described herein, including the UPF controller 323 and the UPF controller 424, may be implemented or structured in a variety of ways. For example, one or more applications described herein may be implemented as modules or components of a single application. Furthermore, one or more applications described herein may execute on a shared or separate computing device, or a combination thereof. For example, multiple applications described herein may execute on the same computing device 700 or on multiple computing devices 700 of the same computing environment 403.

[0129] Unless otherwise indicated, disjunctive language, such as the phrase "at least one of X, Y, or Z," is understood in context as it is generally used to indicate that an item, term, etc. may be either X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z). Thus, such disjunctive language is generally not intended, and should not be intended, to imply that a particular embodiment requires that at least one of X, at least one of Y, or at least one of Z, respectively, be present.

[0130] It should be emphasized that the above-described embodiments of the present disclosure are merely possible examples of implementations described for a clear understanding of the principles of the present disclosure. Many variations and modifications may be made to the above-described embodiment(s) without substantially departing from the spirit and principles of the present disclosure. All such modifications and variations are intended to be included herein within the scope of the present disclosure and protected by the following claims.

[0131] For illustrative purposes, examples of various embodiments of the present disclosure are set forth in the following clauses, which describe some embodiments of the present disclosure, but are not intended to limit the disclosure to only the examples provided.

[0132] Clause 1. An S-system comprising: a Radio Access Network (RAN) providing network connectivity to a plurality of client devices; a core network implementing one or more network functions for the RAN; and a user plane function including a user plane component and a control plane component, the user plane component operating in the RAN and the control plane component operating in the core network to handle network traffic with the plurality of client devices.

[0133] Clause 2 - The system of clause 1, wherein the RAN is operated by a cloud service provider for a customer and the core network is implemented in at least one computing device of a cloud provider network operated by the cloud service provider.

[0134] Clause 3 - The system of clause 1 or 2, further comprising a user plane function controller configured to monitor multiple instances of the user plane components and status information of the multiple instances of the user plane components, and to distribute network traffic from the multiple client devices connected to the radio access network that is processed by a particular user plane component of the multiple instances based at least in part on the status information.

[0135] Clause 4 - The system of clause 3, wherein the user plane function controller is further configured to at least scale a quantity of the multiple instances of the user plane components of the user plane function based at least in part on the status information.

[0136] Clause 5 - The system of any of clauses 1 to 4, wherein an additional instance of the user plane component operates in the core network and is configured to process network traffic from another plurality of client devices connected to a different RAN that does not operate the user plane component.

[0137] Clause 6 - A system as described in any one of clauses 1 to 5, wherein the user plane component is configured to receive data packets that are not encapsulated by the GPRS Tunneling Protocol (GTP) and a further instance of the user plane component is configured to receive data packets encapsulated using GTP.

[0138] Clause 7 - The system described in any of clauses 1 to 6, wherein the control plane component is configured to select the user plane component from a plurality of user plane components for the network traffic from a particular client device of the plurality of client devices based at least in part on a network slice associated with the network traffic or at least one of quality of service parameters associated with the network traffic.

[0139] Clause 8 - A system as described in any of clauses 1 to 7, wherein the user plane component of the user plane functionality is implemented in at least one of a first computing device including a radio unit, or a second computing device implementing at least one of a distributed unit (DU) or centralized unit (CU) functionality for the radio access network.

[0140] Clause 9 - The system of any one of clauses 1 to 8, wherein the network traffic between a first client device of the plurality of client devices and a second client device of the plurality of client devices is configured to bypass the core network.

[0141] Clause 10 - A computer-implemented method comprising: receiving data from a client device connected to a radio access network of a wireless network; forwarding the data to a user plane component of a user plane function, the user plane component being implemented in a computing device of the radio access network; processing the data by the user plane component; and routing the processed data to a data network.

[0142] Clause 11 - The computer-implemented method of clause 10, wherein forwarding the data to the user plane component further comprises forwarding the data to the user plane component without encapsulating the data using GPRS Tunneling Protocol (GTP).

[0143] Clause 12 - The computer-implemented method of clause 10 or 11, further comprising forwarding other data from the data network destined for the client device to the user plane component without encapsulating the other data using GPRS Tunneling Protocol (GTP), thereby avoiding packet fragmentation resulting from one or more GTP headers causing a packet size of the data to exceed a maximum transmission unit value.

[0144] Clause 13 - A computer-implemented method according to any of clauses 10 to 12, further comprising receiving other data from the client device connected to the radio access network, encapsulating the other data using GPRS Tunneling Protocol (GTP), and forwarding the other data to different user plane components of the User Plane Function, the different user plane components being implemented in different computing devices of a core network of the wireless network.

[0145] Clause 14 - The computer-implemented method of any of clauses 10 to 13, further comprising configuring operation of a user plane component of the user plane function in response to configuration data received from a control plane component of the user plane function implemented in a different computing device of a core network of the wireless network.

[0146] Clause 15 - The computer-implemented method of any of clauses 10 to 14, further comprising determining to forward the data to a user plane component implemented in the computing device of the radio access network instead of forwarding the data to a different user plane component of the user plane function implemented in a different computing device of a core network of the wireless network based at least in part on a quality of service parameter associated with the data or at least one of the client devices.

[0147] Clause 16 - The computer-implemented method of any of clauses 10-15, further comprising: determining to forward the data to the user plane component implemented in the computing device of the radio access network instead of forwarding the data to a different user plane component of the user plane function implemented in a different computing device of a core network of the wireless network based at least in part on configuration data received from a user plane function controller that distributes network traffic among a plurality of user plane components of the user plane function implemented in the radio access network.

[0148] Clause 17 - A computer-implemented method comprising: receiving status information from a plurality of user plane component instances for user plane functions implemented in respective computing devices in a radio access network of a wireless network; and allocating network traffic from a plurality of client devices connected to the radio access network to respective ones of the plurality of user plane component instances based at least in part on the status information.

[0149] Clause 18 - The computer-implemented method of clause 17, wherein allocating the network traffic is based at least in part further on a quality of service parameter associated with the network traffic or at least one of a plurality of network slices associated with the network traffic.

[0150] Clause 19 - The computer-implemented method of clause 17 or 18, further comprising receiving other status information from another plurality of user plane component instances for the user plane function implemented in a computing device in a core network of the wireless network, and allocating the network traffic to the plurality of user plane component instances and the respective ones of the other user plane component instances based at least in part on the other status information.

[0151] Clause 20 - The computer-implemented method of any of clauses 17 to 20, wherein the status information includes at least one of a performance indicator or a heartbeat signal.

Claims

1. 1. A system comprising: A radio access network (RAN) of a wireless network, the RAN providing network connectivity to a plurality of client devices; a core network implementing one or more network functions for the RAN; a user plane function including a user plane component and a control plane component, the user plane component operating in the RAN and the control plane component operating in the core network, for handling network traffic to and from the plurality of client devices; a plurality of instances of the user plane component; a user plane function controller configured to monitor status information of the multiple instances of the user plane component and to distribute network traffic from the multiple client devices connected to the radio access network that is processed by a particular user plane component of the multiple instances based at least in part on the status information; Equipped with The Radio Access Network (RAN) comprises at least receiving data from a particular client device among a plurality of client devices connected to the wireless access network; forwarding data to the user plane component, the user plane component being implemented in a computing device of the radio access network; processing said data by said user plane component; Routing the processed data to a data network. The system is configured as follows.

2. 2. The system of claim 1, wherein the RAN is operated by a cloud service provider for a customer and the core network is implemented in at least one computing device of a cloud provider network operated by the cloud service provider.

3. 2. The system of claim 1, wherein the user plane function controller is further configured to at least scale a quantity of the multiple instances of the user plane components of the user plane function based at least in part on the status information.

4. 2. The system of claim 1, wherein an additional instance of the user plane component operates in the core network and is configured to process network traffic from another plurality of client devices connected to a different RAN that does not operate the user plane component.

5. 5. The system of claim 4, wherein the user plane component is configured to receive data packets that are not encapsulated by the Grand Prism Transmission System (GPRS) Tunneling Protocol (GTP), and an additional instance of the user plane component is configured to receive data packets encapsulated using GTP.

6. 2. The system of claim 1, wherein the control plane component is configured to select the user plane component from a plurality of user plane components for the network traffic from a particular client device based at least in part on at least one of a network slice associated with the network traffic or a quality of service parameter associated with the network traffic.

7. 2. The system of claim 1, wherein the user plane components of the user plane functionality are implemented in at least one of a first computing device including a radio unit or a second computing device implementing at least one of a distributed unit (DU) or centralized unit (CU) functionality for the radio access network.

8. The system of claim 1 , wherein the network traffic between a first client device of the plurality of client devices and a second client device of the plurality of client devices is configured to bypass the core network.

9. 2. The system of claim 1, wherein the user plane function controller is further configured to allocate the network traffic further based on at least a portion of a plurality of network slices associated with the network traffic.

10. The system of claim 1 , wherein the status information includes at least one of a performance indicator or a heartbeat signal.

11. 1. A computer-implemented method, comprising: The wireless network receiving data from a client device connected to a radio access network of the wireless network; forwarding the data to a user plane component of a user plane function, the user plane component being implemented in a computing device of the radio access network; processing the data by the user plane component; and routing the processed data to a data network; configuring operation of the user plane component of the user plane function in response to configuration data received from a control plane component of the user plane function implemented in a second computing device of a core network of the wireless network; The computer-implemented method includes performing the following steps:

12. The computer-implemented method of claim 11, wherein the wireless network forwarding the data to the user plane component further comprises forwarding the data to the user plane component without encapsulating the data using the GPRS Tunneling Protocol (GTP).

13. The computer-implemented method of claim 11, further comprising the wireless network forwarding other data from the data network destined for the client device to the user plane component without encapsulating the other data using GPRS Tunneling Protocol (GTP), thereby avoiding packet fragmentation resulting from one or more GTP headers causing a packet size of the data to exceed a maximum transmission unit value.

14. The wireless network, receiving other data from the client devices connected to the wireless access network; encapsulating said other data using the GPRS Tunneling Protocol (GTP); and forwarding the other data to different user plane components of the User Plane Function, the different user plane components being implemented in different computing devices of a core network of the wireless network. The computer-implemented method of claim 11 , further comprising:

15. The computer-implemented method of claim 11, further comprising: the wireless network determining to forward the data to a user plane component implemented on the computing device of the radio access network instead of forwarding the data to a different user plane component of the user plane function implemented on a different computing device of a core network of the wireless network based at least in part on a quality of service parameter associated with the data or at least one of the client devices.

16. The computer-implemented method of claim 11, further comprising: determining, based at least in part on configuration data received from a user plane function controller that allocates network traffic among multiple user plane components of the user plane function implemented in the radio access network, to forward the data to the user plane component implemented in the computing device of the radio access network instead of forwarding the data to a different user plane component of the user plane function implemented in a different computing device of a core network of the wireless network.

17. 1. A computer-implemented method, comprising: The wireless network receiving data from a client device connected to a radio access network of the wireless network; selecting a user plane component of a user plane function from a plurality of user plane components for the data from the client device based on at least one of a network slice associated with the data or a quality of service parameter associated with the data; forwarding the data to the user plane component, the user plane component being implemented in a computing device of the radio access network; processing the data by the user plane component; and Routing the processed data to a data network. A computer-implemented method comprising:

18. The wireless network, receiving status information from a plurality of user plane component instances for user plane functions implemented in respective computing devices in the radio access network; and allocating network traffic from a plurality of client devices coupled to the wireless access network to respective ones of the plurality of user plane component instances based at least in part on the status information; 20. The computer-implemented method of claim 17, further comprising:

19. The computer-implemented method of claim 17, wherein the wireless network forwarding the data to the user plane component further comprises forwarding the data to the user plane component without encapsulating the data using the GPRS Tunneling Protocol (GTP).

20. The wireless network, receiving other data from the client devices connected to the wireless access network; encapsulating said other data using the GPRS Tunneling Protocol (GTP); and forwarding the other data to different user plane components of the User Plane Function, the different user plane components being implemented in different computing devices of a core network of the wireless network.

20. The computer-implemented method of claim 17, further comprising:

Citation Information

Patent Citations

  • Method and apparatus for business transition

    JP2021509245A

  • Communication method, apparatus, and system

    US20200221540A1

  • Enabling NAT for user plane traffic

    WO2021009166A1