Extending cloud-based virtual private networks to user devices over wireless networks

Extending cloud-based VPCs to wireless networks with cloud-native 5G and network slicing addresses scalability and management challenges, providing flexible and reliable connectivity for diverse applications by dynamically allocating resources and ensuring QoS.

JP2024541412APending Publication Date: 2024-11-08AMAZON TECH INC
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2024529439
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2021-11-24
Filing Date
2022-11-25
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Existing wireless networks face challenges in providing flexible, reliable, and cost-effective connectivity solutions that meet strict quality of service (QoS) requirements for diverse applications, such as IoT devices and real-time communications, due to limitations in scalability, management complexity, and vendor-proprietary hardware dependencies.

Method used

The extension of cloud-based virtual private networks (VPCs) to user equipment on wireless networks, utilizing cloud-native 5G networks with network slicing and microservices architecture, allows for dynamic scaling, resource allocation, and management of QoS parameters, decoupling hardware from software to enhance flexibility and performance.

Benefits of technology

This approach enables reliable, low-latency, and high-bandwidth connectivity for diverse applications, supporting new use cases like factory IoT, augmented reality, and autonomous navigation by dynamically allocating resources and ensuring QoS through cloud-native infrastructure and network functions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024541412000001_ABST
    Figure 2024541412000001_ABST
Patent Text Reader

Abstract

Various embodiments are disclosed for extending a cloud-based virtual private network to user equipment on a wireless network. In one embodiment, a request for service from a client device is received from the wireless network. In response to receiving the request from the client device for service from the wireless network, the client device is provided with access to a virtual private cloud network over the wireless network. Encapsulated network traffic is forwarded from the client device to the virtual private cloud network over a network link between the wireless network and a cloud provider network that hosts one or more resources on the virtual private cloud network.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to and the benefit of U.S. patent application Ser. No. 17 / 535,564, entitled "EXTENDING CLOUD-BASED VIRTUAL PRIVATE NETWORKS TO USER EQUIPMENT ON RADIO-BASED NETWORKS," filed Nov. 24, 2021, the entirety of which is incorporated herein by reference. [Background technology]

[0002] 5G is the fifth generation technology standard for broadband cellular networks and is planned to eventually replace the fourth generation (4G) standard Long Term Evolution (LTE). 5G technology will expand the cellular market beyond smartphones by providing significantly increased bandwidth to provide last-mile connectivity to desktops, set-top boxes, laptops, Internet of Things (IoT) devices, and more. Some 5G cells can use frequency spectrum similar to 4G, while other 5G cells can use millimeter wave frequency spectrum. The coverage area of ​​millimeter wave cells is relatively small, but they offer much higher throughput than 4G. Summary of the Invention

[0003] Many aspects of the present disclosure can be better understood with reference to the following drawings. The components in the drawings are not necessarily to scale, emphasis instead being placed upon clearly illustrating the principles of the present disclosure. Moreover, in the drawings, like reference numbers designate corresponding parts in the several views. [Brief description of the drawings]

[0004] [Figure 1] 1 is a drawing of an example communications network that may be deployed and managed in accordance with various embodiments of the present disclosure. [Figure 2A] 1 illustrates an example of a network environment including a cloud provider network according to some embodiments of the disclosure, and further including various provider substrate extensions of the cloud provider network that may be used at various locations within the communications network of FIG. [Figure 2B] 2 illustrates an example of cellularization and geographic distribution of the communication network of FIG. 1 in accordance with some embodiments of the present disclosure. [Figure 3A] 2B illustrates an example of the network environment of FIG. 2A including geographically distributed provider substrate extensions in accordance with some embodiments of the present disclosure. [Figure 3B] 2B illustrates an example of the network environment of FIG. 2A including a virtual private cloud network, according to some embodiments of the present disclosure. [Figure 4] 2B is a schematic block diagram of the network environment of FIG. 2A in accordance with various embodiments of the present disclosure. [Diagram 5] 5 is a flowchart illustrating example functionality implemented as part of the wireless network and virtual private cloud network in the network environment of FIG. 4 in accordance with various embodiments of the present disclosure. [Figure 6] 5 is a flowchart illustrating example functionality implemented as part of a virtual private cloud gateway running on a wireless network in the network environment of FIG. 4 in accordance with various embodiments of the present disclosure. [Figure 7] 5 is a flowchart illustrating example functionality implemented as part of a virtual private cloud application running on a client device in the network environment of FIG. 4 in accordance with various embodiments of the present disclosure. [Figure 8] 5 is a schematic block diagram providing an example of a computing environment for use in the network environment of FIG. 4, in accordance with various embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0005] The present disclosure relates to extending cloud-based virtual private networks to user equipment over wireless networks. A virtual private cloud (VPC) is a custom-defined virtual network within another network, such as a cloud provider network. A VPC can provide an underlying network layer for cloud services, such as a compute cloud or edge cloud, or for customer applications or workloads running on the cloud. A VPC can be defined by at least its address space, internal structure (e.g., computing resources that make up the VPC), and transit paths. VPC resources are typically hosted and provisioned within the cloud provider network, but customer-owned networks can be connected to the VPC through gateways. In hosting a VPC, the cloud provider network implements a logical construct for provisioning the VPC using the cloud provider network's physical and, optionally, virtual resources.

[0006] A wireless network includes one or more radio access networks (RANs) and associated core networks to provide network connectivity to wireless devices. The RANs can operate using Wi-Fi, 4G, 5G, sixth generation (6G), and other access technologies. A wireless network can support network slicing to ensure that quality of service (QoS) requirements are met for customers and their applications. An individual network slice of a wireless network satisfies one or more QoS parameters by reserving network capacity of both the RAN and associated core network for a set of applications or devices, potentially for a defined period of time. The QoS parameters may relate to latency, bandwidth, burst bandwidth, number of network hops, etc., and may be defined in terms of maximum, minimum, average, or other thresholds.

[0007] Various embodiments of the present disclosure extend the concept of VPCs to user equipment on wireless networks such that user equipment, such as smartphones, Internet of Things (IoT) devices, and / or other wireless devices connected to a particular wireless network, can be placed on a VPC, thereby allowing the device to access other resources on the VPC. In this manner, user equipment connected to a wireless network can communicate with other user equipment on the wireless network that are on a VPC, and the same user equipment can communicate with other devices and resources on a VPC in a cloud provider network. Furthermore, in some embodiments, individual network slices on a wireless network can each provide access to a different VPC, or different sub-networks (subnets) of a VPC, which may be associated with different resources governed by different access controls. In some scenarios, a wireless device is in data communication with a VPC via the wireless device's real network interface, rather than through a virtual network interface such as tunneling.

[0008] In some embodiments, an application on the user equipment or an operating system of the user equipment routes network traffic to the VPC by tunneling and packet encapsulation. In such embodiments, the application or operating system may be configured to perform various VPC functions, such as enforcing security group rules for access control and maintaining a VPC routing table that maps VPC network addresses to next-hop network addresses. In other embodiments, a component of the wireless network in either the radio access network or the core network acts as a VPC proxy to route network traffic between the user equipment and the VPC by tunneling and packet encapsulation. In the latter embodiment, resource consumption on the user equipment can be reduced because the tunneling and encapsulation is not performed directly on the user equipment.

[0009] A VPC is a virtual network dedicated to a particular customer account (or a set of related customer accounts, such as different customer accounts belonging to the same business organization). A VPC is logically isolated from other virtual networks in the cloud. A customer can launch resources, such as compute instances, into a VPC. When creating a VPC, a customer can specify the VPC's IPv4 and / or IPv6 address range in the form of a Classless Inter-Domain Routing (CIDR) block. A VPC may span all availability zones within a particular region. After creating a VPC, a customer can add one or more subnets to each availability zone or edge location. In accordance with the present disclosure, a customer can also add a VPC's subnet to a network slice. A customer can request, for example, via an API call to the cloud provider network, that a subscriber identity module of a particular client device (or set of client devices) be registered with one or both of a network slice and a VPC. In response, the cloud provider network can assign an Internet Protocol (IP) address from the VPC's IP address range to the subscriber identity module. This allows the client device to access resources of the VPC, subject to further access control, through the use of a network slice of a wireless network.

[0010] Access control can refer to security groups or network access control lists. Security groups (also referred to in various implementations as network security groups, application security groups, cloud security groups, or compute engine firewall rules) act as a virtual firewall for virtual machine instances to control inbound and outbound traffic. Customers can define security groups as policies that can be applied to specific instances. Customers can assign one or more security groups to an instance when launching the instance in a VPC. Security groups may work at the instance level rather than the subnet level. Thus, each instance in a subnet can be assigned to a different set of security groups. For each security group, customers can add rules that control inbound traffic to the instance and another set of rules that control outbound traffic. Security groups can be stateful in that return traffic is automatically allowed.

[0011] Customers can also configure network access control lists (ACLs) with rules similar to security groups to add an additional layer of security to their VPC. Network ACLs operate at the subnet level, support allow and deny rules, and are automatically applied to all instances in any subnet they are associated with. Network ACLs may not be stateful, in that return traffic must be explicitly allowed by the rules.

[0012] A wireless network may use a core network infrastructure that is dynamically provisioned and used in conjunction with multiple different radio access networks operated by multiple communication service providers. Although wireless networks are provisioned on demand, they can also be dynamically scaled up or down or terminated, providing organizations with the ability to create temporary wireless networks that may exist for a specific period of time or periodically according to a schedule. Additionally, cell sites can be dynamically added or removed from a wireless network on demand. In various scenarios, organizations can use embodiments of the present disclosure to create either private wireless networks for internal use only or wireless networks that are open to third-party customers.

[0013] Previously, wireless network deployments relied on manual deployment and configuration at each step of the process, which proved to be time-consuming and expensive. Additionally, in previous generations, the software was essentially tied to the vendor's proprietary hardware, preventing customers from deploying alternative software. In contrast, 5G decouples the hardware from the software stack, allowing for greater flexibility and allowing components of wireless networks to run on cloud provider infrastructure. Using a cloud delivery model for wireless networks such as 5G networks can facilitate handling network traffic from hundreds to billions of connected devices and computationally intensive applications, while delivering at higher speeds, lower latency, and higher capacity than other types of networks.

[0014] Until now, businesses have had to choose between performance and price when evaluating enterprise connectivity solutions. Cellular networks can offer high performance, excellent indoor and outdoor coverage, and advanced quality of service (QoS) connectivity features, but private cellular networks can be expensive and complex to manage. Ethernet and Wi-Fi require less up-front investment and are easier to manage, but businesses often find they are unreliable, require a lot of work to get optimal coverage, and do not offer QoS features such as guaranteed bit rates, latency, and reliability.

[0015] Enterprises can freely deploy various 5G devices and sensors across the enterprise (factory floors, warehouses, lobbies, and communication centers) and manage these devices, register users, and assign QoS from a management console. Using the disclosed technology, customers can assign a certain bit rate of throughput to every device (such as a camera, sensor, or IoT device), assign reliable low latency connections to devices running on the factory floor, and assign broadband connections to every handheld device. The disclosed service can manage all the software required to provide connectivity that meets the specified constraints and requirements. This enables an entirely new set of applications with stringent QoS requirements or high IoT device density requirements that could not traditionally be run on Wi-Fi networks. Furthermore, the disclosed service can provide application deployment application program interfaces (APIs) that expose and manage 5G features like QoS, enabling customers to build applications that can take full advantage of the network's latency and bandwidth capabilities without needing to understand the network details.

[0016] Additionally, the disclosed service can provide a private zone for running local applications within the cloud provider network. This private zone can be connected to and effectively become part of the broader regional zone, and customers can manage this private zone using the same APIs and tools that are used in the cloud provider network. Similar to availability zones, private zones can be assigned virtual private network subnets. APIs can be used to create subnets and assign them to all zones that customers want to use, including private zones and other existing zones. A management console can provide a simplified process for creating private zones. Virtual machine instances and containers can be launched in private zones just like in regional zones. Customers can configure network gateways to define routes, assign IP addresses, set up network address translation (NAT), etc. Autoscaling allows for scaling capacity of virtual machine instances or containers as needed in the private zone. The same management and authentication APIs can be used in the private zone as in the cloud provider network. In some cases, cloud services available in the regional zone can be accessed remotely from the private zone over a secure connection, allowing access to these cloud services without the need to upgrade or modify local deployments.

[0017] Various embodiments of the present disclosure may also incorporate concepts of elasticity and utility computing from the cloud computing model into wireless and associated core networks. For example, the disclosed techniques may run core and radio access network functions, as well as associated control plane management functions, on cloud provider infrastructure to create a cloud-native core network and / or a cloud-native radio access network (RAN). Such core and RAN network functions may be based on 3rd Generation Partnership Project (3GPP) specifications in some implementations. Providing a cloud-native wireless network allows customers to dynamically scale their wireless networks based on utilization, latency requirements, and / or other factors. Customers may also configure thresholds to receive alerts regarding wireless network usage and excess capacity usage of provisioned infrastructure to more effectively manage the provisioning of new infrastructure or de-provisioning of existing infrastructure based on dynamic networking and workload requirements.

[0018] As one skilled in the art will appreciate in light of this disclosure, certain embodiments may achieve certain advantages, including some or all of the following: (1) improving security of the wireless network by enabling a combination of point-to-point and end-to-end encryption to connect devices on the wireless network to resources available on the virtual private cloud network, (2) improving flexibility of geographically distributed networks by enabling the virtual private cloud network to extend to wireless networks, including 4G and 5G networks, and (3) improving reliability and performance of services on the virtual private cloud network by implementing network slicing on wireless network segments to meet quality of service requirements.

[0019] Among the advantages of the present disclosure is the ability to deploy and chain network functions to deliver end-to-end services that meet specified constraints and requirements. According to the present disclosure, network functions organized into microservices work together to provide end-to-end connectivity. One set of network functions is part of the wireless network, operating at base stations and performing radio signal to IP conversion. Other network functions run in large data centers that perform subscriber-related business logic and route IP traffic to and from the Internet. For applications to use the new features of 5G, such as low latency communications and reserved bandwidth, both of these types of network functions must work together to properly schedule and reserve the wireless spectrum and perform real-time computations and data processing. The technology disclosed herein provides edge location hardware (described further below) integrated with network functions that run throughout the network, from cell sites to Internet breakouts, and orchestrates the network functions to meet required quality of service (QoS) constraints. This enables an entirely new set of applications with stringent QoS requirements that were previously not possible to run on mobile networks, ranging from factory-based Internet of Things (IoT) to augmented reality (AR), virtual reality (VR), game streaming, and autonomous navigation support for connected vehicles.

[0020] The described "elastic 5G" service provides and manages all the hardware, software, and network functions required to build the network. In some embodiments, the network functions may be developed and managed by a cloud service provider, but the described control plane can manage the network functions across different providers so that customers can use a single set of APIs to invoke and manage selected network functions on the cloud infrastructure. The elastic 5G service has the advantage of automating the creation of an end-to-end 5G network, from hardware to network functions, thereby reducing the time to deploy and the operational costs of running the network. By providing APIs that expose the network functions, the disclosed elastic 5G service allows applications to specify the desired QoS as constraints and then simply deploy and chain network functions to deliver an end-to-end service that meets the specified requirements, thus making it easier to build new applications.

[0021] This disclosure describes embodiments related to creating and managing cloud-native 5G core and / or cloud-native 5G RAN and associated control plane components. Cloud-native refers to an approach to building and running applications that leverage the benefits of the cloud computing delivery model, such as dynamic scalability, distributed computing, and high availability (including geographic distribution, redundancy, and failover). Cloud-native refers to the way these applications are created and deployed to make them suitable for deployment in the public cloud. Cloud-native applications can (and often do) run in the public cloud, but can also run in on-premise data centers. Some cloud-native applications can be containerized, e.g., packaging different parts, functions, or subunits of an application into their own containers, which can be dynamically orchestrated so that each part is actively scheduled and managed to optimize resource utilization. These containerized applications can be built using a microservices architecture to improve the overall agility and maintainability of the application.

[0022] In a microservices architecture, an application is arranged as a collection of smaller sub-units ("microservices") that can be deployed and scaled independently of each other and communicate with each other over a network. These microservices are typically fine-grained in that they have a particular technical and functional granularity and often implement lightweight communication protocols. The microservices of an application can perform different functions, may be independently deployable, and may use different programming languages, databases, and hardware / software environments. Decomposing an application into smaller services beneficially increases the modularity of the application, allows individual microservices to be replaced as needed, and parallelizes development by allowing teams to develop, deploy, and maintain their microservices independently of each other. The microservices may be deployed using virtual machines, containers, or serverless functions in some examples. The disclosed core and RAN software may follow a microservices architecture such that the described wireless network is composed of independent sub-units that can be deployed and scaled on demand.

[0023] Referring now to FIG. 1, an example of a communications network 100 deployed and managed in accordance with various embodiments of the present disclosure is shown. The communications network 100 includes a wireless network (RBN) 103, which may correspond to a cellular network, such as a high speed packet access (HSPA) network, a fourth generation (4G) long term evolution (LTE) network, a fifth generation (5G) network, a sixth generation (6G) network, a 4G-5G hybrid core with both 4G and 5G RAN, or other network that provides wireless network access. The wireless network 103 may be operated by a cloud service provider for a business, a non-profit organization, a school system, a government agency, or other organization. The wireless network 103 may use private or public network addresses in various embodiments.

[0024] Various deployments of the wireless network 103 may include one or more of a core network and a RAN network, as well as a control plane for running the core network and / or the RAN network on a cloud provider infrastructure. As mentioned above, these components can be developed in a cloud-native manner, for example using a microservices architecture, such that centralized control and distributed processing are used to efficiently scale traffic and transactions. These components can be based on 3GPP specifications by following a control plane and user plane processing separated application architecture (CUPS architecture).

[0025] The wireless network 103 provides wireless network access to a number of wireless devices 106, which may be mobile devices or fixed location devices. In various examples, the wireless devices 106 may include devices such as smartphones, connected vehicles, IoT devices, sensors, machines (such as in a manufacturing facility), hotspots, etc. The wireless devices 106 may also be referred to as user equipment (UE) or customer premises equipment (CPE).

[0026] In this example, two virtual private cloud networks 107a and 107b of a cloud provider network are shown extended within the wireless network 103 to include wireless devices 106 as client devices. Virtual private cloud network 107a is shown to include four wireless devices 106, while virtual private cloud network 107b is shown to include two wireless devices 106. This shows that a single wireless network 103 can provide access to multiple different virtual private cloud networks 107, possibly enabled on a cell-by-cell basis. Each virtual private cloud network 107 can be assigned one or more network slices for use by the wireless devices 106 configured to meet one or more quality of service requirements.

[0027] Wireless devices 106 grouped within a particular virtual private cloud network 107 can communicate with each other as if they were on a single network segment, even if they are physically connected to different cells or points within the wireless network 103. Furthermore, wireless devices 106 within a particular virtual private cloud network 107 can communicate with services and other resources hosted on the virtual private cloud network 107 as if they were on a local network. Such resources may be located within the wireless network 103 (e.g., at a cell site, an intermediate site, or a central location) or within a regional data center of the cloud provider network.

[0028] The wireless network 103 may include capacity provisioned on one or more radio access networks (RANs) that provide wireless network access to multiple wireless devices 106 through multiple cells 109. The RANs may be operated by different communication service providers. Each of the cells 109 may be equipped with one or more antennas as well as one or more radio units that transmit and receive wireless data signals to and from the wireless devices 106. The antennas may be configured for one or more frequency bands, and the radio units may also be frequency agile or frequency tunable. To focus signals in a particular direction or azimuth range, the antennas may be associated with a particular gain or beamwidth, which may allow for frequency reuse in different directions. Furthermore, the antennas may be horizontally, vertically, or circularly polarized. In some examples, the radio units may utilize multiple-input multiple-output (MIMO) technology to transmit and receive signals. Thus, the RANs implement radio access technologies that enable wireless connections with the wireless devices 106 and provide connectivity with the core network of the wireless network. Components of the RAN may include base stations and antennas covering a given physical area as well as the necessary core network items for managing connections to the RAN.

[0029] Data traffic is routed to the core network through a fiber transport network, which is often composed of multiple hops of layer 3 routers (e.g., aggregation sites). The core network is typically housed in one or more data centers. Typically, the core network aggregates data traffic from end devices, authenticates subscribers and devices, applies personalized policies, manages device mobility, and then routes traffic to operator services or the Internet. For example, the 5G core may be decomposed into several microservice elements, separating the control plane and the user plane. The 5G core may include virtualized software-based network functions (e.g., deployed as microservices) rather than physical network elements, and thus may be instantiated within a multi-access edge computing (MEC) cloud infrastructure. The network functions of the core network may include a user plane function (UPF), an access and mobility management function (AMF), and a session management function (SMF), which are described in more detail below. For data traffic destined for locations external to communications network 100, the network functions typically include a firewall through which traffic can enter or exit communications network 100 to external networks, such as the Internet or a cloud provider network. Note that in some embodiments communications network 100 may include facilities that allow traffic to enter or exit sites further downstream from the core network (e.g., aggregation sites or wireless network 103).

[0030] The UPF provides an interconnection point between the mobile infrastructure and the data network (DN), i.e., General Packet Radio Service (GPRS) Tunneling Protocol (GTP-U) encapsulation and decapsulation of the user plane. The UPF can also provide a session anchor point to provide mobility within the RAN, such as sending one or more end marker packets to the RAN base station. The UPF can also handle packet routing and forwarding, such as steering flows to specific data networks based on traffic match filters. Another function of the UPF includes per-flow or per-application QoS handling, such as uplink (UL) and downlink (DL) transport-level packet marking and rate limiting. The UPF can be implemented as a cloud-native network function using modern microservices techniques, for example, deployable within a serverless framework (which abstracts the underlying infrastructure on which the code runs via managed services).

[0031] The AMF may receive connection and session information from the wireless device 106 or the RAN and may handle connection and mobility management tasks. For example, the AMF may manage handovers between base stations within the RAN. In some examples, the AMF may be considered an access point to the 5G core by terminating traffic for a particular RAN control plane and the wireless device 106. The AMF may implement encryption and integrity protection algorithms.

[0032] The SMF can handle session establishment or modification, for example by creating, updating, and deleting Protocol Data Unit (PDU) sessions and managing session context within the UPF. The SMF can also implement Dynamic Host Configuration Protocol (DHCP) and IP Address Management (IPAM). The SMF can be implemented as a cloud-native network function using modern microservices approaches.

[0033] Various network functions for implementing wireless network 103 may be deployed on distributed computing device 112, which may correspond to a general-purpose computing device configured to perform network functions. For example, distributed computing device 112 may run one or more virtual machine instances that are in turn configured to run one or more services that implement the network functions. In one embodiment, distributed computing device 112 is a ruggedized machine that is deployed at each cell site.

[0034] In contrast, one or more centralized computing devices 115 may perform various network functions at a central site operated by a customer. For example, the centralized computing devices 115 may be centrally located in an air-conditioned server room at the customer premises. The centralized computing devices 115 may run one or more virtual machine instances that are in turn configured to run one or more services that implement the network functions.

[0035] In one or more embodiments, network traffic from wireless network 103 is backhauled to one or more core computing devices 118, which may be located in one or more data centers located remotely from the customer sites. Core computing devices 118 may also perform various network functions, including routing network traffic to and from network 121, which may correspond to the Internet and / or other external public or private networks. Core computing devices 118 may perform functions related to management of communications network 100 (e.g., billing, mobility management, etc.) and transport functions for relaying traffic between communications network 100 and other networks.

[0036] FIG. 2A illustrates an example of a network environment 200 including a cloud provider network 203 and various provider substrate extensions of the cloud provider network 203 that may be used in combination with on-premise customer deployments in the communications network 100 of FIG. 1, according to some embodiments. The cloud provider network 203 (sometimes simply referred to as the "cloud") refers to a pool of network-accessible computing resources (e.g., compute, storage and networking resources, applications, and services), which may be virtualised or bare metal. A cloud can provide convenient on-demand network access to a shared pool of configurable computing resources that can be programmatically provisioned and released in response to customer commands. These resources can be dynamically provisioned and reconfigured to adjust to fluctuating loads. Thus, cloud computing can be viewed as both applications delivered as a service over a publicly accessible network (e.g., the Internet, a cellular communications network) and the hardware and software in the cloud provider's data centre that provides those services.

[0037] The cloud provider network 203 may provide users with an on-demand, scalable computing platform over the network, for example, using compute servers (which provide compute instances using one or both of a central processing unit (CPU) and a graphics processing unit (GPU), optionally with local storage) and block store servers (which provide persistent block storage virtualized to a given compute instance) to allow users to have scalable "virtual computing devices" at their disposal. These virtual computing devices have the attributes of a personal computing device, including hardware (various types of processors, local memory, random access memory (RAM), hard disk, and / or solid state drive (SSD) storage), a selected operating system, network capabilities, and preloaded application software. Each virtual computing device may also virtualize its console input / output (e.g., keyboard, display, and mouse). This virtualization allows users to connect to their virtual computing devices using computer applications, such as browsers, APIs, software development kits (SDKs), etc., to configure and use their virtual computing devices as if they were personal computing devices. Unlike a personal computing device, where a user has a fixed amount of hardware resources available to them, the hardware associated with a virtual computing device can be scaled up or down depending on the resources needed by the user.

[0038] As mentioned above, users can use various interfaces 206 (e.g., APIs) to connect to virtualized computing devices and other cloud provider network 203 resources and services via intermediate network(s) 212 to configure and manage telecommunications networks, such as 5G networks. An API refers to an interface 206 and / or communication protocol between a client device 215 and a server such that if the client makes a request in a predefined format, the client should receive a response in a specific format or initiate a defined action. In the context of a cloud provider network, an API provides a gateway for customers to access the cloud infrastructure by allowing customers to retrieve data from the cloud provider network 203 or perform actions within the cloud provider network 203, enabling the development of applications that interact with resources and services hosted in the cloud provider network 203. APIs can also enable different services in the cloud provider network 203 to exchange data with each other. Users can choose to deploy their own virtual computing systems to provide network-based services for their own use and / or for use by their customers or clients.

[0039] Cloud provider network 203 may include a physical network (e.g., sheet metal boxes, cables, rack hardware) called a substrate. The substrate may be considered a network fabric that includes the physical hardware that runs the provider network's services. The substrate may be isolated from the rest of cloud provider network 203. For example, it may not be possible to route from a substrate network address to an address in the production network that runs the cloud provider's services or to a customer network that hosts customer resources.

[0040] Cloud provider network 203 may also include an overlay network of virtualized computing resources running on the substrate. In at least some embodiments, a hypervisor or other device or process on the network substrate may use encapsulation protocol techniques to encapsulate network packets (e.g., client IP packets) and route the network packets (e.g., client IP packets) on the network substrate between client resource instances on different hosts in the provider network. Encapsulation protocol techniques may be used on the network substrate to route the encapsulated packets (also called network substrate packets) between endpoints on the network substrate via overlay network paths or routes. The encapsulation protocol techniques may be viewed as providing a virtual network topology overlaid on the network substrate. Thus, network packets may be routed along the substrate network according to configurations in the overlay network (e.g., virtual networks, which may be called virtual private clouds (VPCs), port / protocol firewall configurations, which may be called security groups). A mapping service (not shown) may coordinate the routing of these network packets. The mapping service may be a regional distributed lookup service that maps overlay Internet Protocol (IP) and network identifier combinations to substrate IPs, allowing distributed substrate computing devices to look up packet destinations.

[0041] To illustrate, each physical host device (e.g., compute server, block store server, object store server, control server) may have an IP address in the substrate network. Hardware virtualization techniques may enable multiple operating systems to run simultaneously on a host computer, e.g., as virtual machines (VMs) on a compute server. A hypervisor, or virtual machine monitor (VMM), on the host allocates the host's hardware resources to the various VMs on the host and monitors the execution of the VMs. Each VM may be given one or more IP addresses in the overlay network, and the VMM on the host may know the IP addresses of the VMs on the host. The VMM (and / or other devices or processes on the network substrate) may use encapsulation protocol techniques to encapsulate network packets (e.g., client IP packets) and route the network packets (e.g., client IP packets) over the network substrate between virtualized resources on different hosts in the cloud provider network 203. Encapsulation protocol techniques may be used on the network substrate to route the encapsulated packets between endpoints on the network substrate via overlay network paths or routes. The encapsulation protocol technology may be viewed as providing a virtual network topology overlaid on a network substrate. The encapsulation protocol technology may include a mapping service that maintains a mapping directory that maps IP overlay addresses (e.g., IP addresses visible to customers) to substrate IP addresses (IP addresses not visible to customers). This mapping directory may be accessed by various processes on the cloud provider network 203 to route packets between endpoints.

[0042] As described, the traffic and operations of the cloud provider network substrate may be broadly subdivided into two categories, in various embodiments: control plane traffic, which is performed on a logical control plane 218, and data plane operations, which are performed on a logical data plane 221. The data plane 221 represents the movement of user data through the distributed computing system, and the control plane 218 represents the movement of control signals through the distributed computing system. The control plane 218 generally includes one or more control plane components or services distributed across and implemented by one or more control servers. The control plane traffic generally includes management operations such as establishing isolated virtual networks for various customers, monitoring resource usage and health, identifying specific hosts or servers on which requested compute instances will be launched, and provisioning additional hardware as needed. The data plane 221 includes customer resources (e.g., compute instances, containers, block storage volumes, databases, file storage) that are implemented on the cloud provider network 203. The data plane traffic generally includes non-management operations such as transferring data to and from customer resources.

[0043] Control plane components are typically implemented on a separate set of servers from the data plane servers, and the control plane and data plane traffic may be transmitted over separate / different networks. In some embodiments, the control plane and data plane traffic may be supported by different protocols. In some embodiments, messages (e.g., packets) transmitted over the cloud provider network 203 include a flag indicating whether the traffic is control plane or data plane traffic. In some embodiments, the payload of the traffic may be inspected to determine its type (e.g., control plane or data plane). Other techniques for distinguishing between traffic types are possible.

[0044] As described, data plane 221 may include one or more compute servers, which may be bare metal (e.g., single tenant) or may be virtualized by a hypervisor to run multiple VMs (sometimes referred to as “instances”) or micro-VMs for one or more customers. These compute servers may support virtualized computing services (or “hardware virtualization services”) of cloud provider network 203. The virtualized computing services may be part of control plane 218 and allow customers to issue commands via interfaces 206 (e.g., APIs) to launch and manage compute instances (e.g., VMs, containers) for applications. The virtualized computing services may provide virtual computing instances with various computational and / or memory resources. In one embodiment, each of the virtual computing instances may correspond to one of several instance types. An instance type may be characterized by its hardware type, computational resources (e.g., the number, type, and configuration of CPUs or CPU cores), memory resources (e.g., the capacity, type, and configuration of local memory), storage resources (e.g., the capacity, type, and configuration of locally accessible storage), network resources (e.g., the characteristics of its network interfaces and / or network capabilities), and / or other suitable descriptive characteristics. The instance type selection function may be used to select an instance type for a customer, e.g., based (at least in part) on input from the customer. For example, a customer may select an instance type from a set of predefined instance types. As another example, a customer may specify the desired resources of the instance type and / or the requirements of the workload the instance will run, and the instance type selection function may select an instance type based on such specifications.

[0045] The data plane 221 may also include one or more block store servers, which may include persistent storage for storing customer data volumes and software for managing these volumes. These block store servers may support the managed block storage services of the cloud provider network 203. The managed block storage services may be part of the control plane 218 and allow customers to issue commands via interfaces 206 (e.g., APIs) to create and manage volumes for applications running on the compute instances. The block store servers include one or more servers where data is stored as blocks. A block is a sequence of bytes or bits, typically containing an integer number of records with a maximum length of the block size. Blocked data is typically stored in a data buffer, and an entire block is read and written at a time. In general, a volume may correspond to a logical collection of data, such as a set of data maintained on behalf of a user. A user volume, which may be treated as an individual hard drive, ranging in size from 1 GB to 1 terabyte (TB) or more, is composed of one or more blocks stored in a block store server. Although treated as individual hard drives, it will be understood that volumes may be stored as one or more virtualized devices implemented on one or more underlying physical host devices. A volume may be partitioned a small number of times (e.g., up to 16 times), with each partition hosted by a different host. A volume's data may be replicated across multiple devices in the cloud provider network 203 to provide multiple replicas of the volume (such replicas may collectively represent a volume on a computing system).Replicas of volumes in a distributed computing system can beneficially provide automatic failover and recovery, for example, by allowing a user to access either a primary replica of a volume or a secondary replica of a volume synchronized at the block level with the primary replica, such that failure of either the primary or secondary replica does not prevent access to the information in the volume. The role of the primary replica is to facilitate reads and writes (sometimes called "input / output operations" or simply "I / O operations") on the volume and to communicate the writes to the secondary replica (preferably synchronously in the I / O path, although asynchronous replication can also be used). The secondary replica is updated synchronously with the primary replica and can provide a seamless transition during a failover operation, whereby the secondary replica takes over the role of the primary replica and the previous primary replica is designated as the secondary replica or a new replacement secondary replica is provisioned. Although the specific examples herein describe a primary replica and a secondary replica, it should be understood that a logical volume may include multiple secondary replicas. A compute instance can virtualize its I / O to the volume via a client. A client represents instructions that enable a compute instance to connect to and perform I / O operations on remote data volumes (e.g., data volumes stored on a physically separate computing device accessed over a network). A client may be implemented on an offload card of a server that includes a processing unit (e.g., a CPU or GPU) of the compute instance.

[0046] Data plane 221 may also include one or more object store servers, which represent another type of storage in cloud provider network 203. Object storage servers include one or more servers where data is stored as objects in resources called buckets, and may be used to support managed object storage services of cloud provider network 203. Each object typically includes the data stored, a variable amount of metadata that enables various functions of the object storage server regarding analysis of the stored objects, and a globally unique identifier or key that may be used to retrieve the object. Each bucket is associated with a given user account. Customers may store as many objects as they want in their buckets, may write, read, and delete objects in their buckets, and may control access to their buckets and the objects contained therein. Furthermore, in embodiments where several different object storage servers are distributed across different ones of the above-mentioned regions, users may choose the region(s) in which their buckets are stored, e.g., to optimize latency. Customers may use buckets to store various types of objects, such as machine images that can be used to boot VMs, or snapshots that represent point-in-time views of a volume's data.

[0047] Provider substrate extension 224 ("PSE") provides resources and services of cloud provider network 203 within a separate network, such as a telecommunications network, thereby extending the capabilities of cloud provider network 203 to new locations (e.g., for reasons related to latency of communications with customer devices, regulatory compliance, security, etc.). In some embodiments, PSE 224 may be configured to provide capacity to cloud-based workloads executing within the telecommunications network. In some embodiments, PSE 224 may be configured to provide core and / or RAN functions of the telecommunications network and may be configured with additional hardware (e.g., radio access hardware). Some embodiments may be configured to enable both, for example, by allowing unused capacity by the core and / or RAN functions to be used for executing cloud-based workloads.

[0048] As shown, such provider substrate extensions 224 may include cloud provider network managed provider substrate extensions 227 (e.g., formed by servers located in cloud provider managed facilities separate from those associated with the cloud provider network 203), communications service provider substrate extensions 230 (e.g., formed by servers associated with communications service provider facilities), customer managed provider substrate extensions 233 (e.g., formed by servers located on-premise at a customer or partner facility), among other possible types of substrate extensions.

[0049] As shown in the exemplary provider substrate extension 224, the provider substrate extension 224 may similarly include a logical separation between a control plane 236 and a data plane 239, which respectively extend the control plane 218 and the data plane 221 of the cloud provider network 203. The provider substrate extension 224 may be pre-configured, for example by a cloud provider network operator, with an appropriate combination of software and / or firmware elements and hardware to support various types of computing-related resources and to reflect the experience of using the cloud provider network 203 to do so. For example, one or more provider substrate extension location servers may be provisioned by a cloud provider for deployment within the provider substrate extension 224. As noted above, the cloud provider network 203 may offer a set of pre-defined instance types, each with various types and amounts of underlying hardware resources. Each instance type may also be offered in various sizes. The servers may be heterogeneous servers so that customers can continue to use the same instance types and sizes in the provider substrate extension 224 as they use in the region. A heterogeneous server can simultaneously support multiple instance sizes of the same type and can also be reconfigured to host any instance type supported by its underlying hardware resources. Reconfiguration of a heterogeneous server can be performed on the fly using the server's available capacity, i.e., while other VMs are still running and consuming other capacity of the provider substrate extension location server.This can improve utilization of computing resources in edge locations by enabling better packing of instances running on servers, and can also provide a seamless experience for using instances across cloud provider networks 203 and cloud provider network managed provider substrate extensions 227.

[0050] A provider substrate extension server may host one or more compute instances. A compute instance may be a VM or container that packages code and all of its dependencies, allowing an application to run quickly and reliably across computing environments (including, for example, VMs and micro-VMs). Additionally, depending on the customer's desires, the server may host one or more data volumes. In a region of the cloud provider network 203, such volumes may be hosted on dedicated block store servers. However, because the capacity of the provider substrate extension 224 may be significantly smaller than in a region, a suboptimal usage experience may be provided if the provider substrate extension 224 includes such dedicated block store servers. Thus, the block storage service may be virtualized within the provider substrate extension 224 such that one of the VMs runs the block store software to store the data of the volume. Similar to the operation of the block storage service in a region of the cloud provider network 203, the volumes within the provider substrate extension 224 may be replicated for durability and availability. The volumes may be provisioned within their own isolated virtual network within the provider substrate extension 224. The compute instances and any volumes collectively constitute a data plane extension 239 of the provider network data plane 221 within the provider substrate extension 224 .

[0051] Servers within the provider substrate extension 224 may, in some embodiments, host certain local control plane components, such as components that allow the provider substrate extension 224 to continue functioning if the connection back to the cloud provider network 203 is lost. Examples of these components include a migration manager that can move compute instances between provider substrate extension servers as needed to maintain availability, and a key-value data store that indicates the location of volume replicas. In general, however, the control plane 236 functionality of the provider substrate extension 224 remains in the cloud provider network 203 to allow customers to use as much of the resource capacity of the provider substrate extension 224 as possible.

[0052] The migration manager may have a centralized coordination component running within the region as well as a local controller running on the PSE server (and servers at the cloud provider's data center). The centralized coordination component may identify the target edge location and / or target host when a migration is triggered, and the local controller may coordinate the data transfer between the source and target hosts. The described movement of resources between hosts at different locations may take one of several migration forms. Migration refers to moving virtual machine instances (and / or other resources) between hosts within a cloud computing network, or between hosts outside the cloud computing network and hosts within the cloud. There are various types of migrations, such as live migration and reboot migration. During reboot migration, the customer experiences a power down and effective power cycle of the virtual machine instance. For example, the control plane service may coordinate a reboot migration workflow that involves destroying the current domain on the original host and then creating a new domain for the virtual machine instance on the new host. The instance is rebooted by shutting down on the original host and starting it up again on the new host.

[0053] Live migration refers to the process of moving a running virtual machine or application between different physical machines without significantly disrupting the availability of the virtual machine (e.g., such that the virtual machine downtime is not noticeable to the end user). When the control plane executes a live migration workflow, it can create a new "inactive" domain associated with the instance, but the original domain of the instance continues to run as an "active" domain. The virtual machine's memory (including any in-memory state of the running applications), storage, and network connectivity are transferred from the original host with an active domain to a destination host with an inactive domain. The virtual machine may be briefly paused to prevent state changes while transferring memory contents to the destination host. The control plane can migrate the inactive domain to an active domain, demote the original active domain to an inactive domain (also called "flipping"), and then destroy the inactive domain.

[0054] Various types of migration techniques include managing critical phases (times when a virtual machine instance is unavailable to a customer), which should be as short as possible. In the presently disclosed migration techniques, this can be particularly challenging since resources are moved between hosts in geographically separate locations that may be connected via one or more intermediate networks 212. In the case of live migration, the disclosed techniques can dynamically determine how much memory state data to copy in advance (e.g., while the instance is still running on the source host) and how much memory state data to copy after the fact (e.g., after the instance has started running on the destination host) based on, for example, latency between the locations, network bandwidth / usage patterns, and / or which memory pages are most frequently used by the instance. Additionally, the specific times at which memory state data is transferred can be dynamically determined based on network conditions between the locations. This analysis may be performed by a migration management component in the region or by a migration management component running locally at the source edge location. If the instance has access to virtualized storage, both the source and target domains can be attached to the storage simultaneously, allowing uninterrupted access to that data during migration and in the event a rollback to the source domain is required.

[0055] Server software running on the provider substrate extension 224 may be designed by the cloud provider to run on the cloud provider substrate network, and this software may be enabled to run unmodified in the provider substrate extension 224 to create a private replica of the substrate network in the edge location ("shadow substrate"). The local network manager(s) 242 may run on a server of the provider substrate extension 224 and may bridge the shadow substrate to the provider substrate extension 224 network, for example, by acting as a virtual private network (VPN) endpoint(s) between the provider substrate extension 224 and the cloud provider network 203 proxies 245, 248, and by implementing a mapping service (traffic encapsulation and decapsulation) to associate data plane traffic (from the data plane proxy 248) and control plane traffic (from the control plane proxy 245) with the appropriate server(s). By implementing a local version of the provider network's substrate overlay mapping service, the local network manager(s) 242 enable resources in the provider substrate extension 224 to seamlessly communicate with resources in the cloud provider network 203. In some implementations, a single local network manager 242 may perform these actions for all servers hosting compute instances in the provider substrate extension 224. In other implementations, each of the servers hosting compute instances may have a dedicated local network manager 242.In multi-rack edge locations, local network managers 242 may maintain open tunnels between each other so that inter-rack communications can be routed through the local network managers 242 .

[0056] The provider substrate extension locations may utilize secure network tunnels through the provider substrate extension 224 network to the cloud provider network 203, for example, to maintain the security of customer data as it passes through the provider substrate extension 224 network and any other intermediate networks 212 (which may include the public Internet). Within the cloud provider network 203, these tunnels consist of virtual infrastructure components including isolated virtual networks (e.g., in an overlay network), control plane proxies 245, data plane proxies 248, and substrate network interfaces. Such proxies 245, 248 may be implemented as containers running on the compute instances. In some embodiments, each server at the provider substrate extension 224 location that hosts a compute instance may utilize at least two tunnels: one for control plane traffic (e.g., Constrained Application Protocol (CoAP) traffic) and one for encapsulated data plane traffic. A connection manager (not shown) in the cloud provider network 203 manages the lifecycle of the cloud provider network side of these tunnels and their components, for example, by automatically provisioning them as needed and maintaining them in a healthy operational state. In some embodiments, direct connections between the provider substrate extension 224 locations and the cloud provider network 203 can be used for control plane and data plane communications. Compared to VPNs over other networks, direct connections can provide constant bandwidth and more consistent network performance because their network paths are relatively fixed and stable.

[0057] A control plane (CP) proxy 245 can be provisioned in the cloud provider network 203 to represent a particular host(s) at an edge location. The CP proxy 245 is an intermediary between the control plane 218 in the cloud provider network 203 and the control plane targets in the control plane 236 of the provider substrate extension 224. That is, the CP proxy 245 provides an infrastructure for tunneling management API traffic destined for a provider substrate extension server from the regional substrate to the provider substrate extension 224. For example, a virtualized computing service in the cloud provider network 203 can issue a command to a VMM of a server in the provider substrate extension 224 to launch a compute instance. The CP proxy 245 maintains a tunnel (e.g., a VPN) to the local network manager 242 of the provider substrate extension 224. Software implemented in the CP proxy 245 ensures that only eligible API traffic leaves the substrate and returns to the substrate. The CP proxy 245 provides a mechanism to expose remote servers on the cloud provider substrate while protecting the substrate's security material (e.g., encryption keys, security tokens) from leaving the cloud provider network 203. The one-way control plane traffic tunnel imposed by the CP proxy 245 also prevents any (potentially compromised) device from calling back into the substrate. The CP proxy 245 may be instantiated one-to-one with a server in the provider substrate extension 224, or may be capable of managing the control plane traffic of multiple servers within the same provider substrate extension 224.

[0058] Data plane (DP) proxies 248 may also be provisioned in the cloud provider network 203 to represent a particular server(s) in the provider substrate extension 224. The DP proxies 248 act as a shadow or anchor for the server(s) and may be used by services in the cloud provider network 203 to monitor the host's health (including availability, used / free compute and capacity, used / free storage and capacity, and network bandwidth usage / availability). The DP proxies 248 also act as proxies for the server(s) in the cloud provider network 203, thereby allowing isolated virtual networks to span the provider substrate extension 224 and the cloud provider network 203. Each DP proxy 248 may be implemented as a packet forwarding compute instance or container. As shown, each DP proxy 248 may maintain a VPN tunnel with a local network manager 242 that manages traffic to the server(s) that the DP proxy 248 represents. This tunnel may be used to send data plane traffic between the provider substrate extension server(s) and the cloud provider network 203. Data plane traffic traveling between a provider substrate extension 224 and a cloud provider network 203 may pass through a DP proxy 248 associated with that provider substrate extension 224. For data plane traffic traveling from a provider substrate extension 224 to a cloud provider network 203, the DP proxy 248 may receive the encapsulated data plane traffic, verify its accuracy, and allow it to enter the cloud provider network 203. The DP proxy 248 may forward the encapsulated traffic directly from the cloud provider network 203 to the provider substrate extension 224.

[0059] The local network manager(s) 242 can provide secure network connectivity with proxies 245, 248 established in the cloud provider network 203. After connectivity between the local network manager(s) 242 and the proxies 245, 248 is established, the customer may issue commands via the interface 206 to instantiate compute instances (and / or perform other operations using compute instances) using the provider substrate extension resources in much the same way that commands are issued for compute instances hosted in the cloud provider network 203. From the customer's perspective, the customer is able to seamlessly use local resources in the provider substrate extension 224 (and resources in the cloud provider network 203, if desired). Computing instances set up on servers in the provider substrate extension 224 can communicate with both electronic devices in the same network and other resources set up in the cloud provider network 203, if desired. A local gateway 251 may be implemented to provide network connectivity between the provider substrate extension 224 and a network associated with that extension (e.g., a communications service provider network in the example of communications service provider substrate extension 230).

[0060] In some situations, data may need to be transferred between the object storage service and the provider substrate extension (PSE) 224. For example, the object storage service may store machine images used to boot VMs and snapshots representing point-in-time backups of volumes. The object gateway may reside on a PSE server or specialized storage device and provide customers with a configurable, per-bucket cache of the contents of object storage buckets in the PSE 224 to minimize the impact of PSE region latency on customer workloads. The object gateway may also temporarily store snapshot data from snapshots of volumes in the PSE 224 and then synchronize with object servers in the region when possible. The object gateway may also store machine images designated by customers for use in the PSE 224 or on the customer's premises. In some implementations, data in the PSE 224 may be encrypted with a unique key, and the cloud provider may restrict the key from being shared from the region to the PSE 224 for security reasons. Thus, data exchanged between the object store server and the object gateway may utilize encryption, decryption, and / or re-encryption to maintain security boundaries regarding encryption keys or other sensitive data. The conversion intermediary can perform these operations and can create a PSE bucket (on the object store server) to store the snapshot data and the machine image data using the PSE encryption key.

[0061] In this manner, PSE 224 forms an edge location in that it provides cloud provider network 203 resources and services outside of a traditional cloud provider datacenter and closer to customer devices. The edge locations referred to herein can be constructed in a number of ways. In some implementations, an edge location may be an extension of the cloud provider network substrate that includes a limited amount of capacity provided outside of an availability zone (e.g., a small datacenter or other facility of the cloud provider that is located closer to customer workloads and may be far from any availability zones). Such edge locations may be referred to as "far zones" (because they are far from other availability zones) or "near zones" (because they are closer to customer workloads). Near zones may be connected to publicly accessible networks such as the Internet in various ways, e.g., directly, through another network, or through a private connection to a region. Typically, near zones have limited capacity compared to regions, but in some cases, near zones may have significant capacity, such as thousands of racks or more.

[0062] In some implementations, an edge location may be an extension of a cloud provider's network substrate formed by one or more servers located on-premise at a customer or partner facility, such server(s) communicating over a network (e.g., a publicly accessible network such as the Internet) with a nearby availability zone or region of the cloud provider network 203. This type of substrate extension located outside the cloud provider network datacenter may be called an "outpost" of the cloud provider network 203. Some outposts may be integrated into a telecommunications network, for example, as a telecommunications datacenter, a telecommunications aggregation site, and / or a multi-access edge computing (MEC) site with a physical infrastructure that spans across telecommunications base stations in the telecommunications network. In an on-premise example, the limited capacity of the outpost may be available only to the customer that owns the facility (and any other accounts authorized by the customer). In a telecommunications example, the limited capacity of the outpost may be shared among several applications (e.g., gaming, virtual reality applications, healthcare applications) that transmit data to users of the telecommunications network.

[0063] An edge location may include data plane capacity that is at least partially controlled by a control plane of an availability zone near the cloud provider network 203. Thus, an availability zone group may include a "parent" availability zone and any "child" edge locations that are homed to the parent availability zone (e.g., controlled at least in part by its control plane). Certain limited control plane functions (e.g., functions that require low latency communication with customer resources and / or functions that allow the edge location to continue to function when disconnected from the parent availability zone) may also be present in some edge locations. Thus, in the above example, an edge location refers to an extension of at least data plane capacity located at the edge of the cloud provider network 203, close to customer devices and / or workloads.

[0064] In the example of FIG. 1, the distributed computing device 112 (FIG. 1), the centralized computing device 115 (FIG. 1), and the core computing device 118 (FIG. 1) may be implemented as provider substrate extensions 224 of the cloud provider network 203. The placement or location of the provider substrate extensions 224 within the communications network 100 may vary depending on the particular network topology or architecture of the communications network 100. The provider substrate extensions 224 may generally connect anywhere the communications network 100 can break out packet-based traffic (e.g., IP-based traffic). Furthermore, communications between a given provider substrate extension 224 and the cloud provider network 203 typically traverse at least a portion of the communications network 100 securely (e.g., via a secure tunnel, a virtual private network, a direct connection, etc.).

[0065] In 5G wireless network development efforts, edge locations may be considered as a possible implementation of multi-access edge computing (MEC). Such edge locations may be connected to various points in the 5G network that provide data traffic breakout as part of the user plane function (UPF). Even older wireless networks may incorporate edge locations. For example, in a 3G wireless network, an edge location may be connected to a packet-switched network portion of the communications network 100, such as a Serving General Packet Radio Service Support Node (SGSN) or a Gateway General Packet Radio Service Support Node (GGSN). In a 4G wireless network, an edge location may be connected to a Serving Gateway (SGW) or a Packet Data Network Gateway (PGW) as part of the core network or evolved packet core (EPC). In some embodiments, traffic between the provider substrate extension 224 and the cloud provider network 203 may be broken out of the communications network 100 without routing through the core network.

[0066] In some embodiments, the provider substrate extension 224 may connect to multiple communication networks associated with each customer. For example, when two communication networks of each customer share or route traffic through a common point, the provider substrate extension 224 may connect to both networks. For example, each customer may assign a portion of its network address space to the provider substrate extension 224, which may include a router or gateway 251 capable of distinguishing traffic exchanged with each of the communication networks 100. For example, traffic destined for the provider substrate extension 224 from one network may have a different destination IP address, source IP address, and / or virtual local area network (VLAN) tag than traffic received from another network. Traffic from the provider substrate extension 224 destined for a destination in one of the networks may likewise be encapsulated to have the appropriate VLAN tag, source IP address (e.g., from a pool allocated to the provider substrate extension 224 from the destination network address space), and destination IP address.

[0067] FIG. 2B illustrates an example 253 of cellularization and geographic distribution of the communication network 100 (FIG. 1) to provide a highly available user plane function (UPF). In FIG. 2B, a user device 254 communicates with a request router 255 to route requests to one of a number of control plane cells 257a and 257b. Each control plane cell 257 may include a network service API gateway 260, a network slice configuration 262, a function for network service monitoring 264, site planning data 266 (describing customer site requirements such as layout, device types, device quantities, etc.), a network service / function catalog 268, a network function orchestrator 270, and / or other components. To reduce the possibility of a large-scale error affecting a wide range of customers, a large control plane may be divided into cells by providing one or more cells that operate independently, for example, per customer, per network, or per region.

[0068] The network service / function catalog 268 is also referred to as the NF repository function (NRF). In a service-based architecture (SBA) 5G network, the control plane functions and the common data repository can be delivered through a set of interconnected network functions built using a microservices architecture. The NRF can maintain a record of available NF instances and their supported services, allowing other NF instances to subscribe and be notified of registrations from NF instances of a given type. Thus, the NRF can support service discovery by receiving discovery requests from NF instances, detailing which NF instances support a particular service. The network function orchestrator 270 can perform NF lifecycle management, including instantiation, scale out / in, performance measurement, event correlation, and termination. The network function orchestrator 270 can onboard new NFs, manage migration of existing NFs to new or updated versions, identify a suitable set of NFs for a particular network slice or a larger network, and orchestrate NFs across the various computing devices and sites that make up the wireless network 103 (FIG. 1).

[0069] The control plane cell 257 may communicate with one or more cell sites 272, one or more customer local data centers 274, one or more local zones 276, and one or more regional zones 278 via a RAN interface 273. The RAN interface 273 may include an application programming interface (API) that facilitates provisioning or release of capacity in the RAN operated by a third party communication service provider at the cell site 272. The cell site 272 includes computing hardware 280 that executes one or more distributed unit (DU) network functions 282. The customer local data center 274 includes computing hardware 283 that executes one or more DU or central unit (CU) network functions 284, a network controller 285, a UPF 286, one or more edge applications 287 that support customer workloads, and / or other components.

[0070] The local zone 276 may reside in a data center operated by a cloud service provider and may run one or more core network functions 288, such as the AMF, SMF, a network publishing function (NEF) that securely exposes the services and capabilities of other network functions, and a unified data management (UDM) function that manages subscriber data for authorization, registration, and mobility management. The local zone 276 may also run a UPF 286, a service 289 for metric processing, and one or more edge applications 287.

[0071] The regional zone 278 may be located in a data center operated by a cloud service provider and may run one or more core network functions 288, a UPF 286, a network management system, an operations support system (OSS) 290 supporting service delivery, service fulfillment, service assurance, and customer care, an Internet Protocol Multimedia Subsystem (IMS) 291, a business support system (BSS) 292 supporting product management, customer management, revenue management, and / or order management, one or more portal applications 293, and / or other components.

[0072] In this example, the communication network 100 employs a cellular architecture to reduce the blast radius of individual components. At the highest level, the control plane resides in multiple control plane cells 257 to prevent failure of an individual control plane from affecting the entire deployment.

[0073] Within each control plane cell 257, there may be multiple redundant stacks with the control plane shifting traffic to a secondary stack as needed. For example, cell site 272 may be configured to utilize a nearby local zone 276 as its default core network. If the local zone 276 goes down, the control plane can redirect cell site 272 to use a backup stack in regional zone 278. Traffic normally routed from the Internet to the local zone 276 can be shifted to an endpoint in regional zone 278. Each control plane cell 257 may implement a "stateless" architecture that shares a common session database across multiple sites (e.g., availability zones or edge sites).

[0074] FIG. 3A illustrates an exemplary cloud provider network 203 including geographically distributed provider substrate extensions 224 (FIG. 2A) (or "edge locations 303") according to some embodiments. As illustrated, the cloud provider network 203 can be formed as multiple regions 306, which are distinct geographic areas in which a cloud provider has one or more data centers 309. Each region 306 can include two or more availability zones (AZs) connected to each other via a private high-speed network, such as a fiber communication connection. An availability zone refers to an isolated failure domain that includes one or more data center facilities with separate power sources, separate networks, and separate cooling relative to other availability zones. Cloud providers may strive to place availability zones far enough apart from each other within a region 306 to prevent multiple availability zones from being taken offline at the same time due to a natural disaster, major power outage, or other unforeseen event. Customers can connect to resources in the availability zones of the cloud provider network 203 through publicly accessible networks (e.g., the Internet, cellular communications networks, communications service provider networks). Transit centers (TCs) are the main backbone locations that link customers to the cloud provider network 203 and may be co-located with other network provider facilities (e.g., Internet service providers, telecommunications providers). Each region 306 may operate two or more TCs for redundancy. The regions 306 are connected to a global network that includes a private network infrastructure (e.g., fiber connections controlled by the cloud service provider) that connects each region 306 to at least one other region.The cloud provider network 203 can deliver content from points of presence ("PoPs") that are outside of, but networked with, these regions 306 via edge locations 303 and regional edge cache servers. This partitioning and geographic distribution of computing hardware enables the cloud provider network 203 to offer customers global, low-latency resource access with a high degree of fault tolerance and stability.

[0075] The number of edge locations 303 can be much larger compared to the number of regional data centers or availability zones. Such widespread deployment of edge locations 303 can provide low latency connectivity to the cloud for a much larger group of end user devices (compared to end user devices that happen to be very close to a regional data center). In some embodiments, each edge location 303 can peer to a portion of the cloud provider network 203 (e.g., a parent availability zone or regional data center). Such peering allows various components operating within the cloud provider network 203 to manage the computing resources of the edge locations 303. In some cases, multiple edge locations 303 may be located or installed in the same facility (e.g., separate racks of computer systems) and managed by different zones or data centers 309 to provide additional redundancy. It should be noted that although the edge location 303 is generally depicted herein as being within the communications service provider network or wireless network 103 (FIG. 1), in some cases, such as when the cloud provider network facilities are relatively close to the communications service provider facilities, the edge location 303 may remain within the physical premises of the cloud provider network 203 while being connected to the communications service provider network via fiber or other network links.

[0076] Edge locations 303 can be constructed in a number of ways. In some implementations, edge locations 303 may be extensions of the cloud provider network substrate that include a limited amount of capacity provided outside of an availability zone (e.g., in a small data center 309 located close to customer workloads and potentially far from any availability zones, or in other facilities of the cloud provider). Such edge locations 303 may be referred to as local zones (because they are more local or closer to a group of users than traditional availability zones). Local zones may be connected in various ways to a publicly accessible network, such as the Internet, for example, directly, through another network, or through a private connection to a region 306. Typically, local zones have limited capacity compared to a region 306, but in some cases, local zones may have significant capacity, such as several thousand racks or more. Some local zones may use infrastructure similar to a typical cloud provider data center instead of the edge location 303 infrastructure described herein.

[0077] As shown herein, the cloud provider network 203 can be formed as a number of regions 306, each of which represents a geographic area in which the cloud provider clusters data centers 309. Each region 306 can further include multiple (e.g., two or more) availability zones (AZs) connected to each other via a private high-speed network, e.g., fiber communication connections. An AZ can provide an isolated failure domain, including one or more data center facilities with separate power sources, separate networks, and separate cooling from another AZ. The AZs within a region 306 are preferably located in locations far enough apart from each other so that the same natural disaster (or other failure-causing event) does not simultaneously affect or take offline multiple AZs. Customers can connect to the AZs of the cloud provider network 203 via publicly accessible networks (e.g., the Internet, cellular communication networks, etc.).

[0078] Parenting of a given edge location 303 to an AZ or region 306 of a cloud provider network 203 may be based on several factors. One such parenting factor is data sovereignty. For example, an edge location 303 deployed within a telecommunications network in a country may be parented to an AZ or region 306 in that country to keep data originating from that country within that country. Another factor is service availability. For example, some edge locations 303 may have different hardware configurations, such as local non-volatile storage for customer data (e.g., solid state drives), with or without components such as graphics accelerators. Some AZs or regions 306 may lack services to leverage those additional resources, and therefore the edge location 303 may be parented to an AZ or region 306 that supports the use of those resources. Another factor is the latency between the AZ or region 306 and the edge location 303. Deploying edge locations 303 within a communications network can provide latency benefits, but parenting an edge location 303 to a distant AZ or region 306 can introduce significant latency to the edge location's 303 regional traffic, negating those benefits. Thus, edge locations 303 are often parented to nearby (in terms of network latency) AZs or regions 306.

[0079] 3B illustrates an example of a network environment 200 including a virtual private cloud network 107 (FIG. 1) that extends to a cloud provider network 203 to include a client device 215 connected to a wireless network 103, such as a wireless device 106 (FIG. 1) or other user equipment. The cloud provider network 203 may include one or more cloud services 320, one or more firewalls 323, one or more gateways 326, one or more virtual routers 329, and / or other components. The virtual private cloud network 107 may correspond to a software-defined network that exists virtually on or within heterogeneous physical and logical networks. Although one virtual private cloud network 107 is illustrated, it should be understood that one or more client devices 215 may be connected to multiple virtual private cloud networks 107.

[0080] The wireless network 103 may be connected to the cloud provider network 203 via a private network link 332. The private network link 332 may be a dedicated network link, such as an optical fiber link, a microwave link, a T-carrier link, or another type of data link. The private network link 332 may be isolated from the public Internet to provide a higher quality of service, such as target latency characteristics, target bandwidth characteristics, etc. The private network link 332 may terminate at a radio access network portion of the wireless network 103 and / or a core network portion of the wireless network 103. In some cases, some or all of the core network functions of the wireless network 103 are performed in the cloud provider network 203. The use of the private network link 332 may provide an improved quality of service (e.g., improved latency) of connectivity to the virtual private cloud network 107 as compared to a client device 215 connecting to the virtual private cloud network 107 via the public Internet.

[0081] The virtual private cloud network 107 may support access control and security management that may be applied across the entire virtual private cloud network 107 (e.g., ACLs or stateless firewall rules applied across an entire sub-network), as well as access control and security management that may be applied within a subset of the virtual private cloud network 107 (e.g., security groups or stateful firewall rules applied to a machine instance or network interface). The virtual private cloud network 107 may be operated by a cloud provider for a customer, such as a business, educational institution, government, or another organization. The virtual private cloud network 107 may include one or more private network links to one or more on-premise networks operated by the customer that are separate from the cloud provider network 203, and devices on those on-premise networks are also connected to the virtual private cloud network 107.

[0082] Cloud services 320 may be configured to enable a wide variety of functions. In various embodiments, individual cloud services 320 may provide services that allow customers to dynamically launch and manage physical or virtual computing resources, such as machine instances, eventually consistent data storage services where data is stored in buckets, database services supporting key / value and document data structures, distributed message queuing services, workflow management services, and / or other services. Each of cloud services 320 may be associated with a corresponding application programming interface (API) that supports a set of calls or operations that the respective cloud service 320 can perform. Invoking an API may invoke a set of different services or agents to perform operations and process data.

[0083] The firewall 323 may be configured in a rule set that configures what types of network traffic can be transmitted within the virtual private cloud network 107. The firewall 323 may support various functions such as logging, intrusion detection, denial of service protection, malware filtering, etc. The gateway 326 may selectively forward network traffic to allow the virtual private cloud network 107 to communicate with external networks such as the Internet, other virtual private cloud networks, or other resources within the cloud provider network 203 that are not within the virtual private cloud network 107. In some implementations, the gateway 326 may perform network address translation (NAT) to provide publicly routable network addresses to devices with private network addresses on the virtual private cloud network 107. While in FIG. 3B the firewall 323 and gateway 326 are shown as being located within the cloud provider network 203, in other examples the firewall 323 and / or gateway 326 may be implemented at the edge location 303 (FIG. 3A) by the provider substrate extension 224 (FIG. 2A).

[0084] In some embodiments, the client device 215 may use a management agent to run mobile device management software that allows the client device 215 to connect only through the virtual private cloud network 107 and not other networks to prevent data leakage. Alternatively, the mobile device management software may require that sensitive applications communicate only through the virtual private cloud network 107.

[0085] A given virtual router 329 includes a collection of nodes of a multi-layer packet processing service including fast path nodes configured to quickly execute locally cached routing or forwarding actions, and exception path nodes that determine the actions to take for different packet flows based on client-specified policies and connectivity requirements of the isolated networks. In a scenario where dynamic routing is implemented for application data traffic between a pair of isolated networks using a routing information exchange protocol (e.g., a protocol similar to Border Gateway Protocol (BGP)), processing of messages containing dynamic routing information can be offloaded from the virtual router node to protocol processing engines running on other devices, allowing the virtual router node to remain dedicated to its primary task of rule-based packet forwarding.

[0086] Virtual routers 329 implemented in different geographic regions (e.g., using resources located in datacenters in different states, countries, or continents) can be programmatically connected (or "peered") with one another and configured to obtain and automatically exchange dynamic routing information about isolated networks in different regions using a protocol similar to BGP, eliminating the need for clients to painstakingly configure static routes for traffic flowing between isolated networks. Clients can specify various parameters and settings (such as the particular protocol version to use, rules to filter routing information advertised to and from virtual routers 329, etc.) to control how routing information is forwarded between virtual routers 329. Wide area network (WAN) services can be implemented in a provider network using programmatically connected virtual routers with dynamic routing enabled, allowing clients to utilize the provider network's private fiber backbone links (already used for traffic between the provider network's datacenters on behalf of users of various other services) for traffic between the client's premises and cellular network slices distributed around the world, and to manage long-distance traffic using easy-to-use tools with virtualization interfaces. The virtual router 329 may provide connectivity between different network slices of one or more wireless networks 103 that correspond to different subnets of the virtual private cloud network 107.

[0087] In some embodiments, the wireless network 103 includes a VPC gateway 335 that can act as a proxy performing encapsulation / decapsulation and tunneling functions on behalf of the client device 215 so that the client device 215 can communicate with the virtual private cloud network 107. The VPC gateway 335 can identify specific network traffic from the client device 215 to be forwarded to the virtual private cloud network 107 based at least in part on a rule set. For example, IoT telemetry traffic may be characterized by a particular tag or port and may be specified in the rule set to be forwarded to a particular virtual private cloud network 107 having a corresponding telemetry service.

[0088] The VPC gateway 335 may perform network address translation between a network address of a virtual machine instance on the virtual private cloud network 107 and a network address of a computing device on which the virtual machine instance runs in the cloud provider network 203. The VPC gateway 335 may also perform network address translation between a network address of a client device 215 on the virtual private cloud network 107 and a network address of the same client device 215 on the wireless network 103. The VPC gateway 335 may also perform encryption and decryption functions using Internet Protocol Security (IPsec) and / or other protocols. In some cases, the VPC gateway 335 is used due to resource constraints (e.g., processor, memory) on one or more client devices 215. In one embodiment, the VPC gateway 335 is implemented in a customized user plane function in the core network of the wireless network 103.

[0089] The client device 215 may include a VPC application 338 and an operating system 341. The VPC application 338 may perform the functions described in connection with the VPC gateway 335, but executed within the application of the client device 215. For example, such functions may include encapsulation / decapsulation, encryption / decryption, tunneling, and network address translation between the virtual private cloud network 107 and the wireless network 103 or cloud provider network 203. To this end, the VPC application 338 may receive network address translation information from the virtual private cloud network 107. Thus, when an application sends a data packet to a virtual machine instance on the virtual private cloud network 107, the VPC application 338 may encapsulate the data packet with a destination network address that corresponds to a computing device in the cloud provider network 203 on which the virtual machine instance runs. In some embodiments, the functionality of the VPC application 338 may be integrated into the operating system 341 of the client device 215.

[0090] Referring to FIG. 4, a network environment 400 is illustrated according to various embodiments. The network environment 400 includes a computing environment 403, one or more client devices 406, one or more radio access networks (RANs) 409, a spectrum reservation service 410, and one or more wireless networks 103, which are in data communication with each other via a network 412. The network 412 may include, for example, the Internet, an intranet, an extranet, a wide area network (WAN), a local area network (LAN), a wired network, a wireless network, a cable network, a satellite network, or other suitable network, or any combination of two or more such networks. The RAN 409 may be operated by a number of different communication service providers. In some cases, one or more of the RANs 409 may be operated by the cloud provider network 203 (FIG. 2A) or a customer of the cloud provider network 203.

[0091] The computing environment 403 may include, for example, a server computer or any other system that provides computing power. Alternatively, the computing environment 403 may employ multiple computing devices that may be arranged, for example, in one or more server or computer banks or other arrangements. Such computing devices may be located in a single facility or may be distributed among many different geographic locations. For example, the computing environment 403 may include multiple computing devices that together may comprise host computing resources, grid computing resources, and / or any other distributed computing arrangement. In some cases, the computing environment 403 may correspond to elastic computing resources where the allocated capacity of processing, network, storage, or other computing-related resources may change over time. For example, the computing environment 403 may correspond to a cloud provider network 203, where customers are charged according to their usage of computing resources based on a utility computing model.

[0092] In some embodiments, computing environment 403 may correspond to a virtualized private network within a physical network that includes virtual machine instances running on physical computing hardware, for example by a hypervisor. The virtual machine instances and any containers running on those instances may be provided network connectivity through virtualized network components that are enabled by physical network components such as routers and switches.

[0093] Various applications and / or other functions may be executed in the computing environment 403 according to various embodiments. Additionally, various data may be stored in a data store 415 that is accessible to the computing environment 403. The data store 415 may represent multiple data stores 415, as will be appreciated. The data stored in the data store 415 may be associated with the operation of the various applications and / or functional entities described below, for example.

[0094] The computing environment 403 as part of a cloud provider network offering utility computing services includes computing devices 418 and other types of computing devices. The computing devices 418 may correspond to various types of computing devices 418 and may have various computing architectures. The computing architectures may differ by utilizing processors having different architectures, such as x86, x86_64, ARM, Scalable Processor Architecture (SPARC), PowerPC, etc. For example, some computing devices 418 may have x86 processors and other computing devices 418 may have ARM processors. The computing devices 418 may also differ in available hardware resources, such as local storage, graphics processing units (GPUs), machine learning extensions, and other characteristics.

[0095] Computing devices 418 may have various forms of allocated computing capacity 421, which may include virtual machine (VM) instances, containers, serverless functions, etc. A VM instance may be instantiated from a VM image. To this end, a customer may specify that a virtual machine instance should be launched within a particular type of computing device 418 and not other types of computing devices 418. In various examples, one VM instance may run alone on a particular computing device 418, or multiple VM instances may run on a particular computing device 418. Also, a particular computing device 418 may run different types of VM instances, which may provide different amounts of resources available via the computing device 418. For example, one type of VM instance may provide more memory and processing power than other types of VM instances.

[0096] Components executing on the computing environment 403 include, for example, a virtual private cloud (VPC) management service 424, one or more network services 427, as well as other applications, services, processes, systems, engines, or functions not described in detail herein. The VPC management service 424 executes to configure and manage the operation of one or more virtual private cloud networks 107 (FIG. 1) implemented by the network environment 400. To this end, the VPC management service 424 may generate a set of user interfaces or support application programming interfaces (APIs) that allow the virtual private cloud network 107 to be initially created and subsequently modified. The VPC management service 424 may enable configuration of access control lists, security groups, network slices, encryption, decryption, network address allocation and routing, elastic network interfaces, gateways, firewalls, and / or other components of the virtual private cloud network 107.

[0097] Network services 427 include various services that support virtual private cloud network 107 and its underlying physical and logical networks. To this end, network services 427 may perform routing, bridging, translation, firewalling, tunneling, encryption, decryption, logging, and / or other functions that support the flow of network traffic throughout network environment 400.

[0098] Data stored in data store 415 may include, for example, one or more network plans 439, one or more cellular topologies 442, one or more spectrum allocations 445, device data 448, one or more RBN metrics 451, customer billing data 454, radio unit configuration data 457, antenna configuration data 460, network function configuration data 463, one or more network function workloads 466, one or more customer workloads 469, one or more network slices 470, one or more quality of service (QoS) requirements 471, one or more address configurations 472, one or more access control lists 473, one or more encryption configurations 474, one or more firewall rule sets 475, network address translation data 476, one or more security groups 477, and other possible data.

[0099] The network plan 439 is a specification of the wireless network 103 to be deployed for a customer. For example, the network plan 439 may include the location or geographic area of ​​the facility to be covered, the number of cells, device identification and permissions, a desired maximum network latency, a desired bandwidth or network throughput for one or more classes of devices, one or more quality of service parameters for an application or service, one or more routes to be covered by the RBN 103, a schedule of coverage of the RBN 103 or a portion of the RBN 103, a periodic schedule of coverage of the RBN 103 or a portion of the RBN 103, a start time of the RBN 103 or a portion of the RBN 103, and an end time of the RBN 103 or a portion of the RBN 103, which virtual private cloud networks 107 should be supported at various locations of the RBN 103, and / or other parameters that can be used to create the wireless network 103. A customer can manually specify one or more of these parameters via a user interface. One or more of the parameters may be pre-populated as default parameters. In some cases, the network plan 439 may be generated for a customer based at least in part on an automated site survey using an unmanned aerial vehicle. The values ​​of the parameters defining the network plan 439 may be used as a basis for the cloud service provider to bill the customer under a utility computing model. For example, the customer may be charged higher for lower latency targets and / or higher bandwidth targets in a service level agreement (SLA), the customer may be charged per device, per cell, based on the geographic area served, based on spectrum availability, etc. In some cases, the network plan 439 may incorporate thresholds and reference parameters determined at least in part on an automated probe of the customer's existing private network.

[0100] The cellular topology 442 includes an arrangement of multiple cells for a customer, taking into account the location of the cells and, where possible, the reuse of frequency spectrum. The cellular topology 442 can be generated automatically by conducting a site survey. In some cases, the number of cells in the cellular topology 442 can be automatically determined based on the desired geographic area to be covered, the availability of backhaul connectivity at various sites, signal propagation, available frequency spectrum, and / or other parameters. In the case of a wireless network 103, the cellular topology 442 can be developed to cover one or more buildings in an organization campus, one or more schools in a school district, one or more buildings in a university or college system, and other areas.

[0101] Spectrum allocations 445 include frequency spectrum available for allocation to wireless network 103 and frequency spectrum currently allocated to wireless network 103. The frequency spectrum may include spectrum that is publicly accessible without restrictions, spectrum that is privately owned or leased by a customer, spectrum that is owned or leased by a provider, spectrum that is free to use but requires reservation, etc.

[0102] The device data 448 corresponds to data describing the wireless device 106 that is permitted to connect to the over-the-air network 103. This device data 448 may include the corresponding user, account information, billing information, data plan, permitted applications or uses, an indication of whether the wireless device 106 is mobile or fixed, location, current cell, network address, device identifier (e.g., International Mobile Equipment Identity (IMEI) number, Equipment Serial Number (ESN), Media Access Control (MAC) address, Subscriber Identity Module (SIM) number, etc.). In one embodiment, the SIM may be used to map the client device 215 (FIG. 3B) to a particular virtual private cloud network 107. An individual wireless device 106 may use an embedded SIM (eSIM) instead of or in addition to a physical removable SIM. Also, in some cases, the wireless device 106 may have multiple SIMs and / or multiple eSIMs, each of which may be associated with each of multiple virtual private cloud networks 107.

[0103] The RBN metrics 451 include various metrics or statistics indicative of the performance or health of the wireless network 103. Such RBN metrics 451 may include bandwidth metrics, dropped packet metrics, signal strength metrics, latency metrics, etc. The RBN metrics 451 may be aggregated per device, per cell, per customer, etc.

[0104] The customer billing data 454 specifies the charges to be incurred by the customer for the provider's operation of the wireless network 103 for the customer. The charges may include a fixed fee based on the equipment deployed to the customer and / or a usage-based fee determined by tracked usage metrics. In some cases, the customer may have purchased the equipment up front and may only be billed for bandwidth or back-end network costs. In other cases, the customer may not incur upfront costs and may be billed purely on usage. Because the equipment is provided to the customer based on a utility computing model, the cloud service provider can select the optimal configuration of equipment to meet the customer's target performance metrics while avoiding unnecessary hardware over-provisioning.

[0105] The wireless unit configuration data 457 may correspond to configuration settings for wireless units deployed in the wireless network 103. Such settings may include the frequency to use, the protocol to use, modulation parameters, bandwidth, network routing and / or backhaul configurations, etc.

[0106] The antenna configuration data 460 may correspond to the configuration settings of the antenna, including the frequency to be used, the azimuth angle, the vertical or horizontal orientation, the beam tilt, and / or other parameters that may be controlled automatically (e.g., by networked motors and controls on the antenna) or manually by instructing a user to mount the antenna in a particular way or make physical changes to the antenna.

[0107] Network function configuration data 463 corresponds to configuration settings that configure the operation of various network functions of wireless network 103. In various embodiments, the network functions may be deployed on VM instances or containers on computing devices 418 at cell sites, customer aggregation sites, or data centers located remotely from the customers. Non-limiting examples of network functions may include access and mobility management functions, session management functions, user plane functions, policy control functions, authentication server functions, unified data management functions, application functions, network publishing functions, network function repositories, network slice selection functions, and / or others. Network function workloads 466 correspond to machine images, containers, or functions that are to be launched within allocated computing capacity 421 to perform one or more network functions.

[0108] Customer workloads 469 correspond to customer machine images, containers, or functions that may be executed in parallel with or instead of network function workloads 466 within allocated computing capacity 421. For example, customer workloads 469 may provide or support customer applications or services. In various examples, customer workloads 469 are related to factory automation, autonomous robotics, augmented reality, virtual reality, design, monitoring, etc.

[0109] A network slice 470 corresponds to a flow of network traffic specified for one or more particular quality of service requirements 471. A flow may correspond to a flow associated with a particular application running on a particular client device 406, all network traffic from a particular client device 406, flows from all client devices 406 to a particular destination, flows from a particular client device 406 to a particular destination, flows associated with a virtual private cloud network 107, flows associated with a subnet within a virtual private cloud network 107, etc. In one example, a network slice 470 is identified by a source port, a source network address, a destination port, a destination network address, and / or other information. A network slice 470 may be valid for a particular period of time or a particular amount of data, or a network slice 470 may be valid until cancelled or released. In one example, a network slice 470 is allocated on demand to a particular application running on a client device 406. In some scenarios, the network slice 470 may have a specific recurring validity period (e.g., from midnight to 5:00 a.m. every weekday) or the quality of service requirements 471 of the network slice 470 may change based on the recurring period, current cost levels, and / or other factors or events.

[0110] Additionally, network slices 470 may be associated with different security properties to provide differentiated management for the potentially multiple network slices 470 associated with a single virtual private cloud network 107. For example, a first network slice 470 may correspond to a management server with greater access, and a second network slice 470 may correspond to an end-user client device 215 (FIG. 3B) with more restricted access.

[0111] The quality of service requirements 471 may correspond to a minimum or maximum bandwidth, a minimum or maximum latency, a minimum or maximum reliability measure, a minimum or maximum signal strength, etc. The quality of service requirements 471 may be associated with a corresponding cost level, which may include a fixed component, a usage-based component, and / or a congestion-based component. For example, the quality of service requirements 471 may be associated with a monthly recurring fixed cost, a per session or per megabyte cost, and / or a dynamic cost based on congestion at a cell site or a particular network link. In some cases, a customer may select a quality of service requirement 471 that provides a high level of service. However, in other cases, a customer may select a quality of service requirement 471 that provides a low level of cost but a low quality of service at certain times or in certain aspects. For example, a customer may select a quality of service requirement 471 that allows high throughput at night, but low priority throughput otherwise, in order to transmit backup data at low cost over the network.

[0112] Address configuration 472 configures the assignment of network addresses to individual client devices 215 of virtual private cloud network 107. Such network addresses may include IPv4 addresses, IPv6 addresses, and / or other types of addresses. For example, virtual private cloud network 107 may have a private network address space with multiple subnets, and different subnets may be associated with different access control lists 473, security groups, network slices 470, QoS requirements 471, etc. Address configuration 472 provides how and from which network address block network addresses are assigned to a particular device identified in device data 448 (e.g., by SIM). In some cases, address configuration 472 may indicate that a publicly routable network address is assigned.

[0113] Network access control lists 473 (ACLs) operate at the subnet level within virtual private cloud network 107 and support allow and deny rules, corresponding to a type of access control that may be automatically applied to all instances within any associated subnet. Network ACLs 473 may not be stateful in that return traffic must be explicitly allowed by a rule.

[0114] The encryption configuration 474 can define the encryption and decryption used within the virtual private cloud network 107. In some cases, the encryption configuration 474 can define that a combination of point-to-point encryption and end-to-end encryption is used. End-to-end encryption may be implemented by services running on each network host within the virtual private cloud network 107 (e.g., VPC application 338 (FIG. 3B)) or by the initial gateway 326 (FIG. 3B) with which each host communicates. Point-to-point encryption may be implemented for different links within the network, such as, for example, a private link to a customer's on-premise network, or a link between the wireless network 103, such as private network link 332 (FIG. 3B).

[0115] The firewall rule set 475 can define various rules used by the firewall 323 (FIG. 3B) of the virtual private cloud network 107. These rules can configure when network traffic is dropped, what network traffic is logged, denial of service protection, malware protection, and / or other types of firewall rules.

[0116] The network address translation data 476 provides information that maps a network address on the virtual private cloud network 107 (e.g., of a client device 215 or virtual machine instance) to a network address of a computing device on the wireless network 103 or cloud provider network 203.

[0117] Security groups 477 may define specific access controls per instance, per application, or per port. Security groups 477 may refer to different subsets of devices in virtual private cloud network 107 that are subject to management actions by VPC management services 424 that may affect all members of the group. In one embodiment, security groups 477 act as a virtual firewall for virtual machine instances, controlling inbound and outbound traffic. A customer may define security groups 477 as policies that can be applied to specific instances. When a customer launches an instance in virtual private cloud network 107, the customer may assign one or more security groups 477 to the instance. Security groups 477 may work at the instance level rather than at the subnet level, similar to network ACLs 473. Thus, each instance in a subnet may be assigned a different set of security groups 477. For each security group 477, the customer may add rules that control inbound traffic to the instance and another set of rules that control outbound traffic. Security groups 477 may be stateful in that return traffic is automatically allowed.

[0118] Client device 406 represents a plurality of client devices 406 that may be coupled to network 412. Client device 406 may include, for example, a processor-based system such as a computer system. Such a computer system may be embodied in the form of a desktop computer, a laptop computer, a personal digital assistant, a mobile phone, a smartphone, a set-top box, a music player, a web pad, a tablet computer system, a gaming console, an e-reader, a smart watch, a head-mounted display, a voice interface device, or other device. Client device 406 may include a display including, for example, one or more devices such as a liquid crystal display (LCD) display, a gas plasma-based flat panel display, an organic light-emitting diode (OLED) display, an electronic ink (E-ink) display, an LCD projector, or other types of display devices.

[0119] The client device 406 may be configured to execute various applications, such as client applications 436 and / or other applications. The client applications 436 may execute on the client device 406 to access network content provided, for example, by the computing environment 403 and / or other servers, thereby rendering a user interface on a display. To this end, the client applications 436 may comprise, for example, a browser, a dedicated application, etc., and the user interface may include network pages, application screens, etc. The client device 406 may be configured to execute applications in addition to the client applications 436, such as, for example, email applications, social networking applications, word processors, spreadsheets, and / or other applications.

[0120] In some embodiments, the spectrum reservation service 410 provides reservation of frequency spectrum for use by customers in the RAN 409. In one scenario, the spectrum reservation service 410 is operated by an entity, such as a third party, to manage reservations and coexistence in a publicly accessible spectrum. An example of such spectrum may be the Citizens Broadband Radio Service (CBRS). In another scenario, the spectrum reservation service 410 is operated by a telecommunications service provider to sell or sublicense portions of spectrum owned or licensed by the provider.

[0121] 5, a flowchart 500 is shown that provides an example of some operations of the virtual private cloud network 107 (FIG. 1) extended to the wireless network 103 (FIG. 1), according to various embodiments. Of course, the flowchart 500 provides only one example of many different types of functional arrangements that may be employed to implement some operations of the virtual private cloud network 107 described herein. Alternatively, the flowchart 500 may be considered to illustrate an example of elements of a method implemented in the computing environment 403 (FIG. 4), according to one or more embodiments.

[0122] Starting at box 501, the VPC management service 424 (FIG. 4) creates a subnet for the virtual private cloud network 107 of the wireless network 103. For example, the VPC management service 424 can create the subnet in response to receiving an API request to create a subnet from a customer account associated with the virtual private cloud network 107 in the cloud provider network 203 (FIG. 2A). In some cases, the subnet can be created in a network slice 470 (FIG. 4). The network slice 470 can be associated with one or more QoS requirements 471 (FIG. 4) and other parameters that reserve capacity in the wireless network 103 for the network slice 470. The capacity can include bandwidth, network function processing capacity (e.g., resources of a user plane function (UPF) or another network function), resources that provide lower latency, etc.

[0123] In box 502, the VPC management service 424 registers one or more subscriber identity modules (SIMs) or embedded SIMs (eSIMs) associated with a client device 215 ( FIG. 3B ) that is to be allowed to access the virtual private cloud network 107 over the wireless network 103. An identifier associated with the SIM or eSIM may be recorded in the device data 448 ( FIG. 4 ) and / or the security group 477 ( FIG. 4 ). For example, the VPC management service 424 may register the SIM or eSIM with the network slice 470 and / or the virtual private cloud network 107 in response to receiving an API request to register the SIM or eSIM with one or both of the network slice 470 and / or the virtual private cloud network 107. The API request may be received from a customer account associated with the virtual private cloud network 107 or the network slice 470.

[0124] In box 503, a request for service from the wireless network 103 is received from a client device 215. For example, the client device 215 may be a wireless device 106 (FIG. 1) that is powered on and begins negotiating a network connection from a radio access network 409 (FIG. 4) of the wireless network 103. The client device 215 may communicate with a radio unit at a cell site of the radio access network 409.

[0125] In box 506, the wireless network 103 identifies the client device 215. For example, the wireless network 103 can determine an identifier presented by the client device 215 from a SIM or eSIM. In some cases, the client device 215 may have multiple SIMs or eSIMs. Other identifiers may include an International Mobile Equipment Identity (IMEI) number, an equipment serial number (ESN), a media access control (MAC) address, and / or other identifiers.

[0126] In box 507, the VPC management service 424 or a service in the core network of the wireless network 103 determines which virtual private cloud networks 107 the client device 215 is permitted to access. In some cases, multiple virtual private cloud networks 107 may be accessible via the wireless network 103. In one scenario, the wireless network 103 is a private network operated by an organization with a single virtual private cloud network 107. In some cases, the client device 215 may be permitted to access multiple virtual private cloud networks 107. The VPC management service 424 may determine whether the client device 215 has access rights based on a comparison of an identifier (e.g., from the SIM) to identifiers listed in the security group 477. For example, a particular SIM or eSIM of potentially multiple SIMs or eSIMs in the client device 215 may be permitted to access separate virtual private cloud networks 107. Additionally, client devices 215 may be permitted to access the virtual private cloud network 107, but the virtual private cloud network 107 may be enabled or disabled on a per cell site basis within the wireless network 103.

[0127] In box 509, the wireless network 103 assigns network addresses, such as Internet Protocol (IP) addresses, to the client devices 215 according to a rule set based at least in part on the virtual private cloud network 107. Any client device 215 that connects to the wireless network 103 may be assigned a network address, where the assigned network address is selected based at least in part on the address configuration 472 (FIG. 4) associated with the virtual private cloud network 107. That is, other hosts on the virtual private cloud network 107 may potentially be given network addresses in the same address block if they were not connected to the wireless network 103 but were connected to the cloud provider network 203 (FIG. 2A). In some cases, different groups of client devices 215 may be assigned to different subnets within the virtual private cloud network 107, which may enable differentiated handling of access control and / or network slices 470 (FIG. 4) with QoS requirements 471 (FIG. 4).

[0128] In box 512, the VPC management service 424 configures encryption of network traffic to and / or from the client device 215 in the virtual private cloud network 107 according to the encryption configuration 474 (FIG. 4). This may include end-to-end encryption and / or point-to-point encryption. For example, network hosts on the virtual private cloud network 107 may be configured to run agents or services that perform end-to-end encryption and tunneling. Additionally, gateways in the wireless network 103 and virtual private cloud network 107 may be configured to perform point-to-point encryption and tunneling. In some cases, the encryption may use security keys generated from or stored in the SIM of the client device 215.

[0129] In box 513, the VPC management service 424 configures a network slice 470 for the client device 215 according to the rule set or assigns the client device 215, which may have one or more QoS requirements 471 related to latency, bandwidth, etc., to an existing network slice 470 in the wireless network 103 according to the rule set. For example, the wireless network 103 may be deployed in a robotic factory, where a first set of robots operates during the day to perform manufacturing tasks and a second set of robots operates at night to perform cleaning tasks. The first set of robots may be assigned a different subnet of network addresses and a different network slice 470 than the second set of robots. Additionally, the QoS requirements 471 may be changed according to a time schedule, such that, for example, network capacity is available to the second set of robots primarily at night. The network slice 470 can reserve bandwidth in the radio access network 409 and processing capacity in the core network to meet QoS requirements 471 and reduce latency, so that network function workloads 466 can be transferred from the cloud provider network 203 to cell sites or other edge locations 303 (FIG. 3A) of the provider substrate extension 224 (FIG. 2A).

[0130] In box 515, the VPC management service 424 provides the client device 215 with access to the virtual private cloud network 107 via the wireless network 103. Note that this may differ from manually connecting the client device 215 to the virtual private cloud network 107 via a virtual private network (VPN) tunnel or private link in that the client device 215 simply connects to the wireless network 103, and the client device 215 is automatically connected to the virtual private cloud network 107 (as when connecting to a 4G or 5G network). For example, in some implementations, additional authentication with credentials such as a username and password may be avoided. Also, the network address provided to the client device 215 is within the address block of the virtual private cloud network 107, is not a public address, and is not within other subnets of the wireless network 103.

[0131] In box 518, the wireless network 103 and network services 427 route or forward network traffic between one or more cloud services 320 (FIG. 3B) and the client device 215 through the virtual private cloud network 107, potentially using one or more firewalls 323 (FIG. 3B) and one or more gateways 326 (FIG. 3B) that can enforce security groups 477 (FIG. 4) or network ACLs 473 (FIG. 4). The security groups 477 or network ACLs 473 can also control access to one or more resources hosted in the provider substrate extension 224 of the cloud provider network 203 in the wireless network 103. The network traffic may be subject to a network slice 470 that is provisioned to the client device 215 and that applies one or more QoS requirements 471. The network traffic may traverse a private network link 332 (FIG. 3B) between the client device 215 and the cloud provider network 203, rather than the public Internet, thereby improving latency.

[0132] The firewall 323 may enforce rules defined in a firewall rule set 475 (FIG. 4) to filter packets, transcode data, drop or block connections, log traffic, etc. In some scenarios, the cloud service 320, gateway 326, and firewall 323 may be deployed in a provider substrate extension 224 within the wireless network 103 (such as a cell site, a customer site, or other intermediate location) to meet quality of service requirements 471. In such cases, network traffic may remain entirely within the wireless network 103 or within an edge location / local zone within the wireless network 103 and may not enter a regional data center. In some scenarios, the wireless network 103 may enable communication between the client device 215 and other client devices 215 connected to the wireless network 103 on the virtual private cloud network 107. In other scenarios, the client device 215 communicates with cloud services 320 that are on the cloud provider network 203 and the virtual private cloud network 107, rather than the wireless network 103. In one example, a virtual router 329 (FIG. 3B) can be used to connect a first network slice 470 to a second network slice 470, where the first and second network slices 470 are associated with different subnets of the virtual private cloud network 107. Flowchart 500 then ends.

[0133] Turning to Figure 6, a flow chart is shown that provides an example of a portion of the operation of the VPC gateway 335 (Figure 3B) of the wireless network 103 (Figure 1), in accordance with various embodiments. Of course, the flow chart of Figure 6 provides only one example of many different types of functional arrangements that may be employed to implement the portion of the operation of the VPC gateway 335 described herein. Alternatively, the flow chart of Figure 6 may be considered to illustrate an example of elements of a method implemented in the computing environment 403 (Figure 4), in accordance with one or more embodiments.

[0134] Beginning at box 603, the wireless network 103 receives a service request from a client device 215 (FIG. 3B) to the wireless network 103. The client device 215 may be identified by data from a subscriber identity module (SIM) or an embedded SIM. At box 606, the VPC gateway 335 provides the client device 215 with access to the virtual private cloud network 107 (FIG. 3B). For example, the VPC gateway 335 may authenticate the client device 215 based at least in part on the SIM or eSIM, or the client device 215 may provide a certificate, key, password, or other security credentials for authentication. The VPC gateway 335 may assign the client device 215 a network address on the virtual private cloud network 107 from a pool of network addresses corresponding to the virtual private cloud network 107.

[0135] In box 609, the VPC gateway 335 receives network traffic from the client device 215 that is destined for the virtual private cloud network 107. For example, the destination network address may correspond to a virtual machine instance of the cloud service 320 (FIG. 3B). In box 610, the VPC gateway 335 may apply one or more VPC security group 477 (FIG. 4) rules or network ACLs 473 (FIG. 4) obtained via the VPC control plane to determine whether the network traffic is permitted to be forwarded to its destination. For example, the network traffic from the client device 215 may not be permitted to be forwarded to the destination based on a role or user associated with the client device 215 within the organization that operates the virtual private cloud network 107.

[0136] In box 612, the VPC gateway 335 encapsulates the network traffic. For example, the VPC gateway 335 may add one or more headers to the data packets in the network traffic. The VPC gateway 335 may also translate the destination network address to a destination network address of the computing device 418 (FIG. 4) on the cloud provider network 203 (FIG. 3B) that hosts the virtual machine instance. The VPC gateway 335 may perform the translation by referring to the network address translation data 476 (FIG. 4). The VPC gateway 335 may also translate the source network address of the client device 215 on the virtual private cloud network 107 to a source network address on the wireless network 103. The VPC gateway 335 may also encrypt the network traffic.

[0137] In box 615, the VPC gateway 335 forwards the encapsulated network traffic from the wireless network 103 to the cloud provider network 203 via network link 332 (FIG. 3B). The VPC gateway 335 can maintain a VPC routing table populated via the control plane of the virtual private cloud network 107 to make forwarding decisions. In box 618, the VPC gateway 335 receives the encapsulated network traffic from the cloud provider network 203 via network link 332. The encapsulated network traffic is transmitted by one or more hosts on the virtual private cloud network 107.

[0138] In box 621, the VPC gateway 335 decapsulates the network traffic. This may include translating a source network address on the cloud provider network 203 (e.g., of a computing device) to a source network address on the virtual private cloud network 107, and translating a destination network address of the client device 215 on the wireless network 103 to a destination network address on the virtual private cloud network 107. This may include simply removing the headers added by the encapsulation. The VPC gateway 335 may also decrypt the network traffic. In box 624, the VPC gateway 335 forwards the network traffic to the client device 215. Operation on the part of the VPC gateway 335 then ends.

[0139] Turning now to Figure 7, a flow chart is shown providing an example of the operation of a portion of the VPC application 338 (Figure 3B) of the wireless network 103 (Figure 1), according to various embodiments. Of course, the flow chart of Figure 7 provides only one example of many different types of functional arrangements that may be employed to implement the operation of a portion of the VPC application 338 described herein. Alternatively, the flow chart of Figure 7 may be considered to illustrate example elements of a method implemented in the client device 215 (Figure 3B), according to one or more embodiments.

[0140] Beginning at box 703, a client device 215 requests a service from a wireless network 103. The client device 215 may be identified by data from a subscriber identity module (SIM) or an embedded SIM. At box 706, a VPC application 338 connects to a virtual private cloud network 107 (FIG. 3B) over the wireless network 103. For example, the VPC application 338 may authenticate the client device 215 based at least in part on the SIM or eSIM, or the VPC application 338 may provide a certificate, key, password, or other security credentials for authentication. The VPC application 338 may receive a network address on the virtual private cloud network 107 from a pool of network addresses corresponding to the virtual private cloud network 107.

[0141] At box 709, the VPC application 338 receives network traffic from another application on the client device 215 that is destined for the virtual private cloud network 107. For example, the destination network address may correspond to a virtual machine instance of the cloud service 320 (FIG. 3B). At box 710, the VPC application 338 may apply one or more VPC security group 477 (FIG. 4) rules or network ACLs 473 (FIG. 4) obtained via the VPC control plane to determine whether the network traffic is permitted to be forwarded to its destination. For example, the network traffic from the client device 215 may not be permitted to be forwarded to the destination based on a role or user associated with the client device 215 within the organization that operates the virtual private cloud network 107.

[0142] At box 712, the VPC application 338 encapsulates the network traffic. For example, the VPC application 338 may add one or more headers to the data packets in the network traffic. The VPC application 338 may also translate the destination network address to a destination network address of the computing device 418 (FIG. 4) on the cloud provider network 203 (FIG. 3B) that hosts the virtual machine instance. The VPC application 338 may perform the translation by referring to the network address translation data 476 (FIG. 4). The VPC application 338 may also translate the source network address of the client device 215 on the virtual private cloud network 107 to a source network address on the wireless network 103. The VPC application 338 may also encrypt the network traffic.

[0143] At box 715, the VPC application 338 forwards the encapsulated network traffic to the wireless network 103. The VPC application 338 can receive VPC routing table data via the control plane of the virtual private cloud network 107 to make forwarding decisions. At box 718, the VPC application 338 receives the encapsulated network traffic from the wireless network 103. The encapsulated network traffic is transmitted by one or more hosts on the virtual private cloud network 107.

[0144] In box 721, the VPC application 338 decapsulates the network traffic. This may include translating a source network address on the cloud provider network 203 (e.g., of a computing device) to a source network address on the virtual private cloud network 107, and translating a destination network address of the client device 215 on the wireless network 103 to a destination network address on the virtual private cloud network 107. This may include simply removing the headers added by the encapsulation. The VPC application 338 may also decrypt the network traffic. In box 724, the VPC application 338 forwards the network traffic to other applications on the client device 215. For example, the VPC application 338 may use a VPC routing table to make a forwarding decision. The VPC application 338 may also refer to one or more security group 477 rules or network ACLs 473 in the virtual private cloud network 107 to determine whether to forward the network traffic to other applications. The partial operation of the VPC application 338 then ends.

[0145] 8, a schematic block diagram of a computing environment 403 is shown, according to an embodiment of the present disclosure. The computing environment 403 includes one or more computing devices 800. Each computing device 800 includes at least one processor circuit having, for example, a processor 803 and a memory 806, both of which are coupled to a local interface 809. To this end, each computing device 800 may include, for example, at least one server computer or similar device. The local interface 809 may include, for example, a data bus or other bus structure including an associated address / control bus, as will be appreciated.

[0146] Stored in memory 806 are both data and certain components executable by processor 803. Specifically, stored in memory 806 and executable by processor 803 are VPN management service 424, network services 427, and other possible applications. Also stored in memory 806 may be data store 415 and other data. Additionally, an operating system may be stored in memory 806 and executable by processor 803.

[0147] Of course, it is understood that there may be other applications stored in memory 806 and executable by processor 803. Any of the components described herein, if implemented in software, may employ any of a number of programming languages, such as, for example, C, C++, C#, Objective C, Java, JavaScript, Perl, PHP, Visual Basic, Python, Ruby, Flash, or other programming languages.

[0148] Some software components are stored in the memory 806 and are executable by the processor 803. In this regard, the term "executable" refers to a program file in a format that can ultimately be executed by the processor 803. An example of an executable program may be, for example, a compiled program that can be loaded into a random access portion of the memory 806 and converted into machine code in a format executable by the processor 803, source code that can be expressed in a suitable format such as object code that can be loaded into a random access portion of the memory 806 and executed by the processor 803, or source code that can be interpreted by another executable program that generates instructions in the random access portion of the memory 806 to be executed by the processor 803, etc. The executable program may be stored in any portion or component of the memory 806, including, for example, a random access memory (RAM), a read-only memory (ROM), a hard drive, a solid-state drive, a USB flash drive, a memory card, an optical disk such as a compact disk (CD) or a digital versatile disk (DVD), a floppy disk, a magnetic tape, or other memory component.

[0149] Memory 806 is defined herein to include both volatile and non-volatile memory and data storage components. A volatile component is one that does not retain a data value upon loss of power. A non-volatile component is one that retains data upon loss of power. Thus, memory 806 may include, for example, random access memory (RAM), read only memory (ROM), a hard disk drive, a solid state drive, a USB flash drive, a memory card accessed via a memory card reader, a floppy disk accessed via an associated floppy disk drive, an optical disk accessed via an optical disk drive, a magnetic tape accessed via a suitable tape drive, and / or other memory components, or a combination of any two or more of these memory components. Further, RAM may include, for example, static random access memory (SRAM), dynamic random access memory (DRAM), or magnetic random access memory (MRAM), as well as other such devices. ROM may include, for example, a programmable read only memory (PROM), an erasable programmable read only memory (EPROM), an electrically erasable programmable read only memory (EEPROM), or other similar memory devices.

[0150] Also, processor 803 may represent multiple processors 803 and / or multiple processor cores, and memory 806 may represent multiple memories 806 each operating in parallel processing circuits. In such a case, local interface 809 may be a suitable network facilitating communication between any two of the multiple processors 803, between any processor 803 and any of the memories 806, or between any two of the memories 806, etc. Local interface 809 may include additional systems designed to regulate this communication, including, for example, implementing load balancing. Processor 803 may be an electrical structure or some other available structure.

[0151] The VPC management service 424, network service 427, and various other systems described herein may be embodied in software or code executed by general purpose hardware, as discussed above, although, alternatively, the same may be embodied in dedicated hardware or a combination of software / general purpose hardware and dedicated hardware. If embodied in dedicated hardware, each may be implemented as a circuit or state machine using any one or combination of a number of technologies. These technologies may include, but are not limited to, discrete logic circuits having logic gates for performing various logical functions upon application of one or more data signals, application specific integrated circuits (ASICs) having appropriate logic gates, field programmable gate arrays (FPGAs), or other components, and the like. Such technologies are generally well known to those skilled in the art and will not be described in detail herein.

[0152] The flowcharts in Figures 5-7 illustrate the functionality and operation of some embodiments of the wireless network 103 and the virtual private cloud network 107. When embodied in software, each block may represent a module, segment, or portion of code that includes program instructions to perform a particular logical function(s). The program instructions may be embodied in the form of source code that includes human-readable statements written in a programming language, or machine code that includes numerical instructions recognizable by a suitable execution system, such as the processor 803 in a computer system or other system. The machine code may be translated from the source code or the like. When embodied in hardware, each block may represent a circuit or a number of interconnected circuits to perform a particular logical function(s).

[0153] Although the flowcharts of FIGS. 5-7 show a specific execution order, it is understood that the execution order may differ from the order shown. For example, the execution order of two or more blocks may be swapped from the order shown. Also, two or more blocks shown in succession in FIGS. 5-7 may be executed simultaneously or with partial concurrence. Furthermore, in some embodiments, one or more blocks shown in FIGS. 5-7 may be skipped or omitted. Furthermore, any number of counters, state variables, alert semaphores, or messages may be added to the logic flows described herein for purposes such as improving utility, accounting, measuring performance, or providing troubleshooting clues. Of course, all such variations are within the scope of the present disclosure.

[0154] Also, any logic or application described herein, including VPC management services 424 and network services 427, comprising software or code, may be embodied in any non-transitory computer-readable medium for use by or in association with an instruction execution system, such as, for example, processor 803 in a computer system or other system. In this sense, logic may include, for example, statements including instructions and declarations that may be fetched from a computer-readable medium and executed by an instruction execution system. In the context of this disclosure, a "computer-readable medium" may be any medium that can contain, store, or retain logic or applications described herein for use by or in association with an instruction execution system.

[0155] The computer-readable medium may include any one of a number of physical media, such as, for example, magnetic, optical, or semiconductor media. More specific examples of suitable computer-readable media include, but are not limited to, magnetic tape, magnetic floppy diskette, magnetic hard drive, memory card, solid-state drive, USB flash drive, or optical disk. The computer-readable medium may also be, for example, random access memory (RAM), including static random access memory (SRAM) and dynamic random access memory (DRAM), or magnetic random access memory (MRAM). Furthermore, the computer-readable medium may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or other types of memory devices.

[0156] Additionally, any logic or applications described herein, including VPC management services 424 and network services 427, may be implemented or structured in a variety of ways. For example, one or more applications described herein may be implemented as modules or components of a single application. Furthermore, one or more applications described herein may execute on a shared computing device or separate computing devices, or a combination thereof. For example, multiple applications described herein may execute on the same computing device 800 or on multiple computing devices 800 of the same computing environment 403.

[0157] Unless otherwise indicated, disjunctive language such as the phrase "at least one of X, Y, or Z" is understood in context as being generally used to indicate that an item, term, etc. may be either X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z). Thus, such disjunctive language is not generally intended to or suggests that a particular embodiment requires that at least one of X, at least one of Y, or at least one of Z, respectively, be present.

[0158] Embodiments of the present disclosure can be described by one or more of the following clauses.

[0159] Clause 1. A system comprising: a wireless network having a network link to a cloud provider network; and the cloud provider network hosting a virtual private cloud network having an associated Internet Protocol (IP) address range, the cloud provider network including at least one computing device configured to at least: create a subnet of the virtual private cloud network in the wireless network; register a subscriber identity module of a client device with the virtual private cloud network; assign an IP address to the subscriber identity module from the associated IP address range of the virtual private cloud network; and forward network traffic from the virtual private cloud network to the client device via the network link to the wireless network.

[0160] Clause 2. The system of clause 1, wherein the wireless network further comprises a gateway configured to at least decapsulate the network traffic from the virtual private cloud network based at least in part on translating the IP address of the client device on the virtual private cloud network to another network address of the client device on the wireless network.

[0161] Clause 3. The system of clause 2, wherein the gateway is further configured to at least: receive other network traffic from the client device; encapsulate the other network traffic based at least in part on translating a first network address of a virtual machine instance on the virtual private cloud network to a second network address of a particular computing device on the cloud private network, the virtual machine instance running on the particular computing device; and forward the encapsulated other network traffic to the virtual private cloud network via the network link.

[0162] Clause 4. The system described in clauses 1-3, wherein the network link corresponds to a dedicated private link between the wireless network and the cloud provider network.

[0163] Clause 5. The system of clauses 1-4, wherein the client device is configured to decapsulate the network traffic and translate a source IP address on the cloud provider network to a source IP address on the virtual private cloud network.

[0164] Clause 6. A computer-implemented method comprising: receiving a request from a client device for a service from a wireless network; in response to receiving the request from the client device for the service from the wireless network, providing the client device with access to a virtual private cloud network via the wireless network; assigning a network address to the client device from a pool of network addresses associated with the virtual private cloud network; and forwarding encapsulated network traffic from the client device to the virtual private cloud network via a network link between the wireless network and a cloud provider network that hosts one or more resources on the virtual private cloud network.

[0165] Clause 7. The computer-implemented method of clause 6, further comprising: generating the encapsulated network traffic by encapsulating, by a gateway in the wireless network, network traffic received from the client device.

[0166] Clause 8. The computer-implemented method of clause 7, further comprising translating, by the gateway, a first network address of a virtual machine instance on the virtual private cloud network to a second network address of a computing device on the cloud private network, the virtual machine instance running on the computing device.

[0167] Clause 9. The computer implemented method of clauses 7-8, wherein the gateway is implemented in a radio access network of the wireless network.

[0168] Clause 10. The computer implemented method of clauses 7-9, wherein the gateway is implemented in a core network of the wireless network.

[0169] Clause 11. The computer-implemented method of clause 10, wherein the gateway is implemented in a customized user plane function of the core network.

[0170] Clause 12. The computer-implemented method of clauses 6-11, further comprising identifying network traffic from the client device that should be forwarded to the virtual private cloud network based at least in part on a rule set.

[0171] Clause 13. The computer-implemented method of clauses 6-12, wherein providing the client device with access to the virtual private cloud network via the wireless network is based at least in part on a subscriber identity module of the client device.

[0172] Clause 14. The computer-implemented method of clauses 6-13, further comprising configuring a network slice in the wireless network corresponding to the virtual private cloud network.

[0173] Clause 15. The computer-implemented method of clauses 6-14, further comprising providing the client device with access to a plurality of virtual private cloud networks via the wireless network.

[0174] Clause 16. A computer-implemented method comprising: requesting a service by a client device from a wireless network; connecting, by the client device, to a virtual private cloud network over the wireless network; receiving, by the client device, an assignment of a network address from a pool of network addresses corresponding to the virtual private cloud network; receiving network traffic from an application executing on the client device, the network traffic being destined to a first network address of a virtual machine instance on the virtual private cloud network; encapsulating, by the client device, the network traffic based at least in part on converting the first network address to a second network address of a computing device on a cloud provider network, the virtual machine instance executing on the computing device; and forwarding, by the client device, the encapsulated network traffic to the wireless network.

[0175] Clause 17. The computer-implemented method of clause 16, wherein another application running on the client device encapsulates the network traffic or an operating system running on the client device encapsulates the network traffic.

[0176] Clause 18. The computer-implemented method of clauses 16-17, further comprising receiving, by the client device, via a control plane of the virtual private cloud network, at least one of a security group rule or a network access control list (ACL), and determining, by the client device, forwarding of the network traffic based at least in part on application of the at least one of the security group rule or the network ACL to the network traffic.

[0177] Clause 19. The computer-implemented method of clauses 16-18, further comprising receiving network address translation data from the wireless network for translating the first network address to the second network address.

[0178] Clause 20. The computer-implemented method of clauses 16-19, wherein forwarding the encapsulated network traffic further includes forwarding the encapsulated network traffic to the wireless network via a network slice corresponding to the virtual private cloud network.

[0179] It should be emphasized that the above-described embodiments of the present disclosure are merely examples of possible implementations set forth for a clear understanding of the principles of the present disclosure. Many variations and modifications may be made to the above-described embodiment(s) without substantially departing from the spirit and principles of the present disclosure. All such modifications and variations are intended to be included herein within the scope of the present disclosure and protected by the following claims.

Claims

1. a wireless network having a network link to a cloud provider network; the cloud provider network hosting a virtual private cloud network having an associated Internet Protocol (IP) address range; The cloud provider network comprises: creating a subnet of the virtual private cloud network within the wireless network; registering a subscriber identity module of a client device with the virtual private cloud network; assigning the subscriber identity module an IP address from the associated IP address range of the virtual private cloud network; forwarding network traffic from the virtual private cloud network to the client device via the network link to the wireless network; at least one computing device configured to perform at least The system.

2. the wireless network further comprising a gateway configured to at least decapsulate the network traffic from the virtual private cloud network based at least in part on translating the IP address of the client device on the virtual private cloud network to another network address of the client device on the wireless network, the gateway comprising: receiving other network traffic from the client device; and encapsulating the other network traffic based at least in part on translating a first network address of a virtual machine instance on the virtual private cloud network to a second network address of a particular computing device on the cloud private network, the virtual machine instance running on the particular computing device; forwarding the encapsulated other network traffic over the network link to the virtual private cloud network; Further configured to at least The system of claim 1 .

3. The system of claim 1 , wherein the network link corresponds to a dedicated private link between the wireless network and the cloud provider network.

4. The system of claim 1 , wherein the client device is configured to decapsulate the network traffic and translate a source IP address on the cloud provider network to a source IP address on the virtual private cloud network.

5. receiving a request from a client device for a service from a wireless network; in response to receiving the request from the client device for the service from the wireless network, providing the client device with access to a virtual private cloud network over the wireless network; allocating a network address to the client device from a pool of network addresses associated with the virtual private cloud network; forwarding the encapsulated network traffic from the client device to the virtual private cloud network over a network link between the wireless network and a cloud provider network that hosts one or more resources on the virtual private cloud network; 4. A computer-implemented method comprising:

6. encapsulating, by a gateway implemented in a radio access network of the wireless network or in a core network of the wireless network, network traffic received from the client device to generate the encapsulated network traffic; and translating, by the gateway, a first network address of a virtual machine instance on the virtual private cloud network to a second network address of a computing device on the cloud private network, the virtual machine instance running on the computing device.

6. The computer-implemented method of claim 5.

7. The computer-implemented method of claim 6 , wherein the gateway is implemented in a customized user plane function of the core network.

8. The computer-implemented method of claim 5 , further comprising identifying network traffic from the client device that should be forwarded to the virtual private cloud network based at least in part on a rule set.

9. The computer-implemented method of claim 5 , wherein providing the client device with access to the virtual private cloud network over the wireless network is based at least in part on a subscriber identity module of the client device.

10. The computer-implemented method of claim 5 , further comprising configuring a network slice in the wireless network that corresponds to the virtual private cloud network.

11. The computer-implemented method of claim 5 , further comprising providing the client device with access to a plurality of virtual private cloud networks via the wireless network.

12. requesting service from a wireless network by a client device; connecting, by the client device, to a virtual private cloud network via the wireless network; receiving, by the client device, an assignment of a network address from a pool of network addresses corresponding to the virtual private cloud network; receiving network traffic from an application executing on the client device, the network traffic being destined for a first network address of a virtual machine instance on the virtual private cloud network; encapsulating the network traffic based at least in part on translating, by the client device, the first network address to a second network address of a computing device on a cloud provider network, the virtual machine instance running on the computing device; forwarding, by the client device, the encapsulated network traffic to the wireless network; 4. A computer-implemented method comprising:

13. 13. The computer-implemented method of claim 12, further comprising receiving network address translation data from the wireless network for translating the first network address to the second network address, and wherein another application executing on the client device encapsulates the network traffic or an operating system executing on the client device encapsulates the network traffic.

14. receiving, by the client device, at least one of security group rules or a network access control list (ACL) via a control plane of the virtual private cloud network; determining, by the client device, forwarding of the network traffic based at least in part on application of the at least one of the security group rules or the network ACL to the network traffic; The computer-implemented method of claim 12 further comprising:

15. 13. The computer-implemented method of claim 12, wherein forwarding the encapsulated network traffic further comprises forwarding the encapsulated network traffic to the wireless network via a network slice corresponding to the virtual private cloud network.

Citation Information

Patent Citations

  • Control apparatus for gateway in mobile communication system

    WO2017056201A1

  • Secure virtual private mobile and IP network in cloud

    WO2021029910A1