Implementation of the core cellular network stack on cloud infrastructure

Abstract

Techniques for operating a core cellular network stack on a cloud computing environment (e.g., a public cloud infrastructure) are described. For example, a virtualized packet gateway can be run on a virtual machine in the cloud computing environment, and a standard load balancer can distribute network traffic for a cellular network between the virtualized packet gateways. The virtualized packet gateway can be established with a local cache of an external key-value store containing bearers for the cellular network, and the local cache can be used to process received data plane network packets. Bearers can be updated within the cellular network using the external key-value store, and the virtualized packet gateway can obtain and use updated bearer details.

Description

Background Art

[0001] Cellular networks have traditionally been operated by telecommunications providers using customized hardware and software solutions. Although some components of cellular networks can be operated within a cloud infrastructure (e.g., using a public cloud provider), some components and technologies cannot be directly or effectively converted to the cloud. For example, in an LTE cellular network, the control plane network and the user plane network are considered separate networks. The control plane is responsible for establishing, configuring, and removing connections in the user plane. The user plane directly carries user data (network packets from user devices). The serving gateway / packet gateway (abbreviated SPGW) is the component through which user packets flow. In order to handle large amounts of traffic, many SPGW instances may be required. In a cloud implementation, each SPGW instance can run on its own virtual machine. These SPGW instances need to receive control plane messages so that the SPGW instances can correctly process user plane packets.

[0002] Load balancers within a cloud environment are designed to distribute packets across (e.g., evenly across) multiple virtual machines. Typically, a load balancer has little control over which packets go to which virtual machine, other than attempting to direct packets with the same source and destination addresses to the same virtual machine. Because SPGW instances are run behind load balancers in a cloud environment, control plane and user plane data for a given user may be sent to different SPGW instances. In this case, it may be difficult or impossible to send control plane information to or read by an SPGW instance that needs it.

[0003] Thus, there are substantial opportunities to improve technologies related to implementing cellular network components within a cloud environment. Summary of the invention

[0004] This Summary is provided to introduce some concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0005] Techniques for operating a core cellular network stack on a cloud computing environment (e.g., a public cloud infrastructure) are described. For example, a virtualized packet gateway can run on a virtual machine in the cloud computing environment, and a standard load balancer can distribute network traffic for a cellular network among the virtualized packet gateways. The number of virtualized packet gateways can be scaled up (adding new virtualized packet gateways) or scaled down (removing virtualized packet gateways), and the load balancer distributes network traffic accordingly. The virtualized packet gateway can be set up using a local cache of an external key-value store (KVS) containing bearers for the cellular network, and the local cache can be used to process received data plane network packets. Bearers can be updated within the cellular network using the external key-value store, and the virtualized packet gateway can obtain and use the updated bearer details.

[0006] For example, a virtualized packet gateway within a cloud computing environment may perform operations to process a data plane network packet for a cellular network. The virtualized packet gateway may receive a complete copy of an external key-value store, wherein the external key-value store includes bearer details for all current bearers of the cellular network, wherein the bearer details define a network flow associated with a user device using the cellular network. The virtualized packet gateway may save the complete copy of the external key-value store in a local cache. The virtualized packet gateway may receive the data plane network packet from a load balancer. The virtualized packet gateway may process the data plane network packet, wherein the processing includes identifying a bearer associated with the data plane network packet in the local cache.

[0007] As another example, a virtualized packet gateway operating within a cloud computing environment may receive a first data plane network packet associated with a first bearer. Upon determining that the first bearer is not in a local cache, the virtualized packet gateway may retrieve bearer details for the first bearer from an external key-value store and store the retrieved bearer details for the first bearer in the local cache. The virtualized packet gateway may process the first data plane network packet based at least in part on the bearer details for the first bearer stored in the local cache. The virtualized packet gateway may receive a network communication indicating that the first bearer has been updated. In response to receiving the network communication, the virtualized packet gateway may retrieve the updated bearer details of the first bearer from the external key-value store and store the retrieved updated bearer details in the local cache. The virtualized packet gateway may receive a second data plane network packet associated with the first bearer and process the second data plane network packet based at least in part on the updated bearer details of the first bearer stored in the local cache.

[0008] As described herein, a variety of other features and advantages may be incorporated into the technology as desired. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a diagram depicting an example cloud computing environment within which components of a cellular network including a virtualized packet gateway are implemented.

[0010] Figure 2 is a diagram depicting an example cloud computing environment within which components of a cellular network including an SPGW-U are implemented.

[0011] Figure 3 is an example state machine illustrating data plane network packet flow and bearer states.

[0012] Figure 4 is a flow chart of an example method for operating a virtualized packet gateway in a cloud computing environment to process data plane network packets for a cellular network.

[0013] Figure 5 is a flow chart of an example method for operating a virtualized packet gateway in a cloud computing environment to process data plane network packets for a cellular network.

[0014] Figure 6 is a diagram of an example computing system in which some described embodiments may be implemented.

[0015] Figure 7 is an example mobile device that may be used in conjunction with the techniques described herein.

[0016] Figure 8 is an example cloud support environment that you can use with the techniques described in this article. DETAILED DESCRIPTION

[0017] Overview

[0018] As described herein, various techniques and solutions may be applied to operate a core cellular network stack within a cloud computing environment (e.g., a public cloud infrastructure). For example, a virtualized packet gateway may be run on a virtual machine in a cloud computing environment, and a standard load balancer may distribute network traffic of a cellular network between the virtualized packet gateways. The number of virtualized packet gateways may be scaled up (adding new virtualized packet gateways) or scaled down (deleting virtualized packet gateways), and the load balancer distributes the network traffic accordingly.

[0019] In order to provide cellular network services within a cloud computing environment, a key-value store (e.g., a database, a flat file, or another type of data store) is maintained. The key-value store stores bearer details for current bearers of the cellular network. The bearer details define network flows associated with user devices (e.g., mobile phones or other computing devices with cellular network connectivity) using the cellular network. The key-value store is a data store that is separate from other components of the cellular network, such as a virtualized packet gateway. In other words, the key-value store is external to the virtualized packet gateway (also referred to as an external key-value store). In some implementations, the key-value store maintains bearer details for all current bearers of the cellular network (e.g., representing all current network flows active within a cellular network running within a cloud computing environment).

[0020] In the technology described in this article, data plane network packets are processed by a virtualized packet gateway. For example, multiple virtualized packet gateways can be instantiated and run on a virtual machine in a cloud computing environment to process data plane network packets communicated with and from a user device. In some implementations (e.g., implementations that operate an LTE cellular network), the virtualized packet gateway is a serving gateway (SGW), a packet gateway (PGW), or a serving gateway / packet gateway (SPGW). A serving gateway / packet gateway that processes data plane (also called user plane) network packets is also referred to as an SPGW-U. A serving gateway / packet gateway that processes control plane network packets is also referred to as an SPGW-C.

[0021] In traditional cellular networks, telecom providers operate the components of the system. These components can include the following.

[0022] - User Equipment (UE). A device connected to a cellular network. For example, a user equipment may include a mobile phone or other computing device with cellular communication capabilities.

[0023] - Evolved Node B (eNodeB - This component refers to a cellular network base station to which the user equipment is connected via a wireless communication connection (eg, cellular radio).

[0024] - Mobility Management Entity (MME). This component controls the user equipment's access to the cellular network. - Serving Gateway / Packet Gateway (SPGW). Network packets to and from the user equipment flow through the SPGW. The SPGW provides a single anchor point that can be used to relay packets to the user equipment no matter where the user equipment moves (for example, when the user equipment moves from cell tower to cell tower).

[0025] In the context of an LTE cellular network environment, the control plane and the data plane (user plane) are handled separately. The control plane is responsible for establishing, configuring, and tearing down connections in the data plane. The data plane carries network packets to / from user equipment.

[0026] In a cloud computing environment implementation, many SPGW instances may be needed to handle network services, and each SPGW instance runs in its own virtual machine (VM). These virtual machine instances require information from the control plane so that they can correctly process data plane network packets.

[0027] Load balancers in cloud computing environments are designed to distribute network packets between SPGW instances. Typical load balancers will be based on packet header information (e.g., source IP address and destination IP address from the IP header, from User Datagram Protocol (UDP) or Transmission Control Protocol (TCP) header information, and / or from other packet header information). However, there are problems with running SPGW instances within a cloud computing environment using traditional techniques. Specifically, when SPGW instances run behind a load balancer, the control plane and data plane of a given user will typically be directed to different SPGW instances. In this case, the control plane information may not be sent to or read by the SPGW instance that needs it. One possible solution is to have the MME (e.g., the source of the control plane information) broadcast its control data to all SPGW instances. However, this possible solution is inefficient (e.g., requires a large amount of network traffic) and may not be scalable to a small number of SPGWs.

[0028] Using the techniques described herein, cellular networks within a cloud computing environment can be operated more efficiently and reliably. For example, outward expansion can be performed by preloading the contents of a key-value store (e.g., the entire contents of the key-value store) onto a newly instantiated virtualized packet gateway. The virtualized packet gateway can then be prepared to handle any data plane network packets directed to the virtualized packet gateway by the load balancer during the expansion event. As another example, bearers can be moved between virtualized packet gateways. For example, ownership information can be stored (e.g., in a key-value store) indicating which virtualized packet gateway is currently processing the data plane network packets of a given bearer. If the data plane network packets are directed to other virtualized packet gateways (different from the current owner), the other virtualized packet gateways can take ownership (e.g., register themselves in the key-value store to replace the previous owner). In this way, the network flows of the bearer can be moved between virtualized packet gateways (e.g., due to a reduction event or due to some other reasons such as network congestion or failure) without having to close the connection and start a new network flow. As another example, bearers can be updated (e.g., when they are active and do not have to close the bearer and set up a new bearer). For example, a control plane management component (e.g., an MME and / or gateway that handles control plane traffic) can update bearer details in a key-value store. The control plane management component can signal a virtualized packet gateway that owns the updated bearer, and the virtualized packet gateway can obtain the updated bearer details and use the updated bearer details during processing of data plane network packets for the bearer.

[0029] Operating cellular networks in a cloud computing environment

[0030] In the technology described herein, components of a cellular network can be implemented within a cloud computing environment (e.g., a public cloud environment and / or a private cloud environment). For example, a standard load balancer of a cloud environment can be used to distribute cellular network services (e.g., data plane network packets) between a collection (e.g., a cluster) of virtualized packet gateways (e.g., SPGW-U). An external key-value store can store bearer details of current network flows within a cellular network. Expansion of a virtualized packet gateway can be performed by copying the entire contents of the external key-value store to a local cache of a new virtualized packet gateway so that the new virtualized packet gateway can be ready to process any network flow sent to it by a load balancer. The technology also enables bearers to be updated (e.g., for a virtualized packet gateway to receive and apply updated bearer details) and moved (e.g., for moving bearers between virtualized packet gateways). Bearers can be moved and / or updated while the bearer remains active and without shutting down the bearer and setting up a new bearer.

[0031] Figure 1is a diagram depicting an example cloud computing environment 110 within which components of a cellular network (e.g., a virtual cellular network) are implemented. For example, the cloud computing environment 110 may be a public cloud environment or a private cloud environment. The cloud computing environment 110 provides computing resources (e.g., servers, virtual machines, storage resources, network services, etc.) on which the components of the cellular network run.

[0032] like Figure 1 As shown in , user devices 120 and 122 (e.g., mobile phones or other computing devices with cellular network connectivity) are connected to a cellular network via a cellular base station 130 (e.g., a cellular tower). For example, if the cellular network is a long-term evolution (LTE) cellular network, the cellular base station 130 may be an eNodeB. Other types of cellular networks such as 3G cellular networks or 5G cellular networks (e.g., which may have different types of cellular base stations 130) may also be used. Although only two example user devices 120 and 122 are shown, a cellular network may typically support more user devices. Similarly, a cellular network will typically support multiple cellular base stations located in different geographic locations.

[0033] like Figure 1 As shown in , multiple cellular network components are implemented within the cloud computing environment 110. The control plane management 140 component provides control plane services for the cellular network and handles control plane network traffic. For example, the control plane management 140 component can set up network flows (e.g., bearers) for user devices 120 and 122 to access services 180 (e.g., to a public data network (PDN) such as the Internet or a cellular data network). Depending on the type of cellular network, the control plane management 140 component can include many different services. For example, for an LTE cellular network, the control plane management 140 component can include a mobile management entity (MME), a service and / or packet gateway that handles control plane network traffic, a load balancer, etc.

[0034] The key-value store 170 is a data store (e.g., a centralized data store or a distributed data store) containing bearer details for bearers of a cellular network. For example, when the control plane management 140 component receives a request from the user equipment 120 to access a website on the Internet (one of the services 180) or to make a voice call, the control plane management 140 component establishes the bearer (e.g., this may include determining whether the user equipment 120 is authorized to access the service, such as checking the data plan, billing information, etc.). As part of setting up the bearer, the control plane management 140 component stores the bearer details for the bearer in the key-value store 170 (e.g., TEID, QoS value, network bit rate such as download speed, etc.). The bearer represents the type of network traffic (e.g., network flow) of a particular user equipment device. Different bearers can be created for different types of network traffic and / or different applications, including phone calls, website browsing, special traffic (e.g., streaming video, specific VoIP applications, etc.) and / or different quality of service (QoS). A given user equipment device may have multiple currently activated bearers representing different network flows. In some implementations, in order for a user equipment device to use a cellular network, a bearer must first be established for the network flow within the cellular network.

[0035] A virtualized packet gateway 160 (e.g., a plurality of virtualized packet gateways in a cluster running on virtual machines) processes data plane network packets for user devices 120 and 122. For example, the virtualized packet gateway 160 processes data plane network packets transmitted between ("to" and "from") user devices 120 and 122 and a service 180 (e.g., the Internet and / or other external networks).

[0036] When one of virtualized packet gateways 160 receives a first data plane network packet for a given network flow (e.g., a voice call, a visit to a particular website, etc.) from a particular user equipment device, the virtualized packet gateway obtains bearer details for the bearer from key-value store 170 (e.g., the bearer will have been pre-set in the key-value store by control plane management 140 components). The virtualized packet gateway may store the bearer details for the bearer in a local cache (e.g., a local key-value store of the virtualized packet gateway).

[0037] Data plane network packets (as Internet Protocol (IP) network packets) between user devices 120 and 120 and virtualized packet gateway 160 are encapsulated. In some implementations, encapsulation is performed according to a GPRS tunneling protocol (GTP) such as GTP-U. For example, encapsulation provides a way to manage different network flows for a given user device. The tunneling protocol assigns an identifier (e.g., a tunnel endpoint identifier (TEID)) to a bearer associated with a given network flow, and separate identifiers can be used for uplink data (data flowing from the user device to the service 180) and downlink data (data flowing from the service 180 to the user device). For example, a specific bearer can be set up for a specific network flow in which a user device 120 is accessing a specific website, where the bearer details include two identifiers (e.g., two TEIDs) - one identifier is associated with the uplink traffic of the network flow, and one identifier is associated with the downlink traffic of the network flow.

[0038] The load balancer 150 distributes data plane network packets between the virtualized packet gateways 160. The load balancer 150 is a standard load balancer of the cloud computing environment 110 (e.g., a standard load balancer provided by a public cloud service provider) that directs network traffic based on IP header information. Because the load balancer 150 is a standard load balancer, it does not direct network packets based on encapsulated packet information (e.g., it is not a dedicated load balancer that has been configured to work with an encapsulation protocol such as the GTP-U protocol). Although an example load balancer 150 is shown in the figures, multiple load balancers (e.g., multiple load balancer network devices) may be used.

[0039] Because load balancer 150 is a standard load balancer, it will typically direct network packets for a given network flow to the same virtualized packet gateway based on external IP header information (e.g., source and destination IP addresses and / or port numbers). However, since load balancer 150 is not aware of the bearer (e.g., it does not inspect the encapsulated packets or encapsulated headers), it is not possible to determine which virtualized packet gateway will receive the network packets for a given bearer. One possible solution is to pass all bearer details to all virtualized packet gateways. However, this possible solution is inefficient (e.g., uses a large amount of computing resources including network bandwidth, processing resources, and storage).

[0040] Using the techniques described herein, a more efficient solution may be provided. For example, by using the key-value store 170 as a repository of bearer details for bearers of a cellular network, the virtualized packet gateway 160 may obtain bearer details when needed. For example, when one of the virtualized packet gateways 160 receives a data plane network packet associated with an unknown bearer (e.g., a bearer that is not in the local cache of the virtualized packet gateway), the virtualized packet gateway may obtain bearer details for the bearer from the external key-value store 170 and store the bearer details in the local cache of the virtualized packet gateway. This provides improvements in computing resources (e.g., network bandwidth, processing resources, and storage) relative to a solution in which all bearer details are stored at each of the virtualized packet gateways 160.

[0041] Another potential problem with operating cellular network components within a cloud computing environment may occur during expansion of the virtualized packet gateway 160. For example, during an expansion event in which a new virtualized packet gateway is added to the virtualized packet gateway 160 (e.g., an expansion event in which a new virtualized packet gateway instance is instantiated on a virtual machine), the load balancer 150 will begin sending data plane network packets to the new virtual gateway in proportion to the virtualized packet gateways 160 (e.g., evenly dividing the data plane network flows between the virtualized packet gateways 160). If the new virtualized packet gateway begins receiving a large number of network flows for bearers that it has not yet seen, the new virtualized packet gateway may be overloaded (e.g., causing network packets to be delayed or dropped, which may cause network service delays or interruptions). One possible solution is to have the new virtualized packet gateway obtain bearer details from the key-value store 170 for each new bearer as it is encountered (e.g., upon receiving the first network packet associated with a given bearer). However, when a new virtualized packet gateway begins operation and the load balancer 150 begins directing network packets, the new virtualized packet gateway may still become overloaded with the sudden influx of network traffic and the corresponding new bearer details that need to be obtained.

[0042] Using the techniques described herein, such problems of extended events can be reduced or eliminated. For example, after creating a new virtualized packet gateway and before the virtualized packet gateway begins to receive data plane network packets from the load balancer 150, the bearer details can be copied from the key-value store 170 to the local cache of the new virtualized packet gateway. In some implementations, a complete copy of the key-value store 170 is saved to the local cache of the new virtualized packet gateway. Once a copy of the key-value store 170 has been stored at the new virtualized packet gateway, the load balancer 150 begins to guide the data plane network packets to the new virtualized packet gateway. Because the local cache of the new virtualized packet gateway has been filled with bearer details (e.g., pre-filled), any data plane network packets received from the load balancer 150 will have associated bearer details in the local cache, and the new virtualized packet gateway will be able to process them effectively without obtaining bearer details from the key-value store 170. This will achieve reliability (e.g., reducing the chance of network services disrupting user equipment) and savings in network response (e.g., reducing latency).

[0043] Similar problems may occur during the reduction of virtualized packet gateways 160. For example, the remaining virtualized packet gateways 160 may receive a portion of the network flows previously served by the virtualized packet gateway that was disconnected from service. However, in this case, the additional network flows directed to a given remaining virtualized packet gateway through the load balancer 150 should not overload the remaining virtualized packet gateway. Therefore, the remaining virtualized packet gateways can obtain new bearer details for additional network flows from the key-value store 170 when needed.

[0044] When it is necessary to move the bearer between virtualized packet gateways and / or when it is necessary to update the bearer details, another potential problem of operating the cellular network component in the cloud computing environment will occur. For example, during an expansion event, a reduction event, or for some other reasons (e.g., a virtualized packet gateway that is congested or overloaded), it may be necessary to move the bearer representing the ongoing network flow. In some solutions, the bearer is bound to a specific virtualized packet gateway and cannot be moved (e.g., the failure of the virtualized packet gateway will result in the failure of all the bearers and associated network flows served by the virtualized packet gateway). In the technology described in this article, the bearer and the associated network flow can be moved between virtualized packet gateways. For example, if the load balancer 150 guides the network flow of the bearer to other virtualized packet gateways, the other virtualized packet gateways can obtain the bearer details from the key-value storage 170 when receiving the first data plane network packet of the network flow, and serve the bearer. The solution can be executed during an expansion event, a reduction event, or it can be executed in the case of other reasons that cause the network flow to move to other virtualized packet gateways.

[0045] Similar problems may occur when a bearer needs to be updated. For example, due to changes in account status (e.g., bandwidth limitations, types of services accessed, billing issues, etc.), the bearer may need to be updated. In some solutions, the bearer cannot be updated during the network flow (e.g., if a change is required, a new bearer needs to be established, which reduces efficiency and requires additional computing resources). In the technology described in this article, the bearer can be updated. In order to update the bearer, the key-value storage 170 can store ownership information of the bearer, and the ownership information indicates which virtualized packet gateway owns each bearer (e.g., the data plane network packet currently responsible for processing the bearer). For example, when the virtualized packet gateway receives the first data plane network packet associated with a specific bearer (e.g., where the specific bearer is not in the local cache of the virtualized packet gateway, and / or where the packet for the specific bearer is not processed before), the virtualized packet gateway can obtain the bearer details of the specific bearer from the key-value storage 170 and register itself as the owner of the specific bearer in the key-value storage 170 (e.g., the key-value storage library 170 can maintain a separate table and establish an association between the bearer and the virtualized packet gateway). When a bearer needs to be updated, the control plane management 140 component can update the bearer details in the key-value store 170, determine which virtualized packet gateway is the owner of the bearer, and alert the virtualized packet gateway so that the virtualized packet gateway can obtain the updated bearer details from the key-value store 170. This solution allows bearers to be updated during a network flow without stopping and re-establishing a new network flow, which saves time, computing resources, and can reduce or eliminate network interruptions.

[0046] Figure 2 is a diagram depicting an example cloud computing environment 210 within which components of an LTE cellular network (e.g., a virtual LTE cellular network), including an SPGW-U, are implemented. For example, the cloud computing environment 210 may be a public cloud environment or a private cloud environment. The cloud computing environment 210 provides computing resources (e.g., servers, virtual machines, storage resources, network services, etc.) on which the components of the cellular network run. Figure 2 Many of the components described perform the same tasks as described above with respect to Figure 1 Corresponding components are described to have the same or similar functions.

[0047] like Figure 2 As shown in , user devices 220 and 222 (e.g., mobile phones or other computing devices with cellular network connectivity) are connected to the cellular network via eNodeB 230 (e.g., a cellular base station within an LTE cellular network). Other types of cellular networks such as 3G cellular networks or 5G cellular networks may also be used. Although only two example user devices 220 and 222 devices are shown, a cellular network may typically support more user device devices. Similarly, a cellular network will typically support multiple cellular base stations located in different geographical locations.

[0048] like Figure 2 As shown in , multiple cellular network components are implemented within a cloud computing environment 210. Multiple components provide control plane services for the cellular network and handle control plane network traffic, including an MME 240 and a service packet gateway (SPGW-C) 262. For example, the MME 240, together with the SPGW-C 262, can establish network flows (e.g., bearers) for user devices 220 and 222 to access services 290 (e.g., to a public data network (PDN) such as the Internet or a cellular data network). A load balancer 252 (or multiple load balancers in some implementations) can distribute control plane network traffic among multiple SPGW-Cs 262 (e.g., to handle the control plane network traffic load of the cellular network).

[0049] The key-value store 270 is a data store (e.g., a centralized data store or a distributed data store) containing bearer details for bearers for a cellular network. For example, when the MME 240 receives a request from the user equipment 220 to access a website (one of the services 290) on the Internet or to make a voice call, the MME 240 establishes a bearer (e.g., which may include determining whether the user equipment 220 is authorized to access the service, e.g., checking the data plan, billing information, etc.). For example, the MME 240 may instruct the SPGW-C 262 to configure the bearer and store the bearer details of the bearer in the key-value store 270. A bearer represents a type of network service (e.g., a network flow) for a particular user equipment device. Different bearers may be created for different types of network services and / or different applications, including phone calls, website browsing, special traffic (e.g., streaming video, specific VoIP applications, etc.), and / or different qualities of service (QoS). A given user equipment device may have multiple currently active bearers representing different network flows. In some embodiments, in order for a user equipment device to use a cellular network, a bearer must first be established for the network flow within the cellular network.

[0050] The SPGW-U 260 (e.g., multiple SPGW-Us in a cluster running on virtual machines) processes data plane network packets for the user devices 220 and 222. For example, the SPGW-U 260 processes data plane network packets that are transmitted between ("to" and "from") the user devices 220 and 222 and a service 290 (e.g., the Internet and / or other external networks). In some implementations, a network address translation (NAT) service 280 is used for IP address translation between the SPGW-U 260 and the service 290 (e.g., the Internet).

[0051] When one of the SPGW-Us 260 receives a first data plane network packet for a given network flow (e.g., a voice call, a visit to a particular website, etc.) from a particular user equipment device, the SPGW-U obtains the bearer details for the bearer from the key-value store 270 (e.g., the bearer will have been pre-set in the key-value store by the MME 240 and / or the SPGW-C 262). The SPGW-U may store the bearer details for the bearer in a local cache (e.g., a local key-value store of the SPGW-U).

[0052] Data plane network packets (as Internet Protocol (IP) network packets) between user equipment 220 and 220 and SPGW-U 260 are encapsulated. In some implementations, encapsulation is performed according to a GPRS tunneling protocol (GTP), such as GTP-U. For example, encapsulation provides a way to manage different network flows for a given user equipment device. The tunneling protocol assigns an identifier (e.g., a tunnel endpoint identifier (TEID)) to a bearer associated with a given network flow, and separate identifiers may be used for uplink data (data flowing from the user equipment device to the service 290) and downlink data (data flowing from the service 290 to the user equipment device). For example, a specific bearer may be set up for a specific network flow in which user equipment 220 is accessing a specific website, where the bearer details include two identifiers (e.g., two TEIDs), one associated with the uplink traffic of the network flow and one associated with the downlink traffic of the network flow.

[0053] The load balancer 250 distributes data plane network packets between the SPGW-Us 260. The load balancer 250 is a standard load balancer of the cloud computing environment 210 (e.g., a standard load balancer provided by a public cloud service provider), and a standard load balancer directs network traffic based on IP header information. Because the load balancer 250 is a standard load balancer, the load balancer does not direct network packets based on encapsulated packet information (e.g., the network packets are not target-specific load balancers that have been configured to work with encapsulation protocols such as the GTP-U protocol). Although an example load balancer 250 is shown in the figures, multiple load balancers (e.g., multiple load balancer network devices) may be used.

[0054] Because the load balancer 250 is a standard load balancer, it will typically direct network packets for a given network flow to the same SPGW-U based on the outer IP header information (e.g., source and destination IP addresses and / or port numbers). However, because the load balancer 250 is bearer-agnostic (e.g., it does not inspect the encapsulated packets or encapsulated headers), it cannot determine which SPGW-U will receive the network packets for a given bearer. One possible solution is to communicate all bearer details to all SPGW-Us. However, this possible solution is inefficient (e.g., uses a large amount of computing resources including network bandwidth, processing resources, and storage).

[0055] Using the techniques described herein, a more efficient solution may be provided. For example, by using the key-value store 270 as a repository for bearer details for bearers of a cellular network, the SPGW-U 260 may obtain the bearer details when needed. For example, when one of the SPGW-Us 260 receives a data plane network packet associated with an unknown bearer (e.g., a bearer that is not in the SPGW-U's local cache), the SPGW-U may obtain the bearer details for the bearer from the external key-value store 270 and store it in its local cache. This provides improvements in computing resources (e.g., network bandwidth, processing resources, and storage) compared to a solution in which all bearer details are stored at each SPGW-U in the SPGW-U 260.

[0056] Another potential problem with operating cellular network components within a cloud computing environment may occur during expansion of the SPGW-U 262. For example, during an expansion event where a new SPGW-U is added to the SPGW-U 262 (e.g., an expansion event where a new SPGW-U instance is instantiated on a virtual machine), the load balancer 250 will begin sending data plane network packets to the new SPGW-U in proportion to the number of SPGW-Us 262 (e.g., evenly dividing the data plane network flows between the SPGW-Us 262). If the new SPGW-U begins to receive a large number of network flows for bearers that do not yet exist, the new SPGW-U may be overloaded (e.g., causing network packets to be delayed or dropped, which may cause network service delays or interruptions). One possible solution is to have the new SPGW-U obtain bearer details from the key-value store 270 for each new bearer as it is encountered (e.g., upon receiving the first network packet associated with a given bearer). However, when a new SPGW-U starts operating and the load balancer 250 starts directing network packets, the new SPGW-U may still become overloaded with a sudden influx of network traffic and the corresponding new bearer details that need to be obtained.

[0057] Using the techniques described herein, such problems with extended events can be reduced or eliminated. For example, after a new SPGW-U is created and before it begins receiving data plane network packets from the load balancer 250, the bearer details can be copied from the key-value store 270 to the local cache of the new SPGW-U. In some implementations, a complete copy of the key-value store 270 is saved to the local cache of the new SPGW-U. Once a copy of the key-value store 270 has been stored at the new SPGW-U, the load balancer 250 begins to direct the data plane network packets to the new SPGW-U. Because the local cache of the new SPGW-U has been populated (e.g., pre-populated) with bearer details, any data plane network packets received by the new SPGW-U from the load balancer 250 will have associated bearer details in the local cache, and the new SPGW-U will be able to efficiently process data plane network packets without obtaining bearer details from the key-value store 270. This will enable savings in network response (e.g., reducing latency) and reliability (e.g., reducing the chance of network services disrupting user equipment).

[0058] A similar problem may occur during a scale-down of an SPGW-U 262. For example, the remaining SPGW-U 262 may receive a portion of the network flows previously served by the SPGW-U that was out of service. However, in this case, the additional network flows directed to a given remaining SPGW-U by the load balancer 250 should not overload the remaining SPGW-U. Therefore, the remaining SPGW-U may obtain new bearer details for the additional network flows from the key-value store 270 when needed.

[0059] Another potential problem with operating cellular network components within a cloud computing environment may arise when bearers need to be moved between SPGW-Us and / or when bearer details need to be updated. For example, a bearer representing an ongoing network flow may need to be moved during an expansion event, a reduction event, or for some other reason (e.g., network congestion or SPGW-U overload). In some solutions, the bearer is bound to a specific SPGW-U and cannot be moved (e.g., a failure of the SPGW-U will result in the failure of all bearers served by the SPGW-U and the associated network flows). In the technology described herein, bearers and associated network flows can be moved between SPGW-Us. For example, if the load balancer 250 directs the network flow of the bearer to other SPGW-Us, the other SPGW-Us can obtain the bearer details from the key-value store 270 when receiving the first data plane network packet of the network flow, and serve the bearer. This solution can be performed during an expansion event, a reduction event, or due to some other reason that causes the network flow to move to other SPGW-Us.

[0060] Similar problems may occur when a bearer needs to be updated. For example, a bearer may need to be updated due to a change in account status (e.g., bandwidth limitations, type of service being accessed, billing issues, etc.). In some solutions, a bearer cannot be updated during a network flow (e.g., if a change is required, a new bearer needs to be established, which reduces efficiency and requires additional computing resources). In the technology described in this article, a bearer can be updated. In order to update a bearer, the key-value storage 170 can store ownership information for the bearer, and the ownership information indicates which SPGW-U owns each bearer (e.g., the data plane network packet currently responsible for processing the bearer). For example, when the SPGW-U receives the first data plane network packet associated with a specific bearer (e.g., where the specific bearer is not in the local cache of the SPGW-U, and / or, where the packet for the specific bearer has not been processed before), the SPGW-U can obtain the bearer details of the specific bearer from the key-value storage 270 and register itself as the owner of the specific bearer in the key-value storage 170 (e.g., the key-value storage library 270 can maintain a separate table and establish an association between the bearer and the SPGW-U). When a bearer needs to be updated, the MME 240 and / or the SPGW-C 262 may update the bearer details in the key-value store 270, determine which SPGW-U is the owner of the bearer, and alert the SPGW-U so that the SPGW-U may obtain the updated bearer details from the key-value store 270. This solution allows bearers to be updated during a network flow without stopping and re-establishing a new network flow, which saves time, computing resources, and may reduce or eliminate network disruptions.

[0061] In the example cloud computing environment 210, the SPGW-U 260 is described as a combined solution in which the service gateway and packet gateway for data plane traffic are combined into an integrated or co-located service gateway / packet gateway component. However, in some embodiments, the service gateway and the packet gateway are separate components that operate independently. For example, there may be multiple reasons why each UE, SGW, and PGW may be located in different locations. One of the reasons is domain name service (DNS) selection. When the UE is attached, the evolved packet core (EPC) can use a DNS server to select the SGW and PGW separately. The selection algorithm can use different criteria for each algorithm; the SGW can be based on location (usually selecting the SGW closest to the base station), while the PGW can be based on the access point name (APN) (in fact, defining the type of service and network that the UE needs to connect to). Another reason is roaming. There may be a situation where the PGW is located in the local network and the SGW is located in the visited network. For example, this is typical for a mobile virtual network operator (MVNO).

[0062] In embodiments where the SGW and PGW are separate (e.g., located separately, such as running on different computing devices), the SGW and PGW may communicate via an S5 / S8 interface. There may be two features; one based on a GTP tunnel and the other based on Proxy Mobile IPv6 (PMIPv6) (RFC 5213).

[0063] In implementations using a combined SPGW, the S5 / S8 interface is not required. The SPGW-U will map from IP to GTP TEID in the downlink and TEID to IP in the uplink. However, in implementations where the SGW and PGW are separate and use the S5 / S8 interface, the following mapping can be used:

[0064] -PGW-u (GTP-based): Downlink: IP to GTP TEID, Uplink: GTP TEID to IP

[0065] -SGW-u (GTP-based): Downlink: GTP TEID to GTP TEID, Uplink: GTP TEID to GTPTEID

[0066] -PGW-u (based on PMIPv6): Downlink: IP to Generic Routing Encapsulation (GRE) tunnel, Uplink: GRE tunnel to IP

[0067] -SGW-u (based on PMIPv6): Downlink: GRE tunnel to GTP TEID, Uplink: GTP TEID to GRE tunnel

[0068] Figure 3 3 is an example state machine 300 that illustrates data plane (user plane) network packet flows and bearer states (e.g., ownership, additions, deletions, and updates). For example, the example state machine 300 may be implemented by a virtualized packet gateway (e.g., by virtualized packet gateway 160) and / or by an SPGW (e.g., by SPGW-U 260 and / or by SPGW-C 262). In some implementations, there is an instance of a state machine for each bearer.

[0069] With respect to the example state machine 300, there are a number of events that may occur, including:

[0070] 1. Uplink packets arrive at the user plane, i.e. packets are transmitted from the UE towards the PDN.

[0071] 2. Downlink packets arrive at the user plane, i.e. from the PDN to the UE.

[0072] 3. The bearer update packet arrives at the control plane, which can come from MME or KVS.

[0073] 4. Carry unknown packets from KVS to the control plane.

[0074] 5. The bearer deletion packet reaches the control plane from the MME.

[0075] 6. The owner adds a confirmation packet from KVS to the control plane.

[0076] 7. The owner deletion confirmation packet arrives at the control plane from KVS.

[0077] There are a number of variables associated with the example state machine 300, which are defined as follows.

[0078] Am_owner. This variable indicates whether we (e.g., Virtualized Packet Gateway) have informed the KVS owner table that we are the owner of this bearer. The following values ​​are possible.

[0079] 0 KVS has not been informed that we are the owners. Add pending KVS has been informed that we are the owners but have not received confirmation. 1 KVS has been informed that we are the owner and confirmation has been received from KVS. Deletion pending KVS has been told we are no longer the owners but have not received confirmation.

[0080] Read_route_pending. Boolean value that records whether SPGW has requested bearer details from KVS. This variable is used to prevent SPGW from repeatedly requesting bearer details while waiting to read bearer updates from KVS.

[0081] 0 No bearer details read request was sent. 1 Request sent.

[0082] Cached_route_seq. Integer, recording the sequence number present in the bearer update currently stored in our local cache. A value of -1 means that the local cache is invalid.

[0083] The following states depicted are an example state machine 300 .

[0084] State 1: This state indicates that the bearer is unknown and there is no corresponding entry in the local cache.

[0085] States 2, 3, 4, and 5: These are all intermediate states that are passed through when properly configured to handle the user plane.

[0086] State 6: Intermediate state while the hosted knowledge is removed from the local cache.

[0087] State 7: We are in a stable state and are correctly configured to handle user plane data.

[0088] Method for operating a cellular network within a cloud computing environment

[0089] In any examples herein, a method for operating a virtual cellular network within a cloud computing environment may be provided. For example, a virtualized packet gateway (e.g., a service and / or packet gateway) may be implemented on a virtual machine running within the cloud computing environment and process data plane network packets for a cellular network.

[0090] Figure 4 4 is a flow chart of an example method 400 for operating a virtualized packet gateway in a cloud computing environment to process data plane network packets for a cellular network. For example, the example method 400 may be performed by one of the virtualized packet gateways 160 or by one of the SPGW-Us 260.

[0091] At 410, a complete copy of an external key-value store is received. The external key-value store includes bearer details for all current bearers of a cellular network (e.g., all current bearers using a cellular network implemented within a cloud computing environment). The bearer details define a network flow associated with a user device using the cellular network. For example, the external key-value store may be the key-value store 170 or the key-value store 270. At 420, the complete copy of the external key-value store received at 410 is saved in a local cache.

[0092] At 430, a data plane network packet is received from a load balancer. For example, the load balancer can direct the network packet based on Internet Protocol (IP) header information rather than based on encapsulated packet information (eg, not based on a tunnel identifier or TEID).

[0093] At 440, the data plane network packet is processed by the virtualized packet gateway. The processing includes identifying a bearer associated with the data plane network packet in the local cache. For example, the virtualized packet gateway can obtain bearer details from the local cache based on encapsulated network packet header information (e.g., TEID) to identify the network flow (and its associated data plane network packet) and manage it according to its associated bearer (e.g., setting network bit rate, QoS parameters, etc.).

[0094] In some implementations, example method 400 is performed during a scaling event in which a virtualized packet gateway is instantiated on a virtual machine within a cloud computing environment. For example, when the virtualized packet gateway is up and running and saves a complete copy of the key-value store in its local cache, it can indicate to the load balancer (e.g., signal it or respond to its communication) that it is ready to receive data plane network packets. In response, the load balancer can begin directing network traffic to the virtualized packet gateway.

[0095] After the virtualized packet gateway is up and running and processing data plane network packets (e.g., as shown in 440), the virtualized packet gateway may receive a new data plane network packet associated with a new bearer that is not in its local cache. When this occurs, the virtualized packet gateway may obtain bearer details of the new bearer from the external key-value store, store the bearer details in its local cache, and process the new data plane network packet based at least in part on the bearer details of the new bearer. The virtualized packet gateway may also register itself as the owner of the new bearer in the external key-value store.

[0096] In some implementations, the virtualized packet gateway tracks the time when it was last processed for each bearer data plane network packet in the local cache of the virtualized packet gateway. For example, when the virtualized packet gateway processes a packet for a bearer, it can record timing information (e.g., a timestamp or other timing information) in its local cache in association with the bearer. The virtualized packet gateway can use the recorded timing information to clear the bearer from the local cache of the virtualized packet gateway. Although the example method 400 allows the virtualized packet gateway to start operating in an efficient manner using a complete copy of the external key-value store, the virtualized packet gateway will not use many bearers in the key-value store (e.g., the load balancer directs them to other virtualized packet gateways). Therefore, if the data plane network packet is not processed by the virtualized packet gateway for a period of time (e.g., if the packet is not processed within a threshold time period), the virtualized packet gateway can delete the packet from the local cache of the virtualized packet gateway, which can save local storage resources.

[0097] Figure 5 1 is a flow chart of an example method 500 for operating a virtualized packet gateway in a cloud computing environment to process data plane network packets for a cellular network. For example, the example method 500 may be performed by one of the virtualized packet gateways 160 or by one of the SPGW-Us 260.

[0098] At 510, a first data plane network packet associated with a first bearer is received by a virtualized packet gateway. For example, the first data plane network packet can be received from a load balancer at the beginning of a network flow.

[0099] At 520, in response to determining that the first bearer is not in the local cache of the virtualized packet gateway, bearer details of the first bearer are retrieved from the external key-value store. The retrieved bearer details of the first bearer are stored in the local cache at the virtualized packet gateway (e.g., in the local key-value store). For example, the external key-value store can be the key-value store 170 or the key-value store 270.

[0100] At 530, a first data plane network packet is processed based at least in part on the locally stored first bearer details (stored in the local cache).

[0101] At 540, a communication is received indicating that the first bearer has been updated. For example, the communication may be received from the MME or from another component of the cellular network (eg, from a virtualized packet gateway managing a control plane). The communication may be a network control message.

[0102] At 550, in response to the communication indicating that the first bearer has been updated, updated bearer details for the first bearer are retrieved from the external key-value store and stored in the local cache. For example, the first bearer may be updated to reflect a different download bit rate (e.g., based on the user exceeding their monthly limit).

[0103] At 560, a second data plane network packet associated with the first bearer is received by the virtualized packet gateway. The second data plane network packet is a subsequent network packet received from the load balancer as part of the same network flow as the first data plane network packet.

[0104] At 570, a second data plane network packet is processed based at least in part on the locally stored updated bearer details for the first bearer.

[0105] In some implementations, one or more of the following components may be implemented within a cloud computing environment to process data plane network packets. The component may perform one or more of the following and / or other operations. For example, a virtualized packet gateway may be configured via computer executable instructions to store bearer details for a bearer in a local key-value store, wherein the bearer details are obtained from an external key-value store, register ownership of the bearer served by the virtualized packet gateway in the external key-value store, and process network packets based at least in part on the bearer details stored in the local key-value store. A mobile management entity may be configured via computer executable instructions to set a bearer in an external key-value store, update bearer details for the bearer in the external key-value store, and notify the virtualized packet gateway of the updated bearer details. The external key-value store may be configured via computer executable instructions to store bearer details of a current bearer of a cellular network, and store ownership information indicating an ownership relationship between the virtualized packet gateway and the bearer.

[0106] Example Bearer Establishment

[0107] This section shows how to establish a bearer in a specific implementation of an LTE cellular network using the techniques described herein. To illustrate the differences, the bearer establishment sequence is first described for a conventional LTE system and then for the techniques described herein.

[0108] The following sequence illustrates a conventional bearer establishment (without using a cloud computing environment) and the processing of a first user plane data service:

[0109] -UE / eNodeB sends MME bearer establishment request (control plane message).

[0110] -MME sends the bearer establishment request to SPGW.

[0111] -SPGW stores the bearer details from the setup request.

[0112] -SPGW responds to MME, indicating that the bearer establishment is complete.

[0113] -Then, if uplink data comes first:

[0114] -SPGW receives user plane packets from eNodeB.

[0115] -SPGW decapsulates the user plane packets and forwards to the target public data network (PDN).

[0116] -SPGW receives user plane packets from PDN.

[0117] - SPGW looks up the target IP address and finds a match to the UE IP address from the bearer details stored above. If the packet matches the traffic template (TFT) from the bearer details, the packet is encapsulated using the eNodeB TEID from the bearer details.

[0118] -SPGW forwards the encapsulated packet to the eNodeB.

[0119] - or, if downlink data first:

[0120] -The SPGW receives user plane packets from the target PDN.

[0121] -SPGW looks up the target IP address, finds a match, and proceeds with the uplink operation.

[0122] Using the techniques herein, where a virtualized packet gateway (e.g., SPGW-C and / or SPGW-U) is implemented within a cloud computing environment using an external key-value store (also referred to as a shared key-value store), the sequence is as follows:

[0123] -eNodeB sends MME bearer establishment request (control plane message).

[0124] -MME sends the bearer establishment request to the SPGW-C load balancer.

[0125] - For example, the load balancer forwards the request to SPGW-C1 (Instance 1).

[0126] - SPGW-C1 stores the UE IP address, TEID (both SPGW and eNodeB values) and the eNodeB's IP in a shared key-value store.

[0127] -SPGW-C1 responds to MME, indicating that the bearer establishment is complete.

[0128] - The eNodeB sends the user plane packets to the SPGW-U load balancer.

[0129] - For example, the load balancer forwards the user plane packet to SPGW-U2 (instance 2).

[0130] -Then, if uplink data comes first:

[0131] -SPGW-U2 receives user plane packets from the eNodeB.

[0132] - SPGW-U2 looks up the TEID from the packet in the shared key-value store. Gets the eNodeB's TEID in response.

[0133] -SPGW-U2 writes its IP address to the shared key-value store as the "cache owner" for this network flow.

[0134] - SPGW-U2 writes the UE IP address, TEID (both SPGW and eNodeB values) and the eNodeB's IP into the local cache.

[0135] -SPGW-U2 decapsulates the user plane packets and forwards to the target PDN.

[0136] -SPGW-U2 receives user plane packets from PDN.

[0137] - SPGW-U2 looks up the target IP address in its local cache and finds that the target IP address matches the UE IP address from the bearer details stored above. If the packet matches the TFT from the bearer details, the packet is encapsulated using the eNodeB TEID from the bearer details.

[0138] -SPGW-U2 forwards the encapsulated packet to the eNodeB.

[0139] - or, if downlink data first:

[0140] - The SPGW-U load balancer receives user plane packets from the target PDN.

[0141] -Load balancer forwards to SPGW-U2.

[0142] -SPGW-U2 looks up the bearer details in the local cache. No match is found.

[0143] -SPGW-U2 looks up UE IP address in shared key-value store. Gets bearer details in response. Writes to local cache.

[0144] - The IP address that SPGW-U2 writes to the shared key-value store as the "cache owner" for this flow.

[0145] -SPGW-U2 forwards the encapsulated packet to the eNodeB.

[0146] -SPGW-U2 receives user plane packets from eNodeB, decapsulates and forwards to the target PDN.

[0147] Computing System

[0148] Figure 6 A general example of a suitable computing system 600 is depicted in which the described techniques may be implemented. Because the techniques may be implemented in various general-purpose or special-purpose computing systems, computing system 600 is not intended to suggest any limitation as to scope of use or functionality.

[0149] refer to Figure 6 , the computing system 600 includes one or more processing units 610, 615 and memories 620, 625. Figure 6 , the basic configuration 630 is included within the dashed line. Processing units 610, 615 execute computer executable instructions. The processing unit may be a general-purpose central processing unit (CPU), a processor in an application-specific integrated circuit (ASIC), or any other type of processor. The processing unit may also include multiple processors. In a multi-processing system, multiple processing units execute computer executable instructions to increase processing power. For example, Figure 6 A central processing unit 610 and a graphics processing unit or co-processing unit 615 are shown. Tangible memory 620, 625 may be volatile memory (e.g., registers, cache, RAM), non-volatile memory (e.g., ROM, EEPROM, flash memory, etc.), or some combination of both accessible by the processing unit. The memory 620, 625 stores software 680 implementing one or more techniques described herein in the form of computer-executable instructions suitable for execution by the processing unit.

[0150] The computing system may have additional features. For example, the computing system 600 includes storage 640, one or more input devices 650, one or more output devices 660, and one or more communication connections 670. An interconnection mechanism (not shown) such as a bus, controller, or network interconnects the components of the computing system 600. Typically, operating system software (not shown) provides an operating environment for other software executed in the computing system 600 and coordinates the activities of the components of the computing system 600.

[0151] Tangible storage 640 may be removable or non-removable and include disks, magnetic tapes or cassettes, CD-ROMs, DVDs, or any other medium that can be used to store information and that can be accessed within computing system 600. Memory 640 stores instructions for software 680 implementing one or more techniques described herein.

[0152] Input device 650 may be a touch input device such as a keyboard, mouse, pen, or trackball, a voice input device, a scanning device, or another device that provides input to computing system 600. For video encoding, input device 650 may be a camera, video card, TV tuner card, or similar device that accepts video input in analog or digital form, or reads video samples into a CD-ROM or CD-RW of computing system 600. Output device 660 may be a display, printer, speaker, CD burner, or other device that provides output from computing system 600.

[0153] Communication connection 670 enables communication with another computing entity via a communication medium. The communication medium transmits information such as computer executable instructions, audio or video input or output, or other data in a modulated data signal. The modulated data signal is a signal that has one or more of its characteristics set or changed in such a way as to encode information in the signal. By way of example and not limitation, the communication medium may use electrical, optical, RF or other carriers.

[0154] The technology may be described in the general context of computer executable instructions, such as included in program modules, which are executed in a computing system on a target real or virtual processor. Typically, program modules include routines, programs, libraries, objects, classes, components, data structures, etc. that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of program modules may be combined or split between program modules. Computer executable instructions for program modules may be executed within a local or distributed computing system.

[0155] The terms "system" and "device" are used interchangeably herein. Unless the context clearly indicates otherwise, neither term implies any limitation on the type of computing system or computing device. In general, a computing system or computing device may be local or distributed, and may include any combination of special-purpose hardware and / or general-purpose hardware and software that implements the functionality described herein.

[0156] For ease of presentation, the detailed description uses terms such as "determine" and "use" to describe computer operations in a computing system. These terms are high-level abstractions for operations performed by a computer and should not be confused with operations performed by a human. The actual computer operations that correspond to these terms depend on the implementation.

[0157] Mobile devices

[0158] Figure 7 702 is a system diagram depicting an example mobile device 700 including various optional hardware and software components shown generally at 702. Any component 702 in the mobile device may be in communication with any other component, although not all connections are shown for ease of illustration. The mobile device may be any of a variety of computing devices (e.g., a cell phone, a smart phone, a handheld computer, a personal digital assistant (PDA), etc.), and may enable wireless two-way communications with one or more mobile communication networks 704, such as a cellular network, a satellite network, or other networks.

[0159] For example, the example mobile device 700 may be a user equipment device (eg, user equipment 120, 122, 220, or 222) that uses a cellular network (eg, mobile communication network 704). The example mobile device 700 may communicate with the cellular network via a wireless modem 760.

[0160] The illustrated mobile device 700 may include a controller or processor 710 (e.g., a signal processor, microprocessor, ASIC, or other control and processing logic circuit) for performing tasks such as signal encoding, data processing, input / output processing, power control, and / or other functions. An operating system 712 may control the allocation and use of components 702 and support one or more applications 714. The applications may include common mobile computing applications (e.g., email applications, calendars, contact managers, website browsers, messaging applications), or any other computing applications. Functionality 713 for accessing an application store may also be used to obtain and update applications 714.

[0161] The mobile device 700 shown may include a memory 720. The memory 720 may include a non-removable memory 722 and / or a removable memory 724. The non-removable memory 722 may include RAM, ROM, flash memory, a hard disk, or other well-known memory storage technologies. The removable memory 724 may include a flash memory or a subscriber identity module (SIM) card well-known in the GSM communication system, or include other well-known memory storage technologies such as "smart cards". The memory 720 may be used to store data and / or code for running an operating system 712 and application programs 714. Example data may include web pages, text, images, sound files, video data, or other data sets to be sent to one or more network servers and / or received from one or more network servers via one or more wired or wireless networks. The memory 720 may be used to store subscriber identifiers such as an international mobile subscriber identity (IMSI) and equipment identifiers such as an international mobile equipment identifier (IMEI). Such identifiers may be sent to a network server to identify users and equipment.

[0162] The mobile device 700 may support one or more input devices 730 such as a touch screen 732, a microphone 734, a camera 736, a physical keyboard 738, and / or a trackball 740, and one or more output devices 750 such as a speaker 752 and a display 754. Other possible output devices (not shown) may include piezoelectric or other tactile output devices. Some devices may provide more than one input / output function. For example, the touch screen 732 and the display 754 may be combined in a single input / output device.

[0163] The input device 730 may include a natural user interface (NUI). A NUI is any interface technology that enables a user to interact with a device in a "natural" way without artificial constraints imposed by input devices such as a mouse, keyboard, remote control, etc. Examples of NUI methods include methods that rely on voice recognition, touch and stylus recognition, gesture recognition on and adjacent to the screen, air gestures, head and eye tracking, voice and speech, vision, touch, gestures, and machine intelligence. Other examples of NUIs include motion gesture detection using accelerometers / gyroscopes, facial recognition, 3D displays, head, eye and gaze tracking, immersive augmented reality and virtual reality systems, all of which provide more natural interfaces and technologies for sensing brain activity using electric field sensing electrodes (EEG and related methods). Therefore, in a specific example, the operating system 712 or the application 714 may include voice recognition software as part of a voice user interface that allows the user to operate the device 700 via voice commands. In addition, the device 700 may include input devices and software that allow user interaction via spatial gestures of the user, such as detecting and interpreting gestures to provide input to a gaming application.

[0164] As is known in the art, the wireless modem 760 can be coupled to an antenna (not shown) and can support two-way communication between the processor 710 and external devices. The modem 760 is shown generally and can include a cellular modem for communicating with the mobile communication network 704 and / or other radio-based modems (e.g., Bluetooth 764 or Wi-Fi 762). The wireless modem 760 is typically configured to communicate with one or more cellular networks such as a GSM network for data and voice communications between cellular networks within a single cellular network or between a mobile device and a public switched telephone network (PSTN).

[0165] The mobile device may further include at least one input / output port 780, a power source 782, a satellite navigation system receiver 784 such as a global positioning system (GPS) receiver, an accelerometer 786, and / or a physical connector 790 that may be a USB port, an IEEE 1394 (FireWire) port, and / or an RS-232 port. The components 702 shown are not required or all inclusive, and thus any components may be deleted and other components may be added.

[0166] Cloud Support Environment

[0167] Figure 8A general example of a suitable cloud support environment 800 in which the described embodiments, methods, and techniques may be implemented is shown. In the example environment 800, various types of services (e.g., computing services) are provided through a cloud 810. For example, the cloud 810 may include a collection of computing devices that may be centrally located or distributed thereon, thereby providing cloud-based services to various types of users and devices via a network such as the Internet. The implementation environment 800 may be used in different ways to complete computing tasks. For example, some tasks (e.g., processing user input and presenting a user interface) may be performed on a local computing device (e.g., connected devices 830, 840, 850), while other tasks (e.g., storing data used in subsequent processing) may be performed in the cloud 810.

[0168] In the example environment 800, the cloud 810 provides services for connected devices 830, 840, 850 with various screen functions. The connected device 830 represents a device with a computer screen 835 (e.g., a medium-sized screen). For example, the connected device 830 can be a personal computer such as a desktop computer, a laptop computer, a notebook computer, a netbook, etc. The connected device 840 represents a device with a mobile device screen 845 (e.g., a small-sized screen). For example, the connected device 840 can be a mobile phone, a smart phone, a personal digital assistant, a tablet computer, etc. The connected device 850 represents a device with a large screen 855. For example, the connected device 850 can be a TV screen (e.g., a smart TV) or another device connected to the TV (e.g., a set-top box or a game console) or the like. One or more of the connected devices 830, 840, 850 may include touch screen functionality. The touch screen can accept input in different ways. For example, a capacitive touch screen detects touch input when an object (e.g., a fingertip or a stylus) distorts or interrupts the current flowing on the surface. As another example, a touch screen can use an optical sensor to detect touch input when a light beam from the optical sensor is interrupted. For some touch screen detected inputs, physical contact with the screen surface is not required. Devices without screen capabilities can also be used in example environment 800. For example, cloud 810 can provide services for one or more computers (e.g., server computers) without displays.

[0169] Services may be provided by cloud 810 through service provider 820 or through other online service providers (not shown). For example, cloud services may be customized for the screen size, display capabilities, and / or touch screen capabilities of a particular connected device (e.g., connected devices 830, 840, 850).

[0170] In example environment 800, cloud 810 provides the techniques and solutions herein to various connected devices 830, 840, 850, at least in part using service providers 820. For example, service providers 820 can provide a centralized solution for various cloud-based services. Service providers 820 can manage service subscriptions for users and / or devices (e.g., connected devices 830, 840, 850 and / or their respective users).

[0171] Example Implementation

[0172] Although some of the operations in the disclosed method are described in a particular order for convenience of presentation, it should be understood that unless the specific language set forth below requires a particular order, this description includes rearrangement. For example, in some cases, the operations described in sequence can be rearranged or performed simultaneously. In addition, for simplicity, the accompanying drawings may not show various ways in which the disclosed method can be used in combination with other methods.

[0173] Any disclosed method may be implemented as computer executable instructions or computer program products stored on one or more computer readable storage media and executed on a computing device (i.e., any available computing device including a smartphone or other mobile device containing computing hardware). A computer readable storage medium is a tangible medium (such as a DVD or CD, one or more optical disk media of a volatile memory (such as DRAM or SRAM) or a non-volatile memory (such as flash memory or a hard drive)) that can be accessed within a computing environment. As an example and with reference to Figure 6 , the computer readable storage medium includes memories 620 and 625 and memory 640. As an example and with reference to Figure 7 , computer-readable storage media include memory and storage 720, 722, 724. The term "computer-readable storage medium" does not include signals and carrier waves. In addition, the term "computer-readable storage medium" does not include communication connections such as 670, 760, 762 and 764.

[0174] Any computer executable instructions for implementing the disclosed technology and any data created and used during the implementation of the disclosed embodiments can be stored on one or more computer readable storage media. Computer executable instructions can be, for example, a dedicated software application or part of a software application accessed or downloaded via a website browser or other software application (such as a remote computing application). Such software can be, for example, on a single local computer (e.g., any suitable commercial computer) or in a network environment using one or more network computers (e.g., via the Internet, a wide area network, a local area network, a client server network (such as a cloud computing network) or other such network).

[0175] For the sake of clarity, only certain selected aspects of the software-based implementation are described. Other details known in the art are omitted. For example, it should be understood that the disclosed technology is not limited to any particular computer language or program. For example, the disclosed technology can be implemented by software written in C++, Java, Perl, or any other suitable programming language. Similarly, the disclosed technology is not limited to any particular computer or hardware type. The specific details of suitable computers and hardware are known and do not need to be elaborated in this disclosure.

[0176] In addition, any software-based embodiment (e.g., including computer-executable instructions for causing a computer to perform any of the disclosed methods) can be uploaded, downloaded, or remotely accessed via a suitable communication means. Such suitable communication means include, for example, the Internet, the World Wide Web, an intranet, a software application, cables (including optical cables), magnetic communications, electromagnetic communications (including RF, microwave, and infrared communications), electronic communications, or other such communications means.

[0177] The disclosed methods, devices, and systems should not be construed as limiting in any way. Instead, the present disclosure is directed only to all novel and non-obvious features and aspects of the various disclosed embodiments, both individually and in various combinations and sub-combinations with each other. The disclosed methods, devices, and systems are not limited to any particular aspect or feature or combination thereof, nor do the disclosed embodiments require the presence of any one or more specific advantages or solutions to problems.

[0178] The technology from any example can be combined with the technology described in any one or more of the other examples. In view of the many possible embodiments to which the principles of the disclosed technology can be applied, it should be recognized that the illustrated embodiments are examples of the disclosed technology and should not be considered as limiting the scope of the disclosed technology.

Claims

1. One or more computing devices, including: processor; and Memory; The one or more computing devices are configured via computer executable instructions to perform operations for operating a virtualized packet gateway in a cloud computing environment to process data plane network packets for a cellular network, the operations comprising: receiving a complete copy of an external key-value store, wherein the external key-value store includes bearer details for all current bearers of the cellular network, wherein the bearer details define network flows associated with user equipment using the cellular network; Storing the complete copy of the external key-value store in a local cache; receiving a data plane network packet from the load balancer; and The data plane network packet is processed by the virtualized packet gateway, wherein the processing includes identifying a bearer associated with the data plane network packet in the local cache.

2. The one or more computing devices of claim 1, wherein the operations are performed during a scaling event in which the virtualized packet gateway is instantiated within the cloud computing environment.

3. One or more computing devices according to claim 1, wherein the virtualized packet gateway is a service gateway / packet gateway SPGW-U for data plane network packets, and the SPGW-U operates on a virtual machine running in the cloud computing environment.

4. The one or more computing devices of claim 3, wherein the SPGW-U operates as part of a telecommunications provider's Long Term Evolution (LTE) cellular network.

5. The one or more computing devices of claim 1, the operations further comprising: receiving a new data plane network packet from the load balancer, the new data plane network packet being associated with a new bearer not present in the local cache; In response to receiving the new data plane network packet: Obtaining bearer detailed information for the new bearer from the external key-value store; storing the bearer detailed information for the new bearer in the local cache; as well as The new data plane network packet is processed based at least in part on the bearer details for the new bearer.

6. The one or more computing devices of claim 5, the operations further comprising: In response to receiving the new data plane network packet: The virtualized packet gateway is registered in the external key-value store as an owner of the new bearer.

7. One or more computing devices according to claim 1, wherein the load balancer directs network packets based on Internet Protocol (IP) headers and User Datagram Protocol (UDP) or Transmission Control Protocol (TCP) port numbers rather than directing network packets based on encapsulated packet information.

8. The one or more computing devices of claim 1, wherein for each of the one or more bearers in the local cache, the operations further comprise: The bearer is marked in the local cache with an indication of a time when a data plane network packet was last processed for the bearer.

9. The one or more computing devices of claim 8, the operations further comprising: A bearer is purged from the local cache based on the indication of when a data plane network packet was last processed.

10. A method implemented by a computing device for operating a virtualized packet gateway in a cloud computing environment to process data plane network packets for a cellular network, the method comprising: During a scaling event in which the virtualized packet gateway is instantiated within the cloud computing environment, by the virtualized packet gateway: receiving a complete copy of an external key-value store, wherein the external key-value store includes bearer details for all current bearers of the cellular network, wherein the bearer details define network flows associated with user equipment using the cellular network; Storing the complete copy of the external key-value store in a local cache; receiving data plane network packets from a load balancer; as well as The data plane network packet is processed by the virtualized packet gateway, wherein the processing includes identifying a bearer associated with the data plane network packet in the local cache.

11. The method according to claim 10, further comprising: receiving a new data plane network packet from the load balancer, the new data plane network packet being associated with a new bearer not present in the local cache; In response to receiving the new data plane network packet: Obtaining bearer detailed information for the new bearer from the external key-value store; storing the bearer detailed information for the new bearer in the local cache; as well as The new data plane network packet is processed based at least in part on the bearer details for the new bearer.

12. The method according to claim 11, further comprising: In response to receiving the new data plane network packet: The virtualized packet gateway is registered in the external key-value store as an owner of the new bearer.

13. The method of claim 10, wherein the virtualized packet gateway is a service gateway / packet gateway SPGW-U for data plane network packets, and the SPGW-U operates on a virtual machine running in the cloud computing environment.

14. The method of claim 10, wherein the load balancer directs the network traffic based on Internet Protocol (IP) header information rather than directing the network traffic based on encapsulated packet information.

15. A method implemented by a computing device for operating a virtualized packet gateway in a cloud computing environment to process data plane network packets for a cellular network, the method comprising: The virtualized packet gateway consists of: receiving a first data plane network packet associated with a first bearer; When it is determined that the first bearer is not in the local cache, retrieving bearer detailed information for the first bearer from an external key-value store, and storing the retrieved bearer detailed information for the first bearer in the local cache; processing the first data plane network packet based at least in part on the bearer details for the first bearer stored in the local cache; receiving a network communication indicating that the first bearer has been updated; in response to receiving the network communication, retrieving updated bearer details for the first bearer from the external key-value store and storing the retrieved updated bearer details in the local cache; receiving a second data plane network packet associated with the first bearer; as well as The second data plane network packet is processed based at least in part on updated bearer details for the first bearer stored in the local cache.

16. The method of claim 15, wherein the network communication indicating that the first bearer has been updated is received from a mobility management entity (MME) or from a gateway handling control plane network traffic.

17. The method according to claim 15, further comprising: The virtualized packet gateway is registered in the external key-value store as an owner of the first bearer.

18. The method of claim 15, wherein the virtualized packet gateway is a service gateway / packet gateway SPGW-U for data plane network packets, and the SPGW-U operates on a virtual machine running in the cloud computing environment.

19. The method of claim 15, wherein the first data plane network packet and the second data plane network packet are received from a load balancer, wherein the load balancer directs network packets based on an Internet Protocol (IP) header and a User Datagram Protocol (UDP) or Transmission Control Protocol (TCP) port number rather than directing network packets based on encapsulated packet information.

20. The method of claim 15, further comprising: receiving a third data plane network packet associated with the second bearer; When it is determined that the second bearer is not in the local cache, retrieving bearer detailed information for the second bearer from the external key-value store, and storing the retrieved bearer detailed information for the second bearer in the local cache; registering the virtualized packet gateway as an owner of the second bearer in the external key-value store; as well as The third data plane network packet is processed based at least in part on the bearer details for the second bearer stored in the local cache.

Images