Methods, systems, and computer readable media for ingress message rate limitation across distributed producer network function (NF) applications using optimized token bucket algorithm

Abstract

A method of ingress message rate limiting across a distributed producer network function (NF) application using an optimized token bucket algorithm includes implementing a producer NF instance as a distributed producer NF application and implementing a distributed ingress gateway (IGW) for performing ingress message rate limiting on the distributed producer NF application. The method further includes maintaining, for each distributed IGW, a local token bucket for rate restrictions of ingress service-based interface (SBI) request messages received by each distributed IGW; and maintaining a distributed token bucket for refilling the local token bucket. The method further includes receiving an ingress SBI request message at the distributed IGW and consuming, by the distributed IGW, the token from the local token bucket to allow processing of the ingress SBI request message by the distributed producer NF application; and refilling the local token bucket with tokens from the distributed token bucket when the number of tokens in the local token bucket falls below a threshold level.

Description

[0001] Priority Claim

[0002] This application claims the priority benefit of U.S. Patent Application Serial No. 18 / 105,719, filed on February 3, 2023, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] The subject matter described herein relates to ingress message rate limiting at a producer NF. More particularly, the subject matter described herein relates to methods, systems, and computer-readable media for performing ingress message rate limiting across distributed producer NF applications using an optimized token bucket algorithm. Background Art

[0004] In a 5G telecommunications network, the network functions that provide services are referred to as producer NFs or NF service producers. The network functions that consume services are referred to as consumer NFs or NF service consumers. A network function can be a producer NF, a consumer NF, or both, depending on whether the network function consumes, produces, or both consumes and produces services. The terms "producer NF" and "NF service producer" are used interchangeably herein. Similarly, the terms "consumer NF" and "NF service consumer" are used interchangeably herein.

[0005] A given producer NF may have many service endpoints, where a service endpoint is a point of contact for one or more NF instances hosted by the producer NF. A service endpoint is identified by a combination of an Internet Protocol (IP) address and a port number or a fully qualified domain name (FQDN) that resolves to an IP address and a port number on the network node hosting the producer NF. An NF instance is an instance of a producer NF that provides a service. A given producer NF can include multiple NF instances. It should also be noted that multiple NF instances can share the same service endpoint. As used herein, a "distributed producer NF application" refers to a producer NF application implemented in separate clusters, containers, or virtual machines that implement producer NF instances.

[0006] An NF registers with a Network Function Repository Function (NRF). The NRF maintains a profile of NF instances that identifies the services supported by each NF instance. The profile of an NF instance is referred to as an NF profile in 3GPP TS29.510. An NF instance can obtain information about other NF instances that have registered with the NRF through an NF discovery service operation. According to the NF discovery service operation, a consumer NF sends an NF discovery request to the NRF. The NF discovery request includes query parameters that the NRF uses to locate the NF profile of a producer NF that can provide the service identified by the query parameters. An NF profile is a data structure that defines the type of service provided by an NF instance and information about the contact and capacity of the NF instance.

[0007] The Service Communication Proxy (SCP) can also invoke the NF discovery service operation to learn about the available producer NF instances. The situation where the SCP uses the NF discovery service operation to obtain information about the producer NF instances on behalf of the consumer NF is referred to as delegated discovery. The consumer NF connects to the SCP, and the SCP balances the traffic load among the producer NF service instances providing the required services or directly routes the traffic to the destination producer NF instance.

[0008] In addition to the SCP, another example of an intermediate proxy that forwards traffic between the producer and consumer NFs is the Security Edge Protection Proxy (SEPP). The SEPP is a network function used to protect the control plane traffic exchanged between different 5G Public Land Mobile Networks (PLMNs). Thereby, the SEPP performs message filtering, policing, and topology hiding on all Application Programming Interface (API) messages transmitted between PLMNs.

[0009] One problem in 5G and other types of networks is that existing token bucket algorithms are inefficient when performing ingress message rate limiting at distributed producer NF applications. For example, producer NF instances can be implemented as a collection of distributed producer NF applications using different units or partitions of virtual resources such as Kubernetes clusters, Docker containers, or virtual machines. To conserve producer NF resources, it is desirable to perform ingress message rate limiting at each distributed producer NF application. One mechanism for performing such ingress message rate limiting is to use a token bucket algorithm implemented through a distributed cache. When any of the multiple distributed producer NF applications receives an ingress message, the ingress gateway associated with that producer NF application accesses the token bucket implemented in the distributed cache, determines whether the token bucket implemented in the distributed cache indicates sufficient capacity to allow the message to be processed, and if sufficient capacity exists, forwards the message to the associated producer NF application for processing and updates the cache. Accessing the distributed cache for each received message is inefficient and increases the latency of processing ingress messages.

[0010] In view of these and other difficulties, there is a need for improved methods, systems, and computer-readable media for ingress message rate limiting across distributed producer NF applications. Summary of the Invention

[0011] A method for ingress message rate limiting across distributed producer network function (NF) applications using an optimized token bucket algorithm includes implementing producer NF instances as distributed producer NF applications and implementing a distributed ingress gateway (IGW) for performing ingress message rate limiting on the distributed producer NF applications. The method further includes maintaining a local token bucket for each distributed IGW for rate limiting ingress service-based interface (SBI) request messages received by each distributed IGW, and maintaining a distributed token bucket for refilling the local token buckets. The method further includes receiving an ingress SBI request message at the distributed IGW and consuming tokens from the local token bucket by the distributed IGW to allow processing of the ingress SBI request message by the distributed producer NF application and refilling the local token bucket with tokens from the distributed token bucket when the number of tokens in the local token bucket drops below a threshold level.

[0012] According to another aspect of the subject matter described herein, implementing producer NF instances as multiple distributed producer NF applications and implementing a distributed IGW includes implementing the distributed producer NF applications and the distributed IGW using separate virtualized computing resources.

[0013] According to another aspect of the subject matter described herein, implementing the distributed producer NF applications using separate virtualized computing resources includes implementing the distributed producer NF applications using separate clusters, docking station containers, or virtual machines.

[0014] According to another aspect of the subject matter described herein, maintaining a local token bucket for each distributed IGW includes initializing the local token bucket for each distributed IGW with a certain amount of tokens based on historical ingress SBI request message traffic at the distributed IGW.

[0015] According to another aspect of the subject matter described herein, maintaining a distributed token bucket includes implementing the distributed token bucket using a distributed cache accessible to each distributed IGW.

[0016] According to another aspect of the subject matter described herein, receiving an SBI request message and consuming tokens from the local token bucket includes, at a distributed IGW: receiving an ingress SBI request message; determining whether the local token bucket accessible to the distributed IGW has sufficient tokens to allow processing of the ingress SBI request message; and in response to determining that the local token bucket has sufficient tokens to allow processing of the ingress SBI request message, consuming at least one token from the local token bucket and forwarding the ingress SBI request message to a distributed producer NF application for processing of the ingress SBI request message.

[0017] According to another aspect of the subject matter described herein, consuming at least one token from the local token bucket includes consuming at least one token from the local token bucket without accessing the distributed token bucket.

[0018] According to another aspect of the subject matter described herein, refilling the local token bucket from the distributed token bucket includes: in response to determining that the local token bucket does not have sufficient tokens to allow processing of an ingress SBI request message, transmitting a request for tokens to refill the local token bucket to the distributed token bucket; determining whether the distributed token bucket has sufficient tokens to satisfy the request; and in response to determining that the distributed token bucket has sufficient tokens to satisfy the request, refilling the local token bucket with tokens from the distributed token bucket.

[0019] According to another aspect of the subject matter described herein, consuming tokens from the local token bucket to allow processing of an ingress SBI request message includes using the tokens in the local token bucket refilled from the distributed token bucket to allow processing of an ingress SBI request message received by one distributed producer NF instance.

[0020] According to another aspect of the subject matter described herein, a method for ingress message rate limiting includes discarding an SBI request message in response to determining that the distributed token bucket does not have sufficient tokens to satisfy the request.

[0021] A system for ingress message rate limiting across distributed producer network function (NF) applications using an optimized token bucket algorithm is provided. The system includes a producer NF instance that includes at least one processor and a memory. The system also includes a plurality of distributed producer NF applications and a plurality of ingress gateways (IGWs) implemented using the at least one processor, where each distributed IGW can access one of a plurality of local token buckets for rate limiting of ingress service-based interface (SBI) request messages received by each distributed IGW. The system also includes a distributed token bucket for refilling the local token buckets, and the distributed IGWs are configured to receive ingress SBI request messages, consume tokens from the local token buckets to allow processing of the ingress SBI request messages by the distributed producer NF applications, and refill the local token buckets with tokens from the distributed token bucket when the number of tokens in the local token buckets drops below a threshold level.

[0022] According to another aspect of the subject matter described herein, the producer NF includes a plurality of individual virtualized computing resources and the distributed producer NF applications and IGWs are implemented in the individual virtualized computing resources.

[0023] According to another aspect of the subject matter described herein, the individual virtualized computing resources include individual clusters, docking station containers, or virtual machines.

[0024] According to another aspect of the subject matter described herein, the local token bucket for each distributed IGW is initialized with a certain amount of tokens based on the historical ingress SBI request message traffic at the distributed producer NF instance.

[0025] According to another aspect of the subject matter described herein, the producer NF instance includes a distributed cache accessible to each distributed IGW and the distributed token bucket is implemented using the distributed cache.

[0026] According to another aspect of the subject matter described herein, one of the distributed IGWs is configured to: receive an ingress SBI request message; determine whether the local token bucket accessible to the one distributed IGW has sufficient tokens to permit processing of the ingress SBI request message; and in response to determining that the local token bucket has sufficient tokens to permit processing of the ingress SBI request message, consume at least one token from the local token bucket and forward the ingress SBI request message to one of the distributed producer NF applications for processing.

[0027] According to another aspect of the subject matter described herein, the one distributed IGW is configured to consume at least one token from the local token bucket without accessing the distributed token bucket.

[0028] According to another aspect of the subject matter described herein, the one distributed IGW is configured to refill the local token bucket from the distributed token bucket by: in response to determining that the local token bucket does not have sufficient tokens to permit processing of the ingress SBI request message, transmitting a request for tokens to refill the local token bucket to the distributed token bucket; determining whether the distributed token bucket has sufficient tokens to satisfy the request; and in response to determining that the distributed token bucket has sufficient tokens to satisfy the request, refilling the local token bucket with tokens from the distributed token bucket.

[0029] According to another aspect of the subject matter described herein, the one distributed IGW is configured to use the tokens in the local token bucket refilled from the distributed token bucket to permit processing of the ingress SBI request message received by one of the distributed producer NF instances.

[0030] According to another aspect of the subject matter described herein, a non-transitory computer-readable medium storing executable instructions is provided, the executable instructions controlling a computer to perform the following steps when executed by a processor of the computer. The steps include implementing a producer network function (NF) instance as a plurality of distributed producer NF applications. The steps further include implementing a plurality of distributed ingress gateways (IGWs) for performing ingress message rate limiting on the distributed producer NF applications. The steps further include maintaining a local token bucket for each distributed IGW for rate limiting of ingress service-based interface (SBI) request messages received by each distributed IGW. The steps further include maintaining a distributed token bucket for refilling the local token buckets. The steps further include receiving an ingress SBI request message at the distributed IGW and consuming tokens from the local token bucket by the distributed producer NF application to permit processing of the ingress SBI request message by the distributed producer NF application. The steps further include refilling each local token bucket with tokens from the distributed token bucket when the number of tokens in the local token bucket drops below a threshold level.

[0031] The subject matter described herein can be implemented in software in combination with hardware and / or firmware. For example, the subject matter described herein can be implemented as software executed by a processor. In one exemplary implementation, the subject matter described herein can be implemented using a non-transitory computer-readable medium storing computer-executable instructions that control a computer to perform steps when executed by a processor of the computer. Exemplary computer-readable media suitable for implementing the subject matter described herein include non-transitory computer-readable media such as disk memory devices, chip memory devices, programmable logic devices, and application specific integrated circuits. Additionally, the computer-readable media implementing the subject matter described herein can be located on a single device or computing platform or can be distributed across multiple devices or computing platforms. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Exemplary implementations of the subject matter described herein will now be explained with reference to the accompanying drawings, in which:

[0033] Figure 1 is a network diagram illustrating an exemplary 5G system network architecture;

[0034] Figure 2 is a flowchart illustrating a token bucket algorithm for message processing;

[0035] Figure 3 is a network diagram illustrating ingress message rate limiting using a distributed token bucket;

[0036] Figure 4 is a network diagram illustrating ingress message rate limiting using a distributed token bucket implemented using a distributed cache;

[0037] Figure 5 It is a flowchart illustrating the process of performing ingress message rate limiting using a distributed token bucket;

[0038] Figure 6 It is a network diagram illustrating the use of an optimized token bucket algorithm at the producer NF for ingress message rate limiting. This algorithm uses a local token bucket for each distributed producer NF application and a distributed token bucket to refill the local token bucket;

[0039] Figure 7 It is a flowchart illustrating the process performed by the producer NF application that uses a local token bucket and a distributed token bucket to allow processing of messages received by the producer NF application;

[0040] Figure 8 It is a block diagram of an exemplary architecture of a producer NF that uses a local token bucket and a distributed token bucket to allow processing of ingress messages;

[0041] Figure 9 It is a flowchart illustrating the overall process of performing ingress message rate limiting at a producer NF instance implemented as multiple distributed producer NF applications using an optimized token bucket algorithm. Detailed implementation

[0042] Figure 1 It is a block diagram of an exemplary 5G system network architecture. Figure 1 The architecture in it includes an NRF 100 and an SCP 101, which can be located in the same home public land mobile network (HPLMN). As described above, the NRF 100 can maintain profiles of available NF instances and the services they support, and allow consumer NFs or SCPs to subscribe to the registration of new / updated NF instances and be notified of the registration of these instances. The SCP 101 can also support service discovery and selection of NF instances. The SCP 101 can perform load balancing of the connections between consumer and producer NFs.

[0043] The NRF 100 is a repository of NF instance profiles. To communicate with a producer NF instance, a consumer NF or SCP must obtain the NF profile of the producer NF instance from the NRF 100. The NF profile is a JavaScript Object Notation (JSON) data structure defined in 3GPP TS29.510. The NF profile includes attributes indicating the type of service provided, the capacity of the NF instance, and information for contacting the NF instance.

[0044] In Figure 1Among them, any network function can be a consumer NF, a producer NF, or both, depending on whether they request services, provide services, or request and provide services. In the example shown, the NFs include a Policy Control Function (PCF) 102 that performs policy-related operations in the network, a Unified Data Management Function (UDM) 104 that manages user data, and an Application Function (AF) 106 that provides application services.

[0045] Figure 1 The NFs shown in the figure also include a Session Management Function (SMF) 108 that manages the session between the Access and Mobility Management Function (AMF) 110 and the PCF 102. The AMF 110 performs mobility management operations similar to those performed by the Mobility Management Entity (MME) in a 4G network. The Authentication Server Function (AUSF) 112 performs authentication services for user equipment (UE) (such as user equipment (UE) 114) seeking access to the network.

[0046] The Network Slice Selection Function (NSSF) 116 provides network slice services for devices seeking access to specific network capabilities and features associated with a network slice. The NSSF 116 provides an NSSelection service that allows NFs to request information about network slices, and an NSSAIReachability service that enables NFs to update and subscribe to receive updated notifications of Network Slice Selection Assistance Information (NSSAI) reachability information.

[0047] The Network Exposure Function (NEF) 118 provides an Application Programming Interface (API) for application functions seeking to obtain information about Internet of Things (IoT) devices and other UEs attached to the network. The NEF 118 performs a function similar to the Service Capability Exposure Function (SCEF) in a 4G network.

[0048] The Radio Access Network (RAN) 120 connects the user equipment (UE) 114 to the network via a wireless link. A gNB ( Figure 1 not shown in the figure) or other wireless access point can be used to access the Radio Access Network 120. The User Plane Function (UPF) 122 can support various proxy functions for user plane services. An example of such a proxy function is the Multipath Transmission Control Protocol (MPTCP) proxy function. The UPF 122 can also support a performance measurement function that the UE 114 can use to obtain network performance measurements. Figure 1 A Data Network (DN) 124 is also shown in the figure, through which the UE accesses data network services such as Internet services.

[0049] The SEPP 126 filters the incoming traffic from another PLMN and performs topology hiding on the traffic leaving the home PLMN. The SEPP 126 can communicate with the SEPP in an external PLMN that manages the security of the external PLMN. Thus, the traffic between NFs in different PLMNs can traverse two SEPP functions, one for the home PLMN and the other for the external PLMN. The Unified Data Repository (UDR) 128 stores the subscription data for the UE. The Binding Support Function (BSF) 130 manages the binding between the PDU session and the PCF.

[0050] As described above, one problem that can occur in 5G and other types of networks is the inefficiency of the ingress message rate limit across distributed producer NF applications. 5G NFs can use a Kubernetes cloud-native environment cluster to process ingress request messages in a distributed manner. One implementation of the token bucket algorithm for request message rate limiting requires the use of a distributed cache to share a distributed token bucket across the cluster. When using the distributed cache alone, tokens from the distributed token bucket shared using the distributed cache need to be consumed for each SBI request. This means that the distributed cache performs work for each request. The distributed cache processing requires additional resources (CPU, network bandwidth) and introduces latency in request processing. It is necessary to optimize the resources required by the distributed cache and reduce the latency in ingress message processing or the latency introduced by the distributed cache processing. The subject matter described in this document utilizes local token buckets that can be accessed by each of multiple ingress gateways associated with the distributed producer NF application, and a distributed token bucket for refilling the local token buckets. Using the local token buckets to control the rate of message processing at the distributed producer NF application reduces the number of token consumption requests for the distributed token bucket. Reducing the number of token consumption requests for the distributed token bucket reduces the resources required for distributed cache processing and reduces the latency introduced by the distributed cache.

[0051] Generally, the token bucket algorithm only allows the processing of messages when there are tokens in the bucket. The token bucket is filled at a configurable rate during token consumption or is filled asynchronously in response to usage. When a message is allowed to be processed by an entity that regulates its resources using the token bucket, tokens are removed from the bucket. Three configuration parameters are required:

[0052] Bucket capacity;

[0053] The duration that determines how often the bucket is refilled; and

[0054] The tokens (the quantity) that should be added to refill the bucket.

[0055] Tokens / duration defines the traffic rate. The bucket size can be defined as the capacity to handle bursty traffic.

[0056] Figure 2 is a flowchart showing the token bucket algorithm for controlling the rate of message processing. Refer to Figure 2 , the token bucket 200 includes a configured bucket size and a token refresh rate. When a message 202 arrives for processing, at step 204, it is determined whether the token bucket 200 has enough tokens to allow the processing of the message. If it is determined that the token bucket 200 has enough tokens to allow the processing of the message, then the control proceeds to step 206, where the message is processed and one or more tokens are removed from the token bucket 200. If it is determined that there are not enough tokens in the token bucket 200 to allow the processing of the message, then the control proceeds to step 208, where the message is discarded.

[0057] A simple token bucket algorithm (such as Figure 2 shown in) works well in a non-distributed environment. However, in a distributed environment, this algorithm becomes inefficient because it requires access to a distributed cache every time a message arrives at any distributed processing element sharing the token bucket.

[0058] Under 5G NF, ingress request message rate limiting is required to regulate access to 5G NF resources. Ingress request message rate limiting involves counting the number of ingress SBI request messages arriving at the 5G NF within a certain time period and determining whether that number exceeds the configured maximum number of messages allowed to be processed within that time period. Ingress request messages can be filtered and / or rate-limited based on different criteria (such as URI, NF consumer ID, etc.).

[0059] Figure 3 Illustrates two different types of rate limiting at the producer NF - consumer NF set ingress request message rate limiting and global request message rate limiting. Both of these types of rate limiting rely on a distributed cache to count the number of requests, which is very resource-consuming and introduces latency. Refer to Figure 3 , two different consumer NF sets 300 and 302 seek to send messages to the producer NF 304. The consumer NF set 300 includes consumer NF instances 306, 308, and 310. The consumer NF set 302 includes consumer NF instances 312, 314, and 316. The producer NF 304 includes distributed ingress gateways (IGWs) 318 and 320 and distributed producer NF applications 322 and 324 implemented in clusters 326, 328, 330, and 334. Each distributed IGW 318 and 320 can access a distributed cache 336 storing a distributed token bucket.

[0060] When the producer NF 304 receives a message from one of the consumer NF instances 306, 308, 310, 312, 314, and 316, the producer NF 304 accesses the distributed cache 336 to determine if there are sufficient tokens in the token bucket implemented for the set of consumer NFs and a separate global token bucket (both token buckets are implemented using the distributed cache 326) to allow processing of the message. If there are sufficient tokens to allow processing of the message, then the IGW that received the message forwards the message to the producer NF application 322 or 324 for processing. If there are not sufficient tokens to allow processing of the message, then the IGW that received the message discards the message.

[0061] Since the producer NF 304 is required to access the distributed cache 336 for each received message, latency and processing of ingress request messages are increased. Reducing the number of accesses to the distributed cache 336 is desired.

[0062] For ingress request message rate limiting, it is necessary to count the ingress request messages. In a distributed environment, a distributed cache is used to share the count of ingress request messages. Ingress request message rate limiting requires ingress request message counting across clusters, which requires the use of a distributed cache containing a distributed token bucket. Accessing / updating the distributed token bucket is required to reserve / receive tokens for each ingress SBI request, which has resource utilization and latency implications. Updating the number of tokens in the token bucket according to the token bucket algorithm is defined as token consumption. It is necessary to optimize resources and reduce latency by eliminating the need to consume distributed bucket tokens from the distributed cache for each received SBI request message.

[0063] Figure 4 is a network diagram illustrating ingress message rate limiting using a distributed token bucket implemented using a distributed cache. Refer to Figure 4 , the consumer NFs 400 and 402 are configured to send SBI request messages to the producer NF 406. The producer NF 406 includes a distributed producer NF application and an IGW implemented using clusters 408, 410, 412, and 414. In one example, the distributed producer NF application and the IGW are implemented in separate clusters. In an alternative embodiment, a single cluster can be used to implement the IGW and the distributed producer NF application.

[0064] The distributed token bucket 416 is shared by the distributed IGWs implemented in the clusters 408, 410, 412, and 414. When an SBI request message arrives at the producer NF 406, the IWG that receives the message accesses the distributed token bucket 416 and determines whether there are a sufficient number of tokens to allow processing of the message. If the distributed token bucket 416 contains a sufficient number of tokens, then one or more tokens are consumed and the IGW forwards the message to its associated distributed producer NF application, which processes the message. If the distributed token bucket 416 does not contain a sufficient number of tokens, then the message is discarded. Requiring access to the distributed cache for each SBI request message received by the producer NF application is inefficient and increases message latency.

[0065] Figure 5 is a flowchart illustrating ingress message rate limiting using a distributed token bucket. Refer to Figure 5 , a message 500 is received at one of the clusters sharing the distributed token bucket. In step 502, it is determined whether a token can be consumed from the distributed token bucket. If a token can be consumed, then control proceeds to step 504, where processing of the message is allowed. If a token cannot be consumed, then control proceeds to step 506, where the message is discarded. Accessing the distributed token bucket requires access to the distributed cache, which affects resource utilization and latency. There is a need to reduce resource utilization and latency.

[0066] The subject matter described herein also includes local token buckets associated with each of the multiple distributed producer NF applications in addition to the distributed token bucket used to refill the local token buckets and shared across the distributed IGWs. The scope of each local token bucket is limited to a cluster, a container, or a virtual machine, and the IGW accesses the local token bucket without distributed cache processing. Ingress message rate limiting is achieved entirely by consuming tokens from the local token bucket. When the local token bucket is empty or when the number of tokens in the local token bucket is below a threshold, the IGWs having access to the local token bucket transmit a request to refill the local token bucket with a predetermined number of tokens to the distributed token bucket. Thus, the filling of the local token bucket is demand-based rather than operator-configured. To reduce the number of accesses to the distributed token bucket, the request to refill tokens from the distributed token bucket can request more than one token. The filling of the distributed token bucket is operator-configured. Using the local token bucket reduces the load on the distributed token bucket and the distributed cache. The size of the local token bucket can be determined based on historical traffic patterns (such as the amount of traffic received by each local bucket during a configured time period). The size of the local token bucket can determine the number of tokens requested and consumed from the distributed token bucket for each refill request.

[0067] Figure 6It is a network diagram showing the use of an optimized token bucket algorithm at the producer NF for ingress message rate limiting. This algorithm uses local token buckets for each distributed producer NF application and a distributed token bucket to refill the local token buckets. Refer to Figure 6 , consumer NFs 600 and 602 can be configured to send messages to producer NF 604. Producer NF 604 includes distributed producer NF applications implemented using clusters 606, 608, 610, and 612. Each cluster includes a local token bucket 614, 616, 618, and 620. Producer NF 604 also includes a distributed token bucket 622 shared across clusters 606, 608, 610, and 612.

[0068] When a message arrives at producer NF 604, the receiving IGW determines whether there are enough tokens in its local token bucket to allow the associated distributed producer NF application to process the message. If there are enough tokens in the local token bucket, then the IGW forwards the message to the distributed producer NF application for processing and consumes one or more tokens from the local token bucket. If there are not enough tokens in the local token bucket, then the receiving IGW can request to refill or replenish tokens from the distributed token bucket 622. If the distributed token bucket 622 includes enough tokens to satisfy the request from the IGW, then the distributed token bucket 622 provides the requested number of tokens to the IGW, and the tokens are stored in the local token bucket. Then, the IGW forwards the message to the producer NF application for processing and consumes one or more tokens from its local token bucket. If there are not enough tokens in the distributed token bucket 622 to satisfy the request to refill the local token bucket, then the message received from the consumer NF is discarded.

[0069] Discarding the message received from the consumer NF when there are not enough tokens to satisfy the number of tokens required to allow processing of the message implements ingress message rate limiting at the producer NF. The number of tokens required to allow processing of the request can be a configuration parameter set by the network operator. Compared with the rate limiting method that uses only a distributed token bucket, using a local token bucket in combination with a distributed token bucket reduces the frequency and number of accesses to the distributed token bucket. If the distributed token bucket is accessed only when the number of tokens in the local token bucket is below the operator-configured threshold, then further savings in the processing resources of the producer NF are achieved because the distributed token bucket is accessed only when necessary.

[0070] In one example, the IGW may request and receive a predetermined number of tokens from the distributed token bucket. The number of tokens consumed from the distributed token bucket may be based on the traffic previously received by the IGW. The determination of how many tokens to consume from the distributed token bucket may be determined before consuming tokens from the local token bucket. Consuming tokens from the local token bucket by the IGW is a local operation for the cluster, container, or virtual machine in which the IGW executes and does not require invoking the distributed cache that implements the distributed token bucket, thereby reducing the number and frequency of calls to the distributed cache.

[0071] Figure 7 is a flowchart illustrating the processing performed by the IGW using the local token bucket and the distributed token bucket to allow the processing of messages received by the IGW. Refer Figure 7 , when message 700 is implemented Figure 6 and received by the distributed IGW of the token bucket architecture shown in, the distributed IGW performs step 702 to determine if there are available tokens in the local token bucket to allow the processing of the message. In this example, a single token is consumed to allow the processing of one SBI request message. However, any number of tokens may be used or consumed to allow the processing of the SBI request message, depending on the amount of available tokens and the desired ingress message rate limiting granularity. If the local token bucket has tokens, then control proceeds to step 704, where the tokens are consumed and the IGW forwards the message to the associated producer NF application that processes the message. When there are sufficient tokens in the local token bucket to allow the processing of the received SBI request message, the distributed token bucket is not accessed.

[0072] In step 702, if there are not enough tokens in the local token bucket, then control proceeds to step 706, where the IGW reads the value N of the tokens to be requested or consumed from the distributed token bucket (N is an integer set by the network operator). Then control proceeds to step 708, where the IGW determines if the distributed token bucket has the requested N tokens available for consumption. If the distributed token bucket does not have the requested N tokens, then control proceeds to step 710, where the message is dropped.

[0073] If instead the distributed token bucket has N available tokens, then control proceeds to step 712, where the distributed IGW refills its local token bucket with the N tokens consumed from the distributed token bucket. The number of tokens in the distributed token bucket is reduced by the amount N. Then control returns to step 702, where the distributed IGW determines if a single token from the local token bucket is to be consumed. Since the local token bucket has recently been refilled, there are sufficient tokens in the local token bucket to allow the processing of the message, and control proceeds to step 704, where the processing of the message is allowed.

[0074] Figure 8FIG. is a block diagram of an exemplary architecture of a producer NF that uses a local token bucket and a distributed token bucket to allow processing of ingress messages. Refer to Figure 8 , the producer NF 800 includes at least one processor 802 and a memory 804. The producer NF 800 also includes a plurality of distributed producer NF applications 806, 808, and 810 implemented in separate containers, clusters, or virtual machines 812, 814, and 816. "Separate" means that the containers, clusters, or virtual machines 812, 814, and 816 are implemented using different units or partitions of virtualized computing resources. The producer NF 800 also includes distributed IGWs 818, 820, and 822 implemented in separate IGW clusters 824, 826, and 828. Each distributed IGW 818, 820, and 822 can access a local token bucket 830, 832, or 834. The distributed IGWs 818, 820, and 822 share access to a distributed token bucket 836 implemented using a distributed cache 838.

[0075] The distributed IGWs 818, 820, and 822 perform Figure 7 the processes shown in determining whether there are sufficient tokens in the local token bucket and refilling the local token bucket from the distributed token bucket. The distributed producer NF applications 806, 808, and 810 together implement the functionality of a producer NF instance, which can be Figure 1 any NF instance type shown in, including SCP or SEPP. The distributed token bucket 836 can be replenished at a global refresh rate configured by a network operator based on the desired global message processing rate of the producer NF 800. The distributed producer NF applications 806, 808, and 810 and the distributed token bucket 836 can be implemented using computer-executable instructions stored in the memory 804 and executable by the processor 802.

[0076] Figure 9 FIG. is a flowchart of an overall process for performing ingress message rate limiting at a producer NF instance implemented as a plurality of distributed producer NF applications using an optimized token bucket algorithm. Refer to Figure 9 , in step 900, the process includes implementing the producer NF as a plurality of distributed producer NF applications. For example, a producer NF (such as the producer NF 800) can be implemented using a plurality of producer NF applications implemented in separate containers, clusters, or virtual machines, as shown in Figure 8 .

[0077] In step 902, the processing includes implementing multiple ingress gateways (IGWs) for performing ingress message rate limiting on the distributed producer NF applications. For example, the distributed IGWs 818, 820, and 822 can be components of the implementation of the producer NF instance 800 to perform the steps of ingress message rate limiting for the messages processed by the distributed producer NF applications 806, 808, and 810 described herein. In one example, the distributed IGWs 818, 820, and 822 can be implemented in a cluster separate from the distributed producer NF applications 806, 808, and 810. In an alternative example, each of the distributed IGWs 818, 820, and 822 can be implemented in the same cluster as its corresponding distributed producer NF application 806, 808, or 810.

[0078] In step 904, the processing includes maintaining a local token bucket for each distributed IGW for rate limiting the ingress SBI request messages received by that IGW. For example, each distributed IGW can have a local token bucket that stores tokens to be consumed when the ingress messages received by that distributed IGW are forwarded to the distributed producer NF application associated with that distributed IGW.

[0079] In step 906, the processing includes maintaining a distributed token bucket for refilling the local token buckets. For example, the distributed token bucket can be implemented using a distributed cache that is accessible by multiple distributed IGWs to refill their respective local token buckets. The tokens in the distributed token bucket can be shared equally or unequally among the distributed IGWs.

[0080] In step 908, the processing includes receiving an ingress SBI request message at the distributed IGW and consuming tokens from the local token bucket by the distributed IGW to allow the distributed producer NF application to process the ingress SBI request message. For example, each distributed IGW can consume tokens from its local token bucket before attempting to refill its local token bucket from the distributed or shared token bucket.

[0081] In step 910, the processing includes refilling the local token bucket with tokens from the distributed token bucket when the number of tokens in the local token bucket drops below a threshold level. For example, each distributed IGW can access the distributed token bucket when the number of tokens in its local token bucket is insufficient to allow the processing of one or more received SBI request messages. The threshold that triggers the refill event can be set to be higher than the amount of tokens required to allow the processing of a single SBI request message to allow the local token bucket to be refilled before it contains insufficient tokens to allow the processing of SBI request messages. The same or different thresholds can be set for the local token buckets to trigger the refill event.

[0082] Exemplary advantages of the subject matter described herein include reduced resource utilization of the producer NF and reduced latency in processing SBI request messages. In addition, the subject matter described herein can be implemented on any NF operating as a service producer, including an SCP or SEPP.

[0083] The disclosure of each of the following references is hereby incorporated by reference in its entirety.

[0084] References

[0085] 1. 3rd Generation Partnership Project; Technical Specification Group Core

[0086] Network and Terminals; 5G System; Network Function

[0087] Repository Services; Stage 3 (Release 18) 3GPP TS 29.510

[0088] V18.1.0 (2022-12)

[0089] 2. Token Bucket, https: / en.wikipedia.org / wiki / Token_bucket,

[0090] December 8, 2022

[0091] It should be understood that various details of the subject matter described herein can be changed without departing from the scope of the subject matter described herein. In addition, the foregoing description is for illustrative purposes only and not for purposes of limitation, as the subject matter described herein is defined by the claims set forth below.

Claims

1. A method for ingress message rate limiting across distributed producer Network Function (NF) applications using an optimized token bucket algorithm, the method comprising: Implementing a producer NF instance as multiple distributed producer NF applications; Implementing multiple distributed ingress gateways (IGWs) for performing ingress message rate limiting on the distributed producer NF applications; Maintaining a local token bucket for each distributed IGW for rate limiting ingress Service Based Interface (SBI) request messages received by each distributed IGW; Maintaining a distributed token bucket for refilling the local token buckets; Receiving an ingress SBI request message at the distributed IGW and consuming a token from the local token bucket by the distributed IGW to allow processing of the ingress SBI request message by the distributed producer NF application; And When the number of tokens in the local token bucket drops below a threshold level, refilling the local token bucket with tokens from the distributed token bucket.

2. The method according to claim 1, wherein implementing the producer NF instance as multiple distributed producer NF applications and implementing the distributed IGWs includes implementing the distributed producer NF applications and the distributed IGWs using separate virtualized computing resources.

3. The method according to claim 2, wherein implementing the distributed producer NF applications using separate virtualized computing resources includes implementing the distributed producer NF applications using separate clusters, docking station containers, or virtual machines.

4. The method according to any one of the preceding claims, wherein maintaining a local token bucket for each distributed IGW includes initializing the local token bucket for each distributed IGW with a quantity of tokens based on historical ingress SBI request message traffic at the distributed IGW.

5. The method according to any one of the preceding claims, wherein maintaining the distributed token bucket includes implementing the distributed token bucket using a distributed cache accessible to each distributed IGW.

6. The method according to any one of the preceding claims, wherein receiving the SBI request message and consuming a token from the local token bucket includes, at one distributed IGW: Receiving an ingress SBI request message; Determining whether the local token bucket accessible to the one distributed IGW has sufficient tokens to allow processing of the ingress SBI request message; And In response to determining that the local token bucket has sufficient tokens to allow processing of the ingress SBI request message, consuming at least one token from the local token bucket and forwarding the ingress SBI request message to one distributed producer NF application for processing of the ingress SBI request message.

7. The method according to claim 6, wherein consuming at least one token from the local token bucket includes consuming the at least one token from the local token bucket without accessing the distributed token bucket.

8. The method according to claim 6 or 7, wherein refilling the local token bucket from the distributed token bucket includes: In response to determining that the local token bucket does not have sufficient tokens to allow processing of the ingress SBI request message, transmitting a request for tokens for refilling the local token bucket to the distributed token bucket. Determine whether the distributed token bucket has sufficient tokens to satisfy the request; and In response to determining that the distributed token bucket has sufficient tokens to satisfy the request, refill the local token bucket with tokens from the distributed token bucket.

9. The method according to claim 8, wherein consuming tokens from the local token bucket to allow processing of the ingress SBI request message includes using the tokens in the local token bucket refilled from the distributed token bucket to allow processing of the ingress SBI request message received by the one distributed producer NF instance.

10. The method according to claim 8 or 9, including discarding the SBI request message in response to determining that the distributed token bucket does not have sufficient tokens to satisfy the request.

11. A system for ingress message rate limiting across distributed producer network function (NF) applications using an optimized token bucket algorithm, the system comprising: A producer NF instance, the producer NF instance including at least one processor and a memory; A plurality of distributed producer NF applications and a plurality of ingress gateways (IGWs) implemented using the at least one processor, each distributed IGW being able to access one of a plurality of local token buckets for rate limiting of ingress service-based interface (SBI) request messages received by each distributed IGW; and A distributed token bucket for refilling the local token bucket, wherein the distributed IGW is configured to receive an ingress SBI request message, consume tokens from the local token bucket to allow processing of the ingress SBI request message by the distributed producer NF application, and when the number of tokens in the local token bucket drops below a threshold level, refill the local token bucket with tokens from the distributed token bucket.

12. The system according to claim 11, wherein the producer NF includes a plurality of separate virtualized computing resources and the distributed producer NF applications and IGWs are implemented in separate virtualized computing resources.

13. The system according to claim 12, wherein the separate virtualized computing resources include separate clusters, docking station containers, or virtual machines.

14. The system according to any one of claims 11 to 13, wherein the local token bucket for each distributed IGW is initialized with a certain amount of tokens based on the historical ingress SBI request message traffic at the distributed IGW.

15. The system according to any one of claims 11 to 14, wherein the producer NF instance includes a distributed cache accessible to each distributed IGW and the distributed token bucket is implemented using the distributed cache.

16. The system according to any one of claims 11 to 15, wherein one of the distributed IGWs is configured to: Receive an ingress SBI request message; Determine whether the local token bucket accessible to the one distributed IGW has sufficient tokens to allow processing of the ingress SBI request message; and In response to determining that the local token bucket has sufficient tokens to allow processing of the ingress SBI request message, consume at least one token from the local token bucket and forward the ingress SBI request message to one of the distributed producer NF applications for processing.

17. The system according to claim 16, wherein the one distributed IGW is configured to consume the at least one token from the local token bucket without accessing the distributed token bucket.

18. The system according to claim 16 or 17, wherein the one distributed IGW is configured to refill the local token bucket from the distributed token bucket by: transmitting a request for tokens for refilling the local token bucket to the distributed token bucket in response to determining that the local token bucket does not have sufficient tokens to allow processing of an ingress SBI request message; determining whether the distributed token bucket has sufficient tokens to satisfy the request; and refilling the local token bucket with tokens from the distributed token bucket in response to determining that the distributed token bucket has sufficient tokens to satisfy the request.

19. The system according to claim 18, wherein the one distributed IGW is configured to use tokens from the local token bucket refilled from the distributed token bucket to allow processing of an ingress SBI request message received by the one distributed producer NF instance.

20. A non-transitory computer-readable medium having stored thereon executable instructions that, when executed by a processor of a computer, control the computer to perform steps including: implementing a producer network function (NF) instance as a plurality of distributed producer NF applications; implementing a plurality of distributed ingress gateways (IGWs) for performing ingress message rate limiting on the distributed producer NF applications; maintaining a local token bucket for each distributed IGW for rate limiting ingress service-based interface (SBI) request messages received by each distributed IGW; maintaining a distributed token bucket for refilling the local token buckets; receiving an ingress SBI request message at the distributed IGW and consuming tokens from the local token bucket by the distributed IGW to allow processing of the ingress SBI request message by the distributed producer NF applications; and refilling each of the local token buckets with tokens from the distributed token bucket when the number of tokens in the local token bucket drops below a threshold level.

Images