Method, system, and computer-readable medium for using an optimized token bucket algorithm for ingress message rate limiting across distributed producer network function (NF) applications

Abstract

A method for using an optimized token bucket algorithm for ingress message rate limiting across distributed producer network function (NF) applications includes implementing a producer NF instance as a distributed producer NF application and implementing multiple distributed ingress gateways (IGWs) to perform ingress message rate limiting for the distributed producer NF application. The method further includes, for each of the distributed IGWs, maintaining a local token bucket for rate limiting ingress service-based interface (SBI) request messages received by each of the distributed IGWs and maintaining a distributed token bucket for replenishing the local token bucket. The method further includes receiving an ingress SBI request message at the distributed IGW and consuming, by the distributed IGW, tokens from the local token bucket to enable processing of the ingress SBI request message by the distributed producer NF application, and replenishing the local token bucket with tokens from the distributed token bucket when the number of tokens in the local token bucket falls below a threshold level.

Description

[Technical Field]

[0001] Priority claims This application claims the benefit of priority to U.S. Patent Application No. 18 / 105,719, filed February 3, 2023, the entire disclosure of which is incorporated herein by reference.

[0002] Technical Field The subject matter described herein relates to ingress message rate limiting in producer NFs. More particularly, the subject matter described herein relates to methods, systems, and computer-readable media for using an optimized token bucket algorithm for ingress message rate limiting across distributed producer NF applications. [Background technology]

[0003] background In a 5G communication network, a network function that provides a service is called a producer NF or an NF service producer. A network function that consumes a service is called a consumer NF or an NF service consumer. A network function may be a producer NF, a consumer NF, or both, depending on whether the network function is consuming, generating, or consuming and generating the service. The terms "producer NF" and "NF service producer" are used interchangeably herein. Similarly, the terms "consumer NF" and "NF service consumer" are used interchangeably herein.

[0004] A producer NF may have many service endpoints, which are contact points for one or more NF instances hosted by the producer NF. A service endpoint is identified by a combination of an Internet Protocol (IP) address and port number on the network node hosting the producer NF, or a fully qualified domain name (FQDN) that resolves to an IP address and port number. An NF instance is an instance of a producer NF that provides a service. A producer NF may include two or more NF instances. It should also be noted that multiple NF instances can share the same service endpoint. As used herein, "distributed producer NF application" refers to a producer NF application implemented in separate pods, containers, or virtual machines that implement producer NF instances.

[0005] NFs register with a Network Capability Repository Function (NRF). The NRF maintains profiles of available NF instances that identify the services supported by each NF instance. The profile of an NF instance is called an NF profile in 3GPP TS 29.510. An NF instance can obtain information about other NF instances registered with the NRF through an NF discovery service operation. According to the NF discovery service operation, a consumer NF sends an NF discovery request to the NRF. The NF discovery request includes query parameters that the NRF uses to find NF profiles of producer NFs that can provide the service identified by the query parameters. An NF profile is a data structure that defines the types of services provided by the NF instance as well as contact and capability information about the NF instance.

[0006] The Service Communication Proxy (SCP) can also invoke the NF discovery service operation to learn about available producer NF instances. The case where the SCP uses the NF discovery service operation to obtain information about producer NF instances on behalf of a consumer NF is called delegated discovery. The consumer NF contacts the SCP, and the SCP load balances traffic among the producer NF service instances that offer the required service, or routes traffic directly to the destination producer NF instance.

[0007] In addition to the SCP, another example of an intermediate proxy that forwards traffic between producer and consumer NFs is the Security Edge Protection Proxy (SEPP). The SEPP is a network function used to protect the control plane traffic exchanged between different 5G public land mobile networks (PLMNs). To that end, the SEPP performs message filtering, policing, and topology hiding for all application programming interface (API) messages sent between PLMNs.

[0008] One problem in 5G and other types of networks is that existing token bucket algorithms may be insufficient for performing ingress message rate limiting in distributed producer NF applications. For example, a producer NF instance may be implemented as a collection of distributed producer NF applications that use different units or partitions of virtual resources, such as Kubernetes pods, Docker containers, or virtual machines. To conserve producer NF resources, it may be desirable to perform ingress message rate limiting in each of the distributed producer NF applications. One mechanism for performing such ingress message rate limiting is to use a token bucket algorithm implemented with a distributed cache. When any of multiple distributed producer NF applications receives an ingress message, an ingress gateway associated with the producer NF application accesses the token bucket implemented in the distributed cache, determines whether the token bucket implemented in the distributed cache indicates sufficient capacity to allow processing of the message, and, if sufficient capacity exists, forwards the message to the associated producer NF application for processing and updates the cache. Accessing the distributed cache for each received message is insufficient and increases latency in processing ingress messages.

[0009] In view of these and other drawbacks, there is a need for improved methods, systems, and computer-readable media for ingress message rate limiting across distributed producer NF applications. Summary of the Invention [Means for solving the problem]

[0010] overview A method for using an optimized token bucket algorithm for ingress message rate limiting across distributed producer network function (NF) applications includes implementing producer NF instances as distributed producer NF applications and implementing distributed ingress gateways (IGWs) to perform ingress message rate limiting for the distributed producer NF applications. The method further includes, for each of the distributed IGWs, maintaining a local token bucket for rate limiting ingress service-based interface (SBI) request messages received by each of the distributed IGWs and maintaining a distributed token bucket for refilling the local token bucket. The method further includes receiving an ingress SBI request message at the distributed IGWs and consuming, by the distributed IGW, tokens from the local token bucket to enable processing of the ingress SBI request message by the distributed producer NF application, and refilling the local token bucket with tokens from the distributed token bucket when the number of tokens in the local token bucket falls below a threshold level.

[0011] According to another aspect of the subject matter described herein, implementing the producer NF instances as a plurality of distributed producer NF applications and implementing the distributed IGW includes implementing the distributed producer NF applications and the distributed IGW using separate virtualized computing resources.

[0012] According to another aspect of the subject matter described herein, implementing the distributed producer NF application using separate virtualized computing resources includes implementing the distributed producer NF application using separate pods, Docker containers, or virtual machines.

[0013] According to another aspect of the subject matter described in this specification, maintaining a local token bucket for each of the distributed IGWs includes initializing the local token bucket of each of the distributed IGWs with a token amount based on past ingress SBI request message traffic at the distributed IGWs.

[0014] According to another aspect of the subject matter described herein, maintaining the distributed token bucket includes implementing the distributed token bucket using a distributed cache accessible by each of the distributed IGWs.

[0015] According to another aspect of the subject matter described herein, receiving an SBI request message and consuming tokens from a local token bucket includes, at one of the distributed IGWs, receiving an ingress SBI request message, determining whether a local token bucket accessible by the one distributed IGW has sufficient tokens to enable processing of the ingress SBI request message, and in response to determining that the local token bucket has sufficient tokens to enable processing of the ingress SBI request message, consuming at least one token from the local token bucket and forwarding the ingress SBI request message to one of the distributed producer NF applications for processing the ingress SBI request message.

[0016] According to another aspect of the subject matter described herein, consuming at least one token from the local token bucket includes consuming at least one token from the local token bucket without accessing the distributed token bucket.

[0017] According to another aspect of the subject matter described herein, refilling the local token bucket from the distributed token bucket includes communicating a request for tokens to refill the local token bucket to the distributed token bucket in response to determining that the local token bucket does not have enough tokens to enable processing of the ingress SBI request message; determining whether the distributed token bucket has enough tokens to satisfy the request; and refilling the local token bucket with tokens from the distributed token bucket in response to determining that the distributed token bucket has enough tokens to satisfy the request.

[0018] According to another aspect of the subject matter described herein, consuming tokens from the local token bucket to enable processing of the ingress SBI request message includes enabling processing of the ingress SBI request message received by one distributed producer NF instance (the one distributed IGW) using tokens in the local token bucket replenished from the distributed token bucket.

[0019] According to another aspect of the subject matter described herein, a method for ingress message rate limiting includes discarding an SBI request message in response to determining that a distributed token bucket does not have enough tokens to satisfy the request.

[0020] A system using an optimized token bucket algorithm for ingress message rate limiting across distributed producer network function (NF) applications is provided. The system includes a producer NF instance including at least one processor and a memory. The system further includes a plurality of distributed producer NF applications implemented using the at least one processor and a plurality of ingress gateways (IGWs), each having access to one of a plurality of local token buckets for rate limiting ingress service-based interface (SBI) request messages received by each of the distributed IGWs. The system further includes a distributed token bucket for replenishing the local token bucket, wherein the distributed IGW is configured to receive ingress SBI request messages, consume tokens from the local token bucket to enable processing of the ingress SBI request messages by the distributed producer NF applications, and refill the local token bucket with tokens from the distributed token bucket when the number of tokens in the local token bucket falls below a threshold level.

[0021] According to another aspect of the subject matter described herein, the producer NF includes a plurality of separate virtualized computing resources, and the distributed producer NF application and the IGW are implemented in the separate virtualized computing resources.

[0022] In accordance with another aspect of the subject matter described herein, the separate virtualized computing resources include separate pods, docker containers, or virtual machines.

[0023] According to another aspect of the subject matter described herein, the local token bucket of each of the distributed IGWs is initialized with a token amount based on past ingress SBI request message traffic at the distributed producer NF instance.

[0024] According to another aspect of the subject matter described herein, the producer NF instance includes a distributed cache accessible by each of the distributed IGWs, and the distributed token bucket is implemented using the distributed cache.

[0025] According to another aspect of the subject matter described herein, one of the distributed IGWs is configured to receive an ingress SBI request message, determine whether a local token bucket accessible by the one distributed IGW has sufficient tokens to enable processing of the ingress SBI request message, and, in response to determining that the local token bucket has sufficient tokens to enable processing of the ingress SBI request message, consume at least one token from the local token bucket and forward the ingress SBI request message to one of the distributed producer NF applications for processing.

[0026] According to another aspect of the subject matter described herein, one distributed IGW is configured to consume at least one token from a local token bucket without accessing a distributed token bucket.

[0027] According to another aspect of the subject matter described herein, one distributed IGW is configured to replenish the local token bucket from the distributed token bucket by: communicating a request for tokens to replenish the local token bucket to the distributed token bucket in response to determining that the local token bucket does not have enough tokens to enable processing of an ingress SBI request message; determining whether the distributed token bucket has enough tokens to satisfy the request; and replenishing the local token bucket with tokens from the distributed token bucket in response to determining that the distributed token bucket has enough tokens to satisfy the request.

[0028] According to another aspect of the subject matter described herein, one distributed IGW is configured to enable processing of an ingress SBI request message received by one distributed producer NF instance (the one distributed IGW) using tokens in a local token bucket replenished from the distributed token bucket.

[0029] According to another aspect of the subject matter described herein, a non-transitory computer-readable medium having stored thereon executable instructions that, when executed by a processor of a computer, control the computer to perform steps. The steps include implementing producer NF instances as a plurality of distributed producer Network Function (NF) applications. The steps further include implementing a plurality of distributed Ingress Gateways (IGWs) to perform ingress message rate limiting for the distributed producer NF applications. The steps further include maintaining, for each of the distributed IGWs, a local token bucket for rate limiting of ingress Service-Based Interface (SBI) request messages received by each of the distributed IGWs. The steps further include maintaining a distributed token bucket for refilling the local token bucket. The steps further include receiving an ingress SBI request message at the distributed IGW and consuming tokens from the local token bucket by the distributed producer NF application to enable processing of the ingress SBI request message by the distributed producer NF application. The steps further include refilling each of the local token buckets with tokens from the distributed token bucket when the number of tokens in the local token bucket falls below a threshold level.

[0030] The subject matter described herein can be implemented in software combined with hardware and / or firmware. For example, the subject matter described herein can be implemented in software executed by a processor. In one exemplary embodiment, the subject matter described herein can be implemented using a non-transitory computer-readable medium storing computer-executable instructions that, when executed by a computer's processor, control a computer to perform steps. Exemplary computer-readable media suitable for implementing the subject matter described herein include non-transitory computer-readable media such as disk memory devices, chip memory devices, programmable logic devices, and application-specific integrated circuits. In addition, computer-readable media implementing the subject matter described herein can be located on a single device or computing platform or distributed across multiple devices or computing platforms.

[0031] Exemplary embodiments of the subject matter described herein will now be described with reference to the accompanying drawings. [Brief explanation of the drawings]

[0032] [Figure 1] FIG. 1 is a network diagram illustrating an example 5G system network architecture. [Figure 2] 1 is a flowchart illustrating a token bucket algorithm for message processing. [Figure 3] FIG. 1 is a network diagram illustrating ingress message rate limiting using a distributed token bucket. [Figure 4] FIG. 1 is a network diagram illustrating ingress message rate limiting using a distributed token bucket implemented using a distributed cache. [Figure 5] 1 is a flowchart illustrating a process for ingress message rate limiting using a distributed token bucket. [Figure 6] FIG. 1 is a network diagram illustrating ingress message rate limiting at a producer NF using an optimized token bucket algorithm with a local token bucket for each distributed producer NF application and a distributed token bucket to replenish the local token buckets. [Figure 7] 10 is a flowchart illustrating a process performed by a producer NF application to enable processing of messages received by the producer NF application using local and distributed token buckets. [Figure 8] FIG. 10 is a block diagram illustrating an example architecture for a producer NF that enables processing of ingress messages using local and distributed token buckets. [Figure 9] 10 is a flowchart illustrating an overall process for performing ingress message rate limiting in producer NF instances implemented as multiple distributed producer NF applications using an optimized token bucket algorithm. DETAILED DESCRIPTION OF THE INVENTION

[0033] Detailed Description FIG. 1 is a block diagram illustrating an example 5G system network architecture. The architecture of FIG. 1 includes an NRF 100 and an SCP 101, which may be located within the same Home Public Land Mobile Network (HPLMN). As described above, the NRF 100 may maintain profiles of available NF instances and their supported services and enable consumer NFs or SCPs to subscribe to new / updated NF instances and be notified of their registration. The SCP 101 may also support service discovery and NF instance selection. The SCP 101 may perform load balancing of connections between consumer NFs and producer NFs.

[0034] The NRF 100 is a repository of profiles of NF instances. To communicate with a producer NF instance, a consumer NF or SCP must obtain the NF profile of the producer NF instance from the NRF 100. The NF profile may be a Java Script Object Notation (JSON) data structure defined in 3GPP TS 29.510. The NF profile contains attributes that indicate the type of service provided, the capacity of the NF instance, and information for contacting the NF instance.

[0035] 1, any of the network functions may be consumer NFs, producer NFs, or both, depending on whether they are requesting, providing, or both, a service. In the illustrated example, the NFs include a Policy Control Function (PCF) 102 that performs policy-related operations within the network, a Unified Data Manager (UDM) 104 that manages user data, and an Application Function (AF) 106 that provides application services.

[0036] 1 further includes an Access and Mobility Management Function (AMF) 110 and a Session Management Function (SMF) 108 that manages sessions between the PCF 102. The AMF 110 performs mobility management operations similar to those performed by a Mobility Management Entity (MME) in 4G networks. An Authentication Server Function (AUSF) 112 performs authentication services for user equipment (UE), such as user equipment (UE) 114, that seeks access to the network.

[0037] The Network Slice Selection Function (NSSF) 116 provides network slicing services to devices that seek to access specific network capabilities and characteristics associated with a network slice. The NSSF 116 provides the NSSelection service, which allows NFs to request information about network slices and NSSAI reachability services, which allows NFs to update and subscribe to receive communication of updates in Network Slice Selection Assistance Information (NSSAI) reachability information.

[0038] The Network Exposure Function (NEF) 118 provides an application programming interface (API) for application functions that want to obtain information about Internet of Things (loT) devices and other UEs attached to the network. The NEF 118 performs a function similar to the Service Capability Exposure Function (SCEF) in 4G networks.

[0039] The radio access network (RAN) 120 connects the user equipment (UE) 114 to the network via a wireless link. The radio access network 120 may be accessed using a gNB (not shown in FIG. 1) or other wireless access points. The user plane function (UPF) 122 may support various proxy functionalities for user plane services. One example of such proxy functionality is multipath transmission control protocol (MPTCP) proxy functionality. The UPF 122 may also support performance measurement functionality, which may be used by the UE 114 to obtain network performance measurements. Also shown in FIG. 1 is a data network (DN) 124 through which the UE accesses data network services, such as Internet services.

[0040] The SEPP 126 filters incoming traffic from another PLMN and performs topology hiding for traffic egressing from the home PLMN. The SEPP 126 may communicate with a SEPP in a foreign PLMN that manages security for the foreign PLMN. Thus, traffic between NFs in different PLMNs may traverse two SEPP functions, one for the home PLMN and the other for the foreign PLMN. The Unified Data Repository (UDR) 128 stores UE subscription data. The Binding Support Function (BSF) 130 manages binding between PDU sessions and PCFs.

[0041] As mentioned above, one issue that can arise in 5G and other types of networks is inefficiencies in ingress message rate limiting across distributed producer NF applications. 5G NFs can process ingress request messages in a distributed manner using Kubernetes cloud native environment pods. One implementation of a token bucket algorithm for request message rate limiting requires a distributed token bucket to be shared across pods using a distributed cache. When using only a distributed cache, every SBI request requires consuming tokens from a distributed token bucket shared with the distributed cache. This means that the distributed cache does the work for every request. Distributed cache processing requires additional resources (CPU, network bandwidth) and introduces latency in request processing. There is a need to optimize the resources required by the distributed cache to reduce latency in ingress message processing or the latency introduced by distributed cache processing. The subject matter described herein utilizes a local token bucket accessible by each of multiple ingress gateways associated with a distributed producer NF application and a distributed token bucket for refilling the local token bucket. Using a local token bucket to control the rate of message processing in a distributed producer NF application reduces the number of token consumption requests to the distributed token bucket, which reduces the resources required for distributed cache operations and reduces the latency experienced by the distributed cache.

[0042] In general, a token bucket algorithm allows a message to be processed only if there are tokens in the bucket. The token bucket is filled at a configurable rate during token consumption or asynchronously depending on usage. Tokens are removed from the bucket when a message is allowed to be processed by an entity whose resources are being coordinated using the token bucket. Three configuration parameters are required:

[0043] Bucket capacity, A duration to determine how often to refill the bucket, and The (number of) tokens that should be added to replenish the bucket.

[0044] The tokens / duration define the traffic rate. The bucket size can be defined as the capacity to handle traffic bursts.

[0045] 2 is a flowchart illustrating a token bucket algorithm for controlling the rate of message processing. Referring to FIG. 2, token bucket 200 includes a configured bucket size and token refresh rate. When a message 202 arrives for processing, it is determined in step 204 whether token bucket 200 has enough tokens to allow the message to be processed. If it is determined that token bucket 200 has enough tokens to allow the message to be processed, control proceeds to step 206, where the message is processed and one or more tokens are removed from token bucket 200. If it is determined that there are enough tokens in token bucket 200 to allow the message to be processed, control proceeds to step 208, where the message is discarded.

[0046] Simple token bucket algorithms such as that shown in Figure 2 work well in non-distributed environments. However, in distributed environments, such algorithms fall short because they require access to the distributed cache every time a message arrives at any of the distributed processing elements that share the token bucket.

[0047] Ingress request message rate limiting is required for 5G NFs to regulate access to 5G NF resources. Ingress request message rate limiting involves counting the number of ingress SBI request messages that arrive at a 5G NF within a certain period of time and determining whether this number exceeds a configured maximum number of messages allowed to be processed within that period. Ingress request messages can be filtered and / or rate limited based on various criteria, such as URI, NF consumer ID, etc.

[0048] FIG. 3 illustrates two different types of rate limiting at a producer NF: consumer NF set ingress request message rate limiting and global request message rate limiting. Both of these types of rate limiting rely on a distributed cache to count the number of requests, which is resource intensive and causes latency. Referring to FIG. 3, two different consumer NF sets 300 and 302 attempt to send messages to producer NF 304. Consumer NF set 300 includes consumer NF instances 306, 308, and 310. Consumer NF set 302 includes consumer NF instances 312, 314, and 316. Producer NF 304 includes distributed ingress gateways (IGWs) 318 and 320 and distributed producer NF applications 322 and 324, implemented in pods 326, 328, 330, and 334. Each distributed IGW 318 and 320 has access to a distributed cache 336 that stores distributed token buckets.

[0049] When producer NF 304 receives a message from one of consumer NF instances 306, 308, 310, 312, 314, and 316, producer NF 304 accesses distributed cache 336 to determine whether there are enough tokens to enable processing of the message in the token bucket implemented for the consumer NF set and a separate global token bucket, both implemented using distributed cache 326. If there are enough tokens to enable processing of the message, the IGW receiving the message forwards the message to producer NF application 322 or 324 for processing. If there are not enough tokens to enable processing of the message, the IGW receiving the message discards the message.

[0050] The producer NF 304 is required to access the distributed cache 336 for each received message, increasing latency and processing of ingress request messages. It is desirable to reduce the number of accesses to the distributed cache 336.

[0051] Ingress request message rate limiting requires counting ingress request messages. In a distributed environment, a distributed cache is used to share the counting of ingress request messages. Ingress request message rate limiting requires counting ingress request messages across pods, which requires the use of a distributed cache that includes a distributed token bucket. The distributed token bucket needs to be accessed / updated to reserve / receive tokens for every ingress SBI request, which has resource utilization and latency impacts. Updating the number of tokens in the token bucket is defined as token consumption by the token bucket algorithm. By eliminating the need to consume tokens from the distributed bucket in the distributed cache for every received SBI request message, resources should be optimized and latency reduced.

[0052]

[0013] Figure 4 is a network diagram illustrating ingress message rate limiting using a distributed token bucket implemented using a distributed cache. Referring to Figure 4, consumer NFs 400 and 402 are configured to send SBI request messages to producer NF 406. Producer NF 406 includes a distributed producer NF application and an IGW implemented using pods 408, 410, 412, and 414. In one example, the distributed producer NF application and the IGW are implemented in separate pods. In an alternative embodiment, a single pod can be used to implement the IGW and the distributed producer NF application.

[0053] The distributed token bucket 416 is shared by the distributed IGWs implemented in pods 408, 410, 412, and 414. When an SBI request message arrives at the producer NF 406, the IWG receiving the message accesses the distributed token bucket 416 to determine whether there are a sufficient number of tokens to allow the message to be processed. If the distributed token bucket 416 contains a sufficient number of tokens, the tokens are consumed and the IGW forwards the message to its associated distributed producer NF application, where the message is processed. If the distributed token bucket 416 does not contain a sufficient number of tokens, the message is discarded. The required access to the distributed cache for each SBI request message received by a producer NF application is inefficient and increases message latency.

[0054]

[0013] Figure 5 is a flowchart illustrating a process for ingress message rate limiting using a distributed token bucket. Referring to Figure 5, a message 500 is received at one of the pods sharing a distributed token bucket. In step 502, it is determined whether a token can be consumed from the distributed bucket. If a token can be consumed, control proceeds to step 504, where the message is allowed to process. If a token cannot be consumed, control proceeds to step 506, where the message is discarded. Accessing the distributed token bucket requires access to the distributed cache, which impacts resource utilization and latency. Resource utilization and latency need to be reduced.

[0055] The subject matter described herein includes a local token bucket associated with each of multiple distributed producer NF applications, in addition to a distributed token bucket used to replenish the local bucket and shared among the distributed IGWs. Each local token bucket is scoped to a pod, container, or VM, and access to the local bucket by the UGW does not require distributed cache processing. Ingress message rate limiting is achieved solely by consuming tokens from the local token bucket. When the local token bucket is empty or the amount of tokens in the local token bucket falls below a threshold level, the IGW with access to the local token bucket communicates a request to the distributed token bucket to replenish the local token bucket with a predetermined number of tokens. This allows the local token bucket to be filled based on demand, rather than operator configuration. To reduce the number of accesses to the distributed token bucket, a request to replenish tokens from the distributed token bucket can request more than one token. The distributed token bucket is filled based on operator configuration. The use of local token buckets reduces the load on the distributed token bucket and the distributed cache. The size of the local token buckets can be determined based on historical traffic patterns, such as the amount of traffic received by each local bucket during a set period of time, and can determine the number of tokens to request and consume from the distributed token bucket in each replenishment request.

[0056] 6 is a network diagram illustrating ingress message rate limiting at a producer NF using an optimized token bucket algorithm that uses a local token bucket for each distributed producer NF application and a distributed token bucket to replenish the local token buckets. Referring to FIG. 6, consumer NFs 600 and 602 can be configured to send messages to a producer NF 604. The producer NF 604 includes distributed producer NF applications implemented with pods 606, 608, 610, and 612. Each pod includes local token buckets 614, 616, 618, and 620. The producer NF 604 further includes a distributed token bucket 622 that is shared among the pods 606, 608, 610, and 612.

[0057] When a message arrives at the producer NF 604, the IGW receiving the message determines whether there are enough tokens in its local token bucket to enable processing of the message by its associated distributed producer NF application. If there are enough tokens in the local token bucket, the IGW forwards the message to the distributed producer NF application for processing and consumes one or more tokens from the local token bucket. If there are not enough tokens in the local token bucket to enable processing of the message, the IGW receiving the message can request token replenishment or refilling from the distributed token bucket 622. If the distributed token bucket 622 contains enough tokens to satisfy the request from the IGW, the distributed token bucket 622 provides the IGW with the requested number of tokens, which are stored in the local token bucket. The IGW then forwards the message to the producer NF application for processing and consumes one or more tokens from its local token bucket. If there are not enough tokens in the distributed token bucket 622 to satisfy the request to replenish the local token bucket, the message received from the consumer NF is discarded.

[0058] Ingress message rate limiting at a producer NF is achieved by discarding messages received from a consumer NF when there are not enough tokens to satisfy the amount of tokens required to enable processing of the message. The amount of tokens required to enable processing of a request may be a configuration parameter set by the network operator. Using a local token bucket in combination with a distributed token bucket reduces the frequency and number of accesses to the distributed token bucket compared to a rate limiting approach that uses a distributed token bucket alone. If the distributed token bucket is accessed only when the number of tokens in the local token bucket falls below an operator-configured threshold, processing resources at the producer NF are further conserved because the distributed token bucket is accessed only when necessary.

[0059] In one example, the IGW can request and receive a predetermined number of tokens from the distributed token bucket. The number of tokens to consume from the distributed token bucket can be based on previous traffic received by the IGW. The decision on how many tokens to consume from the distributed token bucket can be made in advance, prior to consuming tokens from the local token bucket. Consuming tokens from the local token bucket by the IGW is an operation local to the pod, container, or VM in which the IGW is running and does not require a call to the distributed cache that implements the distributed token bucket, thereby reducing the number and frequency of calls to the distributed cache.

[0060] FIG. 7 is a flowchart illustrating a process performed by an IGW to enable processing of a message received by the IGW using a local token bucket and a distributed token bucket. Referring to FIG. 7, when a message 700 is received by a distributed IGW implementing the token bucket architecture shown in FIG. 6, the distributed IGW performs step 702 to determine whether a token is available in the local token bucket to enable processing of the message. In this example, a single token is consumed to enable processing of one SBI request message. However, depending on the amount of available tokens and the desired granularity of ingress message rate limiting, any number of tokens can be used or consumed to enable processing of the SBI request message. If the local token bucket has tokens, control proceeds to step 704, where the tokens are consumed and the IGW forwards the message to the associated producer NF application, where the message is processed. If there are enough tokens in the local token bucket to enable processing of the received SBI request message, the distributed token bucket is not accessed.

[0061] If in step 702 there are not enough tokens in the local token bucket, control proceeds to step 706, where the IGW reads the value N of tokens to request or consume from the distributed token bucket (N is an integer set by the network operator). Control then proceeds to step 708, where the IGW determines whether the distributed token bucket has the requested N tokens to consume. If the distributed token bucket does not have the requested N tokens, control proceeds to step 710, where the message is discarded.

[0062] If instead the distributed token bucket has N tokens available, control proceeds to step 712, where the distributed IGW replenishes its local token bucket with N tokens consumed from the distributed token bucket. The number of tokens in the distributed token bucket is reduced by the number N. Control then proceeds back to step 702, where the distributed IGW determines whether to consume a single token from the local token bucket. Because the local token bucket was recently replenished, there are tokens available in the local token bucket to enable processing of the message, and control proceeds to step 704, where the message is enabled for processing.

[0063] FIG. 8 is a block diagram illustrating an example architecture for a producer NF that enables processing of ingress SBI request messages using a local token bucket and a distributed token bucket. Referring to FIG. 8, producer NF 800 includes at least one processor 802 and memory 804. Producer NF 800 further includes multiple distributed producer NF applications 806, 808, and 810 implemented in separate containers, pods, or virtual machines 812, 814, and 816. By "separate," we mean that the containers, pods, or virtual machines 812, 814, and 816 are implemented using separate units or partitions of virtualized computing resources. Producer NF 800 further includes multiple distributed IGWs 818, 820, and 822 implemented in separate IGW pods 824, 826, and 828. Each distributed IGW 818, 820, and 822 has access to a local token bucket 830, 832, or 834. Distributed IGWs 818 , 820 , and 822 share access to a distributed token bucket 836 , which is implemented using a distributed cache 838 .

[0064] The distributed IGWs 818, 820, and 822 perform the process shown in Figure 7 to determine whether enough tokens exist in the local token bucket and to replenish the local token bucket from the distributed token bucket. The distributed producer NF applications 806, 808, and 810 collectively implement the functionality of a producer NF instance, which may be any of the NF instance types shown in Figure 1, including an SCP or a SEPP. The distributed token bucket 836 may be replenished at a global refresh rate set by the network operator based on the desired global message processing rate of the producer NF 800. The distributed producer NF applications 806, 808, and 810 and the distributed token bucket 836 may be implemented using computer-executable instructions stored in memory 804 and executable by processor 802.

[0065] 9 is a flowchart illustrating an overall process for performing ingress message rate limiting at a producer NF implemented as a distributed producer NF application using an optimized token bucket algorithm. Referring to FIG. 9, at step 900, the process includes implementing the producer NF as multiple distributed producer NF applications. For example, a producer NF such as producer NF 800 may be implemented with multiple producer NF applications implemented in separate containers, pods, or virtual machines, as shown in FIG. 8.

[0066] In step 902, the process includes implementing multiple ingress gateways (IGWs) to perform ingress message rate limiting for the distributed producer NF applications. For example, distributed IGWs 818, 820, and 822 may be implemented as components of producer NF instance 800 to perform the steps described herein for ingress message rate limiting of messages processed by distributed producer NF applications 806, 808, and 810. In one example, the distributed IGWs 818, 820, and 822 may be implemented in a separate pod from the distributed producer NF applications 806, 808, and 810. In an alternative example, each distributed IGW 818, 820, and 822 may be implemented in the same pod as the respective distributed producer NF application 806, 808, or 810.

[0067] In step 904, the process includes maintaining, for each of the distributed IGWs, a local token bucket for rate limiting of ingress SBI request messages received by the IGW. For example, each distributed IGW may have a local token bucket that stores tokens consumed when an ingress message received by the distributed IGW is forwarded to a distributed producer NF application associated with the distributed IGW.

[0068] In step 906, the process includes maintaining a distributed token bucket for replenishing the local token bucket. For example, the distributed token bucket may be implemented using a distributed cache accessible by multiple distributed IGWs to replenish their respective local token buckets. The tokens in the distributed token bucket may be shared equally or unevenly across the distributed IGWs.

[0069] In step 908, the process includes receiving the ingress SBI request message at the distributed IGW and consuming tokens from a local token bucket by the distributed IGW to enable processing of the ingress SBI request message by the distributed producer NF application. For example, each distributed IGW may consume tokens from its local token bucket before attempting to replenish its local token bucket from the distributed or shared token bucket.

[0070] In step 910, the process includes refilling the local token bucket with tokens from the distributed token bucket when the number of tokens in the local token bucket falls below a threshold level. For example, each distributed IGW can access the distributed token bucket when the number of tokens in the local token bucket is insufficient to enable processing of one or more received SBI request messages. The threshold level for triggering a refill event can be set higher than the amount of tokens required to enable processing of a single SBI request message, thereby allowing the local token bucket to be refilled before it no longer contains enough tokens to enable processing of the SBI request message. The same or different thresholds can be set for the local token buckets to trigger a refill event.

[0071] Example advantages of the subject matter described herein include reduced resource utilization and reduced latency of a producer NF in processing SBI request messages. In addition, the subject matter described herein can be implemented in any NF acting as a service producer, including an SCP or SEPP.

[0072] The disclosure of each of the following references is incorporated herein by reference in its entirety. References 1.3 rdGeneration Partnership Project; Technical Specification Group Core Network and Terminals; 5G System; Network Function Repository Services; Stage 3 (Release 18) 3GPP TS 29.510 V18.1.0 (2022-12) 2. Token Bucket, https: / en.wikipedia.org / wiki / Token_bucket, December 8, 2022 It will be understood that various details of the subject matter described herein can be changed without departing from the scope of the subject matter described herein. Moreover, the above description is intended to be illustrative, not limiting, as the subject matter described herein is defined by the claims set forth below.

Claims

1. 1. A method of using an optimized token bucket algorithm for ingress message rate limiting across a distributed producer network function (NF) application, comprising: Implementing the producer NF instances as multiple distributed producer NF applications; implementing a plurality of distributed Ingress Gateways (IGWs) to perform ingress message rate limiting for the distributed producer NF applications; maintaining, for each of the distributed IGWs, a local token bucket for rate limiting of ingress Service Based Interface (SBI) request messages received by each of the distributed IGWs; maintaining a distributed token bucket for replenishing the local token bucket; receiving an ingress SBI request message at the distributed IGW and consuming tokens from the local token bucket by the distributed IGW to enable processing of the ingress SBI request message by the distributed producer NF application; replenish the local token bucket with tokens from the distributed token bucket when the number of tokens in the local token bucket falls below a threshold level; A method comprising:

2. 2. The method of claim 1 , wherein implementing the producer NF instances as a plurality of distributed producer NF applications and implementing the distributed IGW comprises implementing the distributed producer NF applications and the distributed IGW using separate virtualized computing resources.

3. 3. The method of claim 2, wherein implementing the distributed producer NF application using separate virtualized computing resources comprises implementing the distributed producer NF application using separate pods, docker containers, or virtual machines.

4. 10. The method of claim 9, wherein maintaining a local token bucket for each of the distributed IGWs comprises initializing the local token bucket of each of the distributed IGWs with a token amount based on past ingress SBI request message traffic at the distributed IGW.

5. 10. The method of any preceding claim, wherein maintaining the distributed token bucket comprises implementing the distributed token bucket using a distributed cache accessible by each of the distributed IGWs.

6. Receiving the SBI request message and consuming tokens from the local token bucket comprises, at one of the distributed IGWs: receiving an ingress SBI request message; determining whether the local token bucket accessible by the one distributed IGW has enough tokens to enable processing of the ingress SBI request message; In response to determining that the local token bucket has sufficient tokens to enable processing of the ingress SBI request message, consume at least one token from the local token bucket and forward the ingress SBI request message to one of the distributed producer NF applications for processing the ingress SBI request message; 10. A method according to any preceding claim, comprising:

7. 7. The method of claim 6, wherein consuming at least one token from the local token bucket comprises consuming the at least one token from the local token bucket without accessing the distributed token bucket.

8. Refilling the local token bucket from the distributed token bucket comprises: In response to determining that the local token bucket does not have sufficient tokens to enable processing of the ingress SBI request message, communicating to the distributed token bucket a request for tokens to replenish the local token bucket; determining whether the distributed token bucket has enough tokens to satisfy the request; replenish the local token bucket with tokens from the distributed token bucket in response to determining that the distributed token bucket has sufficient tokens to satisfy the request; The method of claim 6 or 7, comprising:

9. 9. The method of claim 8, wherein consuming tokens from the local token bucket to enable processing of the ingress SBI request message comprises using the tokens in the local token bucket replenished from the distributed token bucket to enable processing of the ingress SBI request message received by one distributed producer NF application (the one distributed IGW).

10. 10. The method of claim 8 or 9, comprising discarding the SBI request message in response to determining that the distributed token bucket does not have enough tokens to satisfy the request.

11. 1. A system using an optimized token bucket algorithm for ingress message rate limiting across distributed producer network function (NF) applications, comprising: a producer NF instance including at least one processor and memory; a plurality of distributed producer NF applications implemented using the at least one processor and a plurality of distributed ingress gateways (IGWs), each of the distributed IGWs having access to one of a plurality of local token buckets for rate limiting of ingress Service Based Interface (SBI) request messages received by each of the distributed IGWs; a distributed token bucket for replenishing the local token bucket, wherein the distributed IGW is configured to receive an ingress SBI request message, consume tokens from the local token bucket to enable processing of the ingress SBI request message by the distributed producer NF application, and replenish the local token bucket with tokens from the distributed token bucket when a number of tokens in the local token bucket falls below a threshold level.

12. 12. The system of claim 11, wherein the producer NF comprises a plurality of separate virtualized computing resources, and the distributed producer NF application and IGW are implemented on the separate virtualized computing resources.

13. The system of claim 12 , wherein the separate virtualized computing resources comprise separate pods, docker containers, or virtual machines.

14. The system of any one of claims 11 to 13, wherein the local token bucket of each of the distributed IGWs is initialized with a token amount based on past ingress SBI request message traffic at the distributed IGW.

15. The system of any one of claims 11 to 14, wherein the producer NF instance includes a distributed cache accessible by each of the distributed IGWs, and the distributed token bucket is implemented using the distributed cache.

16. One of the distributed IGWs is receiving an ingress SBI request message; determining whether the local token bucket accessible by the one distributed IGW has sufficient tokens to enable processing of the ingress SBI request message; in response to determining whether the local token bucket has sufficient tokens to enable processing of the ingress SBI request message, consume at least one token from the local token bucket and forward the ingress SBI request message to one of the distributed producer NF applications for processing. The system according to any one of claims 11 to 15, configured as follows:

17. 17. The system of claim 16, wherein the one distributed IGW is configured to consume the at least one token from the local token bucket without accessing the distributed token bucket.

18. The one distributed IGW is In response to determining that the local token bucket does not have sufficient tokens to enable processing of the ingress SBI request message, communicating to the distributed token bucket a request for tokens to replenish the local token bucket; determining whether the distributed token bucket has enough tokens to satisfy the request; replenish the local token bucket with tokens from the distributed token bucket in response to determining that the distributed token bucket has sufficient tokens to satisfy the request; 18. The system of claim 16 or 17, configured to replenish the local token bucket from the distributed token bucket by

19. 20. The system of claim 18, wherein the one distributed IGW is configured to enable processing of the ingress SBI request message received by one distributed producer NF instance (the one distributed IGW) using the tokens in the local token bucket replenished from the distributed token bucket.

20. When executed by a computer processor, Implementing producer Network Function (NF) instances as a plurality of distributed producer NF applications; implementing a plurality of distributed Ingress Gateways (IGWs) to perform ingress message rate limiting for the distributed producer NF applications; maintaining, for each of the distributed IGWs, a local token bucket for rate limiting of ingress Service Based Interface (SBI) request messages received by each of the distributed IGWs; maintaining a distributed token bucket for replenishing the local token bucket; receiving an ingress SBI request message at the distributed IGW and consuming tokens from the local token bucket by the distributed IGW to enable processing of the ingress SBI request message by the distributed producer NF application; refilling each of the local token buckets with tokens from the distributed token bucket when the number of tokens in the local token bucket falls below a threshold level; A non-transitory computer-readable medium having stored thereon executable instructions for controlling the computer to perform steps including:

Images