Packet handling in heterogeneous networks with different data plane network function implementations

Abstract

A Load Optimizer (LO) node (100) is configured to optimize a load associated with a given traffic flow by switching the data packets of the data flow between first and second instances of a Network Function (NF) (270, 290). Both the first and second instances of the NF perform the same functionality but are respectively implemented on different node types. Each received data packet comprises load balancing (LB) information that includes a LBKey (92) associating the data packet with one or more other data packets in a data flow and a Heavy Hitter (HH) indicator (94) that indicates whether the data packet is a HH data packet or a non- HH data packet. Then, based on the LBKey and the HH indicator, the LO node steers each data packet to one of a plurality of different implementations of the NF instance based on the values of the LBKey and the HH indicator fields carried by the data packet header. The first and second NF instances are implemented on respective first and second nodes types with the second node type being different from the first node type. Additionally, the LO node dynamically handles the migration of a data flow from an HH data flow to a non-HH data flow (and vice-versa) responsive to the changing of the HH indicator.

Description

[0001] PACKET HANDLING IN HETEROGENEOUS NETWORKS WITH DIFFERENT DATA PLANE NETWORK FUNCTION IMPLEMENTATIONS

[0002] TECHNICAL FIELD

[0003] This application relates generally to load optimization in communication networks, and particularly to load optimization of Heavy Hitter (HH) data flows between different Network Functions (NFs) implemented on different types of nodes.

[0004] BACKGROUND

[0005] Currently, packet core networks are experiencing a “softwarization” trend in which software, rather than traditional hardware circuitry, is used to address functionality issues in the network. This trend is driven by the increasing need to deliver new, customizable functionality to the users of the data network in a timely manner. This means, however, that future data networks will be more heterogeneous than currently-existing data networks. Thus, in such future data networks, different variants of certain Network Function (NF) instances may co-exist on the data plane (DP) to provide more customized performance for specific applications.

[0006] SUMMARY

[0007] The present disclosure provides a method and corresponding apparatus for forwarding data packets of a given data flow to one of two different implementations (i.e. , “node types”) of a NF instance on a site. The different NF implementations may be, for example, a software-based NF instance or a hardware-based NF instance. The forwarding is accomplished based on the characteristics of the data flow to which the data packets belong (i.e., by inspecting the headers of incoming data and one or more forwarding policies). The present disclosure also includes mechanisms for seamlessly migrating the state of the traffic (i.e., data flows) between the different implementations of the NF instances.

[0008] In more detail, embodiments of the present disclosure derive the identification and the characteristics of a given data flow from the headers of incoming data packets. Such may include information about the data flow, or any other traffic aggregate to which the data packet belongs (e.g., total traffic of an User Equipment (UE), application traffic, Quality of Service (QoS) data flow, TCP / IP flow, etc.), throughput demand (e.g., a heavy-hitter (HH) flag indicating that the data packet belongs to a HH data flow), or any other fields in the packet header that can be used to derive the HH state of the given data flow. In at least one embodiment, the information carried by the headers of the data packets may be generated by a node that is either upstream from, or downstream to, the apparatus performing the disclosed method.

[0009] As defined herein, a “HH data flow” is a large-volume data flow (e.g., a TCP connection or any other traffic aggregate) that is associated with, or is generated by, a HH. More particularly, HH data flows, which are often referred to by those of ordinary skill in the art as “elephant flows,” comprise a large volume of data (i.e., high volume traffic), and thus, comprise a large number of data packets. When compared to non-HH data flows (i.e., data flows that are not generated by HHs), HH data flows use more network resources.

[0010] Accordingly, in a first aspect, the present disclosure provides a method, implemented at a Load Optimizer (LO) node, for load optimization between first and second instances of a Network Function (NF). Each of the first and second instances of the NF perform the same functionality but are respectively implemented on different node types. In this embodiment, the method calls for the LO node to receive data packets. Each data packet comprises load balancing (LB) information that includes a LB key that associates the data packet with one or more other data packets in a data flow and a Heavy Hitter (HH) indicator that indicates whether the data packet is a HH data packet associated with a HH data flow, or a non-HH data packet associated with a non-HH data flow. So received, the method calls for the LO node to steer each data packet to one of the first and second instances of the NF based on the LB key and the HH indicator. The first instance of the NF is implemented on a first node of a first node type and the second instance of the NF is implemented on a second node of a second node type that is different from the first node type.

[0011] A second aspect of the present disclosure provides a network node for load optimization between first and second instances of a Network Function (NF). In this aspect, each of the first and second instances of the NF perform the same functionality but are respectively implemented on different node types (i.e., different implementations of the same NF). Particularly, the network node is configured to receive data packets. Each data packet comprises load balancing (LB) information that includes a LB key that associates the data packet with one or more other data packets in a data flow and a Heavy Hitter (HH) indicator that indicates whether the data packet is a HH data packet associated with a HH data flow, or a non- HH data packet associated with a non-HH data flow. So received, this aspect of the present disclosure steers each data packet to one of the first and second instances of the NF based on the LB key and the HH indicator. The first instance of the NF is implemented on a first node of a first node type and the second instance of the NF is implemented on a second node of a second node type that is different from the first node type.

[0012] A third aspect of the present disclosure provides a network node for load optimization between first and second instances of a Network Function (NF). Each of the first and second instances of the NF perform the same functionality but are implemented, respectively, on different node types (i.e., different implementations of the same NF). The network node comprises processing circuitry and memory circuitry. The memory is configured to store a control program that, when executed by the processing circuitry, configures the processing circuitry to receive data packets. Each data packet comprises load balancing (LB) information that includes a LB key associating the data packet with one or more other data packets in a data flow and a Heavy Hitter (HH) indicator that indicates whether the data packet is a HH data packet associated with a HH data flow, or a non-HH data packet associated with a non-HH data flow. The control program, when executed by the processing circuitry, further controls the processing circuitry 404 to steer each data packet to one of the first and second instances of the NF based on the LB key and the HH indicator. In one embodiment, the first instance of the NF is implemented on a first node of a first node type and the second instance of the NF is implemented on a second node of a second node type that is different from the first node type.

[0013] A fourth aspect of the present disclosure provides a computer program comprising instructions that, when executed on processing circuitry of a network node, causes the network node to receive data packets. Each data packet comprises load balancing (LB) information that includes a LB key that associates the data packet with one or more other data packets in a data flow and a Heavy Hitter (HH) indicator that indicates whether the data packet is a HH data packet associated with a HH data flow, or a non-HH data packet associated with a non-HH data flow. So received, the method calls for the LO node to steer each data packet to one of the first and second instances of the NF based on the LB key and the HH indicator. The first instance of the NF is implemented on a first node of a first node type and the second instance of the NF is implemented on a second node of a second node type that is different from the first node type.

[0014] In a fifth aspect of the present disclosure, the present embodiments provide a non- transitory computer-readable storage medium comprising a computer program stored thereon. The computer program comprises executable instructions that, when executed by processing circuitry of a network node, causes the network node to receive data packets. Each data packet comprises load balancing (LB) information that includes a LB key that associates the data packet with one or more other data packets in a data flow and a Heavy Hitter (HH) indicator that indicates whether the data packet is a HH data packet associated with a HH data flow, or a non- HH data packet associated with a non-HH data flow. So received, the method calls for the LO node to steer each data packet to one of the first and second instances of the NF based on the LB key and the HH indicator. The first instance of the NF is implemented on a first node of a first node type and the second instance of the NF is implemented on a second node of a second node type that is different from the first node type.

[0015] BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a block diagram illustrating the different types of NFs that may be implemented on a DP node according to embodiments of the present disclosure.

[0017] Figures 2A-2B are block diagrams illustrating a Load Optimizer (LO), some of its constituent functions, and its operation, according to embodiments of the present disclosure.

[0018] Figure 3 is a block diagram illustrating the LO functionality with tagged packets in both the upstream and downstream directions, according to embodiments of the present disclosure.

[0019] Figure 4 is a flow diagram illustrating Heavy Hitter (HH) offloading processing logic implemented by an LO, according to embodiments of the present disclosure. Figure 5A is a flow diagram illustrating a process performed by the LO for handling a change in a data flow from a non-HH state to a HH state, according to embodiments of the present disclosure.

[0020] Figure 5B is a flow diagram illustrating a process performed by the LO for handling a change in a data flow from a HH state to a non-HH state, according to embodiments of the present disclosure.

[0021] Figure 6 is a flow diagram illustrating the logic for transferring the dynamic state of data packets from a non-HH state in which a data packet is processed by an active non-HH NF instance, to a HH state in which the data packet is processed by its passive HH NF Instance, according to embodiments of the present disclosure.

[0022] Figure 7 is a flow diagram illustrating a method for performing LO according to embodiments of the present disclosure.

[0023] Figure 8 is a flow diagram illustrating a method for determining a current HH state of a data packet in a data flow according to embodiments of the present disclosure.

[0024] Figures 9A-9D are flow diagrams, implemented by an LO, for performing the processing logic of Figure 4 according to embodiments of the present disclosure.

[0025] Figure 10 is a flow diagram illustrating a method for processing State Migration (SM) packets according to embodiments of the present disclosure.

[0026] Figure 11 is a functional block diagram illustrating a network node configured to function as a LO according to embodiments of the present disclosure.

[0027] DETAILED DESCRIPTION

[0028] As stated above, different variants of certain Network Function (NF) instances will likely co-exist on the data plane (DP) of future data networks to provide users with a more customized performance for specific applications. For example, some of the NF instances will be implemented on future data networks as software functions on general-purpose hardware (e.g., one or more Central Processing Units (CPUs). Other NF instances, however, will be implemented on the future data networks using hardware (HW) circuitry (e.g., Data Processing Units (DPUs), so-called smart Network Interface Cards (smartNICs), Graphics Processing Units (GPUs), Application-Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs), etc.). In the context of the present disclosure, the NF instances that are implemented as software functions on one or more CPUs are referred to as “CPU-based” NF instances, while the NF instances that are implemented as hardware circuitry are referred to as “HW- accelerated” NF instances. Regardless of whether a given NF instance is a CPU-based NF instance or a HW-accelerated NF instance, however, each has its advantages and disadvantages.

[0029] For example, highly complex NF instances typically require dynamic programmability. Such NF instances are therefore best suited to be implemented as CPU-based NF instances. However, the CPUs that implement such CPU-based NF instances are generally considered “commodity hardware.” That is, the CPUs are not specifically designed for processing data packets, and as such, the latency of CPU-based NF instances can be largely unpredictable. On the other hand, executing NF instances using HW circuitry (i.e., HW-accelerated NF instances) can considerably accelerate the execution of specific DP functions. Thus, HW-accelerated NF instances are especially useful for latency critical applications, but they can also reduce energy consumption.

[0030] To handle the complexity of these heterogeneous DP targets (i.e., CPU-based and HW- accelerated NF instances), embodiments of the present disclosure provide new mechanisms, which are similar to the cloud, for making the DP “cloud-native.” That is, the mechanisms described herein provide communication networks, their operators, and end-users with higher flexibility, target / hardware independence, and transparent load optimization.

[0031] Typically, the deployment or partial deployment of a particular NF instance to a hardware block (i.e., HW circuitry) or software block (i.e., a CPU) is managed by the application itself. One example of such a NF instance is Open vSwitch (OVS), which uses a so-called “micro flow cache” (uFC) mechanism to speed up execution of data flows that have already been classified. Initially, the uFC was implemented as a separate table within (or accessible to) a main NF instance that preceded the main pipeline. The main NF instance includes a cache containing “Heavy Hitter” (HH) data flows to offload the main pipeline. Subsequently, the OVS community realized that such software blocks are rather easy to implement as hardware blocks. Therefore, the "smartNIC offload” became widely available in which certain NF instance functions are offloaded from a server processor (e.g., CPU), thereby freeing-up CPU cycles to drive application performance. Regardless, though, the contents of the uFC are still managed by the application.

[0032] Handling the deployment of CPU-based and HW-accelerated NF instances in heterogeneous packet-based networks, however, can be problematic. By way of example only, one issue that heterogeneous packet-based networks pose is that both CPU-based and HW- accelerated NF instances typically require custom control and traffic routing. Thus, to create and deploy a given NF instance, service developers must identify, address, and account for the nuances of the network infrastructure. This makes it difficult, however, for the developers to focus solely on the business logic of the NF instance. Nevertheless, with currently available limited accelerators, such difficulties were acceptable.

[0033] Programmable hardware devices (e.g., ASICs, FPGAs, smartNICs, etc.) then made it simpler to deploy the same functionality (i.e., different instances of the same NF) on different software and hardware targets. In such systems, the transparent selection of what particular data flows to process on which particular target (i.e., software or hardware) is beneficial. Particularly, it offers advantages that are similar to those provided by cloud-based applications (e.g., automatic scaling, failover, etc.). More particularly, multiple instances of the same packet processing functionality (i.e. , the same NF instance) are implemented and deployed. For example, HW-accelerated NF instances could be used to handle the so-called “Heavy-Hitter” (HH) users (i.e., users that generate a high volume of traffic). This is because the data flows associated with HH users may heavily load a CPU and can fully utilize the CPU cores. Therefore, processing HH data flows using CPU-based NF instances can degrade the throughput of CPU-based NF instances. However, HW- accelerated NF instances can only handle a limited number of users. Therefore, CPU-based NF instances are still needed to handle a potentially large number of users that generate a low to moderate amount of traffic.

[0034] For example, consider a situation with a peak traffic load of 1 .2Tbps generated by 12,000 users - 1 ,000 of which are HH users. Such situations can be efficiently handled by using a CPU-based NF instance having a capacity of 11 ,000 users and 0.2Tbps to handle users generating low-to-moderate amounts of traffic, and a HW-accelerated NF instance having a capacity of 1 ,000 users and 1Tbps (assuming that the users generating the HH traffic of 1Tbps are properly directed to the HW-accelerated NF instance. Note that if only the CPU-based or the HW-accelerated NF instances were used, handling the same traffic mix (i.e., 11 ,000 users / 0.2 Tbps and 1 ,000 users / 1Tbps) would require 6 CPU-based NF instances or 12 HW-accelerated NF instances, respectively).

[0035] However, in a network having a mixture of different DP implementations (i.e., CPUbased and HW-accelerated) of the same NF, new issues arise. For example, how to select a specific type of NF implementation (i.e., CPU-based or HW-accelerated) to handle a specific traffic flow. Further, because the characteristics of the same traffic flow may dynamically change (e.g., change from a non-HH data flow to a HH data flow, or vice versa), it should be possible to select the specific type of NF implementation for a given data flow, including the state transfer of the data flow between old and new nodes, even though such selection may require a remapping of the data flow traffic from one implementation of an NF instance (e.g., a CPU-based NF instance or a HW-accelerated NF instance) to another, different implementation of the NF instance (e.g., the other of a CPU-based or HW-accelerated NF instance). This latter aspect is particularly important considering the state migration of the data flow in HH scenarios. Specifically, because HH traffic is more or less continuously generated, network operators cannot rely on gaps in the HH traffic to perform the state migration.

[0036] Accordingly, embodiments of the present disclosure provide a mechanism for migrating the state of the data flows in networks having a mixture of CPU-based and HW-accelerated NF instance implementations. As used herein the term “data flow” is a generic term that refers to any group of data packets that are to receive the same treatment. For example, for a HH Mobile Broadband (MBB) user, the term “data flow” would refer to the data packets communicated by a given UE. In more detail, the present disclosure provides a method and corresponding apparatus for DP load optimization between two types of NFs. Both types of NFs have identical functionality; however, each is implemented differently. By way of example only, a first type of NF may be a CPU-based NF instance, while the second type of NF may be a HW-accelerated NF instance. Each implementation represents different sets of constraints (e.g., memory, processing, processing latency, etc.).

[0037] In some embodiments, the present disclosure provides a Data-Plane Load Optimizer (DP-LO) configured to determine, at run-time and based on packet header information, how different user traffic is mapped to the different types of NFs (e.g., nodes / workers) at a given site. More particularly, the DP-LO utilizes the packet header information to identify flow characteristics that may indicate that a given data flow is a HH data flow. In situations where the characteristics of the data flow changes (i.e., where the characteristics indicate a change in the data flow from being a non-HH data flow to a HH data flow, or vice versa), the DP-LO is configured to perform transparent and consistent state migration between the different types of nodes in the DP (e.g., the nodes executing CPU-based NF instances and the nodes executing the HW-accelerated NF instances).

[0038] A DP-LO configured according to embodiments of the present disclosure provides benefits and advantages that conventional mechanisms cannot or do not provide. For example, a DP-LO configured according to the present embodiments provides an optimized run-time solution for forwarding DP traffic (i.e., data flows) to different types of NFs even though the characteristics of that traffic will change dynamically from time-to-time. This enables the optimized use of the different types of DP nodes / workers at a site, as well as the usage of different types of NFs (i.e., CPU-based NF instances, HW-accelerated NF instances, etc.).

[0039] In another advantage, the DP-LO is configured according to the present embodiments to implement its functions in a stand-alone and transparent manner. That is, the NF instances themselves do not implement the DP-LO logic. Rather, the NF instances can focus on implementing their corresponding business logic in connection with executing a given service. Additionally, neither the Control Plane (CP) nor the Management Plane (MP) will be aware of the dynamic conditions that exist in the DP. Therefore, neither entity will participate in determining how to optimize traffic forwarding between the different types of NFs. This particular advantage is similar to those achieved with cloud-based applications (e.g., automatic scaling, failover, etc.).

[0040] As a result, a DP-LO configured according to the embodiments described herein improves the performance of specific traffic flows (e.g., data flows associated with latency sensitive applications). Further, due to it providing higher network node utilizations, a DP-LO configured according to the present embodiments can significantly increase energy efficiency.

[0041] Turning now to the drawings, Figure 1 is a block diagram illustrating the different types of NFs that may be implemented on a DP node 10 according to embodiments of the present disclosure. The DP node 10 may be, for example, one or more nodes in a in a packet core network and comprises one or more software-based design targets 20 (e.g., CPU-based NF instances) and / or one or more hardware-based design targets 30 (e.g., HW-accelerated NF instances). Each of the one or more software-based design targets 20 may be implemented on one or more CPUs 22. Additionally, each of the hardware-based design targets 30 may be implemented as hardware circuitry represented here as one or more FPGAs 32, one or more ASICs 34, and the like. As stated above, the different variants (i.e., the software-based design targets 20 and the hardware-based design targets 30) have different strengths and weaknesses. For example, while the software-based design targets 20 cannot perform as fast as their hardware-based design targets 30, they are much more easily programmed for different functions and features. And while the hardware-based design targets 30 are not easily programmable, they can perform their functions faster than the software-based design targets 20.

[0042] One application of the method described herein is the handling of HH data flows (i.e., data flows associated with applications, services, and / or users that generate a high volume of user data). As such, the details of the present embodiments are described in the context of HH and non-HH data flows. Those of ordinary skill in the art should readily appreciate, however, that this is merely for illustrative purposes and that the present disclosure is in no way limited simply to the handling of HH and non-HH data flows.

[0043] Additionally, those of ordinary skill in the art will realize that the DP-LO described herein may be implemented as a hardware circuit, a software function, or both. However, one embodiment of the present disclosure implements the DP-LO as a packet processing digital circuit that considers constrained hardware capabilities (e.g., a programmable packet processing ASIC like INTEL TOFINO).

[0044] Further, the present disclosure assumes that the data packets in a telecommunication network (traveling from a Radio Access Network (RAN) to the CORE Network (CN)) go through different NFs (i.e., different packet processing “middleboxes”). Each NF may have different capabilities and can be deployed on various hardware devices including, but not limited to, commodity servers (e.g., server computers running ebpf / xdp or dpdk-based software), FPGAs and / or ASIC-based smartNICs (e.g., XILINIX ALVEO, INTEL STRATIX 10, NETRONOME AGILIO, etc.), Infrastructure Processing Units (IPUs) (e.g., INTEL MOUNT EVANS), Data Processing Units (DPUs) (e.g., nVidia® Bluefield®), and switching ASICs (e.g., INTEL TOFINO).

[0045] The different NFs may also have various limitations and / or constraints. Such constraints may, for example, limit the number of UEs that are to be handled, as well as the throughput in the packet rate or bitrate. Other limitations and / or constraints may relate to memory constraints, processing constraints, and processing latency constraints. Nevertheless, each of the multiple instances of the same NF are required to handle the mix of user place traffic in a system with the requisite service quality regardless of whether the NF instance is implemented on the same or different target hardware.

[0046] As stated above, embodiments of the present disclosure are configured to identify HH users / traffic groups. The HH users / traffic groups have a much larger throughput demand than do non-HH users / groups, which typically generate only very light network activity. Thus, the demand of non-HH users / traffic groups / flows can be served by CPU-based NF instances, while the HH users / traffic groups / flows can be efficiently processed by HW-accelerated NF instances executing on constrained hardware (e.g., INTEL TOFINO with limited SRAM).

[0047] Additionally, the present embodiments assume that the different implementations of the NFs (e.g., the CPU-based and HW-accelerated NF instances) offer the same Control Plane (CP) Application Programming Interface (API), such as the well-known P4Runtime API, for example.

[0048] As stated above, one aspect of the present disclosure provides an automated data plane mechanism that separates the load optimization functionality for offloading the HH data flows from the data plane logic of the NF instances. This allows developers to create, test, and deploy their NF implementations without having to consider or address the complexities associated with balancing the load between HH and non-HH traffic.

[0049] Figures 2A-2B are block diagrams illustrating a Load Optimizer (LO) 40, some of its constituent functions, and its operation, according to embodiments of the present disclosure. More specifically, Figures 2A-2B illustrate LO 40 in which an NF control plane (i.e., NF-CP 50) “sees” a single logical data plane component (i.e., Logical NF-DP 60) to be controlled. According to the present embodiments, the NF-CP 50 is also unaware of the load optimization mechanism (i.e., LO-CP 62) implemented by the Logical NF-DP 60 (Figure 2B). In particular, the LO-DP 62 is configured to receive input data packets. So received, the LO-DP 62 identifies which of those data packets are associated with non-HH data flows and which are associated with HH data flows. Incoming data packets that are associated with non-HH data flows are then forwarded by the LO-DP 62 to a non-HH NF-DP 66 (e.g., a CPU-based NF instance), while incoming data packets that are associated with HH data flows are forwarded to a HH NF-DP 68 (e.g., a HW-accelerated NF instance).

[0050] Regardless of whether the data packets are associated with a HH data flow or a non-HH data flow, however, the LO-DP 62 and the NF instances 66, 68 communicate with the NF-CP 50 via a LO-CP 64 and the P4Runtime API. The LO-CP 64 hides the implementation details of the HH load offloading from the NF-CP 50 by providing a single view of a Logical NF-DP 60 instance. This includes hiding the implementation details of the LO-DP 62, and the non-HH and HH NF instances 66, 68. And, as will be described in more detail later, the embodiments disclosed herein can change the data flow states such that the data packets of a given stream can be appropriately forwarded between non-HH and HH NF instances 66, 68 transparently and in an on-demand fashion. Thus, the LO 40 is a middlebox that is configured for processing network packets and then forwarding them to one of the NF instances - i.e. , non-HH NF instance 66 (e.g., a CPUbased NF instance), or HH NF instance 68 (e.g., a HW-accelerated NF instance). In an advantageous embodiment, the packet processing functionality of LO 40 is implemented by programmable HW circuitry (e.g., INTEL TOFINO) such that the LO-DP 62 component implements the packet processing logic in a packet processing digital circuit, and the LO-CP 64 component executes on a general-purpose CPU. In the present embodiments, the LO-CP 64 is responsible for the read, write, configure, and handle processes associated with various objects (e.g., match-action tables, registers, counters, notifications, etc.) in the LO-DP 62. Additionally, the LO-CP 64 and the LO-DP 62 are preferably located such that they are physically close to each other. By way of example only, the LO-DP 62 may be implemented by a switch ASIC, while the LO-CP 64 executes on the switch CPU.

[0051] In the present embodiments, before being processed by LO 40, the network packets are tagged at some point in the network. The tags, which may be applied upstream or downstream from the LO 40, carry information about how the LO-DP 62 is to handle the incoming packet. For example, the information may indicate how traffic is grouped (e.g., per user, per application, per QoS class, etc.). Alternatively, or additionally, the information may indicate the activity of a given group of users or data flows (e.g., HH users or non-HH user, active user or passive user, etc.). Regardless of what the information carried by the tags indicates, however, appropriate tagging may or may not require the performance of certain techniques (e.g., HH detection) that are not germane to the present disclosure.

[0052] Figure 3 is a block diagram illustrating the LO functionality with tagged packets in both the upstream and downstream directions, according to embodiments of the present disclosure. As seen in Figure 3, the Radio Access Networks (RANs) 82, 84, 86 (referred to herein collectively as “RAN 80”), forward data packets to a HH detection node 90 that is positioned upstream from the LO 100. Similarly, a HH detection node 120 that is positioned downstream from LO 100 sends data packets it receives from an external network 130 to the LO 100. The Allocated Resources 110 comprise different DP NF implementations between which the LO 100 distributes the traffic.

[0053] In more detail, both the upstream and downstream HH detection nodes 90, 120, tag the data packets with two fields. The first field is a load balancing (LB) key 92 that is used by the LO 100 to identify packets that should be grouped together (e.g., the traffic of a particular UE, application, data flow, etc.). According to the present embodiments, packets with the same LBKey 92 are handled by the same DP NF instance. The second field is a HH-indicator 94 (e.g., a flag) that indicates whether the data packet is or is not associated with a HH (e.g., a user or UE or service application that generates a higher volume of traffic than do other users, UEs, or service applications). According to the present disclosure, the HH-indicator 94 is only set in cases where the intensity of the traffic of the group identified by LBKey 92 is larger than the intensity of most of the other traffic groups (i.e. , users, UEs, applications, data flows, etc.). Regardless, the LO 100 is configured to distribute traffic between the different DP NF implementations (i.e., Allocated Resources 110) based on the values in the LBKey 92 and HH- indicator 94 fields.

[0054] However, the HH state (i.e., as being either HH traffic or non-HH traffic) is a characteristic of the traffic and can dynamically change over time. For example, a user (e.g., a UE) engaging in some web browsing is considered to be a non-HH. Thus, the traffic produced by that user is tagged as a non-HH (or alternatively, not tagged as a HH). However, if that user begins playing a cloud-rendered game with 4K video, the traffic is considered to be, and is marked as, a HH. After finishing the game, the volume traffic generated by the user returns to normal levels. When this happens, the data packets generated by the user will no longer be tagged by the upstream / downstream nodes 90, 120 with information indicating the data packet as being a HH.

[0055] This embodiment of the present disclosure specifically details the LBKey 92 and the HH indicator 94 as being the two fields with which the data packet delivered to LO 100 are tagged. However, those of ordinary skill in the art should readily appreciate that this is merely for illustrative purposes. According to the present embodiments, other fields not specifically shown herein may be included as well, depending on the applied load optimization methodology. Additionally, embodiments of the present disclosure are also configured to selectively apply the HH offloading techniques described herein to data flows with specific QoS classes. Such may require, for example, the handling of throughput-hungry data flows while still achieving a good overall QoS.

[0056] Figure 4 is a flow diagram illustrating a method 140 for HH offloading processing logic according to embodiments of the present disclosure. It should be noted here that method 140 illustrates four different situations and their corresponding paths; however, each of those paths share some commonalities.

[0057] For example, in this embodiment, method 140 is performed by LO 100, and more specifically, the LO-DP 62, which is implemented as a processing digital circuit (i.e., hardware) that comprises:

[0058] • A Packet Parser configured to parse incoming tagged data packets to retrieve the LBKey 92 and HH indicator 94 fields;

[0059] • An Exact Match-Action Lookup Table (i.e., a hash table stored in SRAM). In this embodiment, the hash table is referred to as “StoredHHState.” The lookup will return “TRUE” if the value of the parsed LBKey 92 is in the table and if the traffic group of LBKey 92 is HH. However, if LBKey 92 is not in the table, the lookup will return “FALSE” indicating that the traffic group of LBKey 92 is non-HH;

[0060] • A Register Array indicated by “REG” in Figure 4. In this embodiment, REG is configured to store W values and can be indexed from 0 to N-1. As will be described in more detail below, REG is responsible for tracking whether a notification indicating a change in the HH state of a data flow (i.e., from HH to non-HH or vice versa) was sent to LO-CP 641; and

[0061] • GetHHNode, which is another exact match-action lookup table (i.e., another hash table in SRAM). In this embodiment, GetHHNode maps the LBKeys of HH packet groups to a routable location identifier (e.g., MAC, IPv4, IPv6 addresses, MPLS label, etc.), a physical or a virtual port, or a next table.

[0062] Additionally, the functions implemented in method 140 apply to each of the four different paths. For example, as seen in Figure 4, LO-DP 62 receives and parses an incoming data packet that has been tagged to include LBKey 92 and HH indicator 94. Once parsed, LO-DP 62 determines the current HH state of the data packet by inspecting the HH indicator 94 (box 142). The tagged data packet may be a HH data packet that is part of a HH data flow, or a non-HH data packet that is not part of a HH data flow (e.g., that is part of a low or medium hitter data flow).

[0063] Regardless of whether the tagged data packet is or is not an HH data packet, however, LO-DP 62 applies an exact match lookup for LBKey 92 on StoredHHState (boxes 144, 146). This enables LO-DP 62 to determine the previously known HH state of the previously received data packet in the same data flow. Particularly, if the current HH state of the tagged data packet is different than the previously known HH state of the data flow stored in StoredHHState, it indicates that a HH data flow has changed from being a HH data flow to being a non-HH data flow (or vice versa).

[0064] As previously stated, method 140 of Figure 4 illustrates four different situations or cases - each with a corresponding path through the flowchart. The first situation occurs when, as indicated by HH indicator 94, the current HH state of the tagged incoming packet is determined to be a HH packet (box 142) and matches the previously known HH state stored in StoredHHState (box 144). In this case, there is no need to notify LO-CP of a change in the HH state of the data flow because there has been no change in the HH state of the data flow. Therefore, LO-DP 62, which is a component of LO 100, applies the GetHHNode table to determine the location / address of the HH node (i.e., the HW-accelerated NF instance 68 or the node on which HW-accelerated NF instance 68 executes) dedicated for LBKey 92 (box 148) and forwards the data packet to that HH node (box 180).

[0065] The second situation occurs when, as indicated by HH indicator 94, the current HH state of the tagged incoming packet is determined to be a HH packet (box 142) but does not match the previously known HH state stored in StoredHHState (box 144). This indicates a change in the HH state of the data flow from being non-HH to being HH. In this case, LO-DP 62 may notify the LO-CP 64 of the change in the HH state of the data flow, but only if the LO-CP 64 has not already been notified. Thus, LO-DP 62 will update REG to reflect the change in the HH state. Particularly, LO-DP 62 will first read the key value (SKEY) stored in REG to determine if LBKey 92 is in REG (box 150). To read the value SKEY, LO-DP 62 uses LBKey 92 as an index.

[0066] SKEY = REG[hash[LBkey]]

[0067] If the value SKEY matches the value of LBKey 92 (box 152), LO-CP 64 has already been notified of the change in the HH state of the data flow from the non-HH state to the HH state. Therefore, LO-DP 62 will forward the data packet to a non-HH NF-DP instance 66 or to the node on which the non-HH NF-DP instance 66 executes (box 170). In one embodiment, as previously described, the non-HH NF-DP instance 66 is a CPU-based NF instance. However, if the value SKEY does not match the value of LBKey 92 (box 152), then the LO-CP 64 has not yet been notified of the change in the HH state of the data flow. Accordingly, LO-DP 62 stores the value of LBKey 92 in REG (box 154) and notifies the LO-CP 64 that the HH state of the data flow has changed from being non-HH to being HH (box 158) before forwarding the data packet to the non-HH NF-DP instance 66 (box 170).

[0068] It should be noted here that even though the data flow associated with the received data packet has changed its HH state (e.g., from non-HH to HH as in this embodiment, or from HH to non-HH as in other embodiments), the data packet is still forwarded, at least initially, to the non- HH NF-DP instance 66 (box 170). This is because the new node (in this case, the HH NF-DP instance 68) will need time to receive the states corresponding to the data flow that determine how to process the data packets associated with that data flow. Therefore, the state transfer of the data flow from the old node is performed before the traffic is redirected. This provides the time delay needed by the new node to determine how to process the incoming data packets associated with the data flow with the newly-changed HH state.

[0069] The third situation occurs when, as indicated by the HH indicator 94, the current HH state of the tagged incoming packet is determined to be a non-HH packet (box 142) and matches the previously known HH state stored in StoredHHState (box 146). As above, this indicates a change in the HH state of the data flow, and more specifically, a change from the HH state to the non-HH state. Therefore, LO-DP 62 will notify LO-CP 64 of the change in the HH state, if needed, but will also update REG to reflect the change in the HH state.

[0070] Specifically, LO-DP 62 will first read the key value (SKEY) stored in REG to determine if LBKey 92 is in REG (box 160). To read the value SKEY, LO-DP 62 uses LBKey 92 as an index.

[0071] SKEY = REG[hash[LBkey]]

[0072] If the value SKEY matches the value of LBKey 92 (box 162), there is no reason to notify LO-CP 64 as LO-CP 64 is already aware of the change in the HH state. LO-DP 62 will therefore simply forward the data packet to the appropriate HH node (box 180). If the value SKEY does not match the value of LBKey 92 (box 162), then the LO-CP 64 has not yet been notified of the change in the HH state. In these situations, LO-DP 62 stores the value of LBKey 92 in REG (box 164) to indicate that the LO-CP 64 has been notified of the change in the HH state of the data flow, notifies the LO-CP 64 of the change in the HH state (box 166), and forwards the data packet to the appropriate HH node (box 180).

[0073] As above, there is a time delay that is needed, at least initially, between notifying the LO- CP 64 of the detected change in the HH state of the data flow from HH to non-HH and redirecting received data packets having the same LBKey 92 to the non-HH node handling the non-HH data flow associated with the data packets. This time delay provides the time needed by the new node to receive the state information related to how to process the incoming data packets associated with the data flow with the newly-changed HH state.

[0074] The fourth situation occurs when, as indicated by HH indicator 94, the current HH state of the tagged incoming packet is determined to be a non-HH packet (box 142), and does not match the previously known HH state stored in StoredHHState (box 146). In these cases, LOOP 62 simply forwards the data packet to the appropriate HH node (box 180).

[0075] As stated above, the LO-DP 62 notifies LO-CP 64 about the changes it detects in the HH-state of a data flow associated with the LBKey 92 (i.e., all data packets that have the same LBKey 92). This notification allows the LO-CP 64 to change the HH states between non-HH and HH instances, as well as prepare the data plane tables in LO-DP 62.

[0076] Figure 5A is a flow diagram illustrating a method 190 performed by the LO 100, and more specifically, the LO-CP 64, for handling the changes in the HH state of a data flow from a non-HH state to a HH state, according to embodiments of the present disclosure. As seen in Figure 5A, the LO-DP 62 first sends a message or signal to the LO-CP 64 using the P4Runtime API (box 192). The message or signal notifies the LO-CP 64 that the data flow associated with LBKey 92 has changed from non-HH to HH, and thus, functions as a trigger to begin the reconfigurations. The LO-CP 64 then moves the HH state from non-HH to HH for LBKey 92 (box 202). In this embodiment, for example, the states can either be read from the data plane instance of the non-HH NF instance (box 212) or populated by the LO-CP 64 using information from its local knowledge base (e.g., a database). If, however, the data plane is already configured to handle HH data packets associated with LBKey 92 (box 210), the LO-CP 64 activates the HH path in the LO-DP 62 by first adding an entry (box 204) into the database (e.g., GETHHNODE box 196) for LBKey 92 identifying the location of the particular HH node dedicated to processing the HH data packets associated with LBKey 92. The LO-CP 64 then adds an entry for LBKey 92 (box 206) in the StoredHHState table (box 198) before removing LBKey 92 from REG (box 200) at index HASH[LBKEY] (box 208).

[0077] It should be noted here that in at least one embodiment of the present disclosure, the old non-HH NF instance in the data plane can be suspended or removed, depending on the system.

[0078] Figure 5B is a flow diagram illustrating a method 220 performed by the LO 100, and more particularly, the LO-CP 64, for handling a change in the HH state of a data flow associated with LBKey 92 from a HH state to a non-HH state, according to embodiments of the present disclosure. Similarly to method 190 in Figure 5A, the LO-DP 62 notifies the LO-CP 64 that the HH state of the data flow associated with LBKey 92 is now non-HH (box 224). After obtaining the notification, the LO-CP 64 moves the HH state of the data flow from HH NF-DP instance (box 232) to whichever non-HH NF-DP instance (box 242) is configured to handle data packets having LBKey 92. As above, the HH state can either be obtained from the HH NF-DP Instance (box 244) executing in the data plane or be populated by the LO-CP 64 from information contained in its local knowledge base (e.g., database). It should be noted here that this step of method 220 may require the LO-CP 64 and / or the LO_DP 62 to perform additional actions. Such actions may include, but are not limited to, reviving or launching the non-HH NF-DP instance for LBKey 92.

[0079] Once the HH states on the non-HH path are configured, the non-HH path in LO-DP 62 is activated. More particularly, the LO-CP 64 removes (box 234) LBKey 92 from the StoredHHState table (box 224) and removes the entries for LBKey 92 from REG at index HASH[LBKEY] (boxes 236, 228). Next, LO-CP 64 removes (box 238) the entries for LBKey 92 from the HH NF-DP tables (e.g., GetHHNode) (box 230). In so doing, the LO-DP 62 will not be able to apply GetHHNode to determine the location of the non-HH Instance configured to process the data packets having LBKey 92. Finally, the LO-CP 64 removes the entries for LBKey 92 from the data plane instance of HH NF Instance (box 240).

[0080] It should be noted here that the present embodiments also provide functionalities that extend the LO-based HH offloading technique described herein. For example, in some situations, stateful data plane objects may require state aggregation for the LO-CP 64. In these cases, the present embodiments may use the same counter instances on different HH and / or non-HH targets that is summed by the LO-CP 64. Additionally, or alternatively, registered NF instances could be synched between the HH and non-HH NF instances.

[0081] As previously stated, the LO 100 of the present embodiments is configured to consistently manage the state migration of a flow of data packets between HH and non-HH NF instances (or vice versa) in cases where the packet processing digital circuit (i.e. , the LO-DP 62 part of LO 100) comprises only static state information stored in match-action tables (e.g., hash tables in SRAM or ternary / lpm tables in Ternary Content Addressable Memory (TCAM)). However, the state migration is also important (e.g., for the sake of consistency) in situations where the LO-DP 62 has dynamic states (e.g., states stored in registers of the LO 100) that may be updated by each packet being processed by LO-DP 62. In the present disclosure, it is assumed that the dynamic states are linked to packet groups associated with LBKey, and further, that there are no global dynamic states that require synchronization.

[0082] As shown in Figures 5A-5B, LO-CP 64 handles state transition from one NF Instance to another NF Instance when a packet group associated with LBKey changes its HH state from the non-HH-state to the HH state, or vice versa. In these cases, the static state transitions can be managed by the LO-CP 64 as they do not change frequently. However, dynamic states (e.g., stored in registers) updated by packet arrivals cannot be moved through the CP in a consistent way because the control plane processes work on a much longer timescale (e.g., 1 ms - 10ms) than the packet processing times (10 ns - 1000 ns). To solve this issue, the present embodiments extend the LO-DP 62 instance implementations. Particularly, the LO-DPs 62 are configured according to the present embodiments to support the following:

[0083] 1 . the handling of SM packets;

[0084] 2. the functionality for storing / encoding states of a packet group associated with LBKey into the SM packet;

[0085] 3. the functions for restoring / loading states of the packet group LBKey to the data plane objects (e.g., registers) from the SM packet; and

[0086] 4. the ability to forward data packets to another LO-DP 62 instance (HH or non-HH) without processing the data packets; and

[0087] 5. the ability to notify the LO-CP 64.

[0088] Figure 6 is a flow diagram illustrating a method 250, implemented by LO 100 for migrating data packets associated with a given LBkey 92 from a non-HH NF Instance 270 instance to a HH NF Instance 290 instance with dynamic states, according to embodiments of the present disclosure. In this embodiment, the LO 100 comprises a LO-DP 62, a LO-CP 64, a non-HH NF-DP Instance 270 configured to process non-HH data packets, and a HH NF-DP Instance 290 configured to process HH data packets. By way of example only, the non-HH NF- DP Instance 270 may comprise a CPU-based NF instance and the HH NF-DP Instance 290 may comprise a HW-accelerated NF instance.

[0089] As seen in Figure 6, the LO-CP 64 moves the HH state of data flow associated with the LBKey 92 from a non-HH state to an HH state (box 252) and then loads a set of static rules for LBKey 92 to the HH NF-DP Instance 290 (box 254). If the LO-CP 64 determines that the non- HH NF-DP Instance 270 cannot handle dynamic states (box 256), LO-CP 64 moves the states of the data packets associated with LBKey 92 (e.g., from non-HH to HH) and the process is complete (box 258). Otherwise (box 256), the LO-CP 64 generates and sends an “empty” SM packet identifying the LBKey 92 to the non-HH NF-DP Instance 270 (box 260). The non-HH NF- DP Instance 270 is the active instance of the NF for the data packets associated with LBKey 92.

[0090] Upon receipt, the non-HH NF-DP Instance 270 first determines whether it received a data packet to process or an SM packet (box 272). If the received packet is not an SM packet, then it is a data packet for the non-HH NF Instance 270 to process. Before processing the received packet, however, the non-HH NF-DP Instance 270 first determines whether the HH state of the received data packet indicates a change in its HH State and should be forwarded to the to the HH NF instance 290 that handles data packets associated with LBKey 92 (box 274). To accomplish this function, the non-HH NF Instance 270 checks a forwarding mode field for data packets associated with LBKey 92 (box 274). If the forwarding mode is set to FALSE, the non-HH NF Instance 270 will process the data packet normally (box 276). However, if the forwarding mode is set to TRUE, the non-HH NF Instance 270 will forward the received data packet to the HH NF Instance 290. In this embodiment, the non-HH NF Instance 270 will simply forward the data packet without processing the data packet further, thereby avoiding further HH state changes to the data packet, and thus, the data packets associated with LBKey 92.

[0091] However, if the data packet received by the non-HH NF Instance 270 is an empty SM packet (box 272), the non-HH NF Instance 270 is triggered to perform certain functions. In this embodiment, for example, receiving an empty SM data packet triggers the non-HH NF Instance 270 to set the forwarding mode for data packets associated with LBKey 92 (i.e. , which is determined by inspecting the LBKey 92 received with the empty SM packet) to TRUE (box 278). Additionally, the receipt of the empty SM data packet triggers the non-HH NF Instance 270 to populate the empty SM packet with dynamic state information (box 280), such as the contents of the REG element indexed by LBKey 92, for example, before forwarding the now-populated SM packet to the HH NF Instance 290 (box 282). The HH NF Instance 290 to which the SM packet is forwarded is the passive instance for data packets associated with LBKey 92.

[0092] Upon receipt, the HH NF Instance 290 will first determine if the data packet it received is an SM packet or a data packet to be processed by the HH NF Instance 290 (box 302). If it is determined that the received packet is not an SM packet, HH NF Instance 290 will determine (box 304) whether the received packet should be forwarded to the non-HH NF Instance 270 that handles data packets associated with LBKey 92 (box 306) or processed by the HH NF Instance 290 (box 308). However, if it is determined that the received packet is an SM packet (box 302), the HH NF Instance 290 is triggered to switch to a non-forwarding mode for packet groups / data flows associated with LBKey 92 (box 294) and loads the dynamic states carried by the SM packet into data plane objects (e.g., registers) (box 296). So loaded, the HH NF Instance 290 sends a notification signal (e.g., the populated SM message) to LO-CP 64 indicating the completion of state transitions (box 300) and the process ends. In situations where the populated SM message is not used to notify the LO-CP 64, the SM message is simply dropped.

[0093] Figure 6 illustrates an embodiment for transferring the dynamic state of the data packets associated with LBKey 92 from a non-HH state in which the packet is processed by its active non-HH NF instance, to a HH state in which the packet is processed by its passive HH NF Instance. However, those of ordinary skill in the art should realize that the present disclosure is not so limited and is configured to perform method 250 when the dynamic state of the data packets associated with LBKey 92 changes from a HH state to a non-HH state.

[0094] Additionally, although not specifically shown here, the present disclosure provides other well-known mechanisms to ensure that the SM packets are received by the other instance. Such mechanisms help avoid the loss of SM packets. For example, in one technique, the present embodiments send multiple copies of the SM packets. This technique is generally sufficient in an environment where the two NF instances (i.e., the non-HH NF-DP Instance 270 and the HH NF-DP Instance 290) are co-located at the same site. In an alternative technique, the receiving NF instance (e.g., the HH NF-DP Instance 290 in Figure 6) is configured to implement an acknowledgement function explicitly or implicitly acknowledging the receipt of the SM packets to the non-HH NF-DP Instance 270. For example, the instance sending the SM packet (e.g., the non-HH NF-DP Instance 270 of Figure 6) could be configured to store the SM packet and start a timer. Once the timer expires, the instance that sent the SM packet to the other NF Instance would then resend the SM packet to the other NF Instance.

[0095] It should be noted here that the present embodiments are fully compliant with an O-RAN implementation. For example, the LO-DP 62 and the nodes responsible for marking the data packets with the LBKey 92 and the HH indicator 94 fields may be implemented using separate hardware circuitry (e.g., separate nodes) or they may be implemented on the same node.

[0096] Figure 7 is a flow diagram illustrating a method 310 for performing LO according to embodiments of the present disclosure. In this embodiment, method 310 is implemented at a LO node (e.g., LO 100 comprising a LO-DP 62 component and a LO-CP 64 component) configured to optimize the load between first and second instances of a NF (e.g., the non-HH NF-DP Instance 270 and the HH NF-DP Instance 290). As previously described, both the non- HH and HH NF-DP Instances are configured to perform the same functionality but are respectively implemented on different node types. For example, the non-HH NF-DP Instance 270 may be implemented as a CPU-based NF instance (e.g., a software function), while the HH NF-DP Instance 290 may be implemented as a HW-accelerated NF Instance (e.g., hardware circuitry).

[0097] As seen in Figure 7, LO 100 receives data packets from upstream and / or downstream nodes. Each data packet comprises load balancing (LB) information that includes (1 ) a LBKey that associates the data packet with one or more other data packets in a data flow and (2) a Heavy Hitter (HH) indicator that indicates whether the received data packet is a HH data packet that is associated with a HH or a non-HH data packet that is not associated with a HH (box 312). LO 100 then determines, based on the HH Indicator, whether the received data packet is a HH data packet associated with a HH data flow, or a non-HH data packet associated with a non-HH data flow (box 314). LO 100 then steers each data packet to one of the first and second instances of the NF based on the LBKey and the HH indicator (box 316). In this embodiment, the first instance of the NF (e.g., the non-HH NF-DP Instance 270) is implemented on a first node of a first node type and the second instance of the NF (e.g., the HH NF-DP Instance 290) is implemented on a second node of a second node type. The first and second node types are different from each other. For example, in one embodiment, LO 100 steers the received data packets to the first node responsive to the data packet being a non-HH data packet and to the second node responsive to the data packet being a HH data packet. Thereafter, responsive to detecting a change in the HH Indicator (e.g., from non-HH to HH or vice versa), LO 100 changes steering of the data packets from the one of the first and second NF instances to the other of the first and second NF instances (box 318).

[0098] Figure 8 is a flow diagram illustrating a method 320 for determining a current HH state of a data packet in a data flow according to embodiments of the present disclosure. In this embodiment, method 320 is implemented by LO 100, and more specifically, by the LO-DP 62 component of LO 100.

[0099] As seen in Figure 8, LO 100 first determines the current HH state of the data packet based on the value of the HH indicator (box 322). As previously described, the current HH state of the data packet indicates whether the data flow associated with the data packet (i.e. , the data flow to which the data packet belongs) is an HH data flow or a non-HH data flow. LO 100 also determines a stored HH state for a previously received data packet associated with the data flow (box 324). The stored HH state may be derived, for example, from the REG registry and indicates whether the data flow associated with the previously received data packet is an HH data flow or a non-HH data flow. LO 100 then compares the current HH state of the data packet to the stored HH state for the previously received data packet (box 326) and determines a change in the HH state of the data flow based on the results of the comparison (box 328).

[0100] Figures 9A-9D are flow diagrams, implemented by LO 100, for performing the processing logic of Figure 4 according to embodiments of the present disclosure. As above, LO 100 comprises the functionality of the LO-DP 62 and the LO-CP 64.

[0101] More particularly, Figure 9A illustrates a method 330 for processing a data packet when the comparison detailed in Figure 8 shows that both the current HH state and the stored HH state indicate that the received data packet and the previously received data packet are associated with a HH data flow. Particularly, the LO-DP 62 first determines a location of the HH instance that is configured to process data packets having the LBKey 92 (box 332). Because LO-DP 62 did not detect a change in the HH status of the data flow, LO-DP 62 refrains (box 334) from notifying the LO-CP 64 about a change in the HH state of the data flow and forwards (box 336) the received data packet to the HH instance that is configured to process data packets having the LBKey 92 (e.g., the HH NF-DP Instance 290).

[0102] Figure 9B illustrates a method 340 implemented by the LO-DP 62 component of LO 100 for processing a received data packet when the comparison detailed in Figure 8 indicates a change in the HH state of the data flow (e.g., when the comparison shows that the current HH state indicates that the received data packet is associated with a HH data flow, but the stored HH state indicates that the previously received data packet is associated with a non-HH data flow). Thus, method 340 is implemented responsive to detecting a change of HH states from HH to non-HH (i.e., when a received data packet indicates that the data flow to which it belongs is no longer a HH data flow, but instead, is now a non-HH data flow).

[0103] As seen in method 340, LO 100 first reads a state key (SKEY) stored in memory to determine whether the LO-CP 64 has already been notified about the change in the HH state of the data flow to which the received data packet belongs (box 342). LO 100 then determines whether the value of the state key stored in memory (i.e., the stored LBKey of the previously received data packet) matches the value of the LBKey 92 of the received data packet (box 344). Responsive to determining that the state key does not match the LBKey 92 of the received data packet, LO 100 stores the LBKey 92 of the received data packet into memory to indicate that the LO-CP 64 has already been notified about the change in the HH state of the data flow to which the received data packet belongs (box 346). LO 100 then notifies the LO-CP 64 that the received data packet is associated with a HH data flow (box 348) and forwards the received data packet to the non-HH Instance that is configured to process data packets having the LBKey 92 (box 350). In one embodiment, for example, the non-HH Instance is the second instance of the NF (e.g., the HW-accelerated NF Instance). However, if the comparison of the state key and the LBKey 92 reveals that the two values match (box 344), the received data packet is simply forwarded to the non-HH Instance that is configured to process data packets having the LBKey 92 (box 350).

[0104] Figure 9C illustrates a method 360 implemented by the LO-DP 62 component of LO 100 for processing a received data packet when the comparison detailed in Figure 8 shows that both the current HH state and the stored HH state indicate that the received data packet and the previously received data packet are associated with a non-HH data flow (i.e., that there has been no change in the HH state of the data flow to which the received data packet belongs). As seen in Figure 9C, LO 100 refrains (box 364) from notifying the LO-CP 64 about a change in the HH state of the data flow (box 362) because, as explained above, there has been no change in the HH state of the data flow. Then, LO 100 forwards the received data packet to the HH NF-DP Instance (e.g., the first instance of the NF - i.e., a CPU-based NF Instance) (box 364).

[0105] Figure 9D illustrates a method 370 implemented by the LO-DP 62 component of LO 100 for processing a received data packet when the comparison detailed in Figure 8 shows the current HH state indicates that the received data packet is associated with a non-HH data flow, and the stored HH state indicates that the previously received data packet is associated with a HH data flow). Thus, method 370 is also implemented responsive to detecting a change of HH states from non-HH to HH (i.e., when a received data packet indicates that the data flow to which it belongs is no longer a non-HH data flow, but instead, is now a HH data flow).

[0106] In more detail, LO 100 first reads the state key SKEY from memory to determine whether the LO-CP 64 component of LO 100 has already been notified about the change in the HH state of the data flow to which the received data packet belongs (box 362). Once LO-DP 62 has read the state key, it compares the state key to the LBKey 92 of the received data packet (box 374). If the state key and the LBKey 92 do not match, LO-DP 62 stores the LBKey 92 of the received data packet in memory to indicate that the LO-CP 64 has been notified of the change in the HH state of the data flow (box 376) and notifies the LO-CP 64 that the received data packet now belongs to a non-HH data flow (box 378). So notified, LO-DP 62 then forwards the received data packet to the HH NF-DP instance configured to process data packets having LBKey 92 (e.g., the first instance of the NF such as the CPU-based NF Instance) (box 380). If, however, the comparison reveals that the state key matches the LBKey 92 of the received data packet, LO-DP 62 forwards the received data packet to whatever non-HH NF-DP Instance is configured to process data packets having LBKey 92 (box 382).

[0107] Figure 10 is a flow diagram illustrating a method 390 for processing State Migration (SM) packets according to embodiments of the present disclosure. In particular, the LO-CP 64 component of LO 100 sends (box 392) one or more SM packets to one of the first and second instances of the NF (e.g., one of the non-HH NF-DP Instance 270 and the HH NF-DP Instance 290) notifying that instance of a change in the HH state of the data flow to which a data packet belongs from being an HH data flow (or a non-HH data flow) to being a non-HH data flow (or HH data flow). The NF instance receiving the SM packet then sends the SM packet to the other of the first and second NF instances (i.e., the other of the non-HH NF-DP and the HH NF-DP Instance 270, 290, respectively) (box 394). LO-CP 64 then receives an acknowledgment receipt for the SM packet from the other of the first and second NF instances (box 396).

[0108] As stated above, in one embodiment, the one of the first and second instances of the NF is a non-HH NF-DP instance and the other of the first and second instances of the NF is a HH NF-DP instance. However, in an alternate embodiment, the one of the first and second instances of the NF is a HH NF-DP instance and the other of the first and second instances of the NF is a non-HH NF-DP instance.

[0109] Additionally, or alternatively, the LO-CP 64 component of LO 100 is configured to send multiple copies of an SM packet to the one of the first and second instances of the NF, and eventually, receive an acknowledgment receipt for the multiple copies of the SM packet from the other of the first and second instances of the NF.

[0110] In one embodiment, the SM packet sent by the LO-CP 64 is an empty packet. In other embodiments, however, the SM packet sent by the LO-CP 64 comprises information indicating that the HH state of the data flow has changed from being one of a HH data flow and a non-HH data flow to being the other of the HH data flow and the non-HH data flow.

[0111] In one embodiment, the first instance of the NF is implemented as a software function on the first node and the second instance of the NF is implemented as hardware circuitry on the second node.

[0112] In one embodiment, for each data packet, the method further comprises determining, based on the HH indicator, whether the data packet is a HH data packet that is associated with an HH data flow, or a non-HH data packet that is not associated with a non-HH data flow.

[0113] In one embodiment, the method further comprises determining whether an HH state of the data flow associated with the data packet has changed by determining a current HH state of the data packet from the HH indicator, wherein the current HH state of the data packet indicates whether the data flow associated with the data packet is an HH data flow or a non-HH data flow, determining a stored HH state for a previously received data packet associated with the data flow, wherein the stored HH state indicates whether the data flow associated with the previously received data packet is an HH data flow or a non-HH data flow, comparing the current HH state of the data packet to the stored HH state for the previously received data packet, and determining, based on the comparison, whether the HH state of the data flow has changed.

[0114] In at least one embodiment, the HH state of the data flow has changed when the comparison indicates that the current HH state does not match the stored HH state, and the HH state of the data flow has not changed when the comparison indicates that the current HH state matches the stored HH state.

[0115] In one embodiment, responsive to determining that the HH state of the data flow has not changed and that both the current HH state and the stored HH state indicate that the data flow is a HH data flow, the method further comprises determining a location of the HH Instance configured to process the data packets having the LBKey, and forwarding the data packet to the HH Instance configured to process the data packets having the LB key.

[0116] In one embodiment, one or more data packet forwarding policies are preconfigured polices stored at, or are accessible to, the LO node, and map a HH Indicator to a node type.

[0117] In one embodiment, the one or more data packet forwarding policies are received from a NF in a CP.

[0118] In another embodiment, however, the one or more data packet forwarding policies are received from a Management System.

[0119] In one embodiment, the change in the HH Indicator indicates one of a start of a state transfer of the data flow from being one of a HH data flow and a non-HH data flow to being the other of the HH data flow and the non-HH data flow, and a completion of the state transfer of the data flow from being one of a HH data flow and a non-HH data flow to being the other of the HH data flow and the non-HH data flow.

[0120] In one embodiment, the LO node comprises data packet processing circuitry.

[0121] In one embodiment, the first node type implements a HH NF-DP Instance using hardware circuitry, and the second node type implements a non-HH NF-DP Instance using a software function.

[0122] In another embodiment, however, the first node type implements a non-HH NF-DP Instance using a software function, and the second node type implements a HH NF-DP Instance using hardware circuitry.

[0123] An apparatus can perform any of the methods herein described by implementing any functional means, modules, units, or circuitry. In one embodiment, for example, the apparatuses comprise respective circuits or circuitry configured to perform the steps shown in the method figures. The circuits or circuitry in this regard may comprise circuits dedicated to performing certain functional processing and / or one or more microprocessors in conjunction with memory. For instance, the circuitry may include one or more microprocessors or microcontrollers, as well as other digital hardware, which may include Digital Signal Processors (DSPs), special-purpose digital logic, and the like. The processing circuitry may be configured to execute program code stored in memory, which may include one or several types of memory such as read-only memory (ROM), random-access memory, cache memory, flash memory devices, optical storage devices, etc. Program code stored in memory may include program instructions for executing one or more telecommunications and / or data communications protocols as well as instructions for carrying out one or more of the techniques described herein, in several embodiments. In embodiments that employ memory, the memory stores program code that, when executed by the one or more processors, carries out the techniques described herein.

[0124] Figure 11 is a functional block diagram illustrating a network node 400 configured to function as a LO 100 according to embodiments of the present disclosure. As stated previously, LO 100 comprises two components - i.e., the LO-DP component 62 and the LO-CP component 64. Both components are implemented on a network node, represented herein by node 400 seen in Figure 11 . However, while both components may be implemented on the same node 400, some embodiments of the present disclosure implement the LO-DP 62 and LO-CP 64 distributed across multiple nodes communicatively connected to each other. Therefore, while Figure 11 illustrates a single node 400, those of ordinary skill in the art will readily appreciate that this is for ease of discussion only, and that the present embodiments may comprise a first network node 400 implementing the LO-DP 62 component of LO 100 and a second, different network node 400 implementing the LO-CP 64 component of LO 100.

[0125] As seen in Figure 11 , node 400 of the present disclosure comprises, inter alia, communication circuitry 402, processing circuitry 404, and memory circuitry 406 configured to store a control program 408.

[0126] The communication circuitry 402 is configured to communicate signals and data packets (i.e., data packets, SM packets, etc.) with one or more non-HH and HH NF Instances. For example, such communications may include, but are not limited to, those associated with sending and receiving data packets belonging to a HH data flow or a non-HH data flow, and / or SM data packets that may / may not be empty, as previously described. The processing circuitry 404 is configured according to the present disclosure to control the overall operation of node 400 and processes the signals and data packets sent to or received by node 400. Such processing includes coding and modulation of transmitted data packets, as well as the demodulation and decoding of received data packets. The processing circuit 404 in this embodiment may comprise one or more microprocessors, hardware, firmware, or a combination thereof.

[0127] Memory 406 comprises both volatile and non-volatile memory circuitry for storing computer program code and data needed by the processing circuit 402 for operation. Memory 406 may comprise any tangible, non-transitory computer-readable storage medium for storing data including electronic, magnetic, optical, electromagnetic, or semiconductor data storage. Memory 406 stores a computer program 408 comprising executable instructions that configure the processing circuit 404 to implement any of the methods 140, 190, 220, 250, and 310-390, as described herein. A computer program in this regard may comprise one or more code modules corresponding to the means or units described above. In general, computer program instructions and configuration information are stored in a non-volatile memory, such as a ROM, erasable programmable read only memory (EPROM) or flash memory. Temporary data generated during operation may be stored in a volatile memory, such as a random-access memory (RAM). In some embodiments, computer program 408 for configuring the processing circuit 404 as herein described may be stored in a removable memory, such as a portable compact disc, portable digital video disc, or other removable media. The computer program 408 may also be embodied in a carrier such as an electronic signal, optical signal, radio signal, or computer readable storage medium.

[0128] Those skilled in the art will also appreciate that embodiments herein further include corresponding computer programs. A computer program comprises instructions which, when executed on at least one processor of an apparatus, cause the apparatus to carry out any of the respective processing described above. A computer program in this regard may comprise one or more code modules corresponding to the means or units described above.

[0129] Embodiments further include a carrier containing such a computer program. This carrier may comprise one of an electronic signal, optical signal, radio signal, or computer readable storage medium.

[0130] In this regard, embodiments herein also include a computer program product stored on a non-transitory computer readable (storage or recording) medium and comprising instructions that, when executed by a processor of an apparatus, cause the apparatus to perform as described above.

[0131] Embodiments further include a computer program product comprising program code portions for performing the steps of any of the embodiments herein when the computer program product is executed by a computing device. This computer program product may be stored, for example, on a computer readable recording medium.

[0132] The present embodiments may, of course, be carried out in other ways than those specifically set forth herein without departing from characteristics described herein. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, and all changes coming within the meaning and equivalency range of the appended claims are intended to be embraced therein.

Claims

CLAIMSWhat is claimed is:1 . A method (310), implemented at a Load Optimizer (LO) node (400), for load optimization between first (270) and second (290) instances of a Network Function (NF), wherein each of the first and second instances of the NF perform the same functionality but are respectively implemented on different node types, the method comprising: receiving (312) data packets, wherein each data packet comprises load balancing (LB) information that includes: a LB key (92) that associates the data packet with one or more other data packets in a data flow; and a Heavy Hitter (HH) indicator (94) that indicates whether the data packet is a HH data packet associated with a HH data flow, or a non-HH data packet associated with a non-HH data flow; and steering (316) each data packet to one of the first and second instances of the NF based on the LB key and the HH indicator, wherein the first instance of the NF is implemented on a first node of a first node type and the second instance of the NF is implemented on a second node of a second node type that is different from the first node type.

2. The method of claim 1 , wherein the first instance of the NF is implemented as a software function on the first node and the second instance of the NF is implemented as hardware circuitry on the second node.

3. The method of any of claims 1-2, wherein for each data packet, the method further comprises determining (314), based on the HH indicator, whether the data packet is a HH data packet that is associated with an HH data flow, or a non-HH data packet that is not associated with a non- HH data flow.

4. The method of claim 3, further comprising determining (320) whether an HH state of the data flow associated with the data packet has changed by: determining (322) a current HH state of the data packet from the HH indicator, wherein the current HH state of the data packet indicates whether the data flow associated with the data packet is an HH data flow or a non-HH data flow; determining (324) a stored HH state for a previously received data packet associated with the data flow, wherein the stored HH state indicates whether the data flow associated with the previously received data packet is an HH data flow or a non-HH data flow; comparing (326) the current HH state of the data packet to the stored HH state for the previously received data packet; anddetermining (328), based on the comparison, whether the HH state of the data flow has changed.

5. The method of claim 4: wherein the HH state of the data flow has changed when the comparison indicates that the current HH state does not match the stored HH state; and wherein the HH state of the data flow has not changed when the comparison indicates that the current HH state matches the stored HH state.

6. The method of any of claims 4-5, wherein responsive to determining that the HH state of the data flow has not changed and that both the current HH state and the stored HH state indicate that the data flow is a HH data flow, the method (330) further comprises: determining (332) a location of the HH Instance configured to process the data packets having the LBKey; and forwarding (336) the data packet to the HH Instance configured to process the data packets having the LB key.

7. The method of any of claims 4-5, wherein responsive to determining that the HH state of the data flow has changed from a non-HH data flow to a HH data flow, the method (340) further comprises: reading (342) a state key stored in memory; determining (344) whether the state key matches the LB key of the data packet; responsive to determining that the state key does not match the LB key of the data packet: storing (346) the LB key of the data packet to indicate that a LO-CP has been notified about the change in the HH state of the data flow associated with the data packet; and notifying (348) the LO-CP that the data packet is associated with a HH data flow; and forwarding (350) the data packet to the non-HH Instance configured to process the data packets having the LB key.

8. The method of claim 7, wherein responsive to determining that the state key does match the LB key of the data packet, forwarding (350) the data packet to the non-HH Instance configured to process the data packets having the LB key.

9. The method of any of claims 4-5, wherein responsive to determining that the HH state of the data flow has not changed and that both the current HH state and the stored HH state indicate that the data flow is a non-HH data flow, the method (360) further comprises forwarding (364)the data packet to the non-HH Instance configured to process the data packets having the LB key.

10. The method of any of claims 4-5, wherein responsive to determining that the HH state of the data flow has changed from a HH data flow to a non-HH data flow, the method (370) further comprises: reading (372) a state key stored in memory; determining (374) whether the state key matches the LB key of the data packet; responsive to determining that the state key does not match the LB key of the data packet: storing (376) the LB key of the data packet to indicate that the LO-CP has been notified about the change in the HH state of the data flow associated with the data packet; notifying (378) the LO-CP that the data packet is associated with a non-HH data flow; and forwarding (380) the data packet to the HH Instance configured to process the data packets having the LB key.11 . The method of claim 10, wherein responsive to determining that the state key does not match the LB key of the data packet, forwarding (370) the data packet to the HH Instance configured to process the data packets having the LB key.

12. The method of any of claims 1-11 , wherein one or more data packet forwarding policies are preconfigured polices stored at, or are accessible to, the LO node, and map a HH Indicator to a node type.

13. The method of any of claims 1-12, wherein the one or more data packet forwarding policies are received from a NF in a CP.

14. The method of any of claims 1-12 wherein the one or more data packet forwarding policies are received from a Management System.

15. The method of any of claims 1 -12, further comprising changing (318) the steering of data packets associated with the data flow from the one of the first and second instances of the NF to the other of the first and second instances of the NF responsive to detecting a change in the HH Indicator.

16. The method of claim 15, wherein the change in the HH Indicator indicates one of: a start of a state transfer of the data flow from being one of a HH data flow and a non-HH data flow to being the other of the HH data flow and the non-HH data flow; and a completion of the state transfer of the data flow from being one of a HH data flow and a non-HH data flow to being the other of the HH data flow and the non-HH data flow.

17. The method of any of claims claim 1-16, further comprising sending (392) one or more State Migration (SM) packets to the one of the first and second instances of the NF to notify the one of the first and second instances of the NF of a state transfer of the data flow from being one of a HH data flow and a non-HH data flow to being the other of the HH data flow and the non-HH data flow.

18. The method of claim 17 further comprising receiving (396) an acknowledgment receipt for the SM packet from the other of the first and second instances of the NF.

19. The method of claim 17 further comprising: sending multiple copies of an SM packet to the one of the first and second instances of the NF; and / or receiving an acknowledgment receipt for the multiple copies of the SM packet from the one of the first and second instances of the NF.

20. The method of claim 17, wherein the SM packet is an empty packet.

21. The method of claim 17, wherein the SM packet comprises information indicating that the HH state of the data flow has changed from being one of a HH data flow and a non-HH data flow to being the other of the HH data flow and the non-HH data flow.

22. The method of any of the preceding claims, wherein the LO node comprises data packet processing circuitry.

23. The method of any of the preceding claims wherein the first node type implements a HH NF- DP Instance using hardware circuitry, and wherein the second node type implements a non-HH NF-DP Instance using a software function.

24. The method of any of the preceding claims wherein the first node type implements a non-HH NF-DP Instance using a software function, and wherein the second node type implements a HH NF-DP Instance using hardware circuitry.

25. The method of any of the preceding claims, wherein the LO includes a Data Plane (LO-DP) component and a Control Plane (LO-CP) component.

26. A network node (400) for load optimization between first and second instances of a Network Function (NF) (270, 290), wherein each of the first and second instances of the NF perform the same functionality but are respectively implemented on different node types, the network node configured to: receive (312) data packets, wherein each data packet comprises load balancing (LB) information that includes: a LB key (92) that associates the data packet with one or more other data packets in a data flow; and a Heavy Hitter (HH) indicator (94) that indicates whether the data packet is a HH data packet associated with a HH data flow, or a non-HH data packet associated with a non-HH data flow; and steer (316) each data packet to one of the first and second instances of the NF based on the LB key and the HH indicator, wherein the first instance of the NF is implemented on a first node of a first node type and the second instance of the NF is implemented on a second node of a second node type that is different from the first node type.

27. The network node of claim 26, wherein the network node is further configured to perform the method according to any one of claims 2-25.

28. A network node (400) for load optimization between first and second instances of a Network Function (NF) (270, 290), wherein each of the first and second instances of the NF perform the same functionality but are respectively implemented on different node types, the network node comprising: processing circuitry (404); and memory (406) configured to store a control program (408) that, when executed by the processing circuitry, configures the processing circuitry to: receive (312) data packets, wherein each data packet comprises load balancing (LB) information that includes: a LB key (92) that associates the data packet with one or more other data packets in a data flow; and a Heavy Hitter (HH) indicator (94) that indicates whether the data packet is a HH data packet associated with a HH data flow, or a non-HH data packet associated with a non-HH data flow; and steer (316) each data packet to one of the first and second instances of the NF based on the LB key and the HH indicator, wherein the first instance of the NF isimplemented on a first node of a first node type and the second instance of the NF is implemented on a second node of a second node type that is different from the first node type.

29. The network node of claim 28, wherein the processing circuitry is further configured to perform the method according to any one of claims 2-25.

30. A computer program (408) comprising instructions that, when executed on processing circuitry (404) of a network node (400), causes the network node to perform the method according to any of claims 1-25.31 . A carrier containing the computer program of claim 30, wherein the carrier is one of an electronic signal, optical signal, radio signal, or computer readable storage medium.

32. A non-transitory computer-readable storage medium (406) comprising a computer program (408) stored thereon, the computer program comprising executable instructions that, when executed by processing circuitry (404) of a network node (400), causes the network node to perform the method of any one of claims 1-25.

Images