Hardware-based load balancer with network function scaling

Abstract

A hardware-based implementation for a load balancer distributes packets across a group of packet processing NFs, adapts to dynamic scaling, and does not require special mechanisms in the NF instances or the servers on which they run. Before arriving at the load balancer, the packets are marked or tagged with a key identifying the traffic flow to which the packet belongs and a timestamp that indicates the start of a packet burst to which the packet belongs. The data plane component of the load balancer directs the packets according to the packet markings and its internal configuration. The data plane component notifies a control plane component when a new burst is detected. The control plane component manages the internal configuration of the data plane component and handles the migration of states between NF instances based on notifications from the data plane component.

Description

[0001] HARDWARE-BASED LOAD BALANCER WITH NETWORK FUNCTION SCALING TECHNICAL FIELD

[0002] The present disclosure relates generally to load balancing in communication network and, more particularly, to a hardware-based load balancer with network function scaling.

[0003] BACKGROUND

[0004] Load balancing distributes network traffic dynamically across network resources that support an application. In general, a physical network deployment will typically comprise a plurality of network functions (NFs) that provide the same service. A load balancer maps individual network flows to individual instances of these NFs so that the load is balanced across the deployed NF instances. Typical load balancers act on some keys calculated based on packet characteristics (e.g., 5-tuple) to route traffic consistently among the different NF instances. Consistent routing of traffic is needed in order to minimize cache invalidations and packet reorders within one specific flow. Both problems can occur with random (non-consistent) packet forwarding.

[0005] Another method, used by Kubernetes (k8s) as one example, is flow tracking. Flow tracking is typically used together with a random load balancer that assigns an appropriate worker instance (in k8s it is called POD) to a traffic flow when the first packer arrives and stores the flow parameters in its flow tracking table. In this way, it is possible to combine the advantages of randomness and consistency with the price of increased memory footprint.

[0006] From time to time, there is a need to either scale up or scale down the NF instances providing a particular service. When the configuration of the available NFs instances changes by either adding or removing NF instances, it is typical to apply the new configuration to new traffic flows only in order to provide consistency in the flow handling. In some cases, the traffic load on NFs to be removed for service or maintenance will continue for some long-lasting sessions, which can degrade the user experience, e.g., where it is desired to replace an existing NF with a later version of the same network function. As a result, the scaling event may take a long time to complete.

[0007] For faster scaling, traffic can be redirected from one NF to another when network resources are scaled up or scaled down. But redirection of traffic flows can cause problems with consistency in the way packets in a traffic flow are handled and reordering of packets. It is typical for NFs to maintain states for the traffic flows it is handling. When a traffic flow is redirected to a different network function, the target NF will not have the state information for the traffic flow, which can negatively impact the traffic flow. Some proprietary load balancers migrate state information when a traffic flow is redirected, but it is desirable for load balancers from different vendors to be interoperable and interchangeable. Even where state information is migrated, packet reordering may occur where packets from the new, less loaded NFs overtake packets processed by a more heavily-loaded NFs, which can impact the overall quality of experience for some users.

[0008] Hardware-based load balancers are advantageous because they can provide line-rate operation. In Vladimir Olteanu, et al, Stateless Datacenter Load-Balancing With Beamer, Proceedings of the 15th LISENIX Conference on Networked Systems Design and Implementation (NSDI'18), pp. 125-139 and Jain, N., Mohan et al., Kubernetes Load-Balancing And Related Network Functions Using P4, Proceedings of the Symposium on Architectures for Networking and Communications (2021 ), hardwarebased load balancers have been proposed to steer traffic destined to a given service across a dynamic set of backend machines. These papers describe Programming Protocol-independent Packet Processors (P4) implementations that can run on a programmable application-specific integrated circuit (ASIC) (e.g., Intel Tofino) for packet processing. The described method preserves the state of Transport Control Protocol (TCP) connections by a combination of mechanisms running on both the load-balancing switch and the end hosts. In Jain et al, a P4-based load balancing is proposed for a Kubernetes (K8s) cluster, which also focuses on assigning TCP flows to different service pods / containers. These papers do not address state migration following changes in the configuration of network resources. Further, these papers to not provide a mechanisms to avoid packet reordering, or consistent handling of packet bursts do not handle packet bursts.

[0009] SUMMARY

[0010] The present disclosure provides a hardware-based implementation for a load balancer that distributes packets across a group of packet processing NFs, adapts to dynamic scaling, and does not require special mechanisms in the NF instances or the servers on which they run. Before arriving at the load balancer, the packets are marked or tagged with a key identifying the traffic flow to which the packet belongs and a timestamp that indicates the start of a packet burst to which the packet belongs. The data plane component of the load balancer directs the packets according to the packet markings and its internal configuration. The data plane component notifies a control plane component when a new burst is detected. The control plane component manages the internal configuration of the data plane component and handles the migration of states between NF instances based on notifications from the data plane component.

[0011] The hardware-based implementation of the load balancer enables the load balancing with support for NF instance scaling in programmable data plane hardware, ensuring line-rate operation. Functionality that requires memory and CPU resources is delegated to separate nodes. The logic for the data plane component of the load balancer can be implemented in constrained hardware, such as a Tofino switch or a SmartNIC, resulting in fast processing. No state bookkeeping is required because the key that maps the packet to the logical data flow, the timestamp, and new burst flag are calculated in a separate node. Resource (e.g., static random-access memory (SRAM) and ternary content-addressable memory (TCAM)) consumption in the load balancer is independent of the number of keys (i.e. , flows / users) to be handled so the solution is cost efficient. Further, physical implementation of the load balancing logic is hidden from both NF control plane (NF-CP) and NF data plane (NF-DP), thus enabling a cloud-native data plane realization with auto-scaling properties of NF-DP.

[0012] A first aspect of the disclosure comprises methods implemented by a data plane in load balancer configured to distribute packets among a plurality of NF instances in a resource group. In one embodiment, the method comprises receiving a packet belonging to a traffic flow, the packet including a header containing a flow identifier for the traffic flow and a timestamp associated with a start of a packet burst within the traffic flow to which the packet belongs. The method further comprises directing, based on the timestamp and flow identifier, the packet to a serving NF instance for the traffic flow. The method further comprises sending a notification to the control plane of the load balancer when the packet burst is a determined to be new packet burst and the serving NF instance has changed.

[0013] A second aspect of the disclosure comprises a data plane entity of a load balancer configured to distribute packets among a plurality of NF instances in a resource group. In one embodiment, the data plane entity is configured to receive a packet belonging to a traffic flow. The packet includes a header containing a flow identifier for the traffic flow and a timestamp associated with a start of a packet burst within the traffic flow to which the packet belongs. The data plane entity is further configured to direct, based on the timestamp and flow identifier, the packet to a serving NF instance for the traffic flow. The data plane entity is further configured to send a notification to the control plane of the load balancer when the packet burst is a determined to be new packet burst and the serving NF instance has changed.

[0014] A third aspect of the disclosure comprises a data plane entity of load balancer configured to distribute packets among a plurality of NF instances in a resource group. In one embodiment, the load balancer comprises input / output (I / O) circuitry for receiving and sending packets over the communication network and processing circuitry. The processing circuitry is configured to receive a packet belonging to a packet burst. The packet includes a header containing a flow identifier for the traffic flow and a timestamp associated with a start of a packet burst within the traffic flow to which the packet belongs. The processing circuitry is further configured to direct, based on the flow identifier and the timestamp, the packet to a serving NF instance for the traffic flow. The processing circuitry is further configured to send a notification to the control plane of the load balancer when the packet burst is a determined to be new packet burst and the serving NF instance has changed.

[0015] A fourth aspect of the disclosure comprises a computer program for a load balancer in a communication network. The computer program comprises executable instructions that, when executed by processing circuitry in the load balancer, causes it to perform the method according to the first aspect.

[0016] A fifth aspect of the disclosure comprises a carrier containing a computer program according to the fourth aspect. The carrier is one of an electronic signal, optical signal, radio signal, or a non-transitory computer readable storage medium.

[0017] A sixth aspect of the disclosure comprises methods implemented in a control plane of a load balancer configured to distribute packets among a plurality of NF instances in a resource group. In one embodiment, the method comprises receiving a reconfiguration message indicating a change in the configuration of a resource group including one or more NF instances. The method further comprises, migrating, responsive to the reconfiguration message, state information for one or more traffic flows being distributed by the load balancer from a first NF instance to a second NF instance in the reconfigured resource group. The method further comprises creating, a group identifier for the reconfigured resource group. The method further comprises updating an internal configuration in a data plane of the load balancer to map the traffic flows being distributed by the load balancer to corresponding NFs instances in the reconfigured resource group. A seventh aspect of the disclosure comprises a control plane entity of a load balancer configured to distribute packets among a plurality of NF instances in a resource group. In one embodiment, the control plane entity is configured to receive a reconfiguration message indicating a change in the configuration of a resource group including one or more NF instances. The control plane entity is further configured to, responsive to a reconfiguration message, migrate state information for one or more traffic flows being distributed by the load balancer from a first NF instance to a second NF instance in the reconfigured resource group. The control plane entity is further configured to update an internal configuration in a data plane of the load balancer to map the traffic flows being distributed by the load balancer to corresponding NFs instances in the reconfigured resource group.

[0018] An eighth aspect of the disclosure comprises a control plane entity of a load balancer configured to control the distribution of packets among a plurality of NF instances in a resource group. In one embodiment, the load balancer comprises interface circuitry for receiving and sending packets over the communication network and processing circuitry for processing the packets. The processing circuitry is configured to receive a reconfiguration message indicating a change in the configuration of a resource group including one or more NF instances. The processing circuitry is further configured to, responsive to a reconfiguration message, migrate state information for one or more traffic flows being distributed by the load balancer from a first NF instance to a second NF instance in the reconfigured resource group. The processing circuitry is further configured to update an internal configuration in a data plane of the load balancer to map the traffic flows being distributed by the load balancer to corresponding NFs instances in the reconfigured resource group.

[0019] A ninth aspect of the disclosure comprises a computer program for a load balancer in a communication network. The computer program comprises executable instructions that, when executed by processing circuitry in the load balancer, causes it to perform the method according to the sixth aspect.

[0020] A tenth aspect of the disclosure comprises a carrier containing a computer program according to the ninth aspect. The carrier is one of an electronic signal, optical signal, radio signal, or a non-transitory computer readable storage medium.

[0021] An eleventh aspect of the disclosure comprises methods implemented by a packet marker for marking packets in a traffic flow. The method comprises receiving a packet belonging to a packet burst within the traffic flow. The method further comprises adding a time stamp to a header of the packet indicating a starting time of the packet burst to which the packet belongs.

[0022] An twelfth aspect of the disclosure comprises methods implemented by a packet marker for marking packets in a traffic flow. The packet marker is configured to receive a packet belonging to a packet burst within the traffic flow. The packet marker is further configured to add a time stamp to a header of the packet indicating a starting time of the packet burst to which the packet belongs.

[0023] A thirteenth aspect of the disclosure comprises a packet marker for marking packets in a traffic flow. The packet marker comprises input / output circuitry for receiving and sending packets and processing circuitry. The processing circuitry is configured to receive a packet belonging to a packet burst within the traffic flow. The processing circuitry is further configured to add a time stamp to a header of the packet indicating a starting time of the packet burst to which the packet belongs.

[0024] A fourteenth aspect of the disclosure comprises a computer program for a packet marker in a communication network. The computer program comprises executable instructions that, when executed by processing circuitry in the load balancer, causes it to perform the method according to the eleventh aspect.

[0025] A fifteenth aspect of the disclosure comprises a carrier containing a computer program according to the fourteenth aspect. The carrier is one of an electronic signal, optical signal, radio signal, or a non-transitory computer readable storage medium.

[0026] BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a block diagram illustrating a load balancer in a communication network comprising a plurality of NF instances providing the same network service or application service.

[0028] Figure 2 is a flow chart illustrating a method of distributing packets implemented in the data plane in a load balancer.

[0029] Figure 3 is a flow chart illustrating a method of controlling distribution of packets implemented in the control plane in a load balancer.

[0030] Figure 4 is a flow chart illustrating another method of controlling distribution of packets implemented in the control plane in a load balancer.

[0031] Figure 5 is a flow chart illustrating another method of distributing packets implemented in the data plane in a load balancer.

[0032] Figure 6 is a flow chart illustrating a method of controlling distribution of packets implemented in the control plane in a load balancer. Figure 7 is a flow chart illustrating a method of marking packets for load balancing implemented by packet marker.

[0033] Figure 8 is a functional block diagram of the data plane in a load balancer.

[0034] Figure 9 is a functional block diagram of the control plane in a communication in a load balancer.

[0035] Figure 10 is a functional block diagram of a packet marker.

[0036] Figure 1 1 illustrates both a logical and physical view of the load balancer implemented in a cloud infrastructure.

[0037] DETAILED DESCRIPTION

[0038] The present disclosure provides a hardware-based implementation for a load balancer that distributes packets across a group of packet processing NFs, adapts to dynamic scaling, and does not require special mechanisms in the NF instances or the servers on which they run. Before arriving at the load balancer, the packets are marked or tagged with a key identifying the traffic flow to which the packet belongs and a timestamp that indicates the start of a packet burst to which the packet belongs. The data plane component of the load balancer directs the packets according to the packet markings and its internal configuration. The data plane component notifies a control plane component when a new burst is detected. The control plane component manages the internal configuration of the data plane component and handles the migration of states between NF instances based on notifications from the data plane component.

[0039] The hardware-based implementation of the load balancer enables the load balancing with support for NF instance scaling in programmable data plane hardware, ensuring line-rate operation. Functionality that requires memory and CPU resources is delegated to separate nodes. The logic for the data plane component of the load balancer can be implemented in constrained hardware, such as a Tofino switch or a SmartNIC, resulting in fast processing. No state bookkeeping is required because the key that maps the packet to the logical data flow, the timestamp, and new burst flag are calculated in a separate node. Resource (SRAM and TCAM) consumption in the load balancer is independent of the number of keys (i.e., flows / users) to be handled so the solution is cost efficient. Further, physical implementation of the load balancing logic is hidden from both NF-CP and NF-DP, thus enabling a cloud-native data plane realization with auto-scaling properties of NF-DP.

[0040] Figure 1 illustrates a typical deployment of a load balancer 22 in a communication network 10 where the same service is provided by a plurality of virtual NFs (VNFs) or containerized NFs (CNFs) in a cloud-based network infrastructure. One advantage of NF virtualization (NFV) is the ability to rapidly scale network resources responsive to variations in traffic. The load balancer 22 as herein described is designed to adapt to scaling of network resources while maintaining consistency in flow handling and avoiding packet re-ordering.

[0041] The communication network 10 comprises a core network 12 including plurality of NF instances 14, a network management system (NMS) 16 to manage network resources, and a load balancer 22 to distribute incoming packets among the available NF instances 14. The NF instances 14 may, for example, comprise virtual application servers ASs in a communication network, VNFs or CNFs serving, for example, as virtual user plane functions (UPFs) in a core network of a wireless communication network, or virtual radio access network (RAN) nodes in a wireless communication network. The NMS 16 comprises one or more network nodes in a management domain of the communication network 10 whose job is to manage the network resources. The NMS 16 monitors performance of the NF instances 14 and manages network resources to ensure that service level agreements (SLAs) are being met. The load balancer 22 receives the incoming network traffic and maps the network flows to individual NF instances 14 managed by the NMS 16 so that the load is balanced across the deployed NF instances 14.

[0042] The load balancer 22 includes a control plane component (LB-CP) 24 and a data plane component (LB-DP) 26, which can be implemented in a hardware-constrained switch. LB-DP 26 includes two registers denoted as REG1 and REG2, an InstanceAddress table, and an InstanceGroupTable. These tables are updated by LB- CP 24 when the resource group is reconfigured. A packet marker 28 upstream of the data plane component marks packets with a key identifying a traffic flow to which the packet belongs and a timestamp indicating the starting time of a packet burst to which the packet belongs.

[0043] In a typical deployment, multiple NF instances 14 providing the same service are deployed in the communication network 10. The group of NF instances 14 providing the same service is referred to herein as a resource group or NF instance set. The membership of this resource group can change over time as the total amount of the network traffic varies. During periods of high traffic flows, the resource group can be scaled up by adding additional NF instances 14 to the resource group. As the amount of the network traffic drops, NF instances 14 can be removed to free up network resources and conserve power. The resource group also changes when NF instances 14 are removed for service for maintenance or upgrading. Typically, changes in the resource group are not too frequent (usually on a time scale of minutes or hours)

[0044] The NF instances 14 may have static states, dynamic states, or both. Static states remain constant for long periods of time while dynamic states may change on a per-packet basis. The dynamic state of an NF instance 14 may be stored in register-like constructs in NF data plane (DP) implementations.

[0045] A resource monitor 18 in the NMS 16 monitors performance of the NF instances 14 in a resource group and provides monitoring data and notifications to a resource manager 20. Based on this information provided by the resource monitor 18, the resource manager 20 may decide to add new NF instances 14 to a resource group, remove a NF instance 14 from a resource group, or replace an NF instance 14 in the resource group. The details of the NMS 16 are not material to the present disclosure. Any existing auto-scaling approach can be used in addition to manual scaling. When scaling is needed, the NMS 16 sends the new configuration to the load balancer 22 and launches new NF instances 14 or stops unused NF instances 14 using existing application programming interfaces (APIs), such as virtualized infrastructure management (VIM) on top of a Kubernetes (K8s) cluster running CNFs.

[0046] The load balancer 22 receives incoming packets and, based on information in the packet header, distributes the incoming packets to one of the NF instances 14 in the resource group. To minimize cache invalidations and packet reordering, packets associated with a single traffic flow are directed to the same NF instance 14. A traffic flow may comprise any aggregation of packets in one or more data streams grouped together for similar treatment. When packets associated with a new traffic flow are detected, the load balancer 22 selects a NF instance 14 to serve the traffic flow and stores the flow parameters in a flow tracking table. In this way it is possible to combine the advantages of randomness and consistency at the price of increased memory footprint.

[0047] Each incoming packet arriving at the load balancer 22 is tagged with a key associated with a specific traffic flow. As one example, the packets having the same key may belong to the same packet data unit (PDU) session, the same Quality of Service (QoS) flow, or the same user equipment (UE). The key serves as a flow identifier. The flow tracking table stores associations between specific keys, and specific NF instances 14, which can be identified by a network address (e.g. Internet Protocol (IP) address), label, or other NF identifier (NF ID). As an example, the flow tracking table may store the key associated with a traffic flow and the IP address of a NF instance 14 to which the traffic flow is assigned. This table is updated when a traffic flow is reassigned to a new NF instance 14. When a new packet arrives, the load balancer 22 maps the key of the incoming packet to a corresponding NF instance 14 that is serving the associated traffic flow and forwards the packet to the selected NF instance 14.

[0048] A traffic flow will typically have active phases and passive phases. In an active phase, the packet stream is continuous (meaning without large interruptions), while in a passive phase, the traffic intensity is low (meaning with large interruptions) or zero. In some embodiments, the packets arriving during an active phase are handled by the same NF instance 14.

[0049] When the membership of the resource group changes, there may be a need to redirect packets from one NF instance 14 to another, i.e., when the traffic flow is reassigned to a different NF instance 14. In one embodiment of the present disclosure, packets in a traffic flow are not redirected during an active phase of the traffic flow. Rather, the load balancer 22 waits for a passive phase and redirects the packets in the traffic flow when a new packet burst is detected following a silence period. As used herein, a packet burst is considered to be all of the packets belonging to the same traffic flow received in one active phase, i.e., between consecutive silence periods.

[0050] Figure 2 is a flow chart illustrating one method 100 of forwarding and redirecting a traffic flow implemented by the LB-DP 26. It is assumed that each packet is tagged with a key representing the traffic flow to which it belongs and a timestamp (ts_burst) that denotes the start time of the latest active phase of the traffic flow. In some embodiments, the packets at the start of a new burst can also be marked with a new burst flag.

[0051] The distribution of traffic by LB-DP 26 uses the InstanceAddress table and the InstanceGroup table. These tables represent the current configuration of the resource group and are updated by the LB-CP 24 as will be hereinafter described. The InstanceGroup table stores the instance group identifier (IGID) and instance group size (IGsize) for any given timestamp. IGID identifies a configuration of the resource group and IGsize indicates the number of NF instances in the resource group. A new entry is added to the InstanceGroup table each time the configuration of the resource group changes. The InstanceAddress table stores a mapping of the IGid and instance number iid to an NF address. The instance number iid indicates a specific NF instance in the resource group.

[0052] REG1 and REG2 are register arrays of size N. Each entry in REG1 stores a key for a particular traffic flow. Each entry in REG2 stores an address of an NF instance associated with a key. The size of the registers are constrained so the number of traffic flows may exceed the size of the registers. In this case, the keys contained in the incoming packets are hashed and the hash value is used as an index to lookup up corresponding entries in the registers. Therefore, collisions may occur where two different keys map to the same entry in the register. The implementation described accounts for such collisions, which should be infrequent.

[0053] Returning to Figure 2, the method 100 begins when a packet arrives at the load balancer 22 (block 105). LB-DP 26 initially determines the resource group configuration that was active at the start of the current packet burst (i.e. , current active phase) by invoking the GetJnstanceGroup function to look up IGid and IGsize for the current set of NF instances, i.e., the resource group configuration. The GetJnstanceGroup function takes the timestamp (tjjurst) of the packet as an input and selects an entry in the InstanceGroup table using longest prefix matching (block 1 10). This operation requires ternary content-addressable memory (TCAM) memory space in the LB-DP 26. It should be noted that a Tofino ASIC can implement longest prefix matching by range matches defining time intervals for the instance groups. This implementation of the lookup table assumes time synchronization among all the nodes in the system. Because resource groups for a set of network NF instances 14 change rarely, synchronization in the order of seconds should be sufficient for implementing the InstanceGroup table. The LB-DP 26 also hashes the key contained in the incoming packet to the range 0 - IGsize -1 to obtain an instance identifier (iid) for the NF instance 14 in the instance set assigned to the traffic flow (block 115). LB-DP 26 then invokes Get nstanceAddress with IGid and iid as arguments to get the address of the NF instance assigned to the traffic flow (block 120).

[0054] Before the packet is forwarded, LB-DP 26 checks whether the packet belongs to a new burst (block 125). There are a number of ways of detecting a new burst. In one embodiment, the new burst can be indicated by a new burst flag inserted by the packet marker 28. In other embodiments, the new burst can be detected by comparing the timestamp (tjjurst) in the packet to a time threshold denoted DELTA. It is assumed that NF scaling happens rarely on the time scales of seconds (or at least hundreds of milliseconds) and that reaction time is flexible. On this time scale, it can be assumed that the clocks are synchronized (light synchronization requirement). If the timestamp denotes the beginning of a burst and it is new, ts_burst and now() will be close to each other. A new burst is assumed when now() - ts_burst < DELTA. DELTA is a parameter of the load balancing system and can be tuned according to the actual traffic conditions, topology, etc. DELTA is used to determine whether the packet belongs to the beginning of a burst. The benefit of this approach is that it does not require extra states to be stored in the data plane

[0055] If the packet arriving does not belong to a new burst, LB-DP 26 forwards the packet to the NF instance at iaddr (block 155). If the packet belongs to the beginning of a new packet burst, i.e., after a silence period, the LB-DP 26 checks whether a collision has occurred between keys assigned to different traffic flows and whether the NF instance assigned to the key has changed (block 130, 150). To check for a collision, the LB-DP 26 compares the key contained in the received packet to the key stored at REG1 [Hash(key)] (block 130). Note that the check for a collision can be omitted where there is sufficient space in the register to avoid collisions. If the received key is different, the LB-DP 26 assumes that a collision has occurred and sets REG1 [Hash(key)] equal to the key contained in the received packet (block 135). Next, the LB-DP 26 sets REG2[Hash(key)] equal to iaddr (block 140). After REG1 and REG2 are updated, the LB-DP 26 sends a notification to the LB-CP 24 (block 145). To check whether the NF instance 14 assigned to the traffic flow has changed, the LB-DP 26 compares iaddr obtained at block 120 with the value stored at REG2[Hash(key)]. If the key is the same, but iaddr is not equal to REG2[Hash(key)], the LB-DP 26 assumes that the traffic flow has been reassigned to a different NF instance 14, sets REG2[Hash(key)] equal to iaddr (block 140) and notifies the LB-CP 24 (block 145). The notification sent LB-CP 24 includes the key k received in the packet, which needs to be removed from the NF instances 14 in the resource group no longer assigned to the traffic flow associated with key k. The notification may further include the iaddr for the current serving NF instance 14. Updating the registers REG1 and REG2 as described above prevents the LB-DP 24 from sending more than one new burst notification per new packet burst. After these checks are performed, and the notification, if any, is sent, the LB-DP 26 forwards the packet to the NF instance at iaddr (block 155). It should be noted that the size of the data structures used by the LB-DP 26 are independent of the number of possible keys, resulting in good scalability and low resource consumption in the LB-DP 26. The LB-CP 24 controls the internal state of the LB-DP 26. There are two aspects to the LB-CP 24 operation. First, the LB-CP 24 plays a role in scaling up or scaling down NF instances 14 in the resource group. This aspect includes migration of states from source NF instances 14 to target NF instances in the new reconfigured resource group and managing the internal state of DP 26 responsive to a reconfiguration of the resource group. Second, the LB-CP 24, removes or stops inactive NF instances 14 responsive to notifications from the LB-DP 26.

[0056] Figure 3 illustrates a method 200 of reconfiguring a resource group performed by the LB-CP 24. Note that method 200 can be performed during an active phase of a traffic flow without impacting the forwarding of packets in the current active phase. The method 300 begins when LB-CP 24 receives a new configuration from the resource manager 20 (block 205). The LB-CP 24 creates a new group identifier, IGid, for the reconfigured resource group (block 210). Based on the new configuration, the LB-CP 24 launches new instances (scaling up case) or mark existing NF instances 14 with a “to- be-removed" flag (scaling down case) (block 215). Following this reconfiguration, the LB-CP 24 begins static state migration (block 220). The LB-CP 24 iterates through all active keys k and, for each key k, determines whether the NF instance 14 has changed as a result of the reconfiguration. If so, the LB-CP 24 migrates the state information for the previous NF instance 14 assigned to key k, referred to as the source NF instance 14, to the new NF instance assigned to key k, referred to as the target NF instance 14. After processing each key, the LB-CP 24 checks whether the state migration is complete (block 225). If not, the LB-CP 24 goes to the next key k and repeats the state migration process until the last key is reached.

[0057] When the state migration is complete, the LB-CP 24 loads the instance group identifier IGid into the InstanceGroup table and associates the IGid with a timestamp set equal to the current time, denoted now () (block 230) Note that each NF instance 14 in the reconfigured resource group is associated with an instance identifier iid arrived at by hashing the key k of the traffic flow and is uniquely identified by the tuple [IGid, iid]. The LB-CP 24 also loads the address iaddr for each NF instance 14 into the InstanceAddress table and associates the iaddr for each NF instance 14 with a corresponding tuple [IGid, iid] (block 235). The LB-DP 26 uses this tale to determine the forwarding address for packets in a packet flow. After updating the table used by LB-DP 26, the reconfiguration process ends (block 240) As previously noted, the LB-DP 26 performs longest prefix matching on the timestamp field in the InstanceGroup table so the LB-DP will continue to forward packets to the previous NF instance 14 until a new packet burst is detected. The packets in the new packet burst will include a different timestamp that will match the recently created entry in the InstanceGroup table so the pair [IGid , iid] will change and the LB-DP 26 will send packets in the new packet burst to the NF instance [IGid, iid] in the new resource group assigned to the traffic flow k.

[0058] Figure 4 is a flow chart showing a method 300 implemented by the LD-CP 24 for removing inactive NF instances 14 from the resource group following the detection of a new packet burst for key k by the LB-DP 26. The method 300 begins when the LD-CP 24 receives a notification from the LD-CP 26 that a new packet burst was detected for key k (block 305). At this point, the InstanceGroup and InstanceAddress tables have already been updated so the LB DP 26 is already forwarding packets to the NF instance 14 in the reconfigured resource group assigned to key k. This NF instance 14 could be the same NF instance 14 previously assigned to key k or a different NF instance 14. Therefore, the LB-CP 24 checks whether the NF instance 14 assigned to key k has changed (block 310). If the NF instance 14 assigned to key k is unchanged, the process ends (block 330). If the NF instance 14 has changed, the LB-CP 24 removes key k from the previous NF instance 14, which could still be assigned to other keys k (block 315). After removing the key k, the LB-CP 24 checks whether the active set of keys k served by the NF instance 14 is empty and whether the NF instance 14 is marked for removal (block 320). If both conditions are true, meaning that the NF instance 14 is “inactive” or “idle”, the LB-DP 24 removes or stops the inactive or idle NF instance 14 (block 325) and the process ends (block 330). If the NF instance 14 is serving other keys k or is not marked for removal, the process ends (block 330).

[0059] Figure 5 is a flow chart showing a method 400 implemented by the LB-DP 26 in a load balancer 22 configured to distribute packets among a plurality of NF instances 14 in a resource group. The LB-DP 26 receives a packet belonging to a traffic flow (block 410). The packet includes a header containing a flow identifier for the traffic flow and a timestamp associated with a start time of a packet burst within the traffic flow to which the packet belongs. Based on the flow identifier and timestamp, the LB-DP 26 directs the packet to a serving NF instance 14 for the traffic flow (block 420). The LB-DP 26 sends a notification to the LB-CP 24 of the load balancer 22 when the packet burst is a determined to be new packet burst and the serving NF instance 14 has changed (block 430).

[0060] In some embodiments of method 400, the timestamp represents a start time of the packet burst.

[0061] In some embodiments of method 400, directing, based on the flow identifier and the timestamp, the packet to a serving NF instance 14 for the traffic flow comprises determining an address for the NF instance 14 based on the flow identifier and the timestamp.

[0062] The method of claim 1 or 2, wherein determining an address for the NF instance 14 based on the flow identifier and the timestamp comprises determining, based on the timestamp, a group identifier for a resource group comprising a plurality of NF instance, determining an instance identifier for a NF instance in the resource group based on the flow identifier, and determining the address for the serving NF instance based on the group identifier and the instance identifier.

[0063] In some embodiments of method 400, determining the group identifier for a resource group based on the timestamp comprises using the timestamp as an index to look up the group identifier in a resource group table.

[0064] In some embodiments of method 400, determining an instance identifier for a NF instance in the resource group based on the flow identifier comprises generating the instance identifier by applying a hashing function to the flow identifier.

[0065] In some embodiments of method 400, determining an address for the NF instance based on the group identifier and the instance identifier comprises using the group identifier and instance identifier as an index to look up the instance address in an instance address table.

[0066] In some embodiments of method 400, sending a notification to the control plane of the load balancer comprises sending a notification to the LB-CP 24 including the flow identifier.

[0067] Some embodiments of method 400 further comprise sending a further notification to the LB-CP 24 when the packet burst is a determined to be a new packet burst and a collision between two or more flow identifiers is detected.

[0068] Some embodiments of the method 400 further comprise updating an internal register to indicate that the notification or further notification was sent.

[0069] Figure 6 is a flow chart showing a method 500 implemented by a LB-CP 24 in a load balancer 22 configured to control the distribution packets among a plurality of NF instances 14 in a resource group. The LB-CP 24 receives a reconfiguration message indicating a change in the configuration of a resource group including one or more NF instances 14 (block 510). Responsive to the reconfiguration message, the LB-CP 24 in the load balancer 22 migrates state information for one or more traffic flows being distributed by the load balancer 22 from a first NF instance 14 to a second NF instance 14 in the reconfigured resource group (block 520). The LB-CP 24 further updates an internal configuration in the LB-DP 26 to map the traffic flows being distributed by the load balancer 22 to NF instances 14 in the reconfigured resource group (block 530).

[0070] In some embodiments of method 500, updating an internal configuration in the LB-DP 26 comprises creating a group identifier for the reconfigured resource groups and associating the group identifier with time information. The time information may, for example, indicate a present time at the time the reconfiguration is performed.

[0071] In some embodiments of method 500, updating an internal configuration in the LB-DP 26 of the load balancer 22 further comprises updating a first data structure (e.g., InstanceGroup table) in the LB-DP 26 to include the group identifier for the reconfigured resource group and updating a second data structure (e.g., InstanceAddress table) in the LB-DP 26 to include addresses for NF instances in the reconfigured resource group.

[0072] In some embodiments of method 500, updating a second data structure further comprises associating each of the NF instances 14 in the reconfigured resource group with a tuple comprising the group identifier and an instance identifier for a NF instance in the reconfigured resource group.

[0073] Some embodiments of method 500 further comprise receiving a notification from the LB-DP 26 that a new packet burst for a traffic flow has started (block 540).

[0074] In some embodiments of method 500, the LB-CP 24 triggers, responsive to the notification, the removal of the flow identifier for the traffic flow from one or more network function instances 14 no longer serving the traffic flow (block 550).

[0075] In some embodiments of method 500, the LB-CP 24 triggers the removal of an inactive NF instance 14 from the reconfigured resource group following the reconfiguration when it is determined that the NF instance 14 is no longer serving any traffic flows (block 560).

[0076] Figure 7 is a flow chart showing a method 600 implemented by a packet marker 28 in a communication network upstream from the load balancer 22 as herein described. The packet marker 28 receives a packet belonging to a packet burst within the traffic flow (block 610). The packet marker adds a timestamp to a header of the packet indicating a starting time of the packet burst (block 620).

[0077] In some embodiments of method 600, the packet marker 28 adds flow identifier for the traffic flow to the header of the packet (block 630).

[0078] In some embodiments of method 600, the packet marker 28 further adds a new burst indicator to the packet when the packet is determined to be the first packet in the packet burst (block 640).

[0079] An apparatus can perform any of the methods herein described by implementing any functional means, modules, units, or circuitry. In one embodiment, for example, the apparatuses comprise respective circuits or circuitry configured to perform the steps shown in the method figures. The circuits or circuitry in this regard may comprise circuits dedicated to performing certain functional processing and / or one or more microprocessors in conjunction with memory. For instance, the circuitry may include one or more microprocessor or microcontrollers, as well as other digital hardware, which may include Digital Signal Processors (DSPs), special-purpose digital logic, and the like. The processing circuitry may be configured to execute program code stored in memory, which may include one or several types of memory such as read-only memory (ROM), random-access memory, cache memory, flash memory devices, optical storage devices, etc. Program code stored in memory may include program instructions for executing one or more telecommunications and / or data communications protocols as well as instructions for carrying out one or more of the techniques described herein, in several embodiments. In embodiments that employ memory, the memory stores program code that, when executed by one or more processors, carries out the techniques described herein.

[0080] Figure 8 illustrates an exemplary control plane entity 700 of a load balancer 22 in a communication network configured to distribute packets among a set of network function instances 14 in a resource group. The control plane entity 700 generally comprises input / output (I / O) circuitry 710 for receiving and sending packets, processing circuitry 720, and memory 730.

[0081] The I / O circuitry 710 includes an input 712 for receiving incoming packets and an output 714 for sending packets to the NF instances 14 in a communication network.

[0082] The processing circuitry 720 is configured to distribute packets according to marking in the packets and its internal configuration. The processing circuit 720 may comprise one or more microprocessors, hardware, firmware, or a combination thereof. The processing circuitry 720 in one embodiment is configured to perform the methods herein described. According to an embodiment, the processing circuitry 720 comprises a receiving module 722 configured to receive a packet belonging to traffic flow, the packet including a header containing a flow identifier for the traffic flow and a timestamp associated with a packet burst within the traffic flow to which the packet belongs, a forwarding module 724 configured to direct, based on the flow identifier and the timestamp, the packet to a serving NF instance for the traffic flow, and a notification module 726 configured to send a notification to the control plane of the load balancer when the packet burst is a determined to be new packet burst and the serving NF instance has changed.

[0083] Memory 730 comprises both volatile and non-volatile memory for storing computer program code and data needed by the processing circuit 720 for operation. Memory 730 may comprise any tangible, non-transitory computer-readable storage medium for storing data including electronic, magnetic, optical, electromagnetic, or semiconductor data storage. Memory 730 stores a computer program 740 comprising executable instructions that configure the processing circuit 230 to implement the perform the methods herein described. A computer program 740 in this regard may comprise one or more code modules corresponding to the means or units described above. In general, computer program instructions and configuration information are stored in a non-volatile memory, such as a ROM, erasable programmable read only memory (EPROM) or flash memory. Temporary data generated during operation may be stored in a volatile memory, such as a random access memory (RAM). In some embodiments, computer program 740 for configuring the processing circuitry 720 as herein described may be stored in a removable memory, such as a portable compact disc, portable digital video disc, or other removable media. The computer program 740 may also be embodied in a carrier such as an electronic signal, optical signal, radio signal, or computer readable storage medium.

[0084] Figure 9 illustrates an exemplary control plane entity 800 of a load balancer 22 in a wireless communication network configured to control distribution of packets by the load balancer 22. The control plane entity 800 generally comprises network interface circuitry 810, processing circuitry 820, and memory 830.

[0085] The I / O circuitry 810 comprises circuitry for coupling the load balancer 800 to a communication network and enables communication with one or more network nodes in a communication network. The processing circuitry 820 controls the overall operation of the load balancer 800 and configured the LB-CP 26 to distribute packets. The processing circuit 820 may comprise one or more microprocessors, hardware, firmware, or a combination thereof. The processing circuitry 820 in one embodiment is configured to perform the methods herein described. According to an embodiment, the processing circuitry 820 comprises a receiving module 822 configured to receive a reconfiguration message indicating a change in the configuration of a resource group including one or more NF instances, a migration module 824 configured to migrate, responsive to the reconfiguration message, state information for one or more traffic flows being distributed by the load balancer from a first NF instance to a second NF instance in the reconfigured resource group, and an updating module 826 configured to update an internal configuration in a data plane of the load balancer to map the traffic flows being distributed by the load balancer to corresponding NFs instances in the reconfigured resource group.

[0086] Memory 830 comprises both volatile and non-volatile memory for storing computer program code and data needed by the processing circuit 820 for operation. Memory 830 may comprise any tangible, non-transitory computer-readable storage medium for storing data including electronic, magnetic, optical, electromagnetic, or semiconductor data storage. Memory 830 stores a computer program 840 comprising executable instructions that configure the processing circuit 820 to implement the perform the methods herein described. A computer program 840 in this regard may comprise one or more code modules corresponding to the means or units described above. In general, computer program instructions and configuration information are stored in a non-volatile memory, such as a ROM, erasable programmable read only memory (EPROM) or flash memory. Temporary data generated during operation may be stored in a volatile memory, such as a random access memory (RAM). In some embodiments, computer program 840 for configuring the processing circuitry 820 as herein described may be stored in a removable memory, such as a portable compact disc, portable digital video disc, or other removable media. The computer program 840 may also be embodied in a carrier such as an electronic signal, optical signal, radio signal, or computer readable storage medium.

[0087] Figure 10 illustrates an exemplary packet marker 900 configured to mark packets for load balancing. The packet marker 900 generally comprises input / output (I / O) circuitry 910 for receiving and sending packets, processing circuitry 920, and memory 930. The I / O circuitry 910 comprises circuitry including an input 912 for receiving incoming packets and an output 914 for sending packets toward the load balancer 22.

[0088] The processing circuitry 920 controls the overall operation of the packet marker 900. The processing circuit 920 may comprise one or more microprocessors, hardware, firmware, or a combination thereof. The processing circuitry 920 in one embodiment is configured to perform the methods herein described. According to an embodiment, the processing circuitry 920 comprises a receiving module 922 configured to receive a packet belonging to a packet burst within the traffic flow, and a timestamp module 924 configured to add a time stamp to a header of the packet indicating a starting time of the packet burst to which the packet belongs. Some embodiments may further include a flow ID module 926 configured to add a flow identifier for the traffic flow to the header of the packet and a new burst module 928 configured to add a new burst indication to the first packet in a burst.

[0089] Memory 930 comprises both volatile and non-volatile memory for storing computer program code and data needed by the processing circuit 920 for operation. Memory 930 may comprise any tangible, non-transitory computer-readable storage medium for storing data including electronic, magnetic, optical, electromagnetic, or semiconductor data storage. Memory 930 stores a computer program 940 comprising executable instructions that configure the processing circuit 920 to implement the perform the methods herein described. A computer program 940 in this regard may comprise one or more code modules corresponding to the means or units described above. In general, computer program instructions and configuration information are stored in a non-volatile memory, such as a ROM, erasable programmable read only memory (EPROM) or flash memory. Temporary data generated during operation may be stored in a volatile memory, such as a random access memory (RAM). In some embodiments, computer program 940 for configuring the processing circuitry 920 as herein described may be stored in a removable memory, such as a portable compact disc, portable digital video disc, or other removable media. The computer program 940 may also be embodied in a carrier such as an electronic signal, optical signal, radio signal, or computer readable storage medium.

[0090] Those skilled in the art will also appreciate that embodiments herein further include corresponding computer programs. A computer program comprises instructions which, when executed on at least one processor of an apparatus, cause the apparatus to carry out any of the respective processing described above. A computer program in this regard may comprise one or more code modules corresponding to the means or units described above.

[0091] Embodiments further include a carrier containing such a computer program. This carrier may comprise one of an electronic signal, optical signal, radio signal, or computer readable storage medium.

[0092] In this regard, embodiments herein also include a computer program product stored on a non-transitory computer readable (storage or recording) medium and comprising instructions that, when executed by a processor of an apparatus, cause the apparatus to perform as described above.

[0093] Embodiments further include a computer program product comprising program code portions for performing the steps of any of the embodiments herein when the computer program product is executed by a computing device. This computer program product may be stored on a computer readable recording medium.

[0094] Figure 1 1 illustrates both a logical and physical view of the load balancer 22 implemented in a cloud infrastructure. Assuming that a network service is described as an execution graph (or a chain) of different NF instances 14. In such a system, the logical view of a NF is simple as shown on the left in Figure 1 1 . The NF control plane (NF-CP) sees a single instance of the NF data plane (NF-DP), and manages the various data plane objects (e.g., match-action tables, etc.). However, an NF-DP implementation may have performance constraints. For example, assume that the throughput demand is increasing and additional instances of the same NF-DP are needed to satisfy the performance requirements. Conventional implementation of load balancing may require the extension of either the NF-CP or NF-DP, or both. In the disclosed embodiment, the implementation of the load balancing logic is hidden from both the NF-CP and NF-DP. The LB-CP 24 acts as mediator between the NF-CP and the instances of NF-DP and coordinates the state migration and the traffic forwarding between the instances in a seamless way.

Claims

CLAIMSWhat is claimed is:1 . A method (400) implemented in a data plane (26, 700) of a load balancer (22) configured to distribute packets among a plurality of network function instances in a resource group, the method (400) comprising: receiving (410) a packet belonging to traffic flow, the packet including a header containing a flow identifier for the traffic flow and a timestamp associated with a packet burst within the traffic flow to which the packet belongs; directing (420), based on the flow identifier and the timestamp, the packet to a serving network function instance for the traffic flow; and sending (430) a notification to the control plane (24, 800) of the load balancer (22) when the packet burst is a determined to be new packet burst and the serving network function instance has changed.

2. The method (400) of claim 1 , wherein the timestamp represents a start time of the packet burst.

3. The method (400) of claim 1 or 2, wherein directing, based on the flow identifier and the timestamp, the packet to a serving network function instance for the traffic flow comprises determining an address for the network function instance based on the flow identifier and the timestamp.

4. The method (400) of claim 3, wherein determining an address for the network function instance based on the flow identifier and the timestamp comprises: determining, based on the timestamp, a group identifier for a resource group comprising a plurality of network function instances; determining an instance identifier for a network function instance in the resource group based on the flow identifier; and determining the address for the serving network function instance based on the group identifier and the instance identifier.

5. The method (400) of claim 4, wherein determining the group identifier for a resource group based on the timestamp comprises using the timestamp as an index to look up the group identifier in a resource group table.

6. The method (400) of claim 4 or 5, wherein determining an instance identifier for a network function instance in the resource group based on the flow identifier comprises generating the instance identifier by applying a hashing function to the flow identifier.

7. The method (400) of any one of claims 4 - 6, wherein determining an address for the network function instance based on the group identifier and the instance identifier comprises using the group identifier and instance identifier as an index to look up the instance address in an instance address table.

8. The method (400) of any one of claims 1 - 7, wherein sending a notification to the control plane (24, 800) of the load balancer (22) comprises sending a notification to the control plane entity including the flow identifier.

9. The method (400) of any one of claims 1 - 8, further comprising sending a further notification to the control plane (24, 800) of the load balancer (22) when the packet burst is a determined to be new packet burst and a collision between two or more flow identifiers is detected.

10. The method (400) of claim 8 or 9, further comprising updating an internal register to indicate that the notification or further notification was sent .

11. A method (500) implemented in a control plane (24, 800) for a load balancer (22) configured to distribute packets among a plurality of network function instances in a resource group, the method (500) comprising: receiving (510) a reconfiguration message indicating a change in the configuration of a resource group including one or more network function instances; responsive to the reconfiguration message, migrating (520) state information for one or more traffic flows being distributed by the load balancer (22) from a first network function instance to a second network function instance in the reconfigured resource group; updating (530) an internal configuration in a data plane (26, 700) of the load balancer (22) to map the traffic flows being distributed by the load balancer (22) to corresponding NFs instances in the reconfigured resource group.

12. The method (500) of claim 11 , wherein updating an internal configuration in a data plane (26, 700) of the load balancer (22) to map the traffic flows being distributed by the load balancer (22) to corresponding NFs instances in the reconfigured resource group comprises: creating a group identifier for the reconfigured resource group and associating the group identifier with time information.

13. The method (500) of claim 12, wherein updating an internal configuration in a data plane (26, 700) of the load balancer (22) to map one or more traffic flows to be served by the reconfigured resource group to NFs instances in the reconfigured resource group comprises: updating a first data structure in the data plane (26, 700) of the load balancer (22) to include the group identifier for the reconfigured resource group and the associated time information; and updating a second data structure in the data plane (26, 700) of the load balancer (22) to include addresses for network function instances in the reconfigured resource group.

14. The method (500) of claim 12 or 13, wherein updating the second data structure further comprises associating each of the network function instances in the reconfigured resource group with a tuple comprising the group identifier and an instance identifier for a network function instance in the reconfigured resource group.

15. The method (500) of claim 11 , further comprising: receiving notification from the data plane (26, 700) of the load balancer (22) that a new packet burst for a traffic flow has started; responsive to the notification, triggering the removal of the flow identifier for the traffic flow from one or more function instance no longer serving the traffic flow.

16. The method (500) of any one of claims 1 1 - 16, further comprising: triggering the removal of an inactive network function instance from the reconfigured resource group following the reconfiguration when it is determined that the NF instance is no longer serving any traffic flows.

17. A method (600) implemented by a packet marker for marking packets in the traffic flow for load balancing, the method (600) comprising: receiving (610) a packet belonging to a packet burst within the traffic flow; adding (620) a time stamp to a header of the packet indicating a starting time of the packet burst to which the packet belongs.

18. The method (600) of claim 15 further comprising: adding (630) a flow identifier for the traffic flow to the header of the packet.

19. The method (600) of claim 15 or 16 further comprising: adding (640) a flow identifier associated with the traffic flow to the header of the packet.

20. A data plane entity (26, 700) for a load balancer (22) in a communication network for distributing packets among a plurality of network function instances in a resource group, the data plane entity (26, 700) being configured to: receive a packet belonging to traffic flow, the packet including a header containing a flow identifier for the traffic flow and a timestamp associated with a packet burst within the traffic flow to which the packet belongs; direct, based on the flow identifier and the timestamp, the packet to a serving network function instance for the traffic flow; and send a notification to the control plane of the load balancer when the packet burst is a determined to be new packet burst and the serving network function instance has changed.21 . The data plane entity (26, 700) of claim 20, further configured to perform the method of any one of claims 2 - 10.

22. A data plane entity (26, 700) for a load balancer (22) in a communication network for distributing packets among a plurality of network function instances (14), the data plane entity (26, 700) comprising: input / output circuitry (710) for receiving and sending packets; and processing circuitry (720) configured to:receive a packet belonging to traffic flow, the packet including a header containing a flow identifier for the traffic flow and a timestamp associated with a packet burst within the traffic flow to which the packet belongs; direct, based on the flow identifier and the timestamp, the packet to a serving network function instance for the traffic flow; and send a notification to the control plane of the load balancer when the packet burst is a determined to be new packet burst and the serving network function instance has changed.

23. The data plane entity (26, 700) of claim 22 wherein the processing circuitry (720) is further configured to perform the method of any one of claims 2 - 10.

24. A computer program comprising executable instructions that, when executed by processing circuitry (720) in a data plane entity (26, 700) of a load balancer (22) in a communication network, causes it to perform the method of any one of embodiments 1 - 10.

25. A carrier containing a computer program (740) of claim 24, wherein the carrier is one of an electronic signal, optical signal, radio signal, or computer readable storage medium (730).

26. A non-transitory computer-readable storage medium (730) containing a computer program comprising executable instructions that, when executed by processing circuitry (720) in data plane entity (26, 700) of load balancer (22) in a communication network causes it to perform the methods of any one of embodiment 1 - 10.

27. A control plane entity (24, 800) for a load balancer (22) in a communication network for controlling the distribution of packets among a plurality of network function instances (14), the control plane entity (24, 800) being configured to: receive a reconfiguration message indicating a change in the configuration of a resource group including one or more network function instances; responsive to the reconfiguration message, migrate state information for one or more traffic flows being distributed by the load balancer from a first network function instance to a second network function instance in the reconfigured resource group;update an internal configuration in a data plane of the load balancer to map the traffic flows being distributed by the load balancer to corresponding NFs instances in the reconfigured resource group.

28. The control plane entity (24, 800) of claim 27 further configured to perform the method of any one of claims 12 - 16.

29. A control plane entity (24, 800) for a load balancer (22) in a communication network for controlling distribution of packets among a plurality of network function instances (14), the control plane entity (24, 800) comprising: network interface circuitry (810) for communication with one or more network nodes in a communication network; and processing circuitry (820) configured to: receive a reconfiguration message indicating a change in the configuration of a resource group including one or more network function instances; responsive to the reconfiguration message, migrate state information for one or more traffic flows being distributed by the load balancer from a first network function instance to a second network function instance in the reconfigured resource group; update an internal configuration in a data plane of the load balancer to map the traffic flows being distributed by the load balancer to corresponding NFs instances in the reconfigured resource group.

30. The control plane entity (24, 800) of claim 29 wherein the processing circuitry (820) is further configured to perform the method of any one of claims 12 - 16.31 . A computer program comprising executable instructions that, when executed by processing circuitry (820) in a control plane entity (24, 800) of load balancer (22) in a communication network, causes it to perform the method of any one of embodiments 1 1 - 16.

32. A carrier containing a computer program (840) of claim 29, wherein the carrier is one of an electronic signal, optical signal, radio signal, or computer readable storage medium (830).

33. A non-transitory computer-readable storage medium (830) containing a computer program comprising executable instructions that, when executed by processing circuitry (820) in a control plane entity (24, 800) of a load balancer (22) in a communication network causes it to perform the methods of any one of embodiment 1 1 - 16.

34. A packet marker (28, 900) in a communication network for marking packets for distribution by a load balancer (22), the packet marker (28, 900) being configured to: receive a packet belonging to a packet burst within the traffic flow; add a time stamp to a header of the packet indicating a starting time of the packet burst to which the packet belongs.

35. The packet marker (28, 900) of claim 34 wherein the processing circuitry (820) is further configured to perform the method of any one of claims 18 - 19.

36. A packet marker (28, 900) in a communication network for marking packets for distribution by a load balancer (22), the packet marker (28, 900) comprising: input / output circuitry (910) for receiving and sending packets; and processing circuitry (920) configured to: receive a packet belonging to a packet burst within the traffic flow; add a time stamp to a header of the packet indicating a starting time of the packet burst to which the packet belongs.

37. The packet marker (28, 900) of claim 36 further configured to perform the method of any one of claims 18 - 19.

38. A computer program comprising executable instructions that, when executed by processing circuitry (920) in a packet marker (28, 900) in a communication network, causes it to perform the method of any one of embodiments 17 - 19.

39. A carrier containing a computer program (940) of claim 38, wherein the carrier is one of an electronic signal, optical signal, radio signal, or computer readable storage medium (830).

40. A non-transitory computer-readable storage medium (930) containing a computer program comprising executable instructions that, when executed by processing circuitry (820) in a packet marker (28, 900)in a communication network causes it to perform the methods of any one of embodiment 17 - 19.

Images