Capacity-aware load packing for Layer 4 load balancers

Abstract

This disclosure relates to a method and system for load balancing traffic based on the capacity of Direct IP (DIP) instances. The method and system determines the capacity of each DIP using latency measurements from each DIP. The method and system uses an integer linear programming problem (ILP) to determine the weight of each DIP using the latency measurements. The weights determine the amount of traffic to be served by each DIP. The method and system provides the weights of each DIP to a load balancer controller and programs the load balancer data plane with these weights.

Description

[Background technology]

[0001] background

[0001] A Layer 4 load balancer (L4LB) is one of the main components of an online service. Individual services expose a small number of virtual IPs (called VIPs) to receive traffic from external services. Internally, services are scaled by running on multiple backend instances (hereinafter referred to as DIPs) of servers, each having its own direct IP. The LB receives traffic arriving at the VIP and distributes it across the DIPs. [Overview of the project]

[0002] overview

[0002] This summary is provided in a simplified form to introduce some of the concepts that will be further described in the following detailed description. This summary is not intended to identify the main or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0003]

[0003] Several implementations relate to methods. This method includes obtaining latency measurements for each of a plurality of direct IPs (DIPs) in a virtual network, wherein the latency measurements provide an indicator of the capacity of each DIP. This method includes using an integer linear programming problem (ILP) to calculate the weight of each DIP using the latency measurements. This method includes providing the weight of each DIP to a Layer 4 load balancer, wherein the weights identify the amount of traffic that each DIP will serve.

[0004]

[0004] Several implementations relate to a device. This device includes a processor, a memory that communicates electronically with the processor, and instructions stored in the memory, the instructions being executable by the processor to obtain latency measurements for each of a plurality of direct IPs (DIPs) in a virtual network, the latency measurements providing an indicator of the capacity of each DIP, to calculate the weight of each DIP using the latency measurements with an integer linear programming problem (ILP), and to provide the weight of each DIP to a Layer 4 load balancer, the weights providing an indicator of the amount of traffic that each DIP will serve.

[0005]

[0005] Several implementations relate to the method. This method includes determining a first weight subset for each of several direct IPs (DIPs) in a virtual network using latency measurements. This method includes obtaining latency measurements for the first weight subset for each DIP. This method includes generating a latency weight curve for each DIP using latency measurements for the first weight subset. This method includes calculating a first weight for each DIP using an integer linear programming problem (ILP), with the latency and weights from the latency weight curve for each DIP as input to the ILP, wherein the first weight is an estimate of the weight for each DIP. This method includes determining a second weight subset for each DIP using the first weight for each DIP. This method includes obtaining latency measurements for the second weight subset for each DIP. This method includes generating a second latency weight curve for each DIP using latency measurements for the second weight subset. This method involves using an ILP to calculate a second weight for each DIP, using the latency and weight from a second latency weight curve for each DIP as input to the ILP. This method includes providing the second weight for each DIP to a load balancer controller, which determines the amount of traffic that the second weight provides to each DIP.

[0006]

[0006] Some implementations relate to a device. This device includes a processor, a memory that electronically communicates with the processor, and instructions stored in the memory, the instructions being: to determine a first weight subset of each DIP of a plurality of direct IPs (DIPs) in a virtual network using latency measurements; to obtain latency measurements for the first weight subset of each DIP; to generate a latency weight curve for each DIP using latency measurements for the first weight subset; and to calculate the first weight of each DIP using an integer linear programming problem (ILP) with the latency and weights from the latency weight curve of each DIP as input to the ILP, wherein the first weight is an estimate of the weight of each DIP. The processor can perform the following: calculate, determine a second weight subset for each DIP using the first weight of each DIP, obtain latency measurements for the second weight subset of each DIP, generate a second latency weight curve for each DIP using the latency measurements for the second weight subset, calculate the second weight of each DIP using an ILP, with the latency and weights from the second latency weight curve of each DIP as input to the ILP, and provide the second weight of each DIP to the load balancer controller, wherein the second weight determines and provides the amount of traffic that each DIP will serve.

[0007]

[0007] Additional features and advantages are described below, some of which may become apparent from that description or may be acquired by practicing the teachings herein. The features and advantages of this disclosure can be realized and obtained by means and combinations particularly indicated in the appended claims. The features of this disclosure may become more apparent from the following description and the appended claims or may be acquired by practicing the disclosure as described below.

[0008] Brief explanation of the drawing

[0008] To illustrate how the above-mentioned features and other features of this disclosure can be obtained, a more specific explanation will be given by referring to the specific implementations shown in the accompanying drawings. For better understanding, similar elements are designated by the same reference numerals throughout the various accompanying drawings. Some of the drawings are schematic or exaggerated representations of the concept, but at least some of the drawings are proportional to the actual size. With understanding that the drawings show several exemplary implementations, these implementations will be described and explained in more specific and detailed terms using the accompanying drawings. [Brief explanation of the drawing]

[0009] [Figure 1A]

[0009] An example of an existing Layer 4 load balancer system having Direct Server Return (DSR) is shown. [Figure 1B]

[0010] An example of an existing proxy mode system for a Layer 4 load balancer is shown. [Figure 2]

[0011] This disclosure provides an example environment for a capacity-aware Layer 4 load balancer system implemented in this manner. [Figure 3]

[0012] This disclosure provides an exemplary method for calculating DIP weights using an integer linear programming (ILP) problem, based on the implementation of this disclosure. [Figure 4]

[0013] An example graph of the weights used for latency measurement in the implementation of this disclosure is shown. [Figure 5]

[0014] An example graph of the latency weight curve based on the implementation of this disclosure is shown. [Figure 6]

[0015] An example graph of weights calculated by an integer linear programming (ILP) problem using the implementation of this disclosure is shown. [Figure 7]

[0016] This disclosure provides an exemplary method for capacity-aware Layer 4 load balancing based on its implementation. [Figure 8]

[0017] This disclosure provides an exemplary method for calculating DIP weights used for load balancing using an integer linear programming (ILP) problem, based on the implementation of this disclosure. [Modes for carrying out the invention]

[0010] Detailed explanation

[0018] A Layer 4 load balancer (L4LB) is one of the primary components of online services. Individual services expose a small number of virtual IPs (called VIPs) to receive traffic from external services. Internally, services scale by running on multiple backend instances (referred to as DIPs below) of servers, each with its own direct IP. The LB receives traffic arriving at the VIP and distributes it across the DIPs.

[0011]

[0019] Online services deploy multiple servers with DIPs, primarily for scaling and providing high availability. DIPs operate behind a load balancer (LB) that exposes one or more virtual IPs (VIPs) to receive traffic from outside the service. The L4LB uses TCP / IP fields to split traffic between the DIPs. VIPs offer several advantages in terms of scalability, security, and high availability.

[0012]

[0020] Figure 1A shows an example of an existing Layer 4 load balancer system (L4LB) 100 with Direct Server Return (DSR). The L4LB system 100 operates on multiple instances (called MUX). The MUX intercepts VIP traffic from clients. Most public cloud providers use IP-in-IP encapsulation to route VIP packets to DIP. Packets in the reverse direction use Direct Server Return (DSR) to bypass the MUX.

[0013]

[0021] Figure 1B shows an example of an existing proxy mode system 102 for L4LB. Most third-party L4LB designs use proxy mode, where a TCP connection exists between the client and the MUX, and a corresponding connection exists between the MUX and the DIP. The MUX simply replicates the traffic between such connections. The MUX selects the DIP for new connections using different algorithms such as round-robin, hashing of TCP / IP fields, or least connection, with an emphasis on high-throughput and low-latency packet processing.

[0014]

[0022] Existing L4LB solutions typically assume that DIP capacity is uniform and / or static when distributing incoming traffic across DIPs. However, this assumption has many limitations at best and becomes untenable at worst (especially in virtualized clusters). In virtualized clusters, DIP capacity can change dynamically. One cause of dynamic DIP capacity changes is noise in the neighborhood. VMs on the same host compete for shared resources such as cache and memory buses, resulting in capacity fluctuations. In addition, customers may have DIPs with significantly different capacities within a DIP pool.

[0015]

[0023] In addition, dynamic oversubscription of physical CPU cores (i.e., the number of vCPUs sharing the same physical core changes dynamically according to customer demand) can also cause DIP capacity fluctuations. Reallocation of VMs to different parts of a service (for example, a service owner does not release VMs when not in use, but instead reallocates them to parts of the service) also results in DIPs with different capacities behind the same VIP. Online services using DIPs of different generations or SKUs also results in DIPs with different capacities.

[0016]

[0024] Existing solutions utilize multiple load balancing algorithms, including round-robin, hash-based, and least-connected algorithms. However, existing solutions cannot adapt to different or dynamic DIP capacities. Because existing solutions do not load balance according to DIP capacity, imbalances in CPU utilization occur, leading to increased latency for requests to overutilized DIPs.

[0017]

[0025] The method and system of this disclosure distribute the load of incoming VIP traffic according to the capacity of the DIP. The capacity of the DIP is the maximum throughput (e.g., requests per second) that the DIP can provide before at least one resource (e.g., CPU) becomes a bottleneck.

[0018]

[0026] The method and system of this disclosure model load balancing as a "bin packing" problem, taking a mapping of weights and latency as input to the "bin packing" problem, and using an algorithm combined with logical regression to intelligently and rapidly generate such a mapping. The method and system uses latency as a signal to solve the packing problem and adjusts the weights based on the latency. The method and system specifies weights for splitting traffic between DIPs.

[0019]

[0027] This method and system separates weight calculation from individual LB instances and calculates weights in a central controller. The advantages of calculating weights in a central controller include (a) freeing the LB data plane from weight calculation and supporting improved performance of the LB data plane by providing low latency and high throughput packet processing, and (b) supporting the implementation of a more informed design for weight calculation in the central controller.

[0020]

[0028] In some implementations, the method and system use integer linear programming (ILP) to calculate LB weights and minimize the total end-to-end latency between DIPs in a VIP. In some implementations, the method and system accelerate the construction of weight-to-latency mappings for individual DIPs by taking latency measurements with a small number of weights and estimating the latency of other weights using curve fitting. In some implementations, the method and system roughly estimate the capacity of a DIP by hopping between different weights to reduce the number of latency measurements. In some implementations, the method and system substantially accelerate weight calculation using multi-stage ILP. The method and system adapt to cluster changes such as capacity changes, traffic changes, and DIP failures.

[0021]

[0029] One of the technical advantages of the method and system disclosed is that by load balancing traffic according to the capacity of the DIPs, total latency is kept low and unused capacity is reduced. Another technical advantage of the method and system disclosed is that it minimizes the total latency of traffic across all DIPs of the VIP. Another technical advantage of the method and system disclosed is the ability to quickly calculate the load per backend instance.

[0022]

[0030] Another technical advantage of the method and system disclosed herein is that it requires no prior knowledge of the capacity of the backend instances. In addition, the method and system does not require any MUX, DIP, or agents running on the client. The method and system does not access DIP SKUs, CPUs, or any counters. The method and system receives the IP address of the DIP and intelligently collects information online using algorithms without requiring offline profiling.

[0023]

[0031] Another technical advantage of the methods and systems disclosed herein is their versatility. The methods and systems disclosed herein are compatible with existing Layer 4 LBs. The methods and systems utilize DIP capacity-dependent weighting to enhance the availability, scalability, and cost advantages of existing LB designs, enabling other LBs to recognize the capacity. The disclosure includes several practical applications that provide benefits and / or solve problems associated with load balancing VIP traffic.

[0024]

[0032] Referring to Figure 2, an exemplary environment 200 of a capacity-aware Layer 4 load balancer (L4LB) is shown. Environment 200 includes multiple clients 202 that send online service requests (traffic) to up to n online services (where n is a positive integer). Examples of online services include search engines, email services, video calls, retail services, and healthcare services. Each service exposes a VIP 206 to receive traffic from outside the service (traffic from clients 202). For example, VIP 12061 supports the search engine service, and VIP 22062 supports the email service.

[0025]

[0033] The service is scaled by running on multiple backend servers, each with its own Direct IP (DIP). One example is an online service using 100,000 servers (DIPs) to support services offered by an online service. VIP12061 contains up to x DIPs, where x is a positive integer. VIP22062 contains up to y DIPs, where y is a positive integer. Environment 200 contains up to n VIPs supporting different online services. For example, VIP n 206 n This includes up to z DIPs 22, where z is a positive integer. Requests from client 202 are received by the LB data plane 204 and provided to the VIP 206 of the service supported for that request.

[0026]

[0034] Environment 200 includes a CAPSENSE LB controller 210 that calculates the weight of each DIP 12, 18, and 22 at the central controller by evaluating latency measurements for each DIP 12, 18, and 22. The latency measurements are used by the CAPSENSE LB controller 210 to estimate the capacity of each DIP 12, 18, and 22. The weights are used to determine the amount of traffic to deliver to each DIP 12, 18, and 22 based on the capacity of each DIP 12, 18, and 22.

[0027]

[0035] CAPSENSE Latency Measurement (CLM) components 10, 16, and 20 are present in each VNET. CLM components 10, 16, and 20 periodically measure the latency for requests from each DIP 12, 18, and 22. For example, CLM component 10 is present in VNET1 and measures the latency for each DIP 12 (e.g., from DIP 112 to DIP 22) within VNET1. X Measure the latency 24 for 12). The CLM component 16 resides in VNET2 and each DIP 18 in VNET2 (e.g., from DIP 118 to DIP y Measure the latency of 26 for 18). VNET n This includes CLM component 20, VNET n Each DIP 22 inside (for example, from DIP 122 to DIP Z Measure the latency of 28 for 22).

[0028]

[0036] Latency 24, 26, and 28 are measured per application based on the Uniform Resource Locator (URL) provided by the user. In some implementations, CLM components 10, 16, and 20 measure latency every 5 seconds using the application URL provided by the user. Requests are sent directly to DIPs 12, 18, and 22 without going through the MUX (using the IPs of the DIPs instead of the VIPs), and as a result, CLM components 10, 16, and 20 identify DIPs 12, 18, and 22 and eliminate interference from the MUX.

[0029]

[0037] The measured values of latencies 24, 26, and 28 are stored in latency store 208. In some implementations, the measured values of latencies 24, 26, and 28 are stored together with VIP 206 as a key, and the value is a list of <DIP, latency, time> tuples.

[0030]

[0038] CAPSENSE LB controller 210 reads latencies 24, 26, and 28 from latency store 208 and calculates weights 30, 32, and 34 for DIPs 12, 18, and 22. CAPSENSE LB controller 210 uses latencies 24, 26, and 28 to calculate weights according to the capacitances of DIPs 12, 18, and 22. In some implementations, latencies 24, 26, and 28 provide estimated values of the capacitances of DIPs 12, 18, and 22. CAPSENSE LB controller 210 programs weights 30, 32, and 34 for DIPs 12, 18, and 22 using the capacitances and minimizes the total latency across all DIPs 12, 18, and 22 of VIP 206.

[0031]

[0039] In some implementations, CAPSENSE LB controller 210 includes an integer linear programming (ILP) module that calculates optimal weights 30, 32, and 34 for DIPs 12, 18, and 22. CAPSENSE LB controller 210 uses ILP to calculate weights 30, 32, and 34 in order to minimize the total latency. CAPSENSE LB controller 210 executes ILP for each VIP 206. ILP packs the load for each capacitance of DIPs 12, 18, and 22. An exemplary equation that CAPSENSE LB controller 210 uses for ILP is shown in Equation (1) below,

Number

[0032]

[0040] In some implementations, the following constraints apply to equation (1):

number

[0033]

[0041] l of each DIP to be used as input to the ILP equation (1) d,wCalculating the value of can be a time-consuming process. In some implementations, the CAPSENSE LB controller 210 includes a weight calculation module for latency measurements that determines a subset of weights (e.g., a small number of weights) for latency measurements. In some implementations, the CAPSENSE LB controller 210 uses a subset of latency measurements to determine the input l to the ILP equation (1). d,w The CAPSENSE LB controller 210 obtains the value of the weight. The CAPSENSE LB controller 210 performs latency measurements on a small number of weights and constructs a latency weight curve using curve fitting with polynomial regression (x axis = weight, y axis = latency). The CAPSENSE LB controller 210 uses the latency weight curve to estimate the latency for other weight values ​​for which the CAPSENSE LB controller 210 did not directly perform latency measurements.

[0034]

[0042] In some implementations, the weight calculation module for latency measurement within the CAPSENSE LB controller 210 includes a rough estimate of the DIP capacity for the weights. The CAPSENSE LB controller 210 uses this weight estimate in constructing the latency weight curve.

[0035]

[0043] In some implementations, the CAPSENSE LB controller 210 uses algorithm 1 to process each DIP (e.g., DIP 112 to DIP) in each iteration of the measurement phase. X Calculate the weights for 12).

number

[0036]

[0044] The input to Algorithm 1 is the weights of the current iteration and the previous iteration, w. now and lol prev It includes w max This represents the maximum weight observed so far without packet drops. The input also includes l0, which is the latency when the weight is 0, and l, which is the latency for the current weight.now This is also included. The CAPSENSE LB controller 210 measures l0 by setting the weight to 0 when a new DIP is added. The output is w as the weight for the next iteration. next This includes isExplorationDone, a boolean value indicating whether the calculation to obtain the latency weight map is complete. For such DIPs (those with this flag set), ILP assigns weights, otherwise w next Calculate.

[0037]

[0045] As written in the first and second lines of Algorithm 1, w now and lol prev If the difference is small, the flag isExplorationDone is set. If no packet drops occur, algorithm 1 indicates that some capacity is still available, and the CAPSENSE LB controller 210 can increase the weight. max (Line 5) is updated, and the weight increases in proportion to the latency (Line 6 of Algorithm 1). w If it is about the same as l0, algorithm 1 shows that there is still more capacity remaining, w next The magnitude of the increase can be made larger. w When l0 is significantly greater than w, algorithm 1 indicates that the capacity is approaching its limit, and w next Slow down the rate of increase. α indicates the rate of increase (set to 1). When packet drops occur (when the capacity reaches its limit), w next is, w now and lol max It is reduced to the average value (line 8 of Algorithm 1). Finally, to improve the search time, based on the observation that when CPU usage reaches 100%, the latency becomes more than 5 times that of l0, w When the value becomes 5 times l0, packet drop is flagged.

[0038]

[0046] The CAPSENSE LB controller 210 uses weight estimates from algorithm 1 when constructing the latency weight curve, which are the inputs to the ILP equation (1). d,w The value is obtained. In some implementations, the CAPSENSE LB controller 210 uses a latency weighting curve to execute the ILP equation (1) in two stages instead of one. Executing the ILP algorithm in two stages improves the execution time of the ILP while maintaining the accuracy of the ILP algorithm.

[0039]

[0047] For example, instead of running ILP with 100 weights for all DIPs, the CAPSENSE LB controller 210, in the first stage, starts from 0 to w max Only 10 values ​​(instead of 0-1) are uniformly provided up to this point. This provides an approximate weight estimate without packet drop. In the second stage, the CAPSENSE LB controller 210 calculates the weights more precisely. d If is the weight of the d-th DIP in the first stage, then the CAPSENSE LB controller 210 is W d From δ to W d Up to +δ (δ=b max Ten values ​​equal to 10% of the total are provided uniformly. For example, if the number of DIPs is 100, the CAPSENSE LB controller 210 performs multi-stage iteration. Otherwise, the CAPSENSE LB controller 210 performs the first stage.

[0040]

[0048] The CAPSENSE LB controller 210 sends weights 30, 32, and 34 to the LB controller 212, which then programs the MUX with the new weights 30, 32, and 34. In some implementations, the MUX performs a weighted round-robin (WRR) using the weights 30, 32, and 34 sent by the LB controller 212.

[0041]

[0049] In some implementations, the CAPSENSE LB controller 210 includes a scheduler module that schedules weights 30, 32, and 34 to be sent to the LB controller 212 over multiple rounds. For example, the scheduler module schedules weights 30, 32, and 34 over multiple rounds in response to the combined weight of weights 30, 32, and 34 exceeding 1. Another example involves the scheduler module prioritizing a particular latency measurement over other latency measurements.

[0042]

[0050] In some implementations, the CAPSENSE LB controller 210's scheduler module uses the new weights 30, 32, and 34 to classify the DIPs 12, 18, and 22 to be scheduled into three priority classes: (1) priority class for overutilized DIP weights, (2) priority class for remaining DIP weights, and (3) priority class for weights being refreshed. Within each priority class, the scheduler module uses a first-in, first-out (FIFO) method. To schedule the DIPs with the new weights, the scheduler module uses a simple greedy algorithm that sorts all DIPs by priority and hops through them one by one until (1) the weight of a scheduled DIP becomes 1, or (2) the algorithm has hopped through all DIPs.

[0043]

[0051] The total weight of scheduled DIPs may be less than 1 (especially in (2)). In such cases, the scheduler module will w s The remaining weights of the DIPs are then calculated. Therefore, the scheduler module uses the remaining DIPs to calculate the weights of the DIPs. s Assign a total weight of 1-w for each classified DIP. s Use an ILP with the modified constraint that the ILP returns an unsatisfactory assignment. If the ILP returns an unsatisfactory assignment, the scheduler module will use the remaining weights (1-w) across all remaining DIPs. s Divide ) equally.

[0044]

[0052] In some implementations, the CAPSENSE LB controller 210 responds quickly to changes in the dynamics of the environment 200. These changes in dynamics include changes in traffic, failures, and / or capacity changes.

[0045]

[0053] One example of a use case involves an increase in total traffic (e.g., an increase in requests from client 202), in which case the amount of traffic to DIPs 12, 18, and 22 increases for the same weights 30, 32, and 34, resulting in higher latency for the same weights 30, 32, and 34. As traffic increases, the CAPSENSE LB controller 210 detects the change using the provided latency measurements 24, 26, and 28 and updates the latency weight curve. For example, the latency was 5 milliseconds when the weight was 0.5 (w1). With the increase in traffic, the latency increases to 7 milliseconds for the same weight, and the weight (w2) for a latency of 7 milliseconds becomes 0.625. To calculate the new weight, the CAPSENSE LB controller 210 multiplies the existing weight by δ, where δ = w1 / w2. The CAPSENSE LB controller 210 multiplies all weights by δ. Similarly, when traffic decreases, the CAPSENSE LB controller 210 increases the weights for the same latency values ​​using the mechanism described above. The CAPSENSE LB controller 210 can periodically refresh the weight latency curve because it corrects drift based on changes in traffic.

[0046]

[0054] Another use case involves situations where the DIP capacity changes dynamically (e.g., changes in VMs located in the same location). The CAPSENSE LB controller 210 updates the latency weight curve in response to such DIP capacity changes. The observed latency is subtracted from the estimated latency by a threshold (O is l d,wIf the difference exceeds ±20% (which is set to ±20%), the CAPSENSE LB controller 210 detects that the capacity has changed. The CAPSENSE LB controller 210 adjusts the weights as detailed above (for example, if the latency changes from 5 milliseconds to 7 milliseconds, the CAPSENSE LB controller 210 uses w1 and w2 in the example above).

[0047]

[0055] Another use case involves the CAPSENSE LB controller 210 detecting a DIP failure or unreachability when it fails to receive a successful ping (e.g., a ping for latency). In such cases, the CAPSENSE LB controller 210 excludes that DIP from the ILP calculation. The ILP is readily available for all DIPs. d,w Therefore, the CAPSENSE LB controller 210 recalculates the weights excluding the DIP that has failed and sends those weights to the LB controller 212.

[0048]

[0056] Another use case involves the CAPSENSE LB controller 210 periodically refreshing the latency weight curve. For example, the CAPSENSE LB controller 210 limits the number of DIPs to be refreshed to 5% of the total capacity. To refresh the latency, the LB controller 210 uses algorithm 1 to measure the latency for the weights.

[0049]

[0057] After the CAPSENSE LB controller 210 recalculates the weights for latency measurement in response to changes in the environment 200, the CAPSENSE LB controller 210 sends the new weights to the LB controller 212, which then programs the new weights in the LB data plane 204. There may be a delay in the latency measurement of the new weights. This delay may originate from the LB controller 212 (programming the data plane 204 takes time). Also, once the LB data plane 204 is programmed with the new weights, only new connections will follow the new weights. Older connections will continue to use the old weights. As a result, the CAPSENSE LB controller 210 will not perform latency measurement until the old connections have terminated. Because the CAPSENSE LB controller 210 does not operate within the LB data plane 204, it does not know whether all old connections have terminated.

[0050]

[0058] In some implementations, the CAPSENSE LB controller 210 uses extreme settings of the DIP to calculate the time between weight setting and latency measurement (called drain time). First, the CAPSENSE LB controller 210 sets the weights high enough to increase latency (time T1). Then, the CAPSENSE LB controller 210 sets the weights to 0 so that no new connections are made to this DIP. The CAPSENSE LB controller 210 continuously measures latency until it reaches l0 (time T2). The CAPSENSE LB controller 210 calculates the drain time as T2-T1.

[0051]

[0059] The CAPSENSE LB controller 210 load balances traffic according to the capacity of DIPs 12, 18, and 22, thereby reducing overall latency and decreasing unused capacity of DIPs 12, 18, and 22. By calculating weights 30, 32, and 34 at the central controller (CAPSENSE LB controller 210), the LB data plane 204 can focus on packet processing with low latency and high throughput.

[0052]

[0060] In some implementations, one or more computing devices (e.g., servers and / or other devices) are used to perform the processing of environment 200. One or more computing devices may include, but are not limited to, server devices, personal computers, mobile devices (such as mobile phones, smartphones, PDAs, tablets, or laptops), and / or non-mobile devices. The features and functions discussed herein in relation to various systems may be implemented on one computing device or across multiple computing devices. For example, the latency store 208, CAPSENSE LB controller 210, and LB controller 212 are implemented on the exact same computing device. Another example includes cases where one or more subcomponents of the latency store 208, CAPSENSE LB controller 210, and / or LB controller 212 are implemented across multiple computing devices. Furthermore, in some implementations, one or more subcomponents of the latency store 208, CAPSENSE LB controller 210, and / or LB controller 212 may be implemented to be processed on different server devices in the same or different cloud computing networks.

[0053]

[0061] In some implementations, the components of Environment 200 communicate with each other using any appropriate communication technology. In addition, although the components of Environment 200 are shown separately, any component or subcomponent can be combined into fewer components (such as a single component) or divided into more components in a form suitable for a particular implementation. In some implementations, the components of Environment 200 include hardware, software, or both. For example, the components of Environment 200 may include one or more instructions stored in a computer-readable storage medium that can be executed by the processor of one or more computing devices. The processor may be a general-purpose single-chip or multi-chip microprocessor (e.g., an advanced RISC (Reduced Instruction Set Computer) machine (ARM)), a dedicated microprocessor (e.g., a digital signal processor (DSP)), a microcontroller, a programmable gate array, etc. Memory communicates electronically with the processor. Memory can be any electronic component capable of storing electronic information. For example, memory can be embodied as random access memory (RAM), read-only memory (ROM), magnetic disk storage media, optical storage media, flash memory devices within RAM, onboard memory built into the processor, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, etc. (including combinations thereof). When executed by one or more processors, computer executable instructions for one or more computing devices can perform one or more of the methods described herein. In some implementations, the components of environment 200 include hardware such as dedicated processing devices for performing specific functions or sets of functions. In some implementations, the components of environment 200 include combinations of computer executable instructions and hardware.

[0054]

[0062] Referring to Figure 3, an exemplary method 300 for calculating DIP weights using ILP is shown. The operation of method 300 is discussed below with reference to Figure 2.

[0055]

[0063] In 302, method 300 includes obtaining latency measurements for all DIPs of the VIP. In some implementations, a CLM (e.g., CLM component 10) on the VIP (e.g., VIP12061) obtains latency measurements for each DIP of the VIP (e.g., from DIP112 on VIP12061) X 12) The latency 26 is measured periodically. The latency measurement (e.g., latency 24) is stored in the latency store 208. The CAPSENSE LB controller 210 reads the latency measurement (e.g., latency 24) from the latency store 208.

[0056]

[0064] In 304, method 300 includes using latency measurements to determine weights for ILP and constructing a weighted latency map. In some implementation forms, the CAPSENSE LB controller 210 uses each DIP (e.g., DIP112 of VIP12061 to DIP X 12) Determine the maximum weight. The maximum weight is the weight observed in each DIP if no packet drops occur. Different DIPs may have different capacities and therefore different maximum weights. For example, the maximum weight of DIP112 is 0.6, and the maximum weight of DIP212 is 0.8. In some implementations, the CAPSENSE LB controller 210 uses algorithm 1 to calculate the maximum weight of each DIP.

[0057]

[0065] The CAPSENSE LB controller 210 determines a weight subset for each DIP up to the maximum weight. For example, the CAPSENSE LB controller 210 determines 10 uniform weight subsets for each DIP, ranging from 0 to the maximum weight. The CAPSENSE LB controller 210 uses this weight subset to construct a weight latency map, which it provides as input to the ILP.

[0058]

[0066] In 306, method 300 includes obtaining latency for weights to the IPL using curve fitting. Using this latency curve, a weight latency map (l_{d,w}) is constructed and provided as input to the IPL. The CAPSENSE LB controller 210 obtains latency measurements for a subset of weights and constructs a latency weight curve using curve fitting with polynomial regression, thereby estimating latency measurements for other weight values ​​for which the CAPSENSE LB controller 210 did not directly perform latency measurements.

[0059]

[0067] In 308, method 300 involves performing ILP using a weighted latency map. The CAPSENSE LB controller 210 calculates the weights of the DIPs using the latency weight curve as the input (l_{d,w}) to the ILP equation (1). The output of the ILP equation (1) after the first pass provides a single weight to assign to the DIPs. For example, this single weight is an estimate of the weight to assign to the DIPs based on their capacity. In some implementations, the CAPSENSE LB controller 210 provides the weights of each DIP to the LB controller 212, which then programs the LB data plane 204 with the weights of each DIP.

[0060]

[0068] In some implementations, method 300 returns to 304 and determines a second set of weights for the ILP using the first weights provided during the first pass of the ILP in 308. The CAPSENSE LB controller 210 uses the first weights to determine a second weight subset for each DIP. The CAPSENSE LB controller 210 determines a second weight subset for each DIP. The second weight subset is within the range of the first weights (e.g., the weight estimates). This range includes some weights above the weight estimates and some weights below the weight estimates. An example of this range is that it includes weights 10% above the weight estimates and weights 10% below the weight estimates. For example, if W is the first weight output from the ILP, then the second weight subset is 0.9 * W to 1.1 * This includes up to W. The CAPSENSE LB controller 210 constructs a weight latency map using a second weight subset and provides it as input to the ILP in the second path.

[0061]

[0069] Method 300 returns to 306 and uses curve fitting to obtain latency for a second set of weights. The second latency curve is used to construct a second weight latency map (l_{d,w}) and is provided as input to the IPL in the second pass. The CAPSENSE LB controller 210 obtains latency measurements for the second weight subset from the latency store 208 and constructs a second latency weight curve using curve fitting with polynomial regression, thereby estimating latency measurements for other weight values ​​for which the CAPSENSE LB controller 210 did not directly perform latency measurements.

[0062]

[0070] Method 300 returns to 308 and performs a second pass of ILP using the second latency weight map. The CAPSENSE LB controller 210 calculates the weights of the DIPs using the second latency weight curve as the input (l_{d,w}) to the ILP equation (1). The output of the ILP equation (1) is the weight of each DIP in the VIP (e.g., from DIP 112 to DIP 112). X W1301 to W for 12 X 30 X The weights obtained from the second output of IPL equation (1) may be more accurate than the estimates obtained from the first output of IPL equation (1). By running the ILP algorithm in two stages, the execution time of the ILP algorithm is improved while maintaining the accuracy of the ILP algorithm.

[0063]

[0071] The CAPSENSE LB controller 210 sends the weight of each DIP to the LB controller 212, and upon receiving this, the LB controller 212 processes the new weight (for example, from DIP 112 to DIP X W1301 to W for 12 X 30 X ) Program the MUX.

[0064]

[0072] Method 300 is for each VIP (for example, from VIP22062 to VIP n 206 n This can be repeated for each DIP of each VIP (for example, from DIP218 to DIP22062 of VIP22062) y From age 18, VIP n 206 n From DIP122 to DIP Z The weights (up to 22) are calculated.

[0065]

[0073] Figure 4 shows an exemplary graph 400 of the weights used for latency measurement. Graph 400 has the number of iterations on the x-axis 402 and the weights on the y-axis 404. Graph 400 shows the weights (e.g., different shapes on different lines) calculated for four different DIPs (DIP-1 406, DIP-17 408, DIP-25 410, DIP-29 412) in the VNET. For example, the CAPSENSE LB controller 210 (Figure 2) uses algorithm 1 to calculate the maximum weights plotted in graph 400 for four different DIPs (DIP-1 406, DIP-17 408, DIP-25 410, DIP-29 412). Graph 400 shows that the weights calculated by the CAPSENSE LB controller 210 differ among different DIPs (DIP-1 406, DIP-17 408, DIP-25 410, DIP-29 412).

[0066]

[0074] Figure 5 shows an exemplary graph 500 of a latency weight curve according to the implementation of the present disclosure. Graph 500 has weights on the x-axis 502 and latency on the y-axis 504. The weights shown on the x-axis correspond to the weights in graph 400. Graph 500 uses polynomial regression to fit latency weight curves (e.g., latency weight curves 506, 508, 510, 512) and estimate latency that was not used in the measurement. For example, the CAPSENSE LB controller 210 (Figure 2) uses the weights in graph 400 to calculate a small number of latency measurements (e.g., five latency measurements) for different DIPs (DIP-1 406, DIP-17 408, DIP-25 410, DIP-29 412). The CAPSENSE LB controller 210 uses polynomial regression to fit curves and estimate latency for different DIPs (DIP-1 406, DIP-17 408, DIP-25 410, DIP-29 412).

[0067]

[0075] Figure 6 shows an exemplary graph 600 of the weights calculated by ILP. Graph 600 includes the DIP index on the x-axis 602 and the assigned weights on the y-axis 604. The CAPSENSE LB controller 210 (Figure 2) uses the latency weight curves shown in graph 500 (e.g., latency weight curves 506, 508, 510, 512) as input (l_{d,w}) to the ILP equation (1) for calculating the weights of DIP. The ILP equation assigned more weights to VMs with more capacity, as shown in graph 600.

[0068]

[0076] Referring to Figure 7, an exemplary method 700 for capacity-aware Layer 4 load balancing is shown. The operation of method 700 will be discussed below with reference to Figure 2.

[0069]

[0077] In 702, method 700 includes obtaining latency measurements for each of the multiple Direct IPs (DIPs) in the virtual network. The latency measurement (e.g., latency 24) is obtained for each DIP (e.g., DIP 112 to DIP X 12) Provides an indicator of capacity. In some implementations, CAPSENSE Latency Measurement (CLM) components 10, 16, and 20 reside within each VNET. CLM components (e.g., CLM component 10) are located within each DIP (e.g., DIP 112 to DIP 10). XPeriodically measure the latency for requests from (12). The latency 24 is measured for each application based on the Uniform Resource Locator (URL) provided by the user. In some implementations, the CLM component 10 measures the latency every 5 seconds using the application URL provided by the user. The measured value of the latency 24 is stored in the latency store 208. In some implementations, the measured value of the latency 24 is stored together with the VIP 206 as a key, and the value is a list of <DIP, latency, time> tuples. The CAPSENSE LB controller 210 reads the latency (e.g., latency 24) from the latency store 208.

[0070]

[0078] In 704, the method 700 includes using ILP to calculate the weight of each DIP using the latency measurement values. The CAPSENSE LB controller 210 uses the latency read from the latency store 208 to calculate the weights of the DIPs (e.g., from DIP 112 to DIP X 12) (e.g., from weight 301 to 30 X ). The latency provides an estimate of the capacity of the DIP (e.g., a higher latency reflects that the DIP is approaching its capacity limit, and a lower latency reflects that there is remaining capacity in the DIP). The CAPSENSE LB controller 210 calculates the weights according to the capacities of the DIPs (e.g., from DIP 112 to DIP X 12) and programs the weights of the DIPs (e.g., from DIP 112 to DIP X 12) (e.g., from weight 301 to 30 X ). The weights (e.g., from weight 301 to 30 X ) specify the amount of traffic to be provided to each DIP (e.g., from DIP 112 to DIP X 12).

[0071]

[0079] In some implementations, the CAPSENSE LB controller 210 uses an integer linear programming (ILP) problem to calculate the weights of each DIP (e.g., DIP1), and the ILP calculates the weights of multiple DIPs (e.g., DIP112 to DIP112). X 12) Minimize the total latency. ILP is DIP (e.g., DIP112 to DIP X 12) The load is packed according to the capacity. In some implementations, the CAPSENSE LB controller 210 uses the ILP equation (1) to pack each DIP (e.g., DIP112 to DIP X 1) Weights (for example, weights 301 to 30 X ) determines. In some implementations, the input to the ILP includes multiple latencies with different weights for each DIP, and the output of the ILP determines each DIP (e.g., from DIP112 to DIP X 12) is a single weight for each DIP. In some implementations, the ILP includes constraints that one weight is applied to each DIP, that the total weight of the weights is equal to 1, that imbalance is allowed, and that minimum and maximum weights are specified across multiple DIPs. In some implementations, the CAPSENSE LB controller 210 assigns a single weight to all DIPs in the VNET (e.g., DIP112 to DIP112). X 12) Calculate the weights to minimize the total latency.

[0072]

[0080] In 706, method 700 includes providing the weights of each DIP to a Layer 4 load balancer. The CAPSENSE LB controller 210 provides each DIP (e.g., DIP 112 to DIP) to a Layer 4 load balancer (e.g., LB controller 212). X 12) Weights 301 to 30 X It provides. The Layer 4 load balancer provides each DIP (e.g., DIP112 to DIP X 12) Weights 301 to 30 X The program was designed to use weights 301 to 30 X Using multiple DIPs (for example, DIP112 to DIP) XDisperse traffic to (12). In some implementations, when the capacity of DIP (e.g., from DIP112 to DIP X 12) changes, the CAPSENSE LB controller 210 updates the weight of DIP (e.g., from DIP112 to DIP X 12) in response to the change in capacity.

[0073]

[0081] In some implementations, the CAPSENSE LB controller 210 uses the priority of each DIP (e.g., from DIP112 to DIP X 12) to schedule the weight of each DIP (e.g., from DIP112 to DIP X 12) (e.g., weights 301 to 30 X ) to the layer 4 load balancer (e.g., LB controller 212). In some implementations, the CAPSENSE LB controller 210 includes a scheduler module that schedules weights 301 to 30 X in multiple rounds and sends them to the LB controller 212. For example, the total weight of weights 301 to 30 X exceeds 1. As another example, the case where the scheduler module prioritizes a specific latency measurement over other latency measurements is included.

[0074]

[0082] Method 700 disperses the load of incoming traffic to the VNET according to the capacity of the DIP.

[0075]

[0083] Referring now to FIG. 8, an exemplary method 800 for calculating the weights of DIPs used for load distribution using an integer linear programming problem (ILP) is shown. The operation of method 800 is discussed below while referring to FIG. 2.

[0076]

[0084] In 802, method 800 includes using latency measurements to determine a first weight subset for each of the multiple Direct IPs (DIPs) in the virtual network. In some implementations, the CAPSENSE LB controller 210 uses latency measurements for each of the multiple DIPs in the VNET (e.g., DIP 112 to DIP 112). X 12) Includes a weight calculation module for latency measurements that determines a first weight subset (e.g., a small number of weights).

[0077]

[0085] In some implementations, a first weight subset estimates the capacity of each DIP. In other implementations, the capacity is estimated based on whether packets were dropped for the DIP. For example, the capacity of a DIP is limited if packets are dropped, and the capacity of a DIP is still available if packets are not dropped.

[0078]

[0086] In some implementations, the CAPSENSE LB controller 210 determines the maximum weight of a DIP. The maximum weight of a DIP is the highest weight value of the DIP if no packets were dropped. The maximum weight of a DIP provides an estimate of the DIP's capacity in terms of weight. In some implementations, the CAPSENSE LB controller 210 uses algorithm 1 to determine each DIP (e.g., from DIP 112 to DIP) in each iteration of the measurement phase. X Calculate the maximum weight in 12).

[0079]

[0087] The CAPSENSE LB controller 210 uses the maximum weight to determine a first weight subset for each DIP. For example, the first weight subset for each DIP includes a uniform distribution of weights from zero to the maximum weight. Different DIPs may have different capacities and different maximum weights, and as a result, the weight subsets for each DIP will be different.

[0080]

[0088] In 804, method 800 includes obtaining latency measurements for a first weight subset of each DIP. The CAPSENSE LB controller 210 then processes each DIP (e.g., from DIP 112 to DIP X 12) Obtain latency (e.g., latency 24) measurements for the first weight subset. For example, the CAPSENSE LB controller 210 reads the latency (e.g., latency 24) for the weights of the first weight subset from the latency store 208. In some implementations, the latency (e.g., latency 24) measurement is obtained from each DIP (e.g., DIP 112 to DIP X 1) Measure the latency to the request from.

[0081]

[0089] In 806, method 800 includes generating a latency weight curve for each DIP using latency measurements for a first weight subset. The CAPSENSE LB controller 210 uses the weights of the first weight subset when constructing latency weight curves for each DIP (e.g., latency weight curves 406, 408, 410, 412). The CAPSENSE LB controller 210 obtains latency measurements for a small number of weights and constructs the latency weight curves using curve fitting with polynomial regression (x-axis = weight, y-axis = latency).

[0082]

[0090] Latency weight curves (e.g., latency weight curves 406, 408, 410, 412) estimate latency measurements for weights other than those included in the first weight subset. The CAPSENSE LB controller 210 uses these latency weight curves to estimate latency for other weight values ​​for which the CAPSENSE LB controller 210 did not directly perform latency measurements. In some implementations, the latency weight curves are adjusted in response to detecting changes in VNET traffic or DIP capacity.

[0083]

[0091] In 808, method 700 includes calculating the first weight of each DIP using an integer linear programming problem (ILP), using the latency and weight from each DIP's latency weight curve as input to the ILP. The CAPSENSE LB controller 210 uses latency weight curves (e.g., latency weight curves 406, 408, 410, 412) to obtain the value of l that is input to the ILP equation (1). The latency and weight from the latency weight curves (e.g., latency weight curves 406, 408, 410, 412) are provided as input to the ILP equation (1). The ILP minimizes the total latency of multiple DIPs. In some implementations, the ILP includes constraints such as applying one weight to each DIP, the total weight of the weights being equal to 1, allowing for imbalance, and specifying the minimum and maximum weights across multiple DIPs. In some implementations, the first weight subset provides a set of uniform weights for each DIP from the minimum weight to the maximum weight of the weights from the latency weight curve as input to the ILP equation (1). The output of the ILP equation (1) after the first pass provides the first weight of each DIP. In some implementations, the first weight is an estimated value of the weight to be assigned to the DIP based on the capacity of the DIP. d,w In 810, method 800 includes determining a second weight subset for each DIP using the first weight of each DIP. The CAPSENSE LB controller 210 determines the second weight subset for each DIP using the first weight of each DIP provided as the output from the ILP equation (1). The CAPSENSE LB controller 210 determines the second weight subset for each DIP within the range of the first weight (e.g., a part above the first weight and a part below the first weight). As an example of the range, it includes a set of weights above the first weight (e.g., 5 equally spaced weights) and a set of weights below the first weight (e.g., 5 equally spaced weights).

[0084]

[0092]

[0085]

[0093] In 812, method 800 includes obtaining latency measurements for a second weight subset of each DIP. The CAPSENSE LB controller 210 reads the latency (e.g., latency 24) for the weights of the second weight subset from the latency store 208.

[0086]

[0094] In 814, method 800 includes generating a second latency weight curve for each DIP using latency measurements for a second weight subset. The CAPSENSE LB controller 210 constructs a weight latency map using the second weight subset and provides it as input to the ILP in the second pass. The CAPSENSE LB controller 210 constructs a second latency weight curve using curve fitting with polynomial regression, thereby estimating latency measurements for other weight values ​​for which the CAPSENSE LB controller 210 did not directly perform latency measurements.

[0087]

[0095] In 816, method 800 includes using ILP to calculate a second weight for each DIP using the latency and weight from the second latency curve of each DIP as input to ILP. The CAPSENSE LB controller 210 calculates the weights of the DIPs using the second latency weight curve as input (l_{d,w}) to ILP equation (1). The output of ILP equation (1) is the weight of each DIP in the VIP (e.g., from DIP 112 to DIP X W1301 to W for 12 X 30 X The weights obtained from the second output of IPL equation (1) may be more accurate than the first weights obtained from the first output of IPL equation (1). By running the ILP algorithm in two stages, the execution time of the ILP algorithm is improved while maintaining the accuracy of the ILP algorithm.

[0088]

[0096] In 820, method 800 includes providing a second weight for each DIP to the load balancer controller. The CAPSENSE LB controller 210 provides each DIP (e.g., DIP 112 to DIP) X 12) Weights (for example, from W1301 to W X 30 X These weights (for example, from W1301 to W) are sent to the LB controller 212. X 30 X ) Each DIP (for example, from DIP112 to DIP X 12) Identify the amount of traffic to provide. The load balancer controller 212 weights the MUX in the load balancer data plane 204 (e.g., W1301 to W X 30 X ) is programmed with these weights (for example, W1301 to W X 30 X ) use multiple DIPs (for example, DIP112 to DIP X 12) Distribute the traffic to this.

[0089]

[0097] In some implementations, the CAPSENSE LB controller 210 ensures that the sum of the weights of each DIP is equal to 1, by checking each DIP (e.g., DIP 112 to DIP) X 12) Weights (for example, from W1301 to W X 30 X The DIPs are scheduled to the load balancer controller 212 in multiple rounds. In some implementations, the CAPSENSE LB controller 210 schedules the weights of each DIP to the load balancer controller 212 by ordering each DIP based on its priority. For example, the priority may include the weight of an overutilized DIP, the weight of a remaining DIP, or the weight of a DIP that is being refreshed.

[0090]

[0098] Method 800 is for each VIP (for example, from VIP22062 to VIP n 206 nThis can be repeated for each DIP of each VIP (for example, from DIP218 to DIP22062 of VIP22062) y From age 18, VIP n 206 n From DIP122 to DIP Z The weights (up to 22) are calculated. Method 800 uses ILP to model load balancing as load packing and DIP (e.g., DIP112 to DIP X Pack the load based on capacity 12) to minimize total latency.

[0091]

[0099] As shown in the preceding discussion, this disclosure uses a variety of terms to describe the features and advantages of the methods and systems. Hereinafter, we add further details regarding the meaning of such terms. For example, as used herein, “machine learning model” refers to a computer algorithm or model (e.g., classification model, clustering model, regression model, language model, object detection model) that can be tuned (e.g., trained) based on training inputs to approximate an unknown function. For example, a machine learning model may refer to a neural network (e.g., a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN)), or other machine learning algorithms or architectures that learn and approximate a complex function based on multiple inputs provided to a machine learning model to produce an output. As used herein, “machine learning system” may refer to one or more machine learning models that collaboratively produce one or more outputs based on corresponding inputs. For example, a machine learning system may refer to any system architecture having multiple separate machine learning components that consider different types of information or inputs.

[0092]

[0100] The techniques described herein may be implemented in hardware, software, firmware, or any combination thereof, unless otherwise specifically stated. Furthermore, any feature described as a module, component, or similar may be implemented integrally within an integrated logical device or separately as interoperable individual logical devices. When implemented in software, the techniques may be at least partially realized by a non-transient processor-readable storage medium containing instructions, which, when executed by at least one processor, perform one or more of the methods described herein. Instructions may be organized into routines, programs, objects, components, data structures, etc., capable of performing specific tasks and / or implementing specific data types, which may be combined or distributed as needed in various implementations.

[0093]

[0101] A computer-readable medium can be any available medium that a general-purpose or dedicated computer system can access. A computer-readable medium that stores computer-executable instructions is a non-temporary computer-readable storage medium (memory device). A computer-readable medium that carries computer-executable instructions is a transmission medium. Therefore, for example, but not limited to, implementations of this disclosure may include at least two completely different types of computer-readable media, namely a non-temporary computer-readable storage medium (memory device) and a transmission medium.

[0094]

[0102] As used herein, non-temporary computer-readable storage media (storage devices) may include RAM, ROM, EEPROM, CD-ROM, solid-state drives ("SSDs") (e.g., RAM-based), flash memory, phase-change memory ("PCM"), other types of memory, other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other media that can be used to store desired program code means in the form of computer-executable instructions or data structures and that are accessible by a general-purpose or dedicated computer.

[0095]

[0103] The steps and / or operations of the methods described herein are interchangeable without departing from the claims. In other words, unless a particular order of steps or operations is required for the proper operation of the described method, the order and / or use of any particular steps and / or operations is modifiable without departing from the claims.

[0096]

[0104] The term "decide" encompasses a wide range of actions. Therefore, "decide" can include calculating, arithmetic, processing, deriving, investigating, searching (e.g., searching in a table, database, data store, or other data structure), confirming, and similar actions. It can also include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), and similar actions. Furthermore, "decide" can include resolving, selecting, choosing, establishing, predicting, reasoning, and similar actions.

[0097]

[0105] Articles such as “a,” “an,” and “the” are intended to indicate that one or more elements are present in the preceding description. The terms “equip,” “include,” and “have” are intended to be inclusive and to mean that additional elements other than those listed may exist. In addition, it should be understood that any reference in this disclosure to “one implementation” or “implementation” is not intended to be construed as excluding the existence of additional implementations that similarly incorporate the described features. For example, any element described in relation to an implementation in this specification can be combined with any element of any other implementation described herein. Numerical values, percentages, ratios, or other values ​​described herein are intended to include the value itself, as well as other values ​​that are “approximately” or “about” the stated value, as understood by those skilled in the art in the field covered by the implementation of this disclosure. Accordingly, the stated values ​​should be interpreted broadly enough to include values ​​that are at least close enough to the stated value in order to perform the desired function or achieve the desired result. The stated values ​​include at least the expected variations in appropriate manufacturing or production processes, and may include values ​​within 5%, 1%, 0.1%, or 0.01% of the stated values.

[0098]

[0106] Those skilled in the art will understand, in light of this disclosure, that equivalent configurations will not deviate from the spirit and scope of this disclosure, and that various changes, substitutions, and modifications can be made to the implementations disclosed herein without deviating from the spirit and scope of this disclosure. Equivalent configurations (including functional “means + function” clauses) are intended to cover the structures described herein as performing the described function and include both structural equivalents that operate in the same manner and equivalent structures that provide the same function. The applicant’s explicit intention is not to invoke means + function or other functional claims in any claim except those in which the term “means for” is described together with the relevant function. Additions, deletions, and modifications to implementations that fall within the meaning and scope of the claims should be accepted by those claims, respectively.

[0099]

[0107] This disclosure may be embodied in other specific forms, without departing from its spirit or characteristics. The implementations described should be considered illustrative and not limiting. Accordingly, the scope of this disclosure is indicated by the appended claims rather than by the foregoing description. Any changes resulting in equivalent meaning and scope of the claims should be accepted within that scope.

Claims

1. Obtaining latency measurements (24) for each of the multiple direct IPs (DIPs) (12) in the virtual network (702), wherein the latency measurements (24) provide an indicator of the capacity of each DIP (702), Using an integer linear programming (ILP) problem, the weights (30) of each DIP (30) are calculated using the latency measurement (24) (704), The Layer 4 load balancer (212) is provided with the weights (30) of each DIP (706), wherein the weights (30) specify the amount of traffic to be provided to each DIP (706) Methods that include...

2. The method according to claim 1, wherein the Layer 4 load balancer programs the weights of each DIP and uses the weights to distribute traffic to the plurality of DIPs.

3. The method according to claim 1, wherein the latency measurement is obtained from a component on each DIP that measures the application latency of each DIP.

4. The method according to claim 1, wherein the ILP minimizes the total latency of the plurality of DIPs, and the ILP is subject to constraints that it applies one weight to each DIP, that the total weight of the weights is equal to 1, that imbalance is permitted, and that a minimum weight and a maximum weight are specified across the plurality of DIPs.

5. The method according to claim 1, wherein the input to the ILP includes a plurality of latencies with different weights for each DIP, and the output of the ILP is a single weight for each DIP.

6. The method according to claim 1, wherein when the capacity of the DIP changes, the weight of the DIP is updated in response to the change in capacity.

7. Using the priority of each DIP, schedule the weight of each DIP to the Layer 4 load balancer. The method according to claim 1, further comprising:

8. Using latency measurements, determine a first weight subset of each DIP in multiple Direct IPs (DIPs) within a virtual network (802), Obtaining latency measurements for the first weight subset of each DIP (804), Using the latency measurements for the first weight subset, a latency weight curve for each DIP is generated (806), Using an integer linear programming problem (ILP), the latency and weights from the latency-weight curve of each DIP are used as inputs to the ILP to calculate a first weight for each DIP (808), wherein the first weight is an estimate of the weight of each DIP (808), Using the first weight of each DIP, a second weight subset of each DIP is determined (810), Obtaining the latency measurement values ​​for the second weight subset of each DIP (812), Using the latency measurements for the second weight subset, a second latency weight curve for each DIP is generated (814), Using the ILP, the second weight of each DIP is calculated using the latency and weight from the second latency weight curve of each DIP as input to the ILP (816), Providing the load balancer controller (212) with the second weight of each DIP (818), wherein the second weight specifies the amount of traffic to be provided to each DIP (818) Methods that include...

9. The method according to claim 8, wherein the load balancer controller programs a MUX in the load balancer data plane with the second weight for each DIP, and the MUX uses the second weight for each DIP to distribute a portion of the traffic to the plurality of DIPs based on the second weight.

10. The method according to claim 8, wherein the first weight subset estimates the capacity of each DIP, the capacity reaching its limit when packets are dropped for a DIP, and the capacity still remaining for a DIP when packets are not dropped for a DIP.

11. The method according to claim 8, wherein the first weight subset provides a uniform set of weights for each DIP up to the maximum weight in which no packet is dropped, and the second weight subset provides a portion of weights greater than the first weight and a portion of weights less than the first weight.

12. The method according to claim 8, wherein latency measurement measures the latency to requests from each DIP.

13. The method according to claim 8, wherein the ILP minimizes the total latency of the plurality of DIPs, and the ILP is subject to constraints that it applies one weight to each DIP, that the total weight of the weights is equal to 1, that imbalance is permitted, and that a minimum weight and a maximum weight are specified across the plurality of DIPs.

14. The method according to claim 8, wherein the latency weight curve is adjusted in response to detecting changes in the virtual network traffic or changes in the DIP capacity.

15. Scheduling the weight of each DIP to the load balancer controller by ordering each DIP based on the priority of each DIP, wherein the priority is one of the weights of the overutilized DIP, the weights of the remaining DIPs, and the weight being refreshed, or To ensure that the sum of the aforementioned weights equals 1, the aforementioned weights of each DIP are scheduled to the load balancer controller in multiple rounds. The method according to claim 8, further comprising:

Images