Method, network device, and non-transitory computer-readable medium for a network

Abstract

Embodiments of the present disclosure generally relate to dynamic host configuration protocol lease assignment. In one example, a dynamic host configuration protocol (DHCP) lease request is received from a client device connected to a network. Based on the DHCP lease request, an internet protocol (IP) address is assigned to the client device for a lease time. A first lease renewal request is received. A probability of utilization of a lease time block is predicted based on historical lease patterns, device characteristics, traffic information, and DHCP information. Based on a combination of the probability of utilization and a reward value, the lease time block is assigned for renewal. For each assignment, the reward value is adjusted based on deployment characteristics and traffic load in the network and network connection duration of the client device. A normalized reward value is determined for the lease time block based on the reward values for the lease time block over multiple assignments.

Description

Background Technology

[0001] Computer networks can include devices that communicate using the Internet Protocol (IP). IP addresses can be assigned to each device on a computer network. IP addresses can be dynamically assigned when a device joins the network, or they can be statically assigned based on the device's hardware / software configuration. Dynamic IP addresses can be assigned to devices by a Dynamic Host Configuration Protocol (DHCP) server. DHCP allows devices to share a range of IP addresses on the network. Attached Figure Description

[0002] Some implementations of this disclosure are described with reference to the following figures.

[0003] Figure 1 This is a block diagram of an example network device used for DHCP lease assignment;

[0004] Figure 2 This is a flowchart of an example method for DHCP lease assignment;

[0005] Figure 3A This is a flowchart of an example method for adjusting the reward value used for DHCP lease assignment;

[0006] Figure 3B This is a flowchart of another example method for adjusting the reward value used for DHCP lease assignment;

[0007] Figure 4 This is a flowchart of another example method for assigning rental time blocks for rental renewals; and

[0008] Figure 5 This is a block diagram of an example storage medium that stores machine-readable instructions used for DHCP lease assignment.

[0009] Throughout the accompanying drawings, the same reference numerals denote similar but not necessarily identical elements. These drawings are not necessarily drawn to scale, and the dimensions of some parts may be exaggerated to more clearly illustrate the examples shown. Furthermore, the drawings provide examples and / or implementations consistent with the specification; however, the specification is not limited to the examples and / or implementations provided in the drawings. Detailed Implementation

[0010] Typically, DHCP is a network management protocol used on IP networks to automatically assign IP addresses and other communication parameters to devices connected to the network using a client-server architecture. DHCP eliminates the need for manual, individual device configuration. A DHCP server can use DHCP to assign IP addresses to devices connected to the network. In one example, a DHCP server can be implemented as a combination of hardware and software in a network switch, router, or a separate computing device. A network can include multiple DHCP servers.

[0011] When client devices such as laptops, computers, and smartphones connect to a network, they can broadcast DHCP Discover messages to obtain information about DHCP servers on the network. A DHCP server can have a pool of IP addresses and can assign IP addresses to requesting client devices. A DHCP server responds to a DHCP Discover message by sending a DHCP Offer message. A DHCP Offer message can include an IP address available for assignment to a client device, a lease time associated with the IP address, and the DHCP server's IP address. The lease time refers to the duration for which the IP address assigned to the client device remains valid. A client device can request to renew its IP address lease. If the client device does not renew the lease before the lease expires, the IP address can be assigned to another device after the lease expires. Upon receiving a DHCP Offer message, a client device can send a DHCP Request message to the DHCP server to accept an IP address that can be leased to the client device. In response to receiving a DHCP Request message, the DHCP server can make an entry with a client identifier such as the client device's Media Access Control (MAC) address and associate the offered IP address with the client identifier and the lease time.

[0012] With the proliferation of devices connecting to networks, managing the IP address pool in a DHCP server can be challenging. Optimal selection of the lease duration for IP addresses assigned to client devices is crucial for efficient use of the IP address pool. Typically, once a DHCP server assigns an IP address to a client device, the address remains assigned for the lease period until it expires, even if the client device may be inactive (e.g., in sleep mode) or may leave the network. Therefore, if the lease duration of an IP address assigned to a client device is longer than the duration the client device remains active or connected to the network, the assigned IP address may remain unnecessarily congested for an extended period, potentially making IP addresses unavailable for new devices joining the network. On the other hand, if the lease duration of an IP address assigned to a client device is shorter than the duration the client device remains connected to the network, the client device can request a lease renewal. If a client device remains connected to the network for an extended period, it can request frequent lease renewals, potentially increasing network traffic and congesting the network. Therefore, in the scenario discussed above, lease allocation can be inefficient, and optimizing DHCP lease durations can be challenging.

[0013] This disclosure discloses a technique for managing DHCP lease times for client devices. In one example, a network device, such as a DHCP server, can receive DHCP lease requests from a client device. In one example, the client device can send a DHCP lease request in response to receiving a DHCP offer message from a DHCP server. The DHCP lease request can indicate acceptance of the IP address offered to the client device in the DHCP offer message. Based on the DHCP lease request, the network device can assign an IP address to the client device during the lease period. In one example, the lease period can be included in the DHCP offer message. The network device can receive a first lease renewal request from the client device. In one example, the client device can send the first lease renewal request when half of the lease period expires. The network device can predict the utilization probability for lease time blocks from multiple lease time blocks based on: historical lease patterns associated with the client device, device characteristics, service information, and DHCP information.

[0014] In one example, prediction can be performed using an Artificial Neural Network (ANN). Based on a combination of the utilization probability and reward value of a leased time block, the network device can allocate leased time blocks to client devices for IP address lease renewal. In one example, the reward value comes from a reinforcement learning (RL) model coupled to the ANN. For each allocation of a leased time block, the network device can adjust the reward value of the leased time block based on factors such as network deployment characteristics, network traffic load, and the network connection duration of the client devices. Based on the reward values ​​of leased time blocks across multiple allocations, the network device can determine a normalized reward value for the leased time block. Using a combination of utilization probabilities adjusted based on reward values ​​allows for better prediction of the utilization probability of leased time blocks, and consequently leads to more accurate allocation of leased time blocks for renewal.

[0015] Furthermore, using this technology, the allocation of lease time blocks can be selectively optimized for different clients by using client-specific parameters to predict utilization probabilities, such as historical lease patterns associated with client devices, device characteristics, service information, and DHCP information. Moreover, since the utilization probability is combined with a reward value adjusted based on network deployment characteristics, network traffic load, and the duration of network connections for client devices, overall network performance is also considered when allocating lease time blocks for renewal. Therefore, using this technology, IP address lease renewals can be effectively managed using a combination of utilization probability and reward value.

[0016] The systems and methods described can be implemented in various network devices that implement DHCP in a network. Although the descriptions in this document refer to a DHCP server implemented in a network environment, these methods and described techniques can be implemented in other types of devices that implement DHCP functionality, such as gateways, routers, etc., with some variations. Various implementations of this topic are described below with reference to several examples.

[0017] refer to Figures 1 to 5 The systems and methods described above are further described. It should be noted that the descriptions and figures, together with the examples described herein, illustrate the principles of the subject matter only and should not be construed as limiting the subject matter. Therefore, it should be understood that various arrangements can be designed that, although not explicitly described or shown herein, embody the principles of the subject matter. Furthermore, all statements herein referencing principles, aspects, and embodiments of the subject matter, as well as specific examples thereof, are intended to cover their equivalents.

[0018] Figure 1 This is a block diagram of an example network device 100 used for DHCP lease assignment. In one example, network device 100 may be a DHCP server, router, gateway, etc. Network device 100 may include processor 102 and memory 104, which may be coupled to each other via a communication link (e.g., a bus). Processor 102 may include one or more central processing units (CPUs) or another(s) of suitable hardware processors, such as a network ASIC. Memory 102 may store machine-readable instructions that can be executed by processor 102. Memory 104 may include any suitable combination of volatile and / or non-volatile memory, such as a combination of random access memory (RAM), read-only memory (ROM), flash memory, and / or other suitable memory.

[0019] Memory 104 stores instructions to be executed by processor 102, including instructions for lease request receiver 106, IP address manager 108, renewal request receiver 110, probability generator 112, lease renewal manager 114, and reward generator 116.

[0020] Processor 102 can execute lease request receiver 106, which causes network device 100 to receive DHCP lease requests from client devices connected to the network. In one example, in response to receiving a DHCP offer message from network device 100, the client device can broadcast a DHCP lease request across the network. The DHCP offer message may include an IP address offered to the client device and a lease time associated with the IP address. The client device can broadcast the DHCP lease request across the network to accept the IP address for the lease time.

[0021] In one example, the lease time in the DHCP offer message can be fixed or predefined. For instance, a 24-hour lease could be offered to a client device connecting to the network for the first time. Predefined lease times can be defined by the network administrator based on the type of network deployment (such as a campus network, public network, private network, etc.) and the type of client device. For example, for an office network, a predefined lease time offered to an office desktop might be longer than the lease time offered to a visitor's smartphone. In another example, the lease time in the DHCP offer message could be determined by the network device using machine learning (ML) techniques based on various parameters, such as the date and time the client device connects to the network, the location of the client device (such as a visitor area, employee area, etc.), and the device type of the client device (such as a laptop, desktop, smartphone, etc.).

[0022] Processor 102 may execute IP address manager 108, which causes network device 100 to allocate IP addresses to client devices within a lease period based on a DHCP lease request. In one example, in response to receiving a DHCP lease request, network device 100 may allocate IP addresses to client devices within the lease period mentioned in the DHCP offer message. In one example, allocating IP addresses may include an identifier that maps the IP address to the client device, such as the client device's MAC address. The client device's MAC address may be included in the DHCP lease request.

[0023] Processor 102 can execute renewal request receiver 110, which causes network device 100 to receive a first lease renewal request from the client device. The first lease renewal request may include the client device's MAC address, the client device's IP address, remaining lease time, etc. In one example, in response to half of the lease time associated with the IP address allocated to the client device expiring, the client device may send a first lease renewal request to network device 100. The lease renewal request instructs the client device to renew the lease on the client device's IP address. In one example, if the lease time allocated to the IP address is 2 hours, the client device may send the first lease renewal request 1 hour after the IP address was allocated.

[0024] Processor 102 can execute probability generator 112, which causes network device 100 to predict the utilization probability of lease time blocks among multiple lease time blocks based on historical lease patterns, device characteristics, service information, and DHCP information associated with client devices. In one example, the maximum duration for which lease renewals can be granted to client devices is divided into equal time slots. For example, the maximum duration for which renewals can be granted could be 24 hours, which could be divided into 48 time slots, each corresponding to a 30-minute lease time. Leased time blocks can represent lease time as multiples of such time slots. In this example, there can be 48 lease time blocks. Therefore, the first lease time block may include 1 time slot, the second lease time block may include 2 time slots, and the forty-eighth lease time block may include 48 time slots. The utilization probability of each lease time block can be calculated.

[0025] Some example parameters that can indicate a client device's historical lease patterns may include the total number of past lease renewals, lease start time, lease expiry time, and the number of past consecutive lease renewals. Some example parameters that can indicate a client device's device characteristics may include the client device's operating system, device type based on device fingerprint, VLAN category (e.g., internal device, guest device, and management device), and client location. Some example parameters that can indicate service information associated with a client device may include the type of network service accessed by the client device based on the service origin (e.g., direct service, recommendation service, search service, etc.), the reputation of services accessed by the client device based on a security profile of hostnames maintained by different Deep Packet Inspection (DPI) classification algorithms (e.g., high risk, low risk, secure, etc.), and the percentage of bandwidth consumed by the client device based on the bit rate of incoming services. Some example parameters that can indicate DHCP information associated with a client device may include a list of DHCP option requests based on the client device’s preferences (e.g., the subnet mask to be applied to the network interface of the client device requesting an IP address, the default router or last resort gateway for the client device’s network interface, the preferred domain name server (DNS) to be included in the IP configuration for name resolution, and the requested lease time for the IP address) and the load on the DHCP server in the network device.

[0026] In one example, an artificial neural network (ANN) is used to predict the utilization probability of each of multiple leased time blocks. In the examples described herein, an artificial neural network (ANN) refers to an information processing system comprising interconnected processing elements modeled as biological neural networks. These interconnected processing elements may be referred to as “artificial neurons” or “nodes.” An ANN may include multiple artificial neurons, which may be organized into multiple layers. In one instance, an ANN may include multiple layers: an input layer, (multiple) hidden layers, and an output layer. In one instance, an ANN may be a feedforward neural network, where the connections between units may not form loops. In a feedforward neural network, information can move in one direction from the input layer through (multiple) hidden layers to the output layer. There may be no loops or recurrences in the network.

[0027] In another example, an ANN can be based on a backpropagation architecture. Backpropagation can be used to train an ANN. When an input vector is presented to the ANN, it propagates forward layer by layer until it reaches the output layer. A loss function can be used to compare the network's output with the expected output, and an error value can be calculated for each artificial neuron in the output layer. The error value can be propagated backward from the output until each artificial neuron has an associated error value that roughly represents its contribution to the original output. Backpropagation can use these error values ​​to calculate the gradient of the loss function with respect to the weights in the network. This gradient can be fed to an optimization method, which can then use it to update the weights to minimize the loss function. As the artificial neural network is trained, neurons in the intermediate layers can organize themselves so that different neurons can learn to recognize different characteristics of the total input. After training with arbitrary input patterns, if a new input contains a pattern similar to the features that individual neurons have learned to recognize during their training, neurons in the network's (multiple) hidden layers can respond with their outputs.

[0028] In the examples described in this paper, parameters indicating historical lease patterns, device characteristics, service information, and DHCP information associated with the client device can be provided as input to the input layer of the ANN. In one example, the parameters can be encoded into numerical representations (multiple) using different encoding techniques (such as hot encoding) before being provided to the input layer. In another example, the parameters can also be processed using different dimensionality reduction techniques before being provided to the input layer. The output layer of the ANN can provide the utilization probability of each lease time block in a plurality of lease time blocks. In one example, the output layer is the ANN layer before normalizing the utilization probability of each lease time block in a plurality of lease time blocks. In one example, as described above, there can be 48 lease time blocks, and the utilization probability of each of these lease time blocks can be predicted.

[0029] Table 1 below shows example outputs from an ANN, including the utilization probability of each leased time block in multiple leased time blocks.

[0030] Table 1

[0031]

[0032] As shown in Table 1, for the first client device with MAC 1, the utilization probability of each of the N leased time blocks is PU=[0.001,0.08,……,0.04]; for the second client device with MAC 2, the utilization probability of each of the N leased time blocks is PU=[0.01,0.002,……,0.07]; for the third client device with MAC 3, the utilization probability of each of the N leased time blocks is PU=[0.007,0.08,……,0.005]; and for the fourth client device with MAC 4, the utilization probability of each of the N leased time blocks is PU=[0.002,0.07,……,0.06]. Therefore, the utilization probability is predicted for each leased time block of each client device connected to the network.

[0033] Processor 102 can execute lease renewal manager 114, which causes network device 100 to allocate lease time blocks from a plurality of lease time blocks to client devices for IP address lease renewal. Lease time blocks are allocated based on a combination of utilization probability and reward value. The reward value of a lease time block can be derived from a reinforcement learning (RL) model coupled to an ANN. The reward value refers to the reinforcement signal associated with an action of the RL model, where, at each iteration, the RL model learns a strategy to maximize the normalized reward value for that action. The normalized reward value refers to adjusting the adjusted reward value of the lease time block to a conceptually common proportion, and in some examples may include averaging the reward value over multiple iterations. In one example, the RL model can be trained based on feedback from the allocation of lease time blocks.

[0034] In one example, an RL model can use an exploration-exploitation technique, where the RL model can operate using the Epsilon-greedy algorithm in either the exploration or exploitation phase. In reinforcement learning, an agent or decision-maker learns how to map situations to actions to maximize rewards. Instead of explicitly signaling which actions to take, the agent discovers which actions yield the greatest reward through trial and error. In exploration, the agent performs trials of new actions in response to previously known or unknown situations. During the exploration phase, the agent may not rely on prior learning from the RL model to select actions. Exploration allows the agent to improve its learning for each action, thus yielding long-term benefits by increasing the accuracy of the mapping between situations and actions. In the "exploration" phase, the agent continuously explores to dynamically find more suitable actions. In exploitation, on the other hand, the agent selects actions to maximize rewards by exploiting its learning. However, if the agent chooses to exploit its learning in all or most iterations, it may not actually acquire the maximum cumulative reward in the long run, as new learning and improved learning accuracy can only be achieved through exploration. Since agents cannot simultaneously explore and exploit, a balance between the exploration and exploitation phases is needed for optimal behavior of the RL model. In one example, the epsilon-greedy algorithm provides a method for balancing exploration and exploitation by randomly selecting exploration and exploitation. According to the epsilon-greedy algorithm, epsilon ( ) refers to the probability of choosing to explore. Therefore, according to the epsilon-greedy algorithm, a random action can be chosen with probability η during exploration, otherwise, a random action can be chosen with probability (1-) during the exploitation phase. The action that generates the maximum reward is selected. During the exploration phase of the RL model, the processor 102 may execute the lease renewal manager 114, which causes the network device 100 to select a random lease time block from multiple lease time blocks for dispatching renewal requests in response to receiving the first lease time block.

[0035] In some examples, network device 100 may associate a reward value with each of a plurality of leased time blocks. For example, for a first client device, in the first iteration, network device 100 may associate a reward value with each of a plurality of leased time blocks, for example, reward value (RV) = [0.0003, 0.002, ..., 0.001], where the reward value for the first leased time block is 0.0003, the reward value for the second leased time block is 0.002, and the reward value for the Nth leased time block is 0.001. In some examples, processor 102 may execute lease renewal manager 114, which causes network device 100 to multiply the utilization probability of a leased time block by a normalized reward value to obtain the renewal probability of the leased time block. Therefore, the renewal probability of each leased time block can be obtained by multiplying the utilization probability of the leased time block by its corresponding normalized reward value. In one example, for the first client device, the renewal probability (RP) ... [..., 0.00004], where the renewal probability of the first lease time block is 0.0000003, the renewal probability of the second lease time block is 0.00016, and the renewal probability of the Nth lease time block is 0.00004. In some examples, during the development phase of the RL model, the processor 102 may execute the lease renewal manager 114, which causes the network device 100 to select the lease time block with the highest renewal probability from a plurality of lease time blocks for allocation. Therefore, in the above example, the maximum value in [0.0000003, 0.00016, ..., 0.00004] is determined, and the lease time block with the highest renewal probability can be selected for allocation.

[0036] Processor 102 can execute reward generator 116 to adjust the reward value of a leased time block for each allocation. For each allocation of a plurality of leased time blocks, the reward value associated with each of the plurality of leased time blocks is adjusted. The reward value is adjusted based on the following: network deployment characteristics, service load in the network, and network connection duration of client devices.

[0037] In one example, processor 102 may execute reward generator 116, which causes network device 100 to determine that the network has a sparse client device deployment, such as in a home network. In another example, network device 100 may determine the deployment density of client devices in the network. If the client devices in the network are not highly concentrated, for example, the deployment density of client devices is less than 1 client / 3 square meters, then network device 100 may determine that the network has a sparse client device deployment. Furthermore, processor 102 may execute reward generator 116, which causes network device 100 to determine that the network has a high traffic load, i.e., a traffic flow rate exceeding a certain threshold. For networks with sparse client device deployment or high traffic load, in response to a client device disconnecting from the network before its allocated lease time expires, processor 102 may execute reward generator 116, which causes network device 100 to increase the reward value by a fraction of the unused lease time. The unused lease time corresponds to the duration from when the client device disconnects from the network to when its allocated lease time expires. In one example, the reward value can be increased by the ratio "unused rental time / allocated rental time".

[0038] In another example, for a network with sparse client device deployment or high traffic load, in response to a network device receiving a second lease renewal request from a client device, processor 102 can execute reward generator 116, which causes network device 100 to reduce the reward value by a predefined penalty. In one example, the predefined penalty is a fixed integer value specified by the network administrator.

[0039] For example, in a network with sparse deployment or high traffic load, if the first client device disconnects from the network before the expiration of its assigned first lease time block, the reward value of 0.0003 associated with the first lease time block can be increased. Considering that the first client device disconnects from the network after half of the first lease time block has expired, the increment of the reward value = (unused lease time / assigned lease time) = (0.5 lease time blocks / 1 lease time block) = 0.5. Therefore, the reward value increases to 0.0003 + 0.5 = 0.50003. Alternatively, if the first client device sends a second lease renewal request to network device 100, indicating that the first client device intends to remain connected to the network after the expiration of the first lease time block, the reward value of the first lease time block is reduced. In one example, the reduction in reward value = 0.0003 - 2 = -0.1997. It can be noted that in the example above, reducing the reward value in response to receiving a second lease renewal request indicates that the reward value is adjusted to support higher lease time allocation in order to monitor the number of lease renewal requests. As a result, it prevents a network already under high traffic load from being clogged by frequent lease renewal requests from client devices. Furthermore, due to the sparse deployment of client devices in the network, allocating longer lease times to connected clients may not have a significant impact on the DHCP pool of available IP addresses.

[0040] In another example, processor 102 may execute reward generator 116, which causes network device 100 to determine that the network has a dense deployment of client devices, such as an office network or a public network. If the client devices in the network are highly concentrated, for example, the deployment density of client devices is greater than 1 client / 1.5 square meters, such as in a conference hall, classroom, lecture hall, auditorium, sports field, etc., then network device 100 can determine that the network has a very dense deployment of client devices. In another example, if the network has a deployment density of more than 1 client / 10 square meters, such as in a conference hall, classroom, lecture hall, auditorium, or sports field, then network device 100 can determine that the network has a dense deployment of client devices. Furthermore, processor 102 may execute reward generator 116, which causes network device 106 to determine that the network has a low service load, i.e., a service flow rate below a certain threshold. For networks with densely deployed client devices or low traffic loads, in response to a client device disconnecting from the network before its allocated lease time expires, processor 102 can execute reward generator 116, which causes network device 100 to reduce the reward value by a fraction of the unused lease time. The unused lease time corresponds to the duration from when the client device disconnects from the network until the allocated lease time expires. In one example, the reward value could be reduced by a ratio of "unused lease time / allocated lease time".

[0041] In another example, for a network with densely deployed client devices or low traffic load, in response to a network device receiving a second lease renewal request from a client device, processor 102 can execute reward generator 116, which causes network device 100 to increase a predefined bonus value. In one example, the predefined bonus is a fixed integer value specified by the network administrator.

[0042] In some examples, in networks with dense deployments or low traffic loads, if a first client device disconnects from the network before the expiration of its allocated first lease time block, the reward value of 0.0003 associated with the first lease time block can be reduced. Considering that the first client device disconnects from the network after half of the first lease time block has expired, the reduction in reward value = (unused lease time / allocated lease time) = (0.5 lease time blocks / 1 lease time block) = 0.5. Therefore, the reward value is reduced to 0.0003 - 0.5 = -0.4997. Alternatively, if the first client device sends a second lease renewal request to network device 100, indicating that the first client device intends to remain connected to the network after the expiration of the allocated first lease time block, the reward value of the first lease time block is increased. In one example, the reward value increases to = 0.0003 + 2 = 2.0003, where the predefined bonus is "2". In the example above, increasing the reward value in response to a client device sending a second lease renewal request indicates that the reward value is adjusted to support shorter lease time allocations, preventing unnecessary congestion of the pool of IP addresses available for allocation, given the already densely deployed client devices on the network. While the example above explaining the adjustment of the reward value is based on the first client device and the first lease time block, in other examples, the reward value for each lease time block can be similarly adjusted for each client device.

[0043] Processor 102 may execute reward generator 116, which causes network device 100 to determine a normalized reward value for a leased time block based on reward values ​​for leased time blocks across multiple assignments. In one example, reward generator 116 may include instructions for implementing a reinforcement learning (RL) model to determine the normalized reward value. In one example, the RL model may use an exploration exploitation technique to determine the reward value, wherein the RL model may operate using an Epsilon-greedy algorithm in either the exploration or exploitation phase. A normalized reward value may be determined for each of the multiple leased time blocks. In one example, the normalized reward value for a leased time block may be calculated as "total reward value of leased time blocks over N iterations / N". In one example, the normalized reward value may be used to determine the renewal probability in subsequent iterations.

[0044] Figure 2 This is a flowchart of an example method 200 for DHCP lease assignment. The system in which method 200 is performed can be similar to the network device 100 described above.

[0045] refer to Figure 2In box 202, method 200 may include receiving a DHCP lease request from a client device connected to the network by a network device. In one example, the client device may send a DHCP lease request in response to receiving a DHCP offer message from a network device (e.g., a DHCP server). In box 204, method 200 may include assigning an IP address to the client device for a lease period based on the DHCP lease request from the client device.

[0046] In box 206, method 200 may include receiving a first lease renewal request from a client device by a network device. In box 208, method 200 may include predicting the probability of utilization for a lease time block from a plurality of lease time blocks based on: historical lease patterns associated with the client device, device characteristics, service information, and DHCP information.

[0047] In box 210, method 200 may include allocating leased time blocks to client devices for IP address lease renewals based on a combination of the utilization probability and reward value of the leased time block. The reward value of the leased time block may be derived from a reinforcement learning (RL) model coupled to an ANN. In box 212, the method may include adjusting the reward value of the leased time block for each allocation. In box 214, method 200 may include determining a normalized reward value for the leased time block based on the reward values ​​of leased time blocks over multiple allocations.

[0048] Figure 3A This is a flowchart of an example method for adjusting the reward value used for renting time blocks. The system in which method 300 is executed can be similar to the network device 100 described above. In one example, method 300 can be implemented in a network with low client density.

[0049] refer to Figure 3A In box 302, method 300 may include determining whether the network has a sparse client device deployment or a high traffic load. In one example, the network device may determine that the network has a sparse deployment based on the deployment density of client devices in the network. A high traffic load may correspond to a traffic flow rate exceeding a certain threshold.

[0050] When it is determined that the network has sparse client device deployments or high service load, in box 304, method 300 may include checking whether the client device disconnected from the network before the allocated lease time block expired. In box 306 (the "Yes" branch of box 304), in response to determining that the client device disconnected from the network before the allocated lease time block expired, method 300 may include increasing the reward value by a small fraction of the unused lease time.

[0051] In box 308 (the "No" branch of box 304), in response to determining that the client device has not disconnected from the network before the allocated lease time block expires, method 300 may include checking whether a renewal request has been received by the network device. In response to determining that a renewal request has been received (the "Yes" branch of box 308), in box 310, method 300 may include reducing the reward value by a predefined penalty. In response to determining that no renewal request has been received (the "No" branch of box 308), in box 312, the lease time may remain unchanged.

[0052] Figure 3B This is a flowchart of an example method 350 for adjusting the reward value used for rented time blocks. The system in which method 350 is implemented can be similar to the network device 100 described above. In one example, method 350 can be implemented in a network with a high client density.

[0053] refer to Figure 3B In box 352, method 350 may include determining whether the network has a dense deployment of client devices or a low traffic load. In one example, the network device may determine that the network has a dense deployment based on the deployment density of client devices in the network. Low traffic load may correspond to a traffic flow rate in the network that is below a certain threshold.

[0054] When it is determined that the network has a dense deployment of client devices or that the network has low traffic load, in box 354, method 350 may include checking whether the client device disconnected from the network before the allocated lease time block expired. In box 356 (the "Yes" branch of box 354), in response to determining that the client device disconnected from the network before the allocated lease time block expired, method 350 may include reducing the reward value by a fraction of the unused lease time.

[0055] In box 358 (the "No" branch of box 354), in response to determining that the client device has not disconnected from the network before the allocated lease time block expires, method 350 may include checking whether a renewal request has been received by the network device. In response to determining that a renewal request has been received (the "Yes" branch of box 358), in box 360, method 350 may include increasing the reward value by a predefined bonus. In response to determining that no renewal request has been received (the "No" branch of box 358), in box 362, the lease time may remain unchanged.

[0056] Figure 4 This is a flowchart of an example method 400 for allocating lease time blocks for lease renewal. The system in which method 400 is executed can be similar to the network device 100 described above.

[0057] refer to Figure 4In box 402, method 400 may include multiplying the utilization probability of a leased time block by a normalized reward value to obtain the renewal probability of the leased time block. Thus, the renewal probability of each of a plurality of leased time blocks can be determined. In box 404, method 400 includes checking whether the RL model is in the development phase. In response to determining that the RL model is in the development phase (the "Yes" branch of box 404), in box 406, method 400 may include selecting the leased time block with the highest renewal probability from the plurality of leased time blocks for allocation.

[0058] In response to determining that the RL model is not in the development phase (the "No" branch of box 404), it can be determined that the RL model is in the exploration phase. In response to determining that the RL model is in the exploration phase, in box 408, method 400 may include selecting a random lease time block from a plurality of lease time blocks for allocation to the client device requesting renewal.

[0059] Figure 5 This is an example computing device 500 implemented according to one or more publicly available examples. The computing device 500 has a hardware processor 501 and machine-readable instructions stored on a machine-readable medium 502 for implementing an example system. In one example, the computing device 500 may be a network device, such as the one referenced above. Figure 1 The network device 100 is described. Figure 5 A computing device 500 is shown, configured to execute the instructions 504, 506, 508, 510, 512, 514, and 516 described below. However, the computing device 500 may also be configured to perform other methods, techniques, functions, or processes described in this disclosure, such as method 300 of FIG. 3. Figure 3B Method 350 and Figure 4 Method 400.

[0060] Processing elements such as processor 501 may include one or more hardware processors, each of which may have single or multiple processor cores. In one embodiment, processor 501 may include at least one shared cache that stores data (e.g., compute instructions) used by one or more other components of processor 501. For example, the shared cache may be local cached data stored in memory for faster access by components constituting the processing element of processor 501. In one or more embodiments, the shared cache may include one or more intermediate caches, such as Level 2 (L2), Level 3 (L3), Level 4 (L4), or other levels of cache, last-level cache (LLC), or combinations thereof. Examples of processors include, but are not limited to, central processing units (CPUs) and microprocessors. Although in Figure 4Not shown, but the processing elements constituting processor 501 may also include one or more other types of hardware processing components, such as graphics processing unit (GPU), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), and / or digital signal processor (DSP).

[0061] Processor 501 may be operatively and communicatively coupled to memory. Memory may be a non-transitory computer-readable medium configured to store various types of data, such as machine-readable storage medium 502. For example, memory may include one or more storage devices, including non-volatile storage devices and / or volatile memory. Volatile memory, such as random access memory (RAM), may be any suitable non-persistent storage device. Non-volatile storage device 820 may include one or more disk drives, optical disk drives, solid-state drives (SSDs), split drives, flash memory, read-only memory (ROM), and / or any other type of memory designed to maintain data for a period of time after a power outage or shutdown operation. In some respects, non-volatile storage devices may be used to store overflow data if the allocated RAM is insufficient to hold all working data. Non-volatile storage devices may also be used to store programs that are selected for execution and loaded into RAM.

[0062] Figure 5 The machine-readable storage medium 502 may include volatile and non-volatile removable and non-removable media, and may be any electronic, magnetic, optical, or other physical storage device that contains or stores executable instructions, data structures, program modules, or other data accessible to the processor, such as firmware, erasable programmable read-only memory (EPROM), random access memory (RAM), non-volatile random access memory (NVRAM), optical disc, solid-state drive (SSD), flash memory chip, etc. The machine-readable storage medium may be a non-transitory storage medium, wherein the term "non-transitory" excludes transient propagation signals.

[0063] Machine-readable medium 502 includes instructions 504, which, when executed by processor 501, cause a network device to receive a DHCP lease request from a client device connected to the network. Instruction 506, when executed by the processor, causes the network device to allocate an IP address to the client device within the lease period based on the DHCP lease request. Furthermore, instruction 508, when executed by processor 501, causes the network device to receive a first lease renewal request from the client device. Instruction 510, when executed by processor 501, causes the network device to predict the probability of utilization for a lease time block from multiple lease time blocks based on: historical lease patterns associated with the client device, device characteristics, service information, and DHCP information. Instruction 512, when executed by processor 501, causes the network device to allocate lease time blocks to client devices for IP address lease renewal based on a combination of the lease time block utilization probability and a reward value. For each allocation of a leased time block, instruction 514, when executed by processor 501, causes the network device to adjust the reward value of the leased time block based on the following factors: network deployment characteristics, network traffic load, and network connection duration of client devices. Furthermore, instruction 516, when executed by processor 501, causes the network device to determine a normalized reward value for the leased time block based on the reward values ​​for leased time blocks across multiple allocations.

[0064] Certain terms are used throughout the specification and claims to refer to specific system components. As those skilled in the art will understand, different parties may use different names to refer to components. This document is not intended to distinguish between components with different names but different functions. In this disclosure and claims, the terms "including" and "comprising" are used in an open-ended manner and should therefore be interpreted as meaning "including but not limited to...". Furthermore, the terms "coupled" or "couples" are intended to indicate indirect or direct wired or wireless connections. Thus, if a first device is coupled to a second device, the connection can be either a direct connection or an indirect connection via other devices and connections. The reference to "based on" is intended to mean "at least partially based on". Thus, if X is based on Y, X can be a function of Y and many other factors.

[0065] The foregoing discussion is intended to illustrate the principles and various implementations of this disclosure. Once the foregoing disclosure is fully understood, many variations and modifications will become apparent to those skilled in the art. The following claims are intended to be interpreted as encompassing all such variations and modifications.

Claims

1. A method for a network, comprising: receiving a dynamic host configuration protocol (DHCP) lease request from a client device connected to the network; allocating an internet protocol (IP) address to the client device for a lease time based on the DHCP lease request; receiving a first lease renewal request from the client device; predicting a utilization probability for a lease time block from a plurality of lease time blocks based on historical lease patterns, device characteristics, traffic information, and DHCP information associated with the client device; assigning the lease time block to the client device for lease renewal of the IP address based on a combination of the utilization probability and a reward value for the lease time block; adjusting the reward value of the lease time block based on deployment characteristics of the network, traffic load in the network, and network connection duration of the client device for each assignment of the lease time block; and determining a normalized reward value for the lease time block based on reward values of the lease time block over a plurality of assignments.

2. The method of claim 1, wherein adjusting the reward value comprises: determining that the network has a sparse client device deployment or the network has a high traffic load; increasing the reward value by a portion of unused lease time in response to the client device disconnecting from the network before an assigned lease time block expires; and decreasing the reward value by a predefined penalty in response to receiving a second lease renewal request from the client device.

3. The method of claim 1, wherein adjusting the reward value comprises: determining that the network has a dense client device deployment or the network has a low traffic load; increasing the reward value by a predefined bonus in response to receiving a second lease renewal request from the client device; and decreasing the reward value by a portion of unused lease time in response to the client device disconnecting from the network before the assigned lease time block expires.

4. The method of claim 1, wherein assigning the lease time block comprises: multiplying the utilization probability for the lease time block by the normalized reward value to obtain a renewal probability for the lease time block; selecting a random lease time block from the plurality of lease time blocks for assignment in an exploration phase of a reinforcement learning model that derives the reward value; and selecting a lease time block with a maximum renewal probability from the plurality of lease time blocks for assignment in a development phase of the reinforcement learning model.

5. The method of claim 1, wherein the historical lease patterns are represented by a number of past lease renewals, a minimum start time, a lease expiration time, and a number of past consecutive lease renewals of the client device.

6. The method of claim 1, wherein the traffic information associated with the client device is represented by a bandwidth consumption of the client device, a reputation of traffic accessed by the client device, and a network traffic category accessed by the client device. ​ 7. The method of claim 1, wherein the device characteristics of the client device are represented by: a device type based on a device fingerprint, an operating system of the client device, a location of the client device, a VLAN category of the client device.

8. The method of claim 1, wherein the DHCP information associated with the client device is represented by a DHCP option request list and loaded on a DHCP server.

9. A network device, comprising: a processor; and a non-transitory computer-readable medium comprising instructions that, when executed by the processor, cause the network device to: receive a dynamic host configuration protocol (DHCP) lease request from a client device connected to a network; allocate an internet protocol (IP) address to the client device for a lease time based on the DHCP lease request; receive a first lease renewal request from the client device; predict a utilization probability for a lease time block from a plurality of lease time blocks based on: historical lease patterns associated with the client device, device characteristics, traffic information, and DHCP information; assign the lease time block to the client device for lease renewal of the IP address based on a combination of the utilization probability of the lease time block and a reward value; for each assignment of the lease time block, adjust the reward value for the lease time block based on: deployment characteristics of the network, traffic load in the network, and network connection duration of the client device; and determine a normalized reward value for the lease time block based on reward values for the lease time block over a plurality of assignments.

10. The network device of claim 9, wherein the instructions to adjust the reward value, when executed by the processor, cause the network device to: determine that the network has a sparse client device deployment or the network has a high traffic load; in response to the client device disconnecting from the network before the assigned lease time block expires, increase the reward value by a portion of unused lease time; and in response to receiving a second lease renewal request from the client device, decrease the reward value by a predefined penalty.

11. The network device of claim 9, wherein the instructions to adjust the reward value, when executed by the processor, cause the network device to: determine that the network has a dense client device deployment or the network has a low traffic load; in response to receiving a second lease renewal request from the client device, increase the reward value by a predefined bonus; and in response to the client device disconnecting from the network before the assigned lease time block expires, decrease the reward value by a portion of unused lease time.

12. The network device of claim 9, wherein the instructions to assign the lease time block, when executed by the processor, cause the network device to: multiply the utilization probability of the lease time block by the normalized reward value to obtain a renewal probability for the lease time block; ​ ​ ​ selecting, during an exploration phase of a reinforcement learning model that derives the reward value, a random rental time block from the plurality of rental time blocks for assignment; and selecting, during a development phase of the reinforcement learning model, a rental time block with a maximum renewal probability from the plurality of rental time blocks for assignment.

13. The network device of claim 9, wherein each rental time block of the plurality of rental time blocks represents rental time as a multiple of a time slot.

14. The network device of claim 9, wherein the historical rental pattern is represented by a number of past rental renewals, a minimum start time, a rental expiration time, and a number of past consecutive rental renewals of the client device.

15. The network device of claim 9, wherein the traffic information associated with the client device is represented by a bandwidth consumption of the client device, a reputation of traffic accessed by the client device, and a network traffic category accessed by the client device.

16. The network device of claim 9, wherein the device characteristics of the client device are represented by a device type based on a device fingerprint, an operating system of the client device, a location of the client device, a VLAN category of the client device.

17. The network device of claim 9, wherein the DHCP information associated with the client device is represented by a DHCP option request list and loaded on a DHCP server.

18. A non-transitory computer-readable medium comprising instructions that, when executed by a processor, cause a network device to: receive a dynamic host configuration protocol (DHCP) rental request from a client device connected to a network; allocate an internet protocol (IP) address to the client device for a rental time based on the DHCP rental request; receive a first rental renewal request from the client device; predict a utilization probability for a rental time block from a plurality of rental time blocks based on historical rental patterns, device characteristics, traffic information, and DHCP information associated with the client device; assign the rental time block to the client device for rental renewal of the IP address based on a combination of the utilization probability and a reward value for the rental time block; for each assignment of the rental time block, adjust the reward value for the rental time block based on deployment characteristics of the network, traffic load in the network, and network connection duration of the client device; and determine a normalized reward value for the rental time block based on reward values for the rental time block over a plurality of assignments.

19. The non-transitory computer-readable medium of claim 18, wherein the instructions to adjust the reward value, when executed by the processor, cause the network device to: determine that the network has a sparse client device deployment or the network has a high traffic load; in response to the client device disconnecting from the network before the assigned rental time block expires, increase the reward value by a portion of unused rental time; and ​ ​ in response to receiving a second lease renewal request from the client device, decreasing the reward value by a predefined penalty.

20. The non-transitory computer-readable medium of claim 18, wherein the instructions to adjust the reward value, when executed by the processor, cause the network device to: determine that the network has a dense client device deployment or the network has a low traffic load; in response to receiving a second lease renewal request from the client device, increase the reward value by a predefined bonus; and in response to the client device disconnecting from the network before the assigned lease time block expires, decrease the reward value by a portion of unused lease time.

Images