Address allocation method and device, electronic equipment, storage medium and computer program product

Abstract

The invention provides an address allocation method and device, electronic equipment, a storage medium and a computer program product, and the method comprises the steps: collecting first data; the first data is used for describing network flow characteristics and the use condition of the first address pool; the first address pool comprises one or more first public network addresses; calling a first reinforcement learning model to process the first data to obtain a first distribution strategy; the first reinforcement learning model is used for determining an optimal action corresponding to the first state in an action space of the first reinforcement learning model based on the input data, and outputting a distribution strategy corresponding to the optimal action; adjusting a first corresponding relation between the first public network address and the private network address based on a first allocation strategy, and performing address allocation based on the adjusted first corresponding relation; the first corresponding relation is used for distributing the first public network address to the network flow initiated by the first private network address.

Description

Technical Field

[0001] This application relates to the field of communication technology, and in particular to an address allocation method, apparatus, electronic device, storage medium, and computer program product. Background Technology

[0002] In related technologies, source network address translation (SNAT) is performed based on modulo operations to allocate addresses in the address pool, thereby mapping private network traffic to public network addresses in the address pool. However, when traffic is unbalanced or the address pool has capacity limitations, the load corresponding to the public network addresses in the address pool is unbalanced, resulting in low network resource utilization. Summary of the Invention

[0003] To address the related technical problems, embodiments of this application provide an address allocation method, apparatus, electronic device, storage medium, and computer program product.

[0004] The technical solution of this application embodiment is implemented as follows: This application provides an address allocation method, the method comprising: Collect first data; the first data is used to describe network traffic characteristics and the usage of the first address pool; the first address pool includes one or more first public network addresses; The first reinforcement learning model is invoked to process the first data to obtain a first allocation policy; the first reinforcement learning model is used to determine the optimal action corresponding to the first state in the action space of the first reinforcement learning model based on the input data, and output the allocation policy corresponding to the optimal action; the first state represents the state corresponding to the input data in the state space of the first reinforcement learning model; Based on the first allocation strategy, the first correspondence between the first public network address and the private network address is adjusted, and address allocation is performed based on the adjusted first correspondence. The first correspondence is used to allocate the first public network address to network traffic initiated by the first private network address, and the first private network address represents a private network address that has a first correspondence with the first public network address.

[0005] In the above scheme, the state space of the first reinforcement learning model is constructed based on the following factors: number of network connections, network bandwidth utilization, network traffic volume, and usage of the first public IP address.

[0006] In the above scheme, the action space of the first reinforcement learning model is constructed based on the first action; the first action represents the action of assigning the first public network address to network traffic.

[0007] In the above scheme, after address allocation based on the adjusted first correspondence, the method further includes: The first reinforcement learning model is updated based on multiple first training samples; the first training samples are constructed based on the first data, the first allocation strategy, the second data, and the third data; the second data is used to describe the network traffic characteristics and the usage of the first address pool after address allocation based on the adjusted first correspondence; the third data is used to describe the network performance and the load balancing of the first address pool after address allocation based on the adjusted first correspondence.

[0008] In the above scheme, each of the plurality of first training samples is represented as a quadruple of information. The quadruple of information corresponding to the first training sample includes: the first state corresponding to the first data, the first action corresponding to the first allocation strategy, the first reward, and the first state corresponding to the second data; the first reward is determined based on the third data.

[0009] In the above scheme, the third data includes one or more of the following: network latency, network throughput, and utilization rate of each first public network address in the first address pool.

[0010] In the above scheme, the first reinforcement learning model is represented as a Deep Deterministic Policy Gradient (DDPG) model.

[0011] This application also provides an address allocation device, including: The data acquisition unit is used to acquire first data; the first data is used to describe network traffic characteristics and the usage of the first address pool; the first address pool includes one or more first public network addresses. The calling unit is used to call the first reinforcement learning model to process the first data and obtain a first allocation policy; the first reinforcement learning model is used to determine the optimal action corresponding to the first state in the action space of the first reinforcement learning model based on the input data, and output the allocation policy corresponding to the optimal action; the first state represents the state corresponding to the input data in the state space of the first reinforcement learning model; The allocation unit is used to adjust the first correspondence between the first public network address and the private network address based on the first allocation strategy, and to allocate addresses based on the adjusted first correspondence; the first correspondence is used to allocate the first public network address to network traffic initiated by the first private network address, and the first private network address represents a private network address that has a first correspondence with the first public network address.

[0012] This application also provides an electronic device, including: a processor and a memory for storing a computer program capable of running on the processor; wherein, when the processor runs the computer program, it performs the steps of any of the methods described above.

[0013] This application also provides a storage medium storing a computer program thereon, characterized in that the computer program, when executed by a processor, implements the steps of any of the above methods.

[0014] This application also provides a computer program product, including a computer program, characterized in that, when the computer program is executed by a processor, it implements the steps of any of the above methods.

[0015] In this embodiment, first data is collected, which describes network traffic characteristics and the usage of a first address pool, including one or more first public network addresses. Then, a first reinforcement learning model is invoked to process the first data to obtain a first allocation strategy. The first reinforcement learning model is used to determine the optimal action corresponding to the first state in the action space of the first reinforcement learning model based on the input data, and outputs the allocation strategy corresponding to the optimal action. The first state represents the state corresponding to the input data in the state space of the first reinforcement learning model. Afterward, based on the first allocation strategy, the first correspondence between the first public network address and the private network address is adjusted, and address allocation is performed based on the adjusted first correspondence. The first correspondence is used to allocate the first public network address to network traffic initiated by the first private network address. The first private network address represents the private network address that has the first correspondence with the first public network address. In the above scheme, the correspondence between private network addresses and public network addresses in the address pool is dynamically adjusted by processing network traffic characteristics and address pool usage through a reinforcement learning model. In this way, public network addresses in the address pool can be dynamically allocated based on network traffic characteristics and address pool usage. Compared with related technologies, the correspondence between private network addresses and public network addresses in the address pool is not fixed. Therefore, even when traffic is unbalanced or the address pool has capacity limitations, the load balancing of public network addresses in the address pool can be guaranteed, thereby improving network resource utilization. Attached Figure Description

[0016] Figure 1 This is a functional diagram of an address pool; Figure 2 This is a schematic diagram of the architecture of a reinforcement learning model; Figure 3 A schematic diagram illustrating the implementation process of an address allocation method provided in this application embodiment; Figure 4 A schematic diagram illustrating model training as provided in an embodiment of this application; Figure 5 A schematic diagram of the architecture of an address allocation system provided for an application embodiment of this application; Figure 6 This is a schematic diagram of the structure of an address allocation device provided in an embodiment of this application; Figure 7 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0017] In modern network architectures, address pools are a key component for achieving high availability networks. See also Figure 1 Through the address pooling function, a Network Address Translation (NAT) gateway can use multiple public IP addresses to provide services to the same private subnet, thereby achieving a balanced distribution of outbound traffic from the private network. For example, when the volume of private network traffic, such as outbound traffic from a private subnet, is large, the address pool can distribute the traffic load through multiple public Internet Protocol (IP) addresses, preventing a single public IP address from being overloaded and improving load balancing.

[0018] In related technologies, address allocation in the SNAT address pool is performed based on modulo operations, thereby mapping private network traffic to public network addresses in the address pool. Specifically, the source address of the device in the private network can be obtained first, and a hash function can be called to process the source address to generate a hash value used as a unique identifier for the source address. The source address can also be understood as the private network address, and the hash value can be used for subsequent load balancing. Then, a modulo operation is performed on the hash value according to the number of public network addresses in the address pool to calculate the index value corresponding to the hash value. The index value can be less than or equal to the number of public network addresses in the address pool. Afterwards, based on the index value, the outbound traffic from the source address is mapped to the corresponding public network address, thereby achieving load balancing through address allocation.

[0019] However, address allocation based on modulo operations results in a fixed correspondence between public IP addresses in the address pool and private IP addresses of devices. Therefore, relatively even load distribution can only be achieved when the traffic characteristics from devices are evenly distributed. When traffic is unbalanced or the address pool has capacity limitations, some public IP addresses can become overloaded, while others are underutilized. This makes effective traffic management and maximizing public IP address utilization difficult, leading to an imbalance in the load across public IP addresses in the address pool and low network resource utilization.

[0020] Based on this, in this embodiment of the application, first data is collected, wherein the first data is used to describe network traffic characteristics and the usage of a first address pool, the first address pool including one or more first public network addresses; then, a first reinforcement learning model is invoked to process the first data to obtain a first allocation strategy, the first reinforcement learning model is used to determine the optimal action corresponding to the first state in the action space of the first reinforcement learning model based on the input data, and output the allocation strategy corresponding to the optimal action, wherein the first state represents the state corresponding to the input data in the state space of the first reinforcement learning model; then, based on the first allocation strategy, the first correspondence between the first public network address and the private network address is adjusted, and address allocation is performed based on the adjusted first correspondence; the first correspondence is used to allocate the first public network address to network traffic initiated by the first private network address, the first private network address representing the private network address that has the first correspondence with the first public network address. In the above scheme, the correspondence between private network addresses and public network addresses in the address pool is dynamically adjusted by processing network traffic characteristics and address pool usage through a reinforcement learning model. In this way, public network addresses in the address pool can be dynamically allocated based on network traffic characteristics and address pool usage. Compared with related technologies, the correspondence between private network addresses and public network addresses in the address pool is not fixed. Therefore, even when traffic is unbalanced or the address pool has capacity limitations, the load balancing of public network addresses in the address pool can be guaranteed, thereby improving network resource utilization.

[0021] To facilitate understanding, before further describing the specific implementation methods of this application, the main concepts involved in the embodiments of this application will be explained.

[0022] Reinforcement learning model: A type of machine learning model that learns the optimal policy through the interaction between an agent and its environment.

[0023] In practical applications, see Figure 2 A reinforcement learning model can consist of five parts: agent, environment, state, action, and reward. The agent can continuously interact with the environment, decide on an action based on the current state, and receive a reward from the environment as feedback, thereby continuously optimizing its behavioral strategy. In this way, by selecting behavior through a reward and punishment mechanism, the model can maximize long-term cumulative rewards.

[0024] State space: In a reinforcement learning model, the set of all possible states that an agent can be in the environment.

[0025] Action space: In a reinforcement learning model, the set of all possible actions that an agent can perform.

[0026] Reward function: In reinforcement learning models, a function used to quantify the immediate feedback obtained by an agent after performing an action. The quantized value corresponding to this immediate feedback is also known as the reward value.

[0027] The present application will now be described in further detail with reference to the accompanying drawings and embodiments.

[0028] This application provides an address allocation method. In practical applications, this method can be applied to one or more of the following: a gateway, a NAT gateway, or a first device for implementing gateway functions.

[0029] See Figure 3 The method includes: Step 301: Collect the first data.

[0030] The first data is used to describe network traffic characteristics and the usage of the first address pool; the first address pool includes one or more first public network addresses.

[0031] In practical applications, the first public IP address in the first address pool can be used to handle network traffic from private IP addresses, and this network traffic can be represented as network requests. The first address pool can be represented as an SNAT address pool. For example, the first public IP address can be represented as a public IP address.

[0032] When network traffic is processed through the first public IP address, the network traffic can be considered as the load of the first public IP address. Processing network traffic through the first public IP address can also be understood as one or more of the following: mapping network traffic to the first public IP address, assigning network traffic to the first public IP address, or assigning the first public IP address to network traffic.

[0033] In practical applications, the current network traffic status and the usage of the first address pool can be monitored in real time, and the real-time monitoring data can be collected to obtain the first data.

[0034] In practical applications, the first data may include multi-dimensional information describing network traffic characteristics and the usage of the first address pool.

[0035] For example, the first data may include one or more of the following: number of network connections, network bandwidth utilization, network traffic volume, and information describing the usage of the first public IP address.

[0036] Step 302: Call the first reinforcement learning model to process the first data and obtain the first allocation strategy.

[0037] The first reinforcement learning model is used to determine the optimal action corresponding to the first state in the action space of the first reinforcement learning model based on the input data, and output the allocation strategy corresponding to the optimal action; the first state represents the state corresponding to the input data in the state space of the first reinforcement learning model.

[0038] In practical applications, the first allocation strategy can be used to indicate how addresses in the first address pool are allocated; that is, it can be used to dynamically determine which first public IP address in the first address pool is allocated to a certain network traffic. The first allocation strategy can also be understood as an address allocation strategy.

[0039] In practical applications, the state space of the first reinforcement learning model can be constructed based on multi-dimensional indicators. Each indicator corresponds to information in one dimension, and this information can be used to describe network traffic characteristics and the usage of the first address pool. In this way, the first reinforcement learning model can more comprehensively evaluate the operational status and changes of the network environment, thereby making decisions adapted to the actual network environment.

[0040] In practical applications, the state space of the first reinforcement learning can also be understood as a multi-dimensional state space.

[0041] In one embodiment, the state space of the first reinforcement learning model is constructed based on the following factors: number of network connections, network bandwidth utilization, network traffic volume, and usage of the first public IP address.

[0042] In practical applications, the network connection count can be used to indicate the number of ongoing network connections. The network connection count can also be understood as the number of sessions; for example, the network connection count can be determined based on the number of 5-tuples of information in the network.

[0043] Network bandwidth utilization can be used to indicate the proportion of bandwidth used in a network link to the total available bandwidth, thus measuring the efficiency of network resource utilization.

[0044] Network traffic volume can be used to indicate the amount of data transmitted over a network per unit of time, thus reflecting the intensity of overall network activity.

[0045] The usage of a first public IP address can be used to reflect its load. For example, the usage of a first public IP address can be described based on one or more of the following: the utilization rate of the first public IP addresses in the first address pool, and traffic information of the network traffic processed by the first public IP address. The utilization rate of the first public IP addresses in the first address pool can be used to indicate the proportion of the first public IP addresses used in the first address pool relative to the total number of first public IP addresses in the first address pool. Traffic information can, for example, include one or more of the following: source address, traffic volume.

[0046] In practical applications, based on the first data, the first state corresponding to the first data can be determined. The first state can be understood as a state of the first agent corresponding to the first reinforcement learning model, and the first state corresponding to the first data can be understood as the current state of the first agent.

[0047] In practical applications, the action space of the first reinforcement learning model can be used to indicate the set of operations that the first reinforcement learning model can perform.

[0048] In one embodiment, the action space of the first reinforcement learning model is constructed based on a first action; the first action represents the action of assigning a first public IP address to network traffic.

[0049] In practical applications, the first action can also be understood as the action of choosing which first public IP address to route the network request corresponding to the network traffic.

[0050] Here, by constructing a first reinforcement learning model based on the first action, the decision scope of the first reinforcement learning model can be limited to actions such as allocating a certain first public network address to network traffic. This allows the first reinforcement learning model to focus on optimizing the allocation strategy of the first public network address, improving the accuracy of the decision, avoiding interference from irrelevant actions, and reducing the complexity of processing.

[0051] In practical applications, the action space of the first reinforcement learning model can be represented as a continuous action space rather than a discrete action space, which can more accurately simulate the actual address allocation action, making the determined first allocation strategy closer to the actual allocation scenario and avoiding the decrease in the accuracy of address allocation due to discretization.

[0052] In one embodiment, the first reinforcement learning model is characterized as a DDPG model.

[0053] In practical applications, the DDPG model can include an Actor network and a Critic network. The Actor network outputs the policy corresponding to the optimal action. The Critic network evaluates the expected reward that the optimal action determined by the Actor network can obtain in a given state, thus assisting in the training of the DDPG model; the Critic network can represent this expected reward through its output Q-value.

[0054] In practical applications, calling the first reinforcement learning model to process the first data may include calling the Actor network to process the first data.

[0055] In practical applications, the DDPG model can be applied to continuous action spaces, thus ensuring the adaptability of the first assignment strategy output by the first reinforcement learning model to the actual assignment scenario. Step 303: Based on the first allocation strategy, adjust the first correspondence between the first public network address and the private network address, and allocate addresses based on the adjusted first correspondence.

[0056] The first correspondence is used to assign the first public network address to network traffic initiated by the first private network address, and the first private network address represents a private network address that has a first correspondence with the first public network address.

[0057] In practical applications, adjusting the first correspondence between the first public network address and the first private network address may include one or more of the following: setting a first correspondence between the first public network address and the first private network address, changing the private network address that has a first correspondence with the first public network address from the second private network address to the first public network address, or canceling the first correspondence between the first public network address and the second private network address.

[0058] In practical applications, by adjusting the first correspondence through the first allocation strategy, the correspondence between the first public network address and the private network address can be dynamically adapted to the actual network environment status and changes. Based on this, address allocation based on the adjusted first correspondence can allocate address resources more flexibly, effectively balance the load of each first public network address in the first address pool, thereby improving the utilization rate of the first public network address and the utilization rate of network resources.

[0059] For example, suppose the first address pool includes a first public network address 1 and a first public network address 2, wherein the first public network address 1 and the private network address 1 have a first correspondence relationship, the first public network address 1 is running at near full load, and the first public network address 2 is relatively idle; based on this, through the first allocation strategy, the first correspondence relationship between the first public network address 1 and the private network address 1 can be canceled, and the first public network address 2 and the private network address 1 can be set to have a first correspondence relationship. In this way, network traffic from the private network address 1 can be mapped to the relatively idle first public network address 2, thereby avoiding overload of the first public network address 1 and improving the load balancing of the first public network addresses in the first address pool.

[0060] In this embodiment, a reinforcement learning model is used to process network traffic characteristics and address pool usage to dynamically adjust the correspondence between private network addresses and public network addresses in the address pool. This allows for dynamic allocation of public network addresses in the address pool based on network traffic characteristics and address pool usage. Compared to related technologies, where the correspondence between private network addresses and public network addresses in the address pool is not fixed, load balancing of public network addresses in the address pool can be guaranteed even when traffic is unbalanced or the address pool has capacity limitations, thereby improving network resource utilization.

[0061] The training method for the first reinforcement learning model will be further explained below.

[0062] In practical applications, the first reinforcement learning model can be trained based on multiple training samples. Each training sample can be represented as a four-tuple of data; for example, the data format of this four-tuple data can be... ,in, It can represent a state of the first agent. It can characterize the first intelligent agent in Actions performed in a certain state It can characterize the first intelligent agent in Perform an action in a certain state The rewards received It can characterize the state of the first agent. The next state after that.

[0063] See Figure 4 The following section uses the DDPG model as an example to illustrate the training process of the first reinforcement learning model. In practical applications, this process mainly includes the following steps: Step 1: Initialization.

[0064] In practical applications, the weight parameters of the Actor network and Critic network can be initialized, as can the weight parameters of the target Actor network and target Critic network, and the experience replay buffer can be initialized. Here, for ease of description, the weight parameters of the Actor network are represented as... The weight parameters of the Critic network are expressed as follows: The weight parameters of the target Actor network are expressed as follows: The weight parameters of the Critic network are expressed as follows: .

[0065] The target Actor network and target Critic network can be understood as the target networks corresponding to the Actor network and Critic network, respectively. The Actor network and Critic network can be understood as the main networks corresponding to the target Actor network and target Critic network, respectively. The target network can be used to improve the stability and convergence of the first reinforcement learning model. The initial values ​​of the weight parameters of the target network can be the same as the initial values ​​of the weight parameters of the main network.

[0066] The experience replay pool can be used to store experience data, which can be understood as training samples.

[0067] Step 2: Experience gathering.

[0068] In practical applications, empirical data can be collected and / or constructed in real network environments. The data format of empirical data can be... Then, the collected experience data can be stored in the experience playback pool.

[0069] In practical applications, Figure 4 The management module can be used to provide allocation strategies for address allocation, and the network system can be considered equivalent to the actual network environment. Empirical data can be collected from the management module and the network system.

[0070] Step 3: Training process.

[0071] In practical applications, the first reinforcement learning model can be trained and iterated multiple times until the set convergence condition is met.

[0072] During each iteration, a batch of empirical data can be randomly drawn from the empirical replay pool. This batch of empirical data can be understood as the training samples used in that iteration. Here, for ease of explanation, the number of empirical data used in each iteration is expressed as... .

[0073] In practical applications, the first reinforcement learning model is used for each piece of empirical data. The following processing can be performed: (1) Use the target Actor network to calculate the action of the next state. .

[0074] (2) Calculate the corresponding Q-value using the target Critic network, that is, .

[0075] (3) Calculate the target Q value. The target Q value can be expressed as: , It can be represented as a discount factor.

[0076] (4) Use the Critic network to calculate the Q-values ​​of the current state and action, that is, .

[0077] (5) Calculate the loss value of the Critic network. The corresponding loss function can be expressed as: .

[0078] (6) Optimize the weight parameters of the Critic network through gradient descent, thereby minimizing the loss function. .

[0079] (7) Use the Actor network to calculate the action of the current state. .

[0080] (8) Calculate the policy gradient of the Actor network. The policy gradient can be expressed as: .

[0081] (9) Optimize the weight parameters of the Actor network through gradient ascent, thereby maximizing the policy gradient. .

[0082] Step 4: Target network update.

[0083] In practical applications, the weight parameters of the Actor and Critic networks can be periodically soft-updated to the target network. The soft-updation method for the weight parameters can be expressed as follows: , in, It can be represented as the step size of a soft update.

[0084] In practical applications, the training samples used to train the first reinforcement learning model can be constructed based on historical data and / or real-time environmental feedback data. That is, the first reinforcement learning model can be trained based on historical data and / or real-time environmental feedback data. Figure 4 The empirical data in the corresponding examples can be collected from historical data and / or real-time environmental feedback data. Real-time environmental feedback data can also be understood as real-time data or feedback data.

[0085] In practical applications, the first reinforcement learning model can be trained before it is called to process the first data. Correspondingly, the first reinforcement learning model used to process the first data can have completed the training.

[0086] Historical data can be understood as data collected before the first reinforcement learning model processes the first data. For example, historical data may include data constructed based on address allocation requirements. Training samples for training the first reinforcement learning model can be constructed based on historical data.

[0087] In practical applications, after the first reinforcement learning model is called to process the first data, the first reinforcement learning model can be retrained to update it and enhance its processing capabilities.

[0088] Real-time environmental feedback data can be understood as data acquired in real time by processing the first data using the first reinforcement learning model. Training samples for updating the first reinforcement learning model can be constructed based on this real-time feedback data. In this way, the first reinforcement learning model can be updated based on its actual processing performance, increasing its adaptability to real-world network environments and enabling it to quickly respond to complex network conditions, further enhancing its processing capabilities.

[0089] In practical applications, corresponding to the situation where the first reinforcement learning model is trained based on real-time environmental feedback data, in one embodiment, After address allocation based on the adjusted first correspondence, the address allocation method provided in this application embodiment further includes: The first reinforcement learning model is updated based on multiple first training samples; the first training samples are constructed based on first data, a first allocation strategy, second data, and third data; the second data is used to describe the network traffic characteristics and the usage of the first address pool after address allocation based on the adjusted first correspondence; the third data is used to describe the network performance and the load balancing of the first address pool after address allocation based on the adjusted first correspondence.

[0090] In practical applications, the first training sample can be understood as a training sample constructed based on real-time feedback data.

[0091] The first data can be used to determine a state of the first agent; that is, it can be used to determine... Figure 4 In the corresponding example The first allocation strategy can be used to determine the action taken by the first agent in the state corresponding to the first data; that is, it can be used to determine... Figure 4 In the corresponding example The second data can be used to determine the next state after the state corresponding to the first data; that is, it can be used to determine... Figure 4 In the corresponding example The third set of data can be used to determine the reward received by the first agent after performing the action corresponding to the first allocation strategy; that is, it can be used to determine... Figure 4 In the corresponding example .

[0092] In one embodiment, each of the plurality of first training samples is represented as a quadruple of information. The quadruple of information corresponding to the first training sample includes: a first state corresponding to the first data, a first action corresponding to the first allocation strategy, a first reward, and a first state corresponding to the second data; the first reward is determined based on the third data.

[0093] In practical applications, the first state corresponding to the first data can be... Figure 4 In the corresponding example The first action corresponding to the first allocation strategy can be... Figure 4 In the example The first reward can correspond to Figure 4 In the example The first state corresponding to the second data can be... Figure 4 In the example .

[0094] In one embodiment, the third data includes one or more of the following: network latency, network throughput, and utilization rate corresponding to each first public IP address in the first address pool.

[0095] In practical applications, network latency can be used to indicate the time interval that a data packet takes to travel from the sender to the receiver during transmission. Network latency can also reflect the network's response speed.

[0096] Network throughput indicates the total amount of data transmitted by a network within a specified time period. It reflects network bandwidth utilization and concurrent processing capacity.

[0097] The utilization rate of each public IP address in the first address pool can be used to describe the frequency or proportion of its use within a set time period. The utilization rate of each public IP address can reflect its activity level and load balancing.

[0098] In practical applications, the third data can be processed based on a predefined reward function to obtain the first reward. This reward function can be used to provide the first agent with corresponding rewards based on network performance after address allocation and the load balancing of the first address pool.

[0099] For example, based on the defined reward function, one or more of the following rewards can be applied: If the variance of the utilization rate of each public network address in the first address pool is less than the first set threshold, the reward value of the first reward is set to 1, that is, a reward of 1 point is provided. If the variance of the utilization rate of each public network address in the first address pool is greater than the first set threshold, the reward value of the first reward is set to 0, that is, no reward is provided. If the network latency exceeds the second set threshold, the reward value of the first reward is set to 0, that is, no reward is provided.

[0100] In this embodiment, the first reward is determined based on third data, that is, the first reward is determined by the network performance after address allocation and the load balancing of the first address pool. In this way, the actual impact of address allocation on the network environment can be accurately reflected. On this basis, the first reinforcement learning model is updated to enhance the processing capability of the first reinforcement learning model, so that more reasonable decisions can be made in the subsequent address allocation process, ensuring the load balancing of the first public network address in the first address pool, thereby improving the utilization rate of network resources.

[0101] The present application will be further described in detail below with reference to application embodiments.

[0102] This application provides an address allocation system that can be used to execute the address allocation method provided in this application. In practical applications, the system can be deployed on one or more of the following: a gateway, a NAT gateway, or a first device for implementing gateway functions.

[0103] See Figure 5 The system may include: a data acquisition module, a reinforcement learning module, a management module, a monitoring and feedback module, and a database module.

[0104] The data acquisition module can be used to monitor indicators such as network traffic and response time, and transmit the data to the reinforcement learning module in real time. The data collected by the data acquisition module may include the first data and the second data in the embodiments of this application.

[0105] The reinforcement learning module can receive data from the data acquisition module, train and optimize the model, output the optimal public network address allocation strategy, and then transmit it to the management module. The reinforcement learning module can be equipped with the first reinforcement learning model in this embodiment, and the public network address allocation strategy output by the reinforcement learning module can be equivalent to the first allocation strategy in this embodiment.

[0106] In practical applications, reinforcement learning modules can combine reinforcement learning models with multiple factors to enhance the accuracy of decisions when making optimal public network address allocation strategies. For example, these multiple factors may include one or more of the following: network topology, service priority, and security policies.

[0107] The management module can dynamically adjust the mapping between private network addresses and public network addresses in the address pool based on the strategies provided by the reinforcement learning module. It then synchronizes the allocation rules corresponding to this mapping to modules and / or devices with gateway capabilities, enabling these allocation rules to be used for allocating public network addresses—that is, making the allocation rules effective. Allocation rules can be understood as mapping rules describing the mapping relationship; synchronizing the allocation rules to modules and / or devices with gateway capabilities can also be understood as delivering the allocation rules to the network system.

[0108] In practical applications, the management module can also send the allocation rules to the monitoring and feedback module. The management module can also send the action information corresponding to the allocation rules back to the reinforcement learning module.

[0109] In practical applications, when the address allocation system is used to allocate addresses in the SNAT address pool, the management module can also be described as the SNAT management module.

[0110] The monitoring and feedback module can be used to monitor the execution of the allocation rules output by the management module and obtain network-related performance indicator data. This performance indicator data can be equivalent to the third data in the embodiments of this application.

[0111] The monitoring and feedback module can transmit performance index data back to the reinforcement learning module as feedback information, which can be used to provide feedback on reward status.

[0112] The reinforcement learning module can construct training samples based on action information sent by the management module, feedback information sent by the monitoring and feedback module, and data collected by the data acquisition module, thereby enabling model optimization based on real-time feedback data.

[0113] In practical applications, the monitoring and feedback module can also detect abnormal situations in address allocation. For example, abnormal situations may include one or more of the following: address resource exhaustion, abnormal traffic. When an abnormality is detected, an automatic recovery mechanism can be triggered in a timely manner, and an early warning can be issued.

[0114] The database module can be used to store information related to status, actions, and rewards as historical data, which can also be used for model training and optimization.

[0115] In the application embodiments of this application, the address allocation system can achieve intelligent address allocation based on the reinforcement learning module, dynamically adjust the correspondence between private network addresses and public network addresses in the address pool, thereby dynamically adjusting the allocation rules used for address allocation. In this way, public network addresses can be allocated efficiently and reasonably. Through the allocation and recycling mechanism of public network addresses, the load balancing degree corresponding to public network addresses is increased, and resource utilization is improved.

[0116] Furthermore, in the application embodiments of this application, a data-driven decision-making mechanism is used to combine real-time and historical data. In this way, the allocation strategy can be adaptively and continuously optimized, making the processing of the address allocation system more efficient, flexible and reliable in dynamic network environments. This can significantly reduce address conflicts and network congestion, and improve service quality.

[0117] In addition, the modular design of the address allocation system makes the system structure clear and easy to maintain, reduces manual intervention, lowers operation and maintenance costs, and avoids the risk of human error.

[0118] Based on the embodiments described above, this application also provides an address allocation device, see [link to previous document]. Figure 6 The device includes: The acquisition unit 61 is used to acquire first data; the first data is used to describe network traffic characteristics and the usage of the first address pool; the first address pool includes one or more first public network addresses; Calling unit 62 is used to call a first reinforcement learning model to process the first data and obtain a first allocation strategy; the first reinforcement learning model is used to determine the optimal action corresponding to the first state in the action space of the first reinforcement learning model based on the input data, and output the allocation strategy corresponding to the optimal action; the first state represents the state corresponding to the input data in the state space of the first reinforcement learning model; The allocation unit 63 is used to adjust the first correspondence between the first public network address and the private network address based on the first allocation strategy, and to perform address allocation based on the adjusted first correspondence; the first correspondence is used to allocate the first public network address to network traffic initiated by the first private network address, and the first private network address represents a private network address that has a first correspondence with the first public network address.

[0119] In one embodiment, the state space of the first reinforcement learning model is constructed based on the following factors: number of network connections, network bandwidth utilization, network traffic volume, and usage of the first public IP address.

[0120] In one embodiment, the action space of the first reinforcement learning model is constructed based on a first action; the first action represents the action of assigning a first public IP address to network traffic.

[0121] In one embodiment, the address allocation device further includes a training unit, the training unit being used for: After address allocation based on the adjusted first correspondence, the first reinforcement learning model is updated based on multiple first training samples. The first training samples are constructed based on the first data, the first allocation strategy, the second data, and the third data. The second data is used to describe the network traffic characteristics and the usage of the first address pool after address allocation based on the adjusted first correspondence. The third data is used to describe the network performance and the load balancing of the first address pool after address allocation based on the adjusted first correspondence.

[0122] In one embodiment, each of the plurality of first training samples is represented as a quadruple of information, and the quadruple of information corresponding to the first training sample includes: a first state corresponding to the first data, a first action corresponding to the first allocation strategy, a first reward, and a first state corresponding to the second data; the first reward is determined based on the third data.

[0123] In one embodiment, the third data includes one or more of the following: network latency, network throughput, and utilization rate of each first public IP address in the first address pool.

[0124] In one embodiment, the first reinforcement learning model is characterized as a DDPG model.

[0125] In practical applications, the acquisition unit 61, the calling unit 62, the allocation unit 63, and the training unit can be implemented by the processor in the address allocation device.

[0126] It should be noted that the address allocation device provided in the above embodiments is only illustrated by the division of the above program modules. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. In addition, the address allocation device and the address allocation method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0127] Based on the hardware implementation of the above program modules, and in order to implement the method of the embodiments of this application, this application also provides an electronic device, referring to... Figure 7 The electronic device includes: Communication interface 1 enables information exchange with other devices; Processor 2 is connected to communication interface 1 to enable information interaction with other devices and, when running a computer program, executes the methods provided by one or more technical solutions in the above embodiments. The computer program is stored in memory 3.

[0128] Specifically, the processor 2 is used to collect first data; the first data is used to describe network traffic characteristics and the usage of the first address pool; the first address pool includes one or more first public network addresses; The first reinforcement learning model is invoked to process the first data to obtain a first allocation policy; the first reinforcement learning model is used to determine the optimal action corresponding to the first state in the action space of the first reinforcement learning model based on the input data, and outputs the allocation policy corresponding to the optimal action; the first state represents the state corresponding to the input data in the state space of the first reinforcement learning model; and, Based on the first allocation strategy, the first correspondence between the first public network address and the private network address is adjusted, and address allocation is performed based on the adjusted first correspondence. The first correspondence is used to allocate the first public network address to network traffic initiated by the first private network address, and the first private network address represents a private network address that has a first correspondence with the first public network address.

[0129] In one embodiment, the state space of the first reinforcement learning model is constructed based on the following factors: number of network connections, network bandwidth utilization, network traffic volume, and usage of the first public IP address.

[0130] In one embodiment, the action space of the first reinforcement learning model is constructed based on a first action; the first action represents the action of assigning a first public IP address to network traffic.

[0131] In one embodiment, the processor 2 is further configured to: After address allocation based on the adjusted first correspondence, the first reinforcement learning model is updated based on multiple first training samples. The first training samples are constructed based on the first data, the first allocation strategy, the second data, and the third data. The second data is used to describe the network traffic characteristics and the usage of the first address pool after address allocation based on the adjusted first correspondence. The third data is used to describe the network performance and the load balancing of the first address pool after address allocation based on the adjusted first correspondence.

[0132] In one embodiment, each of the plurality of first training samples is represented as a quadruple of information, and the quadruple of information corresponding to the first training sample includes: a first state corresponding to the first data, a first action corresponding to the first allocation strategy, a first reward, and a first state corresponding to the second data; the first reward is determined based on the third data.

[0133] In one embodiment, the third data includes one or more of the following: network latency, network throughput, and utilization rate of each first public IP address in the first address pool.

[0134] In one embodiment, the first reinforcement learning model is characterized as a DDPG model.

[0135] It should be noted that the specific processing procedure of communication interface 1 can be understood by referring to the above method.

[0136] Of course, in practical applications, the various components in an electronic device are coupled together through bus system 4. It can be understood that bus system 4 is used to achieve communication and connection between these components. In addition to the data bus, bus system 4 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 7 The general will label all buses as Bus System 4.

[0137] The memory 3 in this embodiment is used to store various types of data to support operation in the electronic device. Examples of such data include any computer program used to operate on the electronic device.

[0138] The methods disclosed in the embodiments of this application can be applied to the processor 2, or implemented by the processor 2. The processor 2 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in the processor 2 or by instructions in the form of software. The processor 2 mentioned above may be a general-purpose processor, a DSP, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 2 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in the memory 3. The processor 2 reads the information in the memory 3 and combines its hardware to complete the steps of the aforementioned method.

[0139] In an exemplary embodiment, the electronic device may be implemented by one or more ASICs, DSPs, PLDs, CPLDs, FPGAs, general-purpose processors, controllers, MCUs, microprocessors, or other electronic components to perform the aforementioned method.

[0140] It is understood that the memory 3 in the embodiments of this application can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memories described in the embodiments of this application are intended to include, but are not limited to, these and any other suitable types of memories.

[0141] In an exemplary embodiment, this application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, such as a memory 3 storing a computer program, which can be executed by the processor 2 of an electronic device to complete the steps described in the aforementioned address allocation method.

[0142] Computer-readable storage media can be FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM, etc.

[0143] In an exemplary embodiment, this application also provides a computer program product, including a computer program that can be executed by a processor 2 of an electronic device to complete the steps described in the aforementioned address allocation method.

[0144] It should be noted that terms such as "first" and "second" are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0145] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, the term "one or more" in this document refers to any combination of at least two of any one or more elements from a set of A, B, and C. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set of A, B, and C. Additionally, the term "one or more" in this document is an exemplary expression and can be replaced with any possible expressions, such as one or more, at least one, or at least one of, etc.

[0146] Furthermore, the technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0147] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application.

Claims

1. An address allocation method, characterized in that, The method includes: Collect first data; the first data is used to describe network traffic characteristics and the usage of the first address pool; the first address pool includes one or more first public network addresses; The first reinforcement learning model is invoked to process the first data to obtain a first allocation policy; the first reinforcement learning model is used to determine the optimal action corresponding to the first state in the action space of the first reinforcement learning model based on the input data, and output the allocation policy corresponding to the optimal action; the first state represents the state corresponding to the input data in the state space of the first reinforcement learning model; Based on the first allocation strategy, the first correspondence between the first public network address and the private network address is adjusted, and address allocation is performed based on the adjusted first correspondence. The first correspondence is used to allocate the first public network address to network traffic initiated by the first private network address, and the first private network address represents a private network address that has a first correspondence with the first public network address.

2. The method according to claim 1, characterized in that, The state space of the first reinforcement learning model is constructed based on the following factors: number of network connections, network bandwidth utilization, network traffic volume, and usage of the first public IP address.

3. The method according to claim 1, characterized in that, The action space of the first reinforcement learning model is constructed based on the first action; the first action represents the action of assigning the first public network address to network traffic.

4. The method according to claim 1, characterized in that, After performing address allocation based on the adjusted first correspondence, the method further includes: The first reinforcement learning model is updated based on multiple first training samples; the first training samples are constructed based on the first data, the first allocation strategy, the second data, and the third data; the second data is used to describe the network traffic characteristics and the usage of the first address pool after address allocation based on the adjusted first correspondence; the third data is used to describe the network performance and the load balancing of the first address pool after address allocation based on the adjusted first correspondence.

5. The method according to claim 4, characterized in that, Each of the plurality of first training samples is represented as a quadruple of information, and the quadruple of information corresponding to the first training sample includes: the first state corresponding to the first data, the first action corresponding to the first allocation strategy, the first reward, and the first state corresponding to the second data; the first reward is determined based on the third data.

6. The method according to claim 4, characterized in that, The third data includes one or more of the following: network latency, network throughput, and utilization rate of each first public IP address in the first address pool.

7. The method according to claim 1, characterized in that, The first reinforcement learning model is characterized as the Deep Deterministic Policy Gradient Algorithm (DDPG) model.

8. An address allocation device, characterized in that, include: The data acquisition unit is used to acquire first data; the first data is used to describe network traffic characteristics and the usage of the first address pool; the first address pool includes one or more first public network addresses. The calling unit is used to call the first reinforcement learning model to process the first data and obtain the first allocation strategy; The first reinforcement learning model is used to determine the optimal action corresponding to the first state in the action space of the first reinforcement learning model based on the input data, and output the allocation strategy corresponding to the optimal action; The first state represents the state of the input data in the state space of the first reinforcement learning model; The allocation unit is used to adjust the first correspondence between the first public network address and the private network address based on the first allocation strategy, and to allocate addresses based on the adjusted first correspondence; the first correspondence is used to allocate the first public network address to network traffic initiated by the first private network address, and the first private network address represents a private network address that has a first correspondence with the first public network address.

9. An electronic device, characterized in that, include: A processor and a memory for storing a computer program capable of running on the processor; wherein, when the processor is used to run the computer program, it performs the steps of the method according to any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images