Wireless network resource allocation method based on constrained reinforcement learning

Abstract

The present invention relates to a constrained reinforcement learning-based wireless network resource allocation method, comprising the following steps: Step S1: dividing the 5G wireless network system into three network slices: enhanced mobile broadband slice, massive machine type communication slice, and ultra-reliable low latency communication slice; Step S2: transforming the 5G wireless network system optimization problem into an augmented Lagrangian problem based on the augmented Lagrangian method; Step S3: combining the augmented Lagrangian and SAC algorithms to construct an algorithmic model for resource allocation based on augmented Lagrangian reinforcement learning, and solving the augmented Lagrangian problem to obtain the optimal allocation solution. The present invention maximizes throughput under limited bandwidth resources.

Description

Technical Field

[0001] The present invention relates to the fields of wireless communications and computer technology, and in particular to a wireless network resource allocation method based on constrained reinforcement learning. Background Art

[0002] With the development of the Industrial Internet of Things (IIoT) and 5G, more and more devices are connected to the network, generating massive amounts of data in a short period of time. Currently, the International Telecommunication Union (ITU) has identified three major application scenarios for 5G: enhanced mobile broadband (eMBB), ultra-reliable low-latency communications (URLLC), and massive machine-type communications (mMTC). Different application scenarios have different network performance requirements. Using network slicing to address data in different scenarios is a major development direction for improving network utilization. Network slicing is the process of dividing a single physical network into multiple independent logical (virtual) networks using software-defined networking and network function virtualization technologies. Slices are isolated from each other, and a single network can provide the appropriate network slices for different scenarios. In the power industry, smart grids require a variety of devices in the power generation, transmission, transformation, distribution, and consumption processes, and network slicing is needed to improve network performance. Summary of the Invention

[0003] In view of this, the object of the present invention is to provide a wireless network resource allocation method based on constrained reinforcement learning, which not only constrains the allocated bandwidth but also satisfies the maximum throughput.

[0004] To achieve the above object, the present invention adopts the following technical solutions:

[0005] A method for allocating wireless network resources based on constrained reinforcement learning comprises the following steps:

[0006] Step S1: Divide the 5G wireless network system into three network slices: enhanced mobile broadband slice, massive machine type communication slice, and ultra-reliable low latency communication slice;

[0007] Step S2: Based on the augmented Lagrangian method, the 5G wireless network system optimization problem is converted into an augmented Lagrangian problem;

[0008] Step S3: Combine the augmented Lagrangian and SAC algorithms to build a SAC algorithm model based on the augmented Lagrangian, solve the augmented Lagrangian problem, and obtain the optimal allocation solution.

[0009] Furthermore, the step S1 is specifically as follows: assuming that each type of device in the 5G wireless network system is assigned to a designated slice;

[0010] The devices in the enhanced mobile broadband slice require a network with ultra-high throughput and use a collection of The massive number of devices in the machine-type communication slice requires a network with large-scale terminals, using a collection The devices in the ultra-reliable low-latency communication slice need a network with ultra-low latency, using a collection of express;

[0011] Then, use and Represents the device set and base station set; for different types of devices, each base station reserves a certain bandwidth, and the available bandwidth in the mth base station is denoted by B m Indicated by b um represents the bandwidth allocated to the u-th device, Considering the mth BS associated with the uth device, use P u Indicates the transmission power, using h um represents the power gain caused by the channel multipath effect of the device connected to the mth base station. Furthermore, the rate requirements of the device in the 5G wireless network system are as follows:

[0012] 1) Devices in enhanced mobile broadband slicing The rate requirements of these devices are expressed as Where R0 represents the set Minimum rate requirements for devices in the

[0013] 2) Devices in Massive Machine-Type Communication Slices Connect to The base station of the device will allocate at least a fixed bandwidth B0 to the device, which is expressed as

[0014] 3) Devices in high-reliability, low-latency communication slices The rate requirement of the device is expressed as Where L indicates that the device is The length of a single data packet sent in the packet, T0 indicates the length of a data packet sent to the collection The maximum transmission delay allowed for a single data packet on a device.

[0015] Furthermore, the step S2 is specifically as follows:

[0016] Using a binary vector x umTo indicate the user's affiliation, the minimum power P0 required for a given device to connect to a base station, and the transmission power of the mth base station is P m , when the distance d between the base station and the device um If the value is greater than a certain value, the base station and the user cannot connect immediately, that is, x um =0, then When the distance d um When it is less than a certain value, that is, when When , the base station and the device can be connected;

[0017] The situation of the u-th device is expressed as:

[0018]

[0019] The total bandwidth allocated by the mth base station is calculated as:

[0020]

[0021] The total rate of each type of equipment is expressed as Log(rate). For the u-th user in , the rate is calculated as:

[0022]

[0023] To measure proportional fairness across devices in the entire system, we express the total rate of each device class as "log(rate)". Furthermore, since the rate requirements of the three network slices differ, a weighted utility function is used to balance their contributions to the overall utility. Therefore, the objective optimization formula is:

[0024]

[0025]

[0026]

[0027]

[0028]

[0029]

[0030] where w u Represent the weights of different slices respectively; Represents the collection of devices in different network slices; R0 represents the collection The minimum rate requirement of the device in the group; B0 indicates the rate allocated to the group The minimum bandwidth of the device in the group; T0 means sending to the group The maximum transmission delay allowed for a single data packet on a device.

[0031] Combined with the Lagrange multiplier vector λ={λ1,…,λ M} and penalty term μ, the unconstrained optimization problem of the enhanced Lagrangian is constructed as

[0032]

[0033] in

[0034] Furthermore, in the 5G wireless network system, the power gain between the base station and its related devices is taken as the state, and the bandwidth allocated by the base station and the connection status between the base station and the device are taken as the action. Finally, based on the above optimization objectives, the reward function is set to

[0035] Furthermore, the augmented Lagrangian SAC algorithm includes a value network, a Q network, and a policy network. The parameters of the three networks are set to φ, ψ, and θ, respectively. Several Q networks are added to update the Lagrangian multiplier vector λ and the penalty term μ, which are then used to update the policy network. Finally, the maximum throughput is achieved while satisfying the bandwidth constraint, as follows:

[0036] (1) Initialize all network parameters, as well as the Lagrange multiplier vector λ and the penalty term μ;

[0037] (2) For each episode;

[0038] a) Observe the environment and input the state into the policy network;

[0039] b) The policy network selects an action and executes it;

[0040] c) Calculate rewards and total allocated bandwidth;

[0041] d) Update the experience replay library;

[0042] (3) For each step of gradient descent;

[0043] a) Randomly select several samples from the experience replay library;

[0044] b) Update parameters φ, ψ, θ, λ;

[0045] c) Update target value network parameters;

[0046] d) If the bandwidth constraint is not met, it needs to be increased;

[0047] (4) Repeat (2)-(3) until the preset requirements are met.

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] In the network slicing system, the present invention designs a hybrid action space, state space and reward function, and combines the augmented Lagrangian with the SAC algorithm to constrain the allocated bandwidth while meeting the maximum throughput. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is a schematic diagram of the network slicing system model structure in an embodiment of the present invention;

[0051] Figure 2 is a schematic diagram of the ALM-SAC algorithm proposed in an embodiment of the present invention;

[0052] Figure 3 This is a comparison chart of the reward performance under different bandwidth constraints of the algorithm proposed in the embodiment of the present invention;

[0053] Figure 4 1 is a diagram showing the constraints of the algorithm proposed in the embodiment of the present invention under different bandwidth constraints;

[0054] Figure 5 It is a performance comparison chart of the proposed algorithm and the benchmark algorithm in the example of the present invention. DETAILED DESCRIPTION

[0055] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0056] Please refer to Figure 1 The present invention provides a wireless network resource allocation method based on constrained reinforcement learning, which is characterized by comprising the following steps:

[0057] Step S1: Divide the devices of the 5G wireless network system into three typical network slices: enhanced mobile broadband slice, massive machine type communication slice and ultra-reliable low latency communication slice;

[0058] In this implementation, the 5G wireless network system includes equipment for cameras, VR, unmanned driving, smart meters, and intelligent distributed distribution automation.

[0059] In the network slicing system, we assume that each type of device is assigned to a designated slice, where the devices in the enhanced mobile broadband slice require a network with ultra-high throughput and are connected using a collection of Indicates that the massive machine-type communication slice part of the equipment requires a network with a large number of terminals, using a collection The devices in the ultra-reliable low-latency communication slice need a network with ultra-low latency, which is represented by the collection Then, we use and Denotes the set of devices and the set of base stations. In this work, for different types of devices, each base station reserves a certain bandwidth, and the bandwidth available in the mth base station is denoted by B m Indicates that the bandwidth allocated to the u-th device is b um express, Considering the mth BS associated with the uth device, use P u Indicates the transmit power, using h um represents the power gain caused by the channel multipath effect of the device connected to the mth base station. Therefore, for the above different network slices, the rate requirements of the three devices are as follows

[0060] 1) Devices in enhanced mobile broadband slicing In these slices, these devices need to transmit real-time high-definition video content remotely, which requires a relatively high network rate. Therefore, the rate requirement of these devices can be expressed as Where R0 represents the set The minimum speed requirement of the device.

[0061] 2) Devices in Massive Machine-Type Communication Slices In these slices, these devices do not have strict requirements on speed and latency. The devices in the can communicate normally and connect to The base station of the device will allocate at least a fixed bandwidth B0 to the device, which is expressed as

[0062] 3) Devices in ultra-reliable low-latency communication slices In these slices, these devices need to monitor, control and quickly isolate the distribution network, so the delay requirement is relatively high. The rate requirement of the device can be expressed as Where L indicates that the device is The length of a single data packet sent in the device. T0 indicates the length of a single data packet sent in the device. The maximum permissible transmission delay for sending a single data packet.

[0063] Step S2: To maximize throughput under limited bandwidth resources, an optimization objective is established. The original optimization problem is then transformed into an augmented Lagrangian problem. For the network slicing system described above, the state space and hybrid action space of the entire system are defined, along with a reward function designed based on the optimization objective.

[0064] In this embodiment, a binary vector x is used um To indicate the user's belonging. Given the minimum power P0 requirement for a device to connect to a base station, and the transmission power of the mth base station is P m , when the distance d between the base station and the deviceum If the value is greater than a certain value, the base station and the user cannot connect immediately, that is, x um =0, then When the distance d um When it is less than a certain value, that is, when When , the base station and the device can be connected. Since a device can only be connected to one base station at a time, the situation of the u-th device can be expressed as: Furthermore, the total bandwidth allocated by the m-th base station can be calculated as in It represents the set of devices that can be served by the m-th base station;

[0065] In order to measure the proportional fairness among devices in the whole system, the total rate of each type of device is expressed as "Log(rate)". For the u-th user in, the rate can be calculated as In this embodiment, the goal is to maximize the total rate while meeting the network requirements of the three types of devices. Since each type of device has different requirements for the network, we add corresponding weights to the rates of different types of devices to further ensure fairness. In addition, to measure the proportional fairness between devices in the entire system, we express the total rate of each type of device as "log(rate)". The optimization formula is expressed as

[0066]

[0067]

[0068]

[0069]

[0070]

[0071]

[0072] Combined with the Lagrange multiplier vector λ={λ1,…,λ M} and the parameter term μ of the penalty term, we construct the unconstrained optimization problem of the enhanced Lagrangian as

[0073]

[0074] in

[0075] For the network slicing system, the power gain between the base station and its related devices is taken as the state, followed by the bandwidth allocated by the base station and the connection status between the base station and the device as the action. Finally, based on the above optimization objectives, the reward function is set to

[0076] Step S3: Combine the augmented Lagrangian and SAC algorithms to build a SAC algorithm model based on the augmented Lagrangian, solve the augmented Lagrangian problem, and obtain the optimal allocation solution.

[0077] In this embodiment, reference Figure 2 , provides an augmented Lagrangian SAC algorithm, considering three networks:

[0078] 1) Value network, used to output the value of the state;

[0079] 2) Q network, used to estimate the value of action state;

[0080] 3) Policy network, used to output action probability distribution.

[0081] The parameters of these three networks are set to φ, ψ, and θ, respectively. To constrain bandwidth, we add multiple Q networks to update the λ and μ parameters, which are then used to update the policy network, ultimately achieving maximum throughput while satisfying bandwidth constraints. The specific process of the proposed augmented Lagrangian SAC algorithm is as follows:

[0082] 1) Initialize all network parameters, as well as the Lagrange multiplier vector λ and the penalty term μ;

[0083] 2) For each iteration;

[0084] 3) For each episode;

[0085] a) Observe the environment and input the state into the policy network;

[0086] b) The policy network selects an action and executes it;

[0087] c) Calculate rewards and total allocated bandwidth;

[0088] d) Update the experience replay library;

[0089] 4) For each step of gradient descent;

[0090] a) Randomly select a bunch of samples from the experience replay library;

[0091] b) Update parameters φ, ψ, θ, λ;

[0092] c) Update target value network parameters;

[0093] d) If the bandwidth constraint is not met, μ needs to be increased;

[0094] 5) End.

[0095] refer to Figure 2 , preferably, the architecture of the LM-SAC algorithm includes a soft state value network V with parameter φ φ , parameter is ψ R Soft Q-value network and a policy network π with parameters θ θ To estimate the allocated bandwidth, we also develop several soft Q-value networks The parameters are In state s, the network slicing system selects an action based on the policy network. After taking action a, the rewards and constraints in each base station are calculated separately to obtain the next state. Finally, these are stored in the experience replay library for subsequent update iterations.

[0096] refer to Figure 3 In this example, the rewards are plotted for the maximum bandwidths available in the BSs, 10, 12.5, and 15 MHz. As can be seen from the figure, ALM-SAC with a bandwidth of 15 MHz can achieve higher throughput and thus receive more rewards due to its greater bandwidth resources. Furthermore, the size of the action space is related to the bandwidth available for allocation, so the stability of the ALM-SAC algorithm decreases when the bandwidth is 15 MHz.

[0097] like Figure 4 , which is a Pareto front comparison diagram of the algorithm in the example of the present invention at different numbers of layers.

[0098] The bandwidth constraints are shown for the maximum bandwidths available in the base station, 10, 12.5, and 15 MHz. It can be clearly seen that the proposed ALM-SAC can meet the bandwidth requirements after 100 steps. This also shows that the proposed algorithm can provide an effective solution to constrained RL problems.

[0099] In this embodiment, if Figure 5 The following figure shows a comparison of the reward performance and constraints of different algorithms.

[0100] The proposed ALM-SAC algorithm is compared with the DDPG algorithm and the Random algorithm. As can be seen from the figure, the ALM-SAC algorithm significantly outperforms the DDPG algorithm and the Random algorithm due to its strong ability to handle both discrete and continuous action spaces. Overall, the proposed ALM-SAC algorithm not only satisfies limited bandwidth but also better finds a reasonable bandwidth allocation strategy.

[0101] The above description is only a preferred embodiment of the present invention. All equivalent changes and modifications made according to the scope of the patent application of the present invention should fall within the scope of the present invention.

Claims

1. A wireless network resource allocation method based on constrained reinforcement learning, characterized in that: The following steps are involved: Step S1: Divide the 5G wireless network system into three network slices: enhanced mobile broadband slice, massive machine type communication slice, and ultra-reliable low latency communication slice; Step S2: Based on the augmented Lagrangian method, the 5G wireless network system optimization problem is converted into an augmented Lagrangian problem; Step S3: Combining the augmented Lagrangian and SAC algorithms, constructing a SAC algorithm model based on the augmented Lagrangian, and solving the augmented Lagrangian problem to obtain the optimal allocation solution; The step S2 is specifically as follows: Using a binary vector x um To indicate the user's affiliation, the minimum power P0 required for a given device to connect to a base station, and the transmission power of the mth base station is P m , when the distance d between the base station and the device um If the value is greater than a certain value, the base station and the user cannot connect immediately, that is, x um =0, then When the distance d um When it is less than a certain value, that is, when When , the base station and the device can be connected; The situation of the u-th device is expressed as: The total bandwidth allocated by the mth base station is calculated as: in It represents the set of devices that can be served by the m-th base station; The total rate of each type of equipment is expressed as Log(rate). The u-th user in the u Calculated as: Assume that the target optimization formula is expressed as: where w u Represent the weights of different slices respectively; Represents the collection of devices in different network slices; R0 represents the collection The minimum rate requirement of the device in the group; B0 indicates the rate allocated to the group The minimum bandwidth of the device in the group; T0 means sending to the group The maximum transmission delay allowed for a single data packet on a device; Combined with the Lagrange multiplier vector λ={λ1,…,λ M } and penalty term μ, the unconstrained optimization problem of the enhanced Lagrangian is constructed as in In the 5G wireless network system, the power gain between the base station and its related devices is taken as the state, and the bandwidth allocated by the base station and the connection status between the base station and the device are taken as the action. The reward function is set as The augmented Lagrangian SAC algorithm includes a value network, a Q network, and a policy network. The parameters of the three networks are set to φ, ψ, and θ, respectively. Several Q networks are added to update the Lagrangian multiplier vector λ and the penalty term μ, which are then used to update the policy network. Finally, the maximum throughput is achieved while satisfying the bandwidth constraint.

2. The method for allocating wireless network resources based on constrained reinforcement learning according to claim 1, wherein: The step S1 specifically includes: assuming that each type of device in the 5G wireless network system is assigned to a designated slice; The devices in the enhanced mobile broadband slice require a network with ultra-high throughput and use a collection of The massive number of devices in the machine-type communication slice requires a network with large-scale terminals, using a collection The devices in the ultra-reliable low-latency communication slice need a network with ultra-low latency, using a collection of express; Then, use and Represents the device set and base station set; for different types of devices, each base station reserves a certain bandwidth, and the bandwidth available in the mth base station is denoted by B m Indicated by b um represents the bandwidth allocated to the u-th device, Considering the mth BS associated with the uth device, use P u Indicates the transmit power, using h um It represents the power gain caused by the channel multipath effect of the device connected to the mth base station.

3. The method for allocating wireless network resources based on constrained reinforcement learning according to claim 1, wherein: The speed requirements of the devices in the 5G wireless network system are as follows: 1) Devices in enhanced mobile broadband slicing The rate requirements of these devices are expressed as Where R0 represents the set Minimum rate requirements for devices in the 2) Devices in Massive Machine-Type Communication Slices Connect to The base station of the device will allocate at least a fixed bandwidth B0 to the device, which is expressed as 3) Devices in high-reliability, low-latency communication slices The rate requirement of the device is expressed as Where L indicates that the device is The length of a single data packet sent in the packet, T0 indicates the length of a data packet sent to the collection The maximum transmission delay allowed for a single data packet on a device.

4. The method for allocating wireless network resources based on constrained reinforcement learning according to claim 1, wherein: The augmented Lagrangian SAC algorithm includes a value network, a Q network, and a policy network. The parameters of the three networks are set to φ, ψ, and θ respectively. Several Q networks are added to update the Lagrangian multiplier vector λ and the penalty term μ, which are then used to update the policy network. Finally, the maximum throughput is achieved while satisfying the bandwidth constraint. The details are as follows: (1) Initialize all network parameters, as well as the Lagrange multiplier vector λ and the penalty term μ; (2) For each episode; a) Observe the environment and input the state into the policy network; b) The policy network selects an action and executes it; c) Calculate rewards and total allocated bandwidth; d) Update the experience replay library; (3) For each step of gradient descent; a) Randomly select several samples from the experience replay library; b) Update parameters φ, ψ, θ, λ; c) Update target value network parameters; d) If the bandwidth constraint is not met, μ needs to be increased; (4) Repeat (2)-(3) until the preset requirements are met.

Images