A dynamic resource allocation optimization method based on reinforcement learning in a heterogeneous network

Abstract

This invention discloses a dynamic resource allocation optimization method based on reinforcement learning in heterogeneous networks. Step 1: Establish a heterogeneous network model, clearly defining energy efficiency as the maximum optimization objective; treat the macro base station as an agent, establish a Markov decision process, and determine the state space, action space, and reward function; divide the coverage area of ​​the macro base station and the bandwidth of the heterogeneous network; train the heterogeneous network model using a D3QN network, set the maximum transmit power of the macro base station and small base stations, calculate the signal-to-interference-plus-noise ratio (SNR) from the macro base station to the user, and minimize the loss function; introduce a double DQN network into the dueling DQN network to obtain the value function and dominance function, determine the optimal heterogeneous network hyperparameters K and β, maximize the energy efficiency of the heterogeneous network, and achieve dynamic resource allocation optimization. This invention proposes a heterogeneous network model with a centralized reinforcement learning framework to achieve global optimization of dynamic resource allocation, and proposes using a D3QN network to optimize load balancing and interference coordination, maximizing the energy efficiency of the heterogeneous network and achieving dynamic resource allocation optimization.

Description

Technical Field

[0001] This invention relates to the field of communication system technology, and in particular to a dynamic resource allocation optimization method based on reinforcement learning in heterogeneous networks. Background Technology

[0002] The increasing number of terminals and the rapid growth in data service demands mean that traditional single-layer networks can no longer meet the needs of today's rapidly evolving technology, posing significant challenges to wireless communication networks. To alleviate this immense pressure, researchers have proposed heterogeneous networks. Consequently, current wireless access networks have evolved into heterogeneous networks composed of macro base stations that meet wide-area access requirements and small base stations that meet high-density access requirements in smaller areas. To support high-speed mobile data services and provide better coverage, next-generation cellular networks are expected to widely deploy microcell or small cell base stations, which can offload some users and traffic from traditional macro base stations. While network capacity and coverage have improved, heterogeneous networks face challenges in load balancing and interference coordination, requiring effective user association and resource allocation methods to address these issues. Summary of the Invention

[0003] This invention provides a dynamic resource allocation optimization method based on reinforcement learning in heterogeneous networks to achieve global optimization of dynamic resource allocation.

[0004] To achieve the above effects, the technical solution of the present invention is as follows:

[0005] A dynamic resource allocation optimization method based on reinforcement learning in heterogeneous networks includes the following steps:

[0006] Step 1: Establish a heterogeneous network model and define energy efficiency as the maximum optimization objective; treat the macro base station as an intelligent agent, establish a Markov decision process, and determine the state space, action space, and reward function;

[0007] Step 2: Divide the macro base station coverage area and heterogeneous network bandwidth; train the heterogeneous network model through the D3QN network, which includes a double DQN network and a dueling DQN network;

[0008] Step 3: Calculate the signal-to-interference-plus-noise ratio (SIR) from the macro base station to the user, set the maximum transmit power for the macro base station and the small base station, and minimize the loss function;

[0009] Step 4: Introduce a double DQN network into the dueling DQN network to obtain the value function and the dominance function, determine the optimal heterogeneous network hyperparameters K and β, maximize the energy efficiency of the heterogeneous network, and achieve dynamic resource allocation optimization.

[0010] Furthermore, in step 1, the heterogeneous network model has L base stations and multiple users n; the base stations include L1 macro base stations and L2 small base stations, and satisfy L1+L2=L.

[0011] Furthermore, after step 1, the following steps are also included: the power of each macro base station and small base station is evenly distributed to the sub-channels; the signal-to-interference-plus-noise ratio is greater than a preset threshold β, where the preset threshold β is a hyperparameter of the heterogeneous network; the dimensionality reduction of the action space is achieved by setting the action space to a series of discrete values ​​of the hyperparameters; resource allocation adopts a channel allocation scheme with partial shared deployment, where K sub-channels are shared by macro base stations and small base stations, the number of sub-channels of macro base stations is MK, and the range of the heterogeneous network hyperparameter K is 0 < K < M.

[0012] Furthermore, in step 2, the division of the macro base station coverage area is specifically as follows: the location of small base station users and macro base station users is determined according to the number and distribution of macro base stations and small base stations; the macro base station coverage area is taken as a macro cell and divided into three sectors, with the same number of small base stations deployed in each sector, and the macro cell is divided into the central area C1, C2, C3 and the cell edge area E1, E2, E3.

[0013] Furthermore, in step 2, the heterogeneous network bandwidth is specifically divided as follows: the heterogeneous network has M sub-channels, each with a bandwidth of W; the entire bandwidth of the heterogeneous network is divided into two main parts, c and e, where bandwidth e includes e1, e2, and e3; the main subcarrier transmit power is... It is divided into three parts, with the bandwidth of each part labeled e1, e2, and e3. Soft frequency reuse technology is used to allocate frequency domain resources for macro base station users, while the entire bandwidth is used for resource allocation for small base station cell users. Each cell uses subcarriers divided into primary subcarriers and secondary subcarriers. The transmission power of the primary subcarrier is greater than that of the secondary subcarrier. The primary subcarrier is used by users at the cell edge, and the secondary subcarrier is used by users at the cell center, thus overcoming the co-channel interference problem in orthogonal frequency division multiple access systems.

[0014] Furthermore, in step 2, training the heterogeneous network model using the D3QN network specifically involves:

[0015] Step 2.1: Based on state s t Action a t and reward r t The generated (s) t a t r t s t+1 ) is stored in the heterogeneous network playback pool memory D;

[0016] Step 2.2: Determine whether the amount of data in the playback pool memory D has reached a certain quantity. If it has, start training the heterogeneous network until convergence; if it has not reached a certain quantity, continue with step 2.1.

[0017] Step 2.3: The agent is a macro base station, and the D3QN network is used as the strategy. The state input of the agent is the measurement value of the entire heterogeneous network, and the action space output is the value of the hyperparameters K and β of the heterogeneous network.

[0018] Step 2.4: The agent continuously interacts with the heterogeneous network and updates the weights of the D3QN network based on the feedback; a reinforcement learning framework is set, with the state defined as the channel gain between all users and each base station, i.e., s. t =G ln , { }, ;

[0019] Step 2.5: The action space is set to the values ​​of the heterogeneous network hyperparameters K and β that the agent needs to select, as a solution to the joint UARA problem, i.e., the action space is a. t ={K1 K p β1 β Q}, where p is the possible value of the number of shared channels in PSD, and Q is the possible value of the signal-to-interference-plus-noise ratio (SINNR) threshold; input state, the agent calculates the probability of each hyperparameter K value and the probability of each hyperparameter β value; action space is ;

[0020] Step 2.6: Set the reward , w is the energy efficiency function. n In heterogeneous networks, all users have the same priority, where user weights are assigned. ;

[0021] Step 2.7: Train the heterogeneous network model according to the number of iterations to obtain the trained heterogeneous network model.

[0022] Furthermore, in step 3, the signal-to-interference-plus-noise ratio (SIR) from the macro base station to the user is specifically calculated as follows: the set of all base stations is represented as... The index of the macro base station is 0, and the remaining L are small base stations; the macro base station with index 0 exclusively occupies the signal-to-interference-plus-noise ratio at the macro base station user on subchannel m.

[0023] ,

[0024] in, G on N0 represents the channel gain from macro base station 0 to user n, and N0 represents the noise power on the sub-channel with bandwidth W.

[0025] Signal-to-interference-plus-noise ratio at the macro base station user on the shared sub-channel with macro base station = 0

[0026] ,

[0027] in,

[0028] Signal-to-interference-plus-noise ratio at small base station users on shared sub-channels of small base stations (1,...,L)

[0029] ,

[0030] in, ; Let represent the channel gain from the small base station to user n, including antenna gain, path loss, and shadow fading, assuming that all sub-channels receive the same channel gain from the base station to the user.

[0031] Furthermore, in step 3, the maximum transmit power of the macro base station and the small base station are respectively expressed as: and ; and They are respectively:

[0032] ;

[0033] ;

[0034] The data rate for any user is: ,

[0035] in, ; Indicates base station; R macro R represents the data rate of macro base station users; pico This represents the data rate of a small cell user; if user n and base station n are... If related, then ,otherwise ; The user scheduling factor is the proportion of time that a base station allocates to its connected users.

[0036] Without sacrificing generality, user scheduling is performed using the local proportional fairness criterion, which means that each base station allocates the same proportion of time to all its connected users; each user should be connected to one and only one base station at any given time.

[0037] Furthermore, in step 3, the loss function is minimized specifically, and the Q-table representing the state mapping to the action space is expressed as follows: Using weights Heterogeneous network approximator Represents an online network; the Q-learning method updates its weights to minimize the loss function. ,

[0038] in, s is the target value in the Double DQN network; t s t+1 Let be the states of the agent at times t and t+1, respectively. , These represent the action space of the energy body at times t and t+1, respectively. The weights are the target network weights; θ and γ represent the network parameters and attenuation factor of the online network, respectively; R t Q t These represent the reward and output of the Double DQN network, respectively; heterogeneous networks are introduced into the dueling DQN network to obtain the value function V(s) and the advantage function. ;

[0039] Will The data is stored in the replay pool memory D. During training, the heterogeneous network samples small batches of data uniformly and randomly from the replay pool memory D to train, reducing the correlation between training instances and avoiding obtaining local optima.

[0040] In step 4, a double DQN network is introduced into the dueling DQN network to obtain the value function and the dominance function. Specifically, in the dueling DQN network, the last layer of the double DQN network is divided into two sub-networks to estimate the value function V(s) and the dominance function of the dueling DQN network, respectively. Use a Double DQN network to output its connection layer. and ,in, Both μ and μ represent the weights of the fully connected layer; combined and The output of the dueling DQN network is obtained. for:

[0041]

[0042] To improve the stability of the D3QN network, the output of the dueling DQN network is... Subtracting a constant ensures that the dominance function of the action space remains unchanged relative to the sequence under the given state, thus narrowing the range of Q-values, i.e.:

[0043] .

[0044] For K shared sub-channels, a frequency reuse factor of 1 is selected to maximize resource utilization. When different users in adjacent cells use the same subcarrier, co-channel interference will occur, and users at the cell edge will be more affected.

[0045] In the above scheme, this invention effectively utilizes time, space, and frequency resources in heterogeneous networks. The D3QN network combines the concepts of Double DQN and Dueling DQN networks. The deployment of small base stations can eliminate coverage gaps and create hotspots, thereby improving coverage in heterogeneous networks.

[0046] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0047] This invention proposes a heterogeneous network model based on a centralized reinforcement learning framework, and proposes to use a D3QN network to optimize load balancing and interference coordination. By training the heterogeneous network model through the D3QN network, the signal-to-interference-plus-noise ratio from the macro base station to the user is calculated, the maximum transmit power of the macro base station and the small base station is set, the loss function is minimized, the energy efficiency of the heterogeneous network is maximized, and dynamic resource allocation optimization is achieved. Attached Figure Description

[0048] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the invention. To better illustrate this embodiment, some components in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0049] Figure 1 This is a schematic diagram of the optimization method provided in an embodiment of the present invention;

[0050] Figure 2a This is a schematic diagram of the macro base station bandwidth provided in an embodiment of the present invention;

[0051] Figure 2b This is a schematic diagram of macro base station power allocation provided in an embodiment of the present invention;

[0052] Figure 3 This is a flowchart illustrating the specific implementation of a heterogeneous network provided in this embodiment of the invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] Before introducing the embodiments of this application, the relevant terms involved in the embodiments of this application are first explained as follows:

[0055] D3QN: Dueling Double DQN, network;

[0056] Macrocell: Also known as macro cell, in the early stages of cellular mobile phone network construction, cells using cellular technology were called "macrocells". Macrocells are large areas, and base station transmitting antennas are usually erected on top of surrounding buildings; typically, there is no direct path between transmission and reception.

[0057] Soft frequency reuse: a technology for solving the regional allocation of spectrum resources in cellular communication systems; frequency reuse is an important technology for solving the regional allocation of spectrum resources in cellular communication systems.

[0058] Example 1

[0059] For easier understanding, please refer to Figure 1 This invention provides an embodiment of a dynamic resource allocation optimization method based on reinforcement learning in heterogeneous networks, comprising the following steps:

[0060] Step 1: Establish a heterogeneous network model, clearly defining energy efficiency as the maximum optimization objective; treat macro base stations as intelligent agents, establish a Markov decision process, and determine the state space, action space, and reward function; the heterogeneous network model has L base stations and multiple users n; the base stations include L1 macro base stations and L2 small base stations, and satisfy L1+L2=L;

[0061] Step 2: Divide the macro base station coverage area and heterogeneous network bandwidth; train the heterogeneous network model through the D3QN network to optimize long-term network efficiency over a period of time; the D3QN network includes double DQN network and dueling DQN network;

[0062] Step 3: Calculate the signal-to-interference-plus-noise ratio (SIR) from the macro base station to the user, set the maximum transmit power for the macro base station and the small base station, and minimize the loss function;

[0063] Step 4: Introduce a double DQN network into the dueling DQN network to obtain the value function and the dominance function, determine the optimal heterogeneous network hyperparameters K and β, calculate the energy efficiency of the heterogeneous network over a period of time, and realize dynamic resource allocation optimization to achieve the global optimum of dynamic resource allocation.

[0064] In this embodiment, the set of all base stations is represented as The index of the macro base station is 0, and the remaining L are small base stations; the maximum transmit power, antenna gain and path loss of the small base stations are the same; N users are evenly distributed in the macro cell.

[0065] Example 2

[0066] Specifically, based on Example 1, the solution will be described in conjunction with specific embodiments to further demonstrate its technical effects. Specifically:

[0067] Specifically, after step 1, the following steps are also included: the power of each macro base station and small base station is evenly distributed to the sub-channel; the signal-to-interference-plus-noise ratio is greater than a preset threshold β, where the preset threshold β is a hyperparameter of the heterogeneous network; and the connection problem of user association and resource allocation is simplified to the selection of heterogeneous network hyperparameters K and β.

[0068] Centralized deep reinforcement learning is employed to address the connectivity issues of user association and resource allocation, with the action space constrained to ensure feasibility. Dimensionality reduction of the action space is achieved by setting the action space as a series of discrete values ​​for hyperparameters. Resource allocation utilizes a partially shared channel allocation scheme, where K sub-channels are shared by macro base stations and small base stations. The number of sub-channels for macro base stations is MK, and the range of the heterogeneous network hyperparameter K is 0 < K < M. This addresses the user association problem, where each user has a limited number of connections to the small base station with the maximum signal-to-interference-plus-noise ratio (SINR).

[0069] Specifically, in step 2, the macro base station coverage area is divided as follows: the location of small base station users and macro base station users is determined based on the number and distribution of macro base stations and small base stations; the heterogeneous network is a two-layer heterogeneous network consisting of one macro base station and several small base stations.

[0070] like Figure 2b As shown, the coverage area of ​​the macro base station is taken as the macro cell and divided into three sectors. The same number of small base stations are deployed in each sector. The macro cell is divided into central areas C1, C2, and C3 and edge areas E1, E2, and E3. The central area is 2 / 3 of the total macro cell area. The reuse factor of the auxiliary subcarrier frequency in the central area is 1, and the reuse factor of the primary subcarrier frequency in the edge area is 3.

[0071] Specifically, such as Figure 2a As shown; in step 2, the heterogeneous network bandwidth is divided as follows: the heterogeneous network has M sub-channels, each with a bandwidth of W; the entire bandwidth of the heterogeneous network is divided into two parts, c and e, where bandwidth e includes e1, e2, and e3; the main subcarrier transmit power is... It is divided into three parts, with the bandwidth of each part labeled e1, e2, and e3. Soft frequency reuse technology is used to allocate frequency domain resources for macro base station users, while the entire bandwidth is used for resource allocation for small base station cell users. Each cell uses subcarriers divided into primary subcarriers and secondary subcarriers. The transmission power of the primary subcarrier is greater than that of the secondary subcarrier. The primary subcarrier is used by users at the cell edge, and the secondary subcarrier is used by users at the cell center, thus overcoming the co-channel interference problem in orthogonal frequency division multiple access systems.

[0072] Specifically, in step 2, training the heterogeneous network model using the D3QN network involves the following steps:

[0073] Step 2.1: Based on state s t Action a t and reward r t The generated (s) t a t r t s t+1 ) is stored in the heterogeneous network playback pool memory D;

[0074] Step 2.2: Determine whether the amount of data in the playback pool memory D has reached a certain quantity. If it has, start training the heterogeneous network until convergence; if it has not reached a certain quantity, continue with step 2.1.

[0075] Step 2.3: The agent is a macro base station, and the D3QN network is used as the strategy. The state input of the agent is the measurement value of the entire heterogeneous network, and the action space output is the value of the hyperparameters K and β of the heterogeneous network.

[0076] Step 2.4: The agent continuously interacts with the heterogeneous network and updates the weights of the D3QN network based on the feedback; the state is set to the channel gain between all users and each base station, i.e., s. t =G ln , { }, ;

[0077] Step 2.5: The action space is set to the values ​​of the heterogeneous network hyperparameters K and β that the agent needs to select, as a solution to the joint UARA problem, i.e., the action space is a. t ={K1 K p β1 β Q}, where p is the possible value of the number of shared channels in PSD, and Q is the possible value of the signal-to-interference-plus-noise ratio (SINNR) threshold; input state, the agent calculates the probability of each hyperparameter K value and the probability of each hyperparameter β value; action space is ;

[0078] Step 2.6: Set the reward , w is the energy efficiency function. n In heterogeneous networks, all users have the same priority, where user weights are assigned. ;

[0079] Step 2.7: Train for 10,000 rounds, with each round looping 100 times, to obtain a trained heterogeneous network model.

[0080] Assume a heterogeneous network consists of one macro base station and six small base stations. The macro base station has a sector radius of 500m, and N users are evenly distributed throughout the macro cell. The moving speed of each user is... The direction of movement is The transmit power of macro base stations The bandwidth is 180kHz, the shadow fading is 8dB, and the path loss is... small base station transmit power The bandwidth is 100 kHz, the shadow fading is 8 dB, and the path loss is... ;

[0081] The heterogeneous network has 128 neurons in its one-dimensional convolutional layer and 256 neurons in its fully connected layer; the action space consists of 10 K values ​​and 10... Value composition; The optimizer for the heterogeneous network is RMSProp, with an actor learning rate of 0.00001 and a critic learning rate of 0.0001.

[0082] Specifically, in step 3, the maximum transmit power of the macro base station and the small base station are expressed as follows: and ; and They are respectively:

[0083] ;

[0084] ;

[0085] The data rate for any user is: ,

[0086] in, ; Indicates base station; R macro R represents the data rate of macro base station users; pico This represents the data rate of a small cell user; if user n and base station n are... If related, then ,otherwise ; The user scheduling factor is the proportion of time that a base station allocates to its connected users.

[0087] Without sacrificing generality, user scheduling is performed using the local proportional fairness criterion, meaning each base station allocates the same proportion of time to all its connected users; each user should be connected to one and only one base station at any given time. In heterogeneous networks, the network utility maximization problem uses a logarithmic form for network rate utility. .

[0088] Specifically, in step 3, the signal-to-interference-plus-noise ratio (SIR) from the macro base station to the user is calculated as follows: the set of all base stations is represented as... The index of the macro base station is 0, and the remaining L are small base stations; the macro base station with index 0 exclusively occupies the signal-to-interference-plus-noise ratio at the macro base station user on subchannel m. for:

[0089] ,

[0090] in, G on N0 represents the channel gain from macro base station 0 to user n, and N0 represents the noise power on the sub-channel with bandwidth W.

[0091] Signal-to-interference-plus-noise ratio at the macro base station user on the shared sub-channel with macro base station = 0

[0092] ,

[0093] in,

[0094] Signal-to-interference-plus-noise ratio at small base station users on shared sub-channels of small base stations (1,...,L)

[0095] ,

[0096] in, ; Let represent the channel gain from the small base station to user n, including antenna gain, path loss, and shadow fading, assuming that all sub-channels receive the same channel gain from the base station to the user.

[0097] Minimizing the loss function specifically involves the Q-learning method, which performs well with relatively small state and action spaces. However, finding the optimal policy within the massive Q-table becomes extremely difficult when state changes are large. Therefore, heterogeneous networks are introduced into the Q-learning framework, with DQN being the most commonly used method. In DQN, heterogeneous networks are used to represent the action space and states; the Q-table mapping the state to the action space is represented as follows: Using weights Heterogeneous network approximator Online networks are represented; DQN suffers from overestimation; Double DQN networks are proposed to address overestimation; D3QN networks utilize target networks alongside online networks to stabilize overall D3QN network performance; Q-learning methods update the weights of heterogeneous networks to minimize the loss function. ,

[0098] in, s is the target value in the Double DQN network; t s t+1 Let be the states of the agent at times t and t+1, respectively. , These represent the action space of the energy body at times t and t+1, respectively. The weights are the target network weights; θ and γ represent the network parameters and attenuation factor of the online network, respectively; R t Q t These represent the reward and output of the Double DQN network, respectively; heterogeneous networks are introduced into the dueling DQN network to obtain the value function V(s) and the advantage function. ;

[0099] In DQN networks, an experience replay strategy is employed to overcome learning instability; The data is stored in the replay pool memory D. During training, the heterogeneous network samples small batches of data uniformly and randomly from the replay pool memory D to train, reducing the correlation between training instances and avoiding obtaining local optima.

[0100] Specifically, the advantage function The action space is described in comparison to other possible action spaces. Therefore, in step 4, a double DQN network is introduced into the dueling DQN network to obtain the value function and the dominance function. Specifically, in the dueling DQN network, the last layer of the double DQN network is divided into two sub-networks to estimate the value function V(s) and the dominance function of the dueling DQN network, respectively. Use a Double DQN network to output its connection layer. and ,in, Both μ and μ represent the weights of the fully connected layer; combined and The output of the duelingDQN network Represented as:

[0101]

[0102] To improve the stability of the D3QN network, the output of the dueling DQN network is... Subtracting a constant ensures that the dominance function of the action space remains unchanged relative to the sequence under the given state, thus narrowing the range of Q-values, i.e.:

[0103] .

[0104] User association and resource allocation are often coupled, making it difficult to solve the problem using non-convex mixed integer programming. Without sacrificing generality, the load balancing and interference coordination problem can be simplified to how each user selects a base station to connect to and how many sub-channels each base station occupies.

[0105] Heterogeneous networks belong to orthogonal frequency division multiple access systems. This invention uses a D3QN network to optimize the problem. D3QN combines the ideas of double DQN and dueling DQN networks. Its difference from dueling DQN lies in the objective value... In terms of calculation methods, D3QN The calculation uses double DQN. The calculation method;

[0106] The following is the pseudocode of the program in an embodiment of the present invention:

[0107] The pseudocode for the D3QN network is:

[0108] Initialize the current Q network parameters Initialize target Network parameters And assign the Q network parameters to network, Initialize the current competing network parameters and Total number of iterations T, decay factor Exploration rate The target Q-network parameter replacement frequency P, the number of samples m for each random sampling, and the initial playback pool memory D.

[0109] Execute in each round:

[0110] Initialize the environment and obtain its status and rewards.

[0111] Every step in each round

[0112] Each state is based on - A greedy strategy selects actions within the action space, using weights as... Heterogeneous network approximator Indicates an online network, obtaining a new state s t+1 and rewards

[0113] Will The samples are stored in the playback pool memory D, and m samples are uniformly and randomly sampled from the playback pool memory D for training.

[0114] Calculate the Q value, i.e.:

[0115]

[0116] Calculate the TD target:

[0117] Minimize the loss function:

[0118] Replace the target parameter in each P step.

[0119] The process ends after all states have been processed.

[0120] Finish

[0121] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A dynamic resource allocation optimization method based on reinforcement learning in heterogeneous networks, characterized in that, Includes the following steps: Step 1: Establish a heterogeneous network model, clearly defining energy efficiency as the maximum optimization objective. The heterogeneous network model consists of macro base stations, small base stations, and users. Treat the macro base station as an intelligent agent, establish a Markov decision process, and determine the state space, action space, and reward function. Step 2: Divide the macro base station coverage area and heterogeneous network bandwidth; train the heterogeneous network model through the D3QN network, which includes a double DQN network and a dueling DQN network; Step 2, specifically training the heterogeneous network model using the D3QN network, involves the following steps: Step 2.1 : Based on state s t , action a t and reward r t , the resulting (s t , a t , r t , s t+1 ) is stored in the heterogeneous network replay pool memory D; Step 2.2: Determine whether the amount of data in the playback pool memory D has reached a certain quantity. If it has, start training the heterogeneous network until convergence; if it has not reached a certain quantity, continue with step 2.

1. Step 2.3: The agent is a macro base station, and the D3QN network is used as the strategy. The state input of the agent is the measurement value of the entire heterogeneous network, and the action space output is the value of the hyperparameters K and β of the heterogeneous network. Step 2.4: The agent continuously interacts with the heterogeneous network and updates the weights of the D3QN network based on the feedback; Set up a reinforcement learning framework, with the state set as the channel gain between all users and each base station; Step 2.5: The action space is set to the values ​​of the heterogeneous network hyperparameters K and β that the agent needs to select, as a solution to the joint UARA problem, i.e., the action space is a. t ={K1 K p β1 β Q }, where p is the possible value of the number of shared channels in PSD, and Q is the possible value of the signal-to-interference-plus-noise ratio (SINNR) threshold; input state, the agent calculates the probability of each hyperparameter K value and the probability of each hyperparameter β value; action space is ; Step 2.6: Set the reward , w is the energy efficiency function. n In heterogeneous networks, all users have the same priority, where user weights are assigned. ; Step 2.7: Train the heterogeneous network model according to the number of iterations to obtain the trained heterogeneous network model; Step 3: Calculate the signal-to-interference-plus-noise ratio (SIR) from the macro base station to the user, set the maximum transmit power for the macro base station and the small base station, and minimize the loss function; Step 4: Introduce a double DQN network into the dueling DQN network to obtain the value function and the dominance function, determine the optimal heterogeneous network hyperparameters K and β, maximize the energy efficiency of the heterogeneous network, and achieve dynamic resource allocation optimization.

2. The dynamic resource allocation optimization method based on reinforcement learning in heterogeneous networks according to claim 1, characterized in that, In step 1, the heterogeneous network model has L base stations and multiple users n; the base stations include L1 macro base stations and L2 small base stations, and satisfy L1+L2=L; the users n include macro base station users and small base station users.

3. The dynamic resource allocation optimization method based on reinforcement learning in heterogeneous networks according to claim 2, characterized in that, Step 1 is followed by the process of distributing the power of each macro base station and small base station equally to the sub-channels; The signal-to-interference-plus-noise ratio (SINR) is greater than a preset threshold β, where the preset threshold β is a hyperparameter of the heterogeneous network. Dimensionality reduction of the action space is achieved by setting the action space to a series of discrete values ​​of hyperparameters; Resource allocation adopts a partially shared channel allocation scheme, with K sub-channels shared by macro base stations and small base stations. The number of sub-channels for macro base stations is MK, and the range of the heterogeneous network hyperparameter K is 0 < K < M.

4. The dynamic resource allocation optimization method based on reinforcement learning in heterogeneous networks according to claim 3, characterized in that, Step 2 involves dividing the macro base station coverage area as follows: determining the location of small base station users and macro base station users based on the number and distribution of macro base stations and small base stations; dividing the macro base station coverage area into three sectors, with the same number of small base stations deployed in each sector; and dividing the macro cell into central areas C1, C2, and C3 and edge areas E1, E2, and E3.

5. The dynamic resource allocation optimization method based on reinforcement learning in heterogeneous networks according to claim 4, characterized in that, Step 2, specifically the division of the heterogeneous network bandwidth, involves the following: the heterogeneous network has M sub-channels, each with a bandwidth of W; the entire bandwidth of the heterogeneous network is divided into two parts, c and e, where bandwidth e includes e1, e2, and e3; the main subcarrier transmit power is... It is divided into three parts, and the bandwidth of the three parts is labeled as e1, e2, and e3. Soft frequency reuse technology is used to allocate frequency domain resources for macro base station users, and the entire bandwidth is used to allocate resources for cell users of small base stations. The subcarriers used by each cell are divided into primary subcarriers and secondary subcarriers. The transmission power of the primary subcarrier is greater than that of the secondary subcarrier. The primary subcarrier is used by users at the cell edge, and the secondary subcarrier is used by users at the cell center.

6. The dynamic resource allocation optimization method based on reinforcement learning in heterogeneous networks according to claim 5, characterized in that, Step 3, calculating the signal-to-interference-plus-noise ratio (SIR) from the macro base station to the user, specifically involves the set of all base stations represented as follows: Among them, the index of macro base stations is 0, and the remaining L are small base stations; The signal-to-interference-plus-noise ratio at the macro base station user on the exclusive sub-channel m of macro base station = 0 for: ， in, G on N0 represents the channel gain from macro base station 0 to user n, and N0 represents the noise power on the sub-channel with bandwidth W. Signal-to-interference-plus-noise ratio at the macro base station user on the shared sub-channel with macro base station = 0 for: ， in, Signal-to-interference-plus-noise ratio at small base station users on shared sub-channels of small base stations (1,...,L) for: ， in, ; Let represent the channel gain from the small base station to user n, including antenna gain, path loss, and shadow fading, assuming that all sub-channels receive the same channel gain from the base station to the user.

7. The dynamic resource allocation optimization method based on reinforcement learning in heterogeneous networks according to claim 6, characterized in that, In step 3, the maximum transmit power of the macro base station and the small base station are respectively expressed as: and ; and They are respectively: ； ； The data rate for any user is: , in, ; Indicates base station; R macro R represents the data rate of macro base station users; pico This represents the data rate of a small cell user; if user n and base station n are... If related, then ,otherwise ; The user scheduling factor is the proportion of time that a base station allocates to its connected users. Without sacrificing generality, user scheduling is performed using the local proportional fairness criterion, which means that each base station allocates the same proportion of time to all its connected users; each user should be connected to one and only one base station at any given time.

8. The dynamic resource allocation optimization method based on reinforcement learning in heterogeneous networks according to claim 7, characterized in that, In step 3, the loss function to be minimized is specifically expressed as the mapping function between states and actions. Using weights Heterogeneous network approximator Representing an online network; the Q-learning method is used to update the weights of the heterogeneous network to minimize the loss function: ， in, The target value in the Double DQN network; s t s t+1 Let be the states of the agent at times t and t+1, respectively. , These are the action spaces of the agent at times t and t+1, respectively. The weights are the target network weights; θ and γ represent the network parameters and attenuation factor of the online network, respectively; R t Q t These represent the reward and output of the DoubleDQN network, respectively; heterogeneous networks are introduced into the dueling DQN network to obtain the value function V(s) and the advantage function. ; Will The data is stored in the replay pool memory D. During training, the heterogeneous network is trained by uniformly and randomly sampling small batches of data from the replay pool memory D.

9. A dynamic resource allocation optimization method based on reinforcement learning in heterogeneous networks according to claim 8, characterized in that, In step 4, a double DQN network is introduced into the dueling DQN network to obtain the value function and the dominance function. Specifically, in the dueling DQN network, the last layer of the double DQN network is divided into two sub-networks to estimate the value function V(s) and the dominance function of the dueling DQN network, respectively. Use a Double DQN network to output its connection layer. and ,in, Both μ and μ represent the weights of the fully connected layer; combined and The output of the dueling DQN network is obtained. for: To improve the stability of the D3QN network, the output of the dueling DQN network is... Subtracting a constant ensures that the dominance function of the action space remains unchanged relative to the sequence under the given state, thus narrowing the range of Q-values, i.e.: 。

Images