A deep reinforcement learning based resource allocation method for cellular heterogeneous networks

Abstract

The application discloses a kind of based on deep reinforcement learning's cellular heterogeneous network resource allocation method, comprising the following steps: S1: constructing three-layer heterogeneous network model, obtains the parameter information of three-layer heterogeneous network model, wherein parameter information at least includes: selectable action list, index set, signal-to-interference-plus-noise ratio, downlink capacity and total transmission capacity;S2: after completion of construction, the joint optimization of user association and channel allocation in three-layer heterogeneous network model is completed by MAPD3QN algorithm.The application can optimize without CSI, and strategy convergence can also be completed in the case of a large number of users and base stations.

Description

Technical Field

[0001] This application relates to the field of wireless network communication technology, and in particular to a resource allocation method for cellular heterogeneous networks based on deep reinforcement learning. Background Technology

[0002] Heterogeneous network technology, a hot research topic in future 6G, can address spectrum shortages by sharing spectrum between macro and micro base stations and increase system capacity by adding micro base stations. However, in heterogeneous networks with a large number of base stations and user devices, severe interference exists due to shared channels and dense equipment. For each user, selecting appropriate base stations and channels can greatly reduce interference. Therefore, solving the joint optimization problem of user association and channel allocation is of great significance. However, traditional algorithms cannot optimize without CSI (Channel State Information) and cannot achieve policy convergence with a large number of users and base stations. Summary of the Invention

[0003] The purpose of this application is to provide a resource allocation method for heterogeneous cellular networks based on deep reinforcement learning, which can be optimized without CSI and can also achieve policy convergence with a large number of users and base stations.

[0004] To achieve the above objectives, this application provides a resource allocation method for cellular heterogeneous networks based on deep reinforcement learning, comprising the following steps: S1: Constructing a three-layer heterogeneous network model and obtaining parameter information of the three-layer heterogeneous network model, wherein the parameter information includes at least: a list of optional actions, an index set, a signal-to-interference-plus-noise ratio, downlink capacity, and total transmission capacity; S2: After completion of the construction, performing joint optimization of user association and channel allocation in the three-layer heterogeneous network model through the MAPD3QN algorithm.

[0005] As shown above, the specific steps of the MAPD3QN algorithm are as follows: S201: Input the list of optional actions in the three-layer heterogeneous network model, where the list of optional actions includes: a list of base stations associated with each user equipment and a list of channels; S202: After assuming the agent, store the state, actions, and Sum Tree memory. Initialize the weight update interval N; S203: Adjust the online network Q using the weights θ of the online network. u (s,a u ;θ) and Double Q network Q Double Q Net (s′,a u Initialize θ); S204: Update the target network's weights θ using the online network's weights θ. - Based on the weights θ of the target network -Initialize the target network Q u (s′,a′ u ;θ - S205: Initialize the state by passing messages to the connected base station; S206: The agent, in state s, uses the ε-greedy algorithm to obtain information from the online network Q. u (s,a u In θ), an action a is randomly selected with probability ε. u S207: The agent sends a request to the selected base station to access the selected channel; if the base station sends a feedback signal of the available channel frequency to the user equipment, the user equipment will receive an immediate reward μ. u (s,a u Otherwise, the base station will not reply with any content, and the user equipment will receive a negative reward; S208: The agent obtains the state s′ by transmitting a message to the connected base station; S209: The agent transmits the data (s, a u ,μ u (s,a u ),s′) with probability p u The highest priority sequence is stored in the SumTree memory memory numbered H; S210: The agent stores the data (s, a) in the SumTree memory memory H. u ,μ u (s,a u ),s′) are divided into B parts according to the smallest batch size, based on probability p u S211: Sample from B parts to obtain multiple samples; S212: Calculate the TD bias η of all samples and update the priority of all nodes in the SumTree memory H; S213: Through the target network Q u (s′,a′ u ;θ - Action a to obtain the maximum Q value u S213: Action a based on the maximum Q value u Learn and obtain updated target networks S214: Update the loss function to obtain the minimized loss function; S215: Every N learning iterations, the agent replaces the weights θ of the target network. - =θ; S216: After completing N learning cycles, obtain the state set s = {1,...,1} for N learning cycles, and complete one round of learning. If the current round is not the last round of learning, then execute S201 again to start a new round of learning until all rounds of learning are completed, and then output the optimal action sequence.

[0006] As mentioned above, the three-layer heterogeneous network model includes at least: B m One macro base station, B p One micro base station, Bf One femtocell base station and U randomly located user equipment.

[0007] As mentioned above, the final reward R from the user device is utilized. u (t) represents the instant reward μ u (s,a u The final reward R for user equipment u The expression for (t) is as follows: R u (t)=w u (t)-ψ u Among them, R u (t) represents the final reward; w u (t) represents the user's revenue; ψ u The cost of choosing an action, ψ u >0.

[0008] As shown above, the absolute deviation |η| represents the magnitude of the TD deviation η, and the expression for the absolute deviation |η| is: |η u |=|μ u (s,a u )+γQ u (s′,a u ′;θ - )-Q u (s,a u ;θ)|;where, |η u | represents the absolute deviation of the u-th user equipment; μ u (s,a u ) is for instant reward; μ u The reward for the u-th user device; a u For the action of the u-th user equipment, a u ∈A u A u For each user device, Q represents the action vector; γ is the discount factor, γ∈[0,1]; u (s,a u ;θ) represents an online network; Q u (s,a u ;θ - ) represents the target network, Q u (s′,a u ′;θ - ) for Q u (s,a u ;θ - A subsequent target network architecture; θ - θ represents the weights of the target network; θ represents the weights of the online network.

[0009] As above, among which, the target network The expression is as follows:

[0010] Where μ represents the reward; a u The action that maximizes Q-value; γ is the discount factor; s′∈S, s∈S, S is the set of states; θ is the weight in the online network; θ - The weights of the target network; μ u (s,a u ) is an instant reward; a u For the action of the u-th user equipment, a u ∈A u A u For each user device, there is an action vector; otherwise, it means that the current state s is not the last state; if s′is terminal, it means that if the current state s′ is the last state.

[0011] As shown above, the expression for minimizing the loss function is as follows:

[0012] in, Among them, Q u (s,a u ;θ) represents an online network; Q u (s,a u ;θ - ) represents the target network; s′∈S, s∈S, S is the set of states; θ represents the weights in the online network; θ - The weights of the target network; μ u (s,a u ) is an instant reward; a u For the action of the u-th user equipment, a u ∈A u A u For each user device, there is an action vector; a′∈A, where action a′ is an action following action a, and A is the action; γ is the discount factor. This represents the target network of a deep neural network; It expresses expectation.

[0013] As shown above, after each learning process, it is determined whether the user equipment meets the minimum quality of service threshold. If it does, the value in the state set is 1; if it does not, the value in the state set is 0. If the user equipment meets the minimum quality of service threshold after N learning processes, the obtained state set is s = {1,...,1}.

[0014] As shown above, the number of user devices in the MAPD3QN algorithm is U=500, the number of learning rounds is E=500, and the number of learning iterations is N=50.

[0015] As shown above, the learning rate δ is 0.005; the discount rate is 0.9; and the ratio of the number of agents to the number of channels is 5:3.

[0016] This application can be optimized without CSI, and can also achieve policy convergence with a large number of users and base stations. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0018] Figure 1 This is a schematic diagram of a three-layer heterogeneous network model;

[0019] Figure 2 The reinforcement learning process for intelligent agents;

[0020] Figure 3 This is a diagram illustrating the iterative process of the Double DQN strategy in a Double Q network.

[0021] Figure 4 A schematic diagram showing various base stations and users;

[0022] Figure 5a A comparison chart of different gradient optimization algorithms;

[0023] Figure 5b Detailed comparison diagrams of different gradient optimization algorithms;

[0024] Figure 6a Comparison of smooth steady-state training for different numbers of hidden layers;

[0025] Figure 6b Detailed comparison images of smooth steady-state training for different numbers of hidden layers;

[0026] Figure 7a Comparison of steady-state training curves for different numbers of neurons;

[0027] Figure 7b Detailed comparison of steady-state training for different numbers of neurons;

[0028] Figure 8a A comparison chart of training step counts at different learning rates;

[0029] Figure 8b Detailed plot showing the comparison of training step counts at different learning rates;

[0030] Figure 8c Detailed plot showing the comparison of training step counts at different learning rates;

[0031] Figure 8d A comparison of the sum and rate of the MAPD3QN algorithm at different learning rates;

[0032] Figure 9a A comparison chart of training steady-state data under different discount rates;

[0033] Figure 9b Detailed plot showing the comparison of training step counts at different learning rates;

[0034] Figure 9c A comparison of the sum rate of the MAPD3QN algorithm at different thresholds;

[0035] Figure 10 A comparison of the impact of different convergence threshold values ​​on training step count;

[0036] Figure 11a Comparison of training smooth steady-state charts for different numbers of channels;

[0037] Figure 11b Detailed comparison of training smooth steady-state data for different numbers of channels;

[0038] Figure 12 Train a smooth steady-state comparison chart for different users and number of channels;

[0039] Figure 13 A comparison chart of different users, channel numbers, and rates;

[0040] Figure 14a Comparison of training smooth step counts for different algorithms;

[0041] Figure 14b A comparison chart of different algorithms and speeds;

[0042] Figure 15 A comparison chart of the sum and rate of different algorithms with different numbers of channels;

[0043] Figure 16 A graph showing the sum and rate of different algorithms with different numbers of users;

[0044] Figure 17 A comparison of the sum rate of different algorithms with different numbers of channels when the QoS threshold is -5dB.

[0045] Figure 18 The graph shows the sum rate comparison of different algorithms with different numbers of users when the QoS threshold is -5dB. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] This application provides a resource allocation method for heterogeneous cellular networks based on deep reinforcement learning, comprising the following steps:

[0048] S1: Construct a three-layer heterogeneous network model and obtain the parameter information of the three-layer heterogeneous network model. The parameter information includes at least: a list of optional actions, an index set, a signal-to-interference-plus-noise ratio, a downlink capacity, and a total transmission capacity.

[0049] Furthermore, such as Figure 1 As shown, the three-layer heterogeneous network model includes at least: B m Macro Base Stations (MBSs), B p Pico Base Stations (PBSs), B f U femto base stations (FBSs) and U randomly located user equipments (UEs), where the subscript s indicates a complex number.

[0050] Specifically, the expressions for all base stations in the three-layer heterogeneous network model are:

[0051]

[0052] in, : indicates 1 to B m One macro base station (MBS); : indicates 1 to B p One micro base station (PBS); : indicates 1 to B f A nanometer base station.

[0053] The index set B for base stations (BSs) is as follows:

[0054]

[0055] Where L is the total number of base stations, L = B m +B p +B f B m B represents the number of macro base stations. p B represents the number of micro base stations.f This is the format for a femtocell base station.

[0056] Assume the base station operates on K shared orthogonal channels. Assume each user equipment (UE) can select at most one base station at any given time, as shown in the following expression:

[0057]

[0058] in, This represents the association vector of the u-th user device. U = [1, U], where U is the number of user devices; This represents the user association vector between the u-th user equipment and the 0th base station; This represents the user association vector between the u-th user equipment and the (L-1)-th base station.

[0059] Assuming each user equipment can select at most one channel at any given time, the expression is as follows:

[0060]

[0061] in, This represents the channel allocation vector of the u-th user equipment at time t on channel k. U = [1, U], K = [1, K], where K is the number of shared orthogonal channels; Let represent the channel allocation vector between the u-th user equipment and the 0th channel; Let represent the channel allocation vector between the u-th user equipment and the k-th channel.

[0062] Define the set P of transmit powers l,u (t) is: u∈U,U=[1,U],l∈B,k∈K,K=[1,K]; This represents the power of the u-th user equipment when it uses the first channel in association with the l-th base station; This represents the power of the u-th user equipment when it uses the k-th channel in association with the l-th base station. The u-th user equipment uses channel C... k (C k (representing the k-th channel) and its associated base station B l (B l The signal-to-interference-plus-noise ratio (SINR) of the link at the l-th base station in set B. The definition is as follows:

[0063]

[0064] in, It is the channel gain on channel k between the l-th base station and the u-th user equipment in set B at time t; It is the channel gain on channel k between the m-th base station and the u-th user equipment in set B at time t; Let be the association vector between the m-th BS and the u-th user equipment at time t; Let be the association vector between the l-th BS and the u-th user equipment at time t; Let represent the channel allocation vector of the u-th user equipment at time t on channel k; Let represent the transmit power vector between the m-th base station and the u-th user equipment on channel k at time t; Let represent the transmit power vector between the l-th base station and the u-th user equipment on channel k; W represents the channel bandwidth, and N0 represents the noise power spectral density. BS belongs to BSs.

[0065] Downlink capacity on channel k between the l-th base station and the u-th user equipment in set B at time t. Represented as:

[0066]

[0067] Therefore, the total transmission capacity r of the u-th user equipment at time t u (t) is represented as:

[0068]

[0069] S2: After the construction is completed, the joint optimization of user association and channel allocation in the three-layer heterogeneous network model is completed by the MAPD3QN algorithm (multi-agent priority experience replay duel network and deep double Q network algorithm).

[0070] Furthermore, as shown in Table 1, the specific steps of the MAPD3QN algorithm are as follows:

[0071] S201: Input the list of optional actions in the three-layer heterogeneous network model, where the list of optional actions includes: a list of base stations and a list of channels associated with each user equipment.

[0072] S202: After assuming the agent, the state, action, and Sum Tree memory are stored. The weight update interval N is initialized.

[0073] Specifically, such as Figure 2As shown, assuming an agent exists in the environment capable of taking actions, let S represent the agent's specific state at a given moment; let A represent the agent's action in the environment; R represent the reward that these actions sometimes bring; and P represent the policy formed through continuous learning (after taking action A0 from the initial state S0, the environment changes, and the agent receives a reward R0 based on state S0 or action A0, entering a new state S1; in the new round of learning, the agent takes a new action A1, and so on, continuously learning in a loop). The ultimate goal of reinforcement learning in the MAPD3QN algorithm is to learn a mapping from environment states to actions, i.e., policy π, through the interaction between the agent and the environment. Therefore, reinforcement learning introduces a state-value function to determine the effect of policy π, thereby maximizing the long-term benefits of the policy. After each round of learning, the state and action need to be initialized; that is, the initial states of the state and action are defined in the program, and values ​​are assigned to the state and action during program execution.

[0074] Furthermore, considering the state transition probability of policy π as p(s′|s,a), s′∈S, s∈S, a∈A, then the state value function V π The expression for (s) is as follows:

[0075]

[0076] Where π(s) represents the policy of state s; p(s′|s,π(s)) represents the transition probability; V π (s′) is the state value function of a state s′ following state s; r∈R; γ is the discount factor, γ∈[0,1].

[0077] Furthermore, the state-value function V π (s) describes the value of a state when following policy π, while the value describing the action of policy π in a given state needs to be defined as the action value function Q. π (s,a). When the state transition probability of policy π is p(s′|s,a), the action value function Q can be obtained. π The expression for (s,a) is as follows:

[0078]

[0079] Where r(s′|s,a) is the expression for the reward R related to the previous state and action; a′∈A, action a′ is an action following action a; Q π (s′,a′) is the action value function for the next state and action.

[0080] Furthermore, after each round of learning, the Sum Tree memory needs to be updated. (Where H is the number of the summation tree) is initialized.

[0081] Furthermore, after each round of learning, the weight update interval N needs to be initialized. The larger the weight update interval N of the target network, the more stable the algorithm will be. The slower the update frequency of the target network, the slower the convergence speed of the algorithm will be.

[0082] S203: Applying weights θ to the online network Q u (s,a u ;θ) and Double Q network Q Double Q Net (s′,a u Initialize using θ).

[0083] Specifically, the ε-greedy algorithm is used to obtain data from the online network Q. u (s,a u Select action a from θ). u Meanwhile, the weights θ in the online network are continuously updated; that is, an action a is randomly selected with probability ε. u After each round of learning, the current online network Q needs to be updated using the weights θ of the updated online network. u (s,a u Initialize using θ).

[0084] like Figure 3 As shown, the Double Q Network (Double Deep Q Network) is the target network of the DQN (Deep Q Network, a reinforcement learning method that combines neural networks and Q-learning) target network. u (s',a';θ - A new Q-network has been added to the DQN target network, which directly selects the largest Q-network. u (s',a';θ - The approach is transformed into DoubleDQN, which first finds the optimal action a through the newly added Q-network. u Then, a u Substituting into the equation, we get Q. u (s',a';θ - DoubleDQN uses a Double Q network with the same weights θ as the online network. Find the action a that yields the maximum Q value. u Then calculate this action a.u The weights of the target network are θ - Target Q network Q Double Q Net,u (s,a u ;θ - The Q value in the () is used to supervise the target Q network using a Double Q network.

[0085] Among them, DQN target network The expression is:

[0086]

[0087] S204: Update the weights θ of the target network using the weights θ of the online network. - Based on the weights θ of the target network - Initialize the target network Q u (s′,a′ u ;θ - ).

[0088] Specifically, θ - Let θ be the weights of the target network. After N learning iterations, the weights θ of the target network are updated using the weights θ of the online network. - .

[0089] Furthermore, the target network Q u (s′,a′ u ;θ - The expression for ) is:

[0090]

[0091] Among them, Q TargetQ Net,u () indicates the addition of a Q-network; Action a represents maximizing the target network. u .

[0092] Specifically, the target network Q u (s′,a′ u ;θ - ) represents the target network of the QDN; the added Q network Q TargetQ Net,u () indicates that double DQN adds a Q-network to the target network in DQN.

[0093] S205: Initialize the state by passing a message to the connected base station.

[0094] Specifically, the message is transmitted to the connected base station, and then distributed to each base station through the connected base station, thereby enabling updates to all base stations.

[0095] S206: The agent, in state s, uses the ε-greedy algorithm to learn from the online network Q. u (s,a u In θ), an action a is randomly selected with probability ε. u .

[0096] S207: The agent sends a request to the selected base station to access the selected channel; if the base station sends a feedback signal of the available channel frequency to the user equipment, the user equipment will receive an immediate reward μ. u (s,a u Otherwise, the base station will not reply with any content, and the user equipment will receive a negative reward.

[0097] Specifically, if a base station sends a feedback signal indicating the available channel frequency to a user equipment, it means that the base station can connect and the connection is successful. Only with a successful connection can the user equipment receive an immediate reward. If the base station does not send a feedback signal indicating the available channel frequency to a user equipment, it means that the base station cannot connect, the connection fails, and the user equipment cannot receive an immediate reward.

[0098] Furthermore, utilizing the user device's final reward R u (t) represents the instant reward μ u (s,a u The final reward R for user equipment u The expression for (t) is as follows:

[0099] R u (t)=w u (t)-ψ u ;

[0100] in,

[0101]

[0102] in,

[0103]

[0104] Among them, R u (t) represents the final reward; w u (t) represents the user's revenue, i.e., the difference between the revenue obtained and the transmission cost; To be with B l The associated total transmission cost, B l Represents the l-th base station in index set B; -ψ u Negative reward; ψ u The cost of choosing an action, ψ u >0; λ l B represents BS l The cost of transmission power; ρu It is the yield to transmission rate; r u The transmission capacity of the u-th user equipment.

[0105] S208: The agent obtains the state s′ by passing a message to the connected base station.

[0106] Specifically, the state s′ is obtained by passing the message to the connected base station, and s←s′ is set, that is, state s′ is the state after state s.

[0107] S209: The agent will transfer data (s, a) u ,μ u (s,a u ),s′) with probability p u The highest priority order is stored in the Sum Tree memory memory numbered H.

[0108] Specifically, a Sum Tree is a tree-like data structure that uses the technique of summation trees to store (s, a) data in a tree structure. u ,μ u (s,a u The summation tree (s′) is stored in the form of data along with the absolute deviation |η| in a Sum Tree memory memory numbered H. The tree is defined to store the data according to the priority of the child nodes, with the parent node generated based on the size of its child nodes (i.e., the parent node is the sum of the sizes of all child nodes) and the root node is the sum of all priority numbers.

[0109] S210: The agent stores the data (s, a) in the Sum Tree memory H. u ,μ u (s,a u ),s′) are divided into B parts according to the smallest batch size, based on probability p u Multiple samples are obtained by sampling from part B.

[0110] S211: Calculate the TD bias η for all samples and update the priority of all nodes in the Sum Tree memory H.

[0111] The technique of prioritizing experience replay is introduced, which assigns higher sampling probabilities to data experiences that are highly efficient in reinforcement learning by prioritizing the data. The magnitude of the TD (Time Difference) bias η determines the importance of the experience; the larger the TD bias, the greater the amount of updates during reinforcement learning and the greater its impact on backpropagation, thus resulting in higher learning efficiency.

[0112] Furthermore, the magnitude of the TD deviation η is represented by the absolute deviation |η|, and the expression for the absolute deviation |η| is:

[0113] |η u |=|μ u (s,a u )+γQ u (s′,a u ′;θ - )-Q u (s,a u ;θ)|;

[0114] Where, |η u | represents the absolute deviation of the u-th user equipment; μ u (s,a u ) is an instant reward; μ u The reward for the u-th user device; a u For the action of the u-th user equipment, a u ∈A u A u For each user device, Q represents the action vector; γ is the discount factor, γ∈[0,1]; u (s,a u ;θ) represents an online network; Q u (s,a u ;θ - ) represents the target network, Q u (s′,a u ′;θ - ) for Q u (s,a u ;θ - A subsequent target network architecture; θ - θ represents the weights of the target network; θ represents the weights of the online network.

[0115] S212: Via target network Q u (s′,a′ u ;θ - Action a to obtain the maximum Q value u .

[0116] Specifically, definition Calculate action a u The weights of the target network are θ - Target network Q Double Q Net,u (s,a u ;θ - The action a that obtains the maximum Q value in the Q-value of the given data. u Prepare for step S213.

[0117] S213: Action a based on the maximum Q value u Learn and obtain updated target networks

[0118] Furthermore, the target network The expression is as follows:

[0119]

[0120] Where μ represents the reward; a u The action that maximizes Q-value; γ is the discount factor; s′∈S, s∈S, S is the set of states; θ is the weight in the online network; θ - The weights of the target network; μ u (s,a u ) is an instant reward; a u For the action of the u-th user equipment, a u ∈A u A u For each user device, the action vector is given; otherwise, it indicates that the current state s is not the last state; if s 'is terminal' means if state s' is the last state.

[0121] S214: Update the loss function to obtain the minimized loss function.

[0122] Specifically, the loss function is updated using gradient descent.

[0123] Furthermore, the expression for minimizing the loss function is as follows:

[0124]

[0125] in,

[0126] Among them, Q u (s,a u ;θ) represents an online network; Q u (s,a u ;θ - ) represents the target network; s′∈S, s∈S, S is the set of states; θ represents the weights in the online network; θ - The weights of the target network; μ u (s,a u ) is an instant reward; a u For the action of the u-th user equipment, a u ∈A u A u For each user device, there is an action vector; a′∈A, where action a′ is an action following action a, and A is the action; γ is the discount factor. Represents the target network of a deep neural network (DNN); It expresses expectation.

[0127] S215: Every N training iterations, the agent replaces the weights θ of the target network. - =θ.

[0128] Specifically, after N learning iterations, the weights θ of the target network are updated using the weights θ in the online network. - This means updating the action-value evaluation criterion, which is the evaluation criterion in the algorithm, and the algorithm learns based on the action-value evaluation criterion.

[0129] S216: After completing N learning iterations, obtain the state set s = {1,...,1} for N learning iterations, completing one round of learning. If the current round is not the last round of learning, then execute S201 again to start a new round of learning, until all rounds of learning are completed, and then output the optimal action sequence.

[0130] Furthermore, after each learning iteration, it is determined whether the user equipment meets the minimum Quality of Service (QoS) threshold. If it does, the value in the state set is 1; otherwise, the value is 0. If the user equipment meets the QoS threshold after N learning iterations, the resulting state set is s = {1,...,1}, completing one round of learning. If this is not the last round of learning, S201 is executed again for a new round of learning, until all rounds of learning are completed, at which point the optimal action sequence is output.

[0131] The self-learning process involves U user devices, each requiring E rounds of learning, with each round consisting of N learning iterations. The specific number of E depends on the actual situation; in this application, E = 500 is preferred. The specific number of N also depends on the actual situation; in this application, N = 50 is preferred.

[0132] The optimal action sequence includes a list of base stations and a list of channels associated with user equipment that meet the minimum quality of service (QoS) threshold.

[0133] Furthermore, the signal-to-interference-plus-noise ratio (SINR) of the u-th user equipment... u Not less than the minimum Quality of Service (Qos) threshold Ω u The expression is as follows:

[0134]

[0135] in, For the u-th user equipment through channel C k (C k (representing the k-th channel) and its associated base station B l (B l The signal-to-interference-plus-noise ratio (SIR) of the link for the l-th base station in set B is represented.

[0136]

[0137]

[0138] Table 1

[0139] Furthermore, the resource allocation method for cellular heterogeneous networks based on deep reinforcement learning also includes: conducting computer simulations, observing the learning progress of the MAPD3QN algorithm based on the simulations, obtaining simulation results, and determining the actual effect of the MAPD3QN algorithm after learning based on the simulation results.

[0140] Specifically, as an example, in the simulation process, a computer with a 10th-generation i7 CPU was selected as the hardware platform, and PyTorch version 3.7 was used as the software platform for the deep neural network. Before conducting the simulation, the parameters of the simulation environment need to be set, as shown in Table 2:

[0141]

[0142] Table 2

[0143] like Figure 4 As shown, in the simulation, the network consists of 2 macro base stations (MBSs), 8 micro base stations (PBSs), 16 femto base stations (FBSs), and 50-500 UEs. The BS radii are 500m, 100m, and 30m, respectively. The number of UEs changes randomly in each simulation, and the Monte Carlo experiment number is set to 20.

[0144] The following experiment uses simulation analysis to obtain the training efficiency under different learning hyperparameters:

[0145] Experiment 1: Number of iterations for different optimization algorithms

[0146] like Figure 5a and Figure 5b As shown in the diagram, the training results under various optimization strategies indicate that when the number of training steps is small, the convergence speeds of adaptive moment estimation (Adam) and mean squared error propagation (RMSProp) are basically the same. However, the detailed diagram reveals the advantage of RMSprop, while the adaptive gradient optimization algorithm (AdaGrad) converges more slowly. Overall, RMSprop has the advantages of fast convergence speed and relatively stable convergence.

[0147] Furthermore, this application selects the RMSProp optimization strategy.

[0148] Experiment 2: Number of iterations for different optimization algorithms

[0149] 1. Different hidden layers

[0150] like Figure 6a and 6b As shown, the training steps of each DNN structure layer show that the best results are achieved when layer=4. At the same time, when the number of hidden layers is too large, the optimization problem tends to local optima, while too few hidden layers make it difficult to extract key information. Therefore, it is necessary to select a suitable hidden layer for a specific problem.

[0151] Furthermore, in joint optimization problems, layer=4 converges faster and is more stable, therefore a four-layer network is the optimal parameter.

[0152] 2. Different Neural Networks

[0153] like Figure 7a and 7b As shown, the 64-32-32-32 network structure converges fastest overall and remains relatively stable after convergence. However, when the number of neurons is too large, the optimization problem may lead to overfitting, increasing training time; conversely, if the number of neurons is too small, it is difficult to extract key information. Therefore, four-layer network structures with 64, 32, 32, and 32 neurons were chosen.

[0154] Experiment 3: Convergence performance under different learning rates

[0155] from Figure 8a , Figure 8b , Figure 8c and Figure 8d It can be seen that, considering the real-time performance and convergence stability of the MAPD3QN algorithm, choosing a learning rate δ of 0.005 results in a better and more stable convergence effect.

[0156] Furthermore, a learning rate of 0.005 was selected for the joint optimization problem of user association and channel selection.

[0157] Experiment 4: Convergence performance under different learning rates

[0158] As shown in Figures 9(a), 9(b) and 9(c), the optimal sum rate is achieved when the discount rate is 0.9.

[0159] Experiment 5: The impact of different QoS thresholds on the system

[0160] Experiment 5 discusses the effect of different convergence Qos thresholds on the convergence stationary sum rate. From Figure 10It can be seen that the performance is not very stable at SINR values ​​of -5dB and 10dB, exhibiting significant fluctuations during the learning process. Ω=5 was chosen as the parameter for subsequent experiments not only because the system capacity is maximized at this point, but also because SINR=5dB is a relatively reasonable threshold for measuring communication quality.

[0161] Experiment 6: The impact of different numbers of channels on the algorithm's convergence and summation rate

[0162] As shown in Figure 11(a), when the number of channels is 20, no strategy can be found to satisfy QoS because the small number of channels results in too many UEs sharing the channels, making it difficult to meet the SINR QoS threshold. As shown in Figure 11(b), a channel count of 30 is the minimum number of channels required to satisfy the QoS requirements of UEs. Therefore, when Agent = 50, the channel count is chosen to be 30. For other agent counts, the ratio of Agent to channel count is maintained at 5:3.

[0163] Experiment 7: Convergence performance when user and channel data are different

[0164] Following the above experiments on setting the algorithm's hyperparameters, Experiment Seven tests the performance of these hyperparameters. For example... Figure 12 As shown, as the number of Agents increases, the number of convergence iterations increases accordingly, but all of them converge very quickly and remain relatively stable after convergence without significant fluctuations.

[0165] Experiment 8: Sum Rate with Different Numbers of Users

[0166] like Figure 13 As shown, when there are 500 agents, the sum rate increases significantly after the algorithm learns, indicating that the algorithm is very effective in increasing the sum rate of the system. After the algorithm converges, this increase in the sum rate stops. At the same time, the sum rate remains relatively stable during continuous iteration. This stability is more obvious when the number of agents is less than 500. This proves that the algorithm's decisions on user association and channel allocation bring stable system performance to the communication system.

[0167] The performance of the proposed MAPD3QN algorithm is evaluated through simulation experiments. This paper compares it with the DQN algorithm, MAPD3QN algorithm, random allocation method, Double DQN algorithm, and greedy allocation algorithm. The hyperparameters of MAPD3QN are shown in Table 3.

[0168]

[0169] Table 3

[0170] Experiment 8: Comparison of convergence of different algorithms when the number of users is 500

[0171] The simulation involved 500 users, representing an extremely high user count. As shown in Figure 14(a), DQN, Double DQN, the Random algorithm, and the Greedy algorithm all failed to ensure all UEs met the QoS threshold, thus failing to achieve final convergence. MADDQN also failed to achieve final stable convergence, primarily because Dueling DQN divides the Q function into A and V parts, allowing for faster extraction of key information compared to DQN and Double DQN. However, even with such a high user count, it is insufficient to learn a stable convergence strategy. In contrast, the MAPD3QN algorithm proposed in this application, supported by priority experience replay technology, achieved a strategy that met the QoS threshold in just 20 iterations and remained stable in subsequent steps. As shown in Figure 14(b), the MAPD3QN algorithm of this application maintains stable and higher rate (system capacity) performance than the other algorithms.

[0172] Experiment 9: Comparison of different algorithms and rates (system capacity) with different numbers of channels when QoS=5.

[0173] This simulation primarily aims to compare the growth of different algorithms when the number of channels varies. For example... Figure 15 As shown, the growth rate of system capacity slows down as the number of channels increases. This is because the number of channels increases, the number of shared channels decreases, and thus the interference in the system also decreases. However, when the number of channels approaches the number of users, the number of shared channels will tend to disappear, meaning that the system capacity gain from increasing the number of channels will become smaller and smaller.

[0174] Experiment 10: Comparison of different algorithms and speeds with different numbers of users when QoS=5.

[0175] like Figure 16 As shown, with a fixed number of channels, the growth rate of system capacity slows down as the number of users increases. This is mainly because as the number of users increases, the sharing of channels also increases, causing significant interference. Although adding new users increases the overall system capacity, the increased interference from shared channels diminishes the benefit to system capacity. The MAD3QN algorithm proposed in this application stabilizes with the increase in users, but it outperforms other algorithms under various user conditions.

[0176] Experiment 11: Comparison of different algorithms and rates (system capacity) with different numbers of channels when the QoS threshold is -5dB.

[0177] like Figure 17As shown, the sum and rate increase with the number of channels, but the growth rate slows down. This is because the interference within the channels decreases with the increase in the number of channels, which is basically the same trend as when QoS = 5dB. It can also be seen that when the number of channels is very small, the sum and rate achieved by the various algorithms are very similar. However, as the number of channels increases, the gap between other algorithms and the MAPD3QN algorithm widens. This is because the number of channels is limited, resulting in fewer selection options. Furthermore, as the action space and state space increase, the advantages of the MAPD3QN algorithm gradually become apparent.

[0178] Experiment 12: Comparison of different algorithms and speeds (system capacity) with different numbers of users and a QoS threshold of -5dB.

[0179] like Figure 18 As shown, the sum rate of different algorithms increases from 10 to 50 users. When channel resources are sufficient, the sum rate increases almost linearly with the increase in the number of users. However, when channel resources are scarce, the increase in system sum rate brought about by the increase in users becomes extremely limited. Nevertheless, the MAPD3QN algorithm still achieves a fairly good system sum rate.

[0180] Experiment 13: Algorithm running time at different QoS thresholds (QoS = 5, -5) (Table)

[0181]

[0182] Table 4. Running time of different algorithms at different QoS thresholds

[0183] As shown in Table 4, it is clear that the algorithm runtime is significantly reduced when the QoS threshold is lowered. Furthermore, because MAPD3QN converges quickly, it achieves the fastest runtime for both QoS thresholds. MADDQN, due to its Dueling architecture, can quickly extract information to complete convergence, resulting in a much shorter runtime compared to DQN and Double DQN.

[0184] This application can be optimized without CSI, and can also achieve policy convergence with a large number of users and base stations.

[0185] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the scope of protection of this application is intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application. Obviously, those skilled in the art can make various alterations and variations to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of protection of this application and its equivalents, this application also intends to include these modifications and variations.

Claims

1. A resource allocation method for cellular heterogeneous networks based on deep reinforcement learning, characterized in that, Includes the following steps: S1: Construct a three-layer heterogeneous network model and obtain the parameter information of the three-layer heterogeneous network model. The parameter information includes at least: a list of optional actions, an index set, a signal-to-interference-plus-noise ratio, a downlink capacity, and a total transmission capacity. S2: After the construction is completed, the joint optimization of user association and channel allocation in the three-layer heterogeneous network model is completed by the MAPD3QN algorithm; The specific steps of the MAPD3QN algorithm are as follows: S201: Input the list of optional actions in the three-layer heterogeneous network model, where the list of optional actions includes: a list of base stations and a list of channels associated with each user equipment; S202: After assuming the agent, the state, action, and Sum Tree memory are stored. The weight update interval N is initialized. S203: Applying weights θ to the online network Q u (s,a u ;θ) and Double Q network Q Double Q Net (s′,a u Initialize θ) S204: Update the weights θ of the target network using the weights θ of the online network. - Based on the weights θ of the target network - Initialize the target network Q u (s′,a′ u ;θ - ); S205: Initialize the state by passing a message to the connected base station; S206: The agent, in state s, uses the ε-greedy algorithm to learn from the online network Q. u (s,a u In θ), an action a is randomly selected with probability ε. u ; S207: The agent sends a request to the selected base station to access the selected channel; if the base station sends a feedback signal of the available channel frequency to the user equipment, the user equipment will receive an immediate reward μ. u (s,a u Otherwise, the base station will not reply with any content, and the user equipment will receive a negative reward. S208: The agent obtains state s′ by transmitting a message to the connected base station; S209: The agent will transfer data (s, a) u ,μ u (s,a u ),s′) with probability p u The highest priority sequence is stored in the SumTree memory memory numbered H; S210: The agent stores the data (s, a) in the SumTree memory H. u ,μ u (s,a u ),s′) are divided into B parts according to the smallest batch size, based on probability p u Multiple samples are obtained by sampling from part B; S211: Calculate the TD bias η for all samples and update the priority of all nodes in the SumTree memory H; S212: Via target network Q u (s′,a′ u ;θ - Action a to obtain the maximum Q value u ; S213: Action a based on the maximum Q value u Learn and obtain updated target networks S214: Update the loss function to obtain the minimized loss function; S215: Every N training iterations, the agent replaces the weights θ of the target network. - =θ; S216: After completing N learning iterations, obtain the state set s = {1,...,1} for N learning iterations, completing one round of learning. If the current round is not the last round of learning, then execute S201 again to start a new round of learning, until all rounds of learning are completed, and then output the optimal action sequence.

2. The resource allocation method for cellular heterogeneous networks based on deep reinforcement learning according to claim 1, characterized in that, A three-layer heterogeneous network model should include at least: B m One macro base station, B p One micro base station, B f One femtocell base station and U randomly located user equipment.

3. The resource allocation method for cellular heterogeneous networks based on deep reinforcement learning according to claim 2, characterized in that, Utilizing the user device's final reward R u (t) represents the instant reward μ u (s,a u The final reward R for user equipment u The expression for (t) is as follows: R u (t)=w u (t)-ψ u ; Among them, R u (t) represents the final reward; w u (t) represents the user's revenue; ψ u The cost of choosing an action, ψ u >0.

4. The resource allocation method for cellular heterogeneous networks based on deep reinforcement learning according to claim 3, characterized in that, The absolute deviation |η| represents the magnitude of the TD deviation η. The expression for the absolute deviation |η| is: |the u |=|μ u (s,a u )+γQ u (s′,a u ′;θ - )-Q u (s,a u ;i)|; Where, |η u | represents the absolute deviation of the u-th user equipment; μ u (s,a u ) is for instant reward; μ u The reward for the u-th user device; a u For the action of the u-th user equipment, a u ∈A u A u For each user device, Q represents the action vector; γ is the discount factor, γ∈[0,1]; u (s,a u ;θ) represents an online network; Q u (s,a u ;θ - ) represents the target network, Q u (s′,a u ′;θ - ) for Q u (s,a u ;θ - A subsequent target network architecture; θ - θ represents the weights of the target network; θ represents the weights of the online network.

5. The resource allocation method for cellular heterogeneous networks based on deep reinforcement learning according to claim 4, characterized in that, Target Network The expression is as follows: Where μ represents the reward; a u The action that maximizes Q-value; γ is the discount factor; s′∈S, s∈S, S is the set of states; θ is the weight in the online network; θ - The weights of the target network; μ u (s,a u ) is an instant reward; a u For the action of the u-th user equipment, a u ∈A u A u For each user device, the action vector is given; otherwise, it indicates that the current state s is not the last state; if s 'isterminal' indicates if state s' is the last state.

6. The resource allocation method for cellular heterogeneous networks based on deep reinforcement learning according to claim 5, characterized in that, The expression for minimizing the loss function is as follows: in, Among them, Q u (s,a u ;θ) represents an online network; Q u (s,a u ;θ - ) represents the target network; s′∈S, s∈S, S is the set of states; θ represents the weights in the online network; θ - The weights of the target network; μ u (s,a u ) is an instant reward; a u For the action of the u-th user equipment, a u ∈A u A u For each user device, there is an action vector; a′∈A, where action a′ is an action following action a, and A is the action; γ is the discount factor. This represents the target network of a deep neural network; It expresses expectation.

7. The resource allocation method for cellular heterogeneous networks based on deep reinforcement learning according to claim 6, characterized in that, After each learning iteration, it is determined whether the user equipment meets the minimum quality of service threshold. If it does, the value in the state set is 1; if it does not, the value in the state set is 0. If the user equipment meets the minimum quality of service threshold after N learning iterations, the obtained state set is s = {1,...,1}.

8. The resource allocation method for cellular heterogeneous networks based on deep reinforcement learning according to claim 7, characterized in that, The MAPD3QN algorithm has 500 user devices (U=500), 500 learning rounds (E=500), and 50 learning attempts (N=50).

9. The resource allocation method for cellular heterogeneous networks based on deep reinforcement learning according to claim 8, characterized in that, The learning rate δ is 0.005; the discount rate is 0.9; and the ratio of the number of agents to the number of channels is 5:3.

Images