A real-time spectrum resource optimization method and system based on deep reinforcement learning

Abstract

The present invention relates to the field of ultra-dense network communication technology, and specifically discloses a real-time spectrum resource optimization method and system based on deep reinforcement learning, wherein the method comprises: constructing a typical ultra-dense network downlink scenario, simulating the user's dynamic movement behavior according to a random walk model; performing system modeling analysis based on the ultra-dense network downlink scenario, generating a joint optimization model with the goal of maximizing the total system throughput; solving the optimal solution of the joint optimization model through a deep reinforcement learning algorithm, and setting the user's state function, the user's action function, and the reward function to train the user to learn the optimal solution. This application constructs a dynamic ultra-dense network environment based on a random walk model by considering the unpredictability of dynamic user movement.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of ultra-dense network communication, and particularly relates to a real-time spectrum resource optimization method and system based on deep reinforcement learning. BACKGROUND

[0002] With the continuous development of mobile communication technology, people's demand for wireless communication is also getting higher and higher. Especially in the era of popularization of intelligent devices and Internet of Things, users are no longer limited to fixed positions, but continue to dynamically move in urban and densely populated areas. Such behavior patterns lead to the high dynamicity and unpredictability of connection demand and data traffic in wireless networks. Under this background, the traditional cellular network, due to its fixed network structure, is difficult to meet the demand of users for high-speed data transmission and low latency. Ultra-dense network (UDN) thus emerges as the times require, providing better network coverage and service quality by deploying a large number of micro base stations. However, such deployment also brings its own challenges. Due to the extremely high density of base stations, when users dynamically move, the interference problem between base stations and between users and base stations becomes particularly prominent. The dynamic behavior of users leads to the continuous change of service demand and signal quality, thereby putting forward higher flexibility and real-time requirements for spectrum resource allocation. Therefore, how to effectively allocate limited spectrum resources in such a dynamic environment to adapt to the real-time movement of users and alleviate interference has become a key problem to be solved.

[0003] Traditional resource optimization methods often rely on prior knowledge and rules, such as graph theory, optimization theory, etc., to design heuristic algorithms or optimization models. For example, a common method is to group base stations with high interference and related base stations into a cluster, and then use graph coloring theory for resource allocation. There are also methods that use a two-stage resource allocation scheme to handle cluster-based resource management problems. In addition, some methods attempt to cluster small cells through interference graphs to reduce computational complexity, or study how to optimize the energy efficiency of dense small cell networks. But a common shortcoming of these traditional methods is that they all rely on complete channel state information (CSI), which may be a challenge to obtain in actual scenarios.

[0004] In contrast, reinforcement learning (RL) provides a new solution direction. It does not rely on complete environment information, but learns the optimal policy through interaction with the environment. For example, some research uses dueling deep Q network (Dueling DQN) to optimize energy efficiency (EE) and system efficiency (SE) simultaneously in ultra-dense network (UDN) systems. Some research uses DQN to optimize the switching strategy of small base stations, thereby improving energy efficiency while ensuring quality of service (QoS). For another example, some methods explore how to use deep Q network to optimize unmanned aerial vehicle-assisted cellular networks and flight trajectories.

[0005] Although reinforcement learning shows great potential in resource optimization, many current researches are still based on static or predetermined network performance and user demand. However, the actual network environment is constantly changing, and the user's location is also moving in real time. This requires resource allocation strategies to be dynamically adjusted according to the real-time location of users and the state of the network. Therefore, future research directions can further explore how to effectively implement reinforcement learning methods to achieve dynamic optimization of resources in a constantly changing environment.

[0006] Therefore, how to effectively implement reinforcement learning methods to achieve dynamic optimization of resources is a technical problem that needs to be solved by those skilled in the art. SUMMARY

[0007] To achieve the purpose of the present application, the present application provides a real-time spectrum resource optimization method based on deep reinforcement learning, comprising:

[0008] Step S1: Construct a typical ultra-dense network downlink scenario, and simulate the behavior of user dynamic movement according to a random walk model;

[0009] Step S2: According to the ultra-dense network downlink scenario, perform system modeling analysis to maximize the total system throughput, and generate a joint optimization model;

[0010] Step S3: Solve the optimal solution of the joint optimization model by a deep reinforcement learning algorithm, and set the state function of the user, the action function of the user and the reward function to train the user to learn the optimal solution.

[0011] In some embodiments, the step S1 comprises:

[0012] Step S11: The user movement node randomly selects a direction and speed to move from the current location to a new location;

[0013] Step S12: the user mobile node moves at a constant time interval or constant travel distance each time, and after the movement ends, the new direction and speed of the user mobile node are calculated;

[0014] Step S13: if the user mobile node reaches the simulation boundary, it is bounced back from the simulation boundary, the bounce-back angle is determined by the incident direction, and the movement continues along this path, and the above steps are repeated until the end.

[0015] In some embodiments, the step S2 includes generating a joint optimization model aiming to maximize the total system throughput, and the joint optimization model is as follows:

[0016]

[0017]

[0018] In the formula, SIRkn represents the signal-to-interference ratio of user k on subchannel n, C1 represents a linear constraint of power, Pkn represents the base station transmission power, P max Pmax represents the maximum base station transmission power; C2 represents that the transmission power cannot be less than 0; C3 represents a binary variable representing the relationship between user k and channel n; C4 represents that each channel can only be allocated to one user, Skn represents the user binary channel allocation vector.

[0019] In some embodiments, the total system throughput is represented as:

[0020]

[0021] In the formula, R total (t) represents the total system throughput at time t, R k (t) represents the throughput of base station k at time t, t represents time t, k represents user base station, and U represents the total user set.

[0022] In some embodiments, the step S3 includes that the state function of the user is:

[0023] s k (t) = {C(t), P(t), d k (t)}

[0024] In the formula, C(t) is the set of all user channel selections, P(t) is the set of all user power selections, d k (t) is the position vector of base station k at time t, s k (t) is the state vector of base station k at time t.

[0025] In some embodiments, the step S3 comprises that the action function of the user is:

[0026]

[0027] wherein, is the base station transmit power at time t, c k (t) is the channel selection vector at time t.

[0028] In some embodiments, the step S3 comprises that the reward function is:

[0029]

[0030] wherein, R(t) is the reward value at time t, is the signal-to-interference-plus-noise ratio of the base station k selecting the channel n at time t.

[0031] To achieve the same purpose, the application also provides a real-time spectrum resource optimization system based on deep reinforcement learning, comprising:

[0032] a behavior simulation module for constructing a typical ultra-dense network downlink scenario and simulating the behavior of dynamic user movement according to a random walk model;

[0033] an optimization model generation module for system modeling analysis according to the ultra-dense network downlink scenario to generate a joint optimization model with the goal of maximizing the total system throughput;

[0034] a model solving module for solving the optimal solution of the joint optimization model through a deep reinforcement learning algorithm and setting the state function of the user, the action function of the user and the reward function to train the user to learn the optimal solution.

[0035] In some embodiments, the behavior simulation module is configured to simulate the behavior of dynamic user movement according to a random walk model by the following steps:

[0036] Step S11: the user movement node randomly selects a direction and a speed to move from the current position to a new position;

[0037] Step S12: the user movement node moves at a constant time interval or a constant travel distance each time, and after the movement is completed, the new direction and speed of the user movement node are calculated;

[0038] Step S13: if the user movement node reaches the simulation boundary, it is bounced back from the simulation boundary, the bounce-back angle is determined by the incident direction, and the movement continues along this path, and steps S11-S13 are repeated until the end.

[0039] In some embodiments, the optimization model generation module is configured to generate a joint optimization model aiming at maximizing the total system throughput, and the joint optimization model is as follows:

[0040]

[0041]

[0042] wherein, denotes the signal-to-interference ratio of user k on subchannel n, C1 denotes a linear constraint of power, denotes the base station transmission power, P max denotes the maximum base station transmission power; C2 denotes that the transmission power cannot be less than 0; C3 denotes a binary variable representing the relationship between user k and channel n; C4 denotes that each channel can only be allocated to one user, denotes a user binary channel allocation vector.

[0043] The above technical solution has the following beneficial effects:

[0044] (1) The present application considers the unpredictability of dynamic user movement, and constructs a dynamic hyper-dense network environment based on a random walk model.

[0045] (2) The present application also considers the non-convexity and complexity of the joint optimization problem and introduces a multi-agent reinforcement learning method, specifically, the FUE (Femto User Equipment) defines a state, an action and a reward function, and uses a DoubleQ strategy (DoubleQ-Learning) and a duel network architecture (Dueling Network Architecture) to approximate the action value function under the current state, thereby obtaining an approximately optimal strategy.

[0046] (3) The simulation experiment results prove that the optimization method proposed in the present application has good learning performance in a dense base station environment, especially in terms of total system throughput, the method proposed in the present application is obviously superior to the traditional DQN method and other traditional resource allocation strategies. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0048] Figure 1A flowchart of a deep reinforcement learning-based real-time spectrum resource optimization method provided by an embodiment of the present application is shown in the figure.

[0049] Figure 2 A structural diagram of a deep reinforcement learning-based real-time spectrum resource optimization system provided by an embodiment of the present application is shown in the figure.

[0050] Figure 3 A hyperdense network scenario model of a deep reinforcement learning-based real-time spectrum resource optimization method provided by an embodiment of the present application is shown in the figure.

[0051] Figure 4 An iteration comparison diagram of a deep reinforcement learning-based real-time spectrum resource optimization method provided by an embodiment of the present application under different neuron numbers of a neural network hidden layer is shown in the figure.

[0052] Figure 5 An iteration comparison diagram of a deep reinforcement learning-based real-time spectrum resource optimization method provided by an embodiment of the present application and other methods is shown in the figure.

[0053] Figure 6 A system throughput comparison diagram of a deep reinforcement learning-based real-time spectrum resource optimization method provided by an embodiment of the present application and other methods under different base station numbers is shown in the figure.

[0054] Figure 7 A system throughput comparison diagram of a deep reinforcement learning-based real-time spectrum resource optimization method provided by an embodiment of the present application and other methods under different channel numbers is shown in the figure. DETAILED DESCRIPTION

[0055] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application.

[0056] Examples of the embodiments are shown in the drawings, in which the same or similar symbols represent the same or similar elements or elements with the same or similar functions throughout. The embodiments described below by reference to the drawings are exemplary and are intended to explain the present application, but cannot be understood as a limitation of the present application.

[0057] Embodiment One

[0058] An embodiment of the present application provides a deep reinforcement learning-based real-time spectrum resource optimization method, as shown in Figure 1 , Figure 3 , which includes

[0059] Step S1: Constructing a typical ultra-dense network downlink scenario, and simulating the behavior of user dynamic movement according to a random walk model.

[0060] In one embodiment of the present application, the step S1 comprises:

[0061] Step S11: The user mobile node randomly selects a direction and a speed to move from the current position to a new position.

[0062] Step S12: Each movement of the user mobile node is performed at a constant time interval or a constant travel distance, and after the end, the new direction and speed of the user mobile node are calculated.

[0063] Step S13: If the user mobile node reaches the simulation boundary, it is bounced back from the simulation boundary, the bounce angle is determined by the incident direction, and the movement continues along this path, and the above steps are repeated until the end.

[0064] Step S2: According to the ultra-dense network downlink scenario, system modeling analysis is performed to maximize the total system throughput, and a joint optimization model is generated.

[0065] Specifically, in reality, users will move in an unpredictable way, and a random walk model (RWMM) is used to simulate this unstable movement. In this movement model, the mobile node randomly selects a direction and a speed to move from the current position to a new position. The new speed and direction are selected from the predefined ranges [V min ,V max ] and [0, 2π] respectively. Each movement of the mobile node is performed at a constant time interval t or a constant travel distance d, and after the end, the new direction and speed are calculated. If the mobile node of this model reaches the simulation boundary, it will be "bounced back" from the simulation boundary, the angle is determined by the incident direction, and then the movement continues along this path, and the above steps are repeated until the end.

[0066] In one embodiment of the present application, a typical ultra-dense network downlink scenario is considered, including a macro base station (MBS) and F femto base stations (FBS), and the macro base station serves as the information center of the entire network and can collect information of the entire network. The MBS and the FBS use orthogonal spectrum resources, so there is no interlayer interference. Assuming that each FBS serves only one user at the same time, and the FBS and the user are both configured with a single antenna, the same set U = {0, 1, 2, 3…F-1} can be used, and the FBS and the user are respectively denoted as f kand k denotes the kth base station and user. The number of available channels in the network is N, and the nth channel is denoted by n e B = {0, 1, 2,..., N-1}.

[0067] Each femto user equipment (FUE) served by a femto base station (FBS) moves randomly within the coverage of its serving base station at time t. For the kth FUE, the binary channel allocation vector is denoted by if the kth FUE uses channel n at time t, and otherwise. For simplicity of discussion, it is assumed in this paper that each UE can select at most one channel at any time t, i.e.,

[0068]

[0069] The FBSs in the scenario share the same spectrum resource, so the users are interfered by co-channel users. Therefore, the signal-to-interference-and-noise ratio (SINR) of the kth FUE on subchannel n at time t can be expressed as:

[0070]

[0071] where is the transmission power of the kth FUE served by the femto base station on subchannel n, and σ 2 is the Gaussian white noise, denotes the path loss from the base station f k serving the kth user on subchannel n at time t, and

[0072]

[0073] where L is the modified path loss constant, d k (t) denotes the straight-line distance from the base station f k serving the kth user to its served user at time t, and a denotes the path loss exponent. denotes the Rayleigh fading of the kth user served by the base station f k on subchannel n at time t.

[0074] Therefore, the total capacity of the kth user at time t can be obtained by Shannon's formula as:

[0075]

[0076] where W is the bandwidth of the subchannel. The total throughput of the system at time t can be expressed as:

[0077]

[0078] where Rtotal (t) represents the total throughput of the system at time t, R k (t) represents the throughput of base station k at time t, t represents time t, k represents a user base station, and U represents a total user set.

[0079] A joint optimization model is generated to maximize the total throughput of the system, and the joint optimization model is as follows:

[0080]

[0081]

[0082] In the formula, SIRk,n represents the signal-to-interference ratio of user k on subchannel n, C1 represents a linear constraint of power, P represents the base station transmit power, P max Pmax represents the maximum base station transmit power; C2 represents that the transmit power cannot be less than 0; C3 represents that the binary variable represents the relationship between user k and channel n; C4 represents that each channel can only be allocated to one user, S represents the user binary channel allocation vector.

[0083] Step S3: An optimal solution of the joint optimization model is solved by a deep reinforcement learning algorithm, and a state function of a user, an action function of the user, and a reward function are set to train the user to learn the optimal solution.

[0084] In one specific embodiment of the present application, each femto base station served user is regarded as an agent, which interacts with an unknown environment to obtain experience for guiding its own strategy. All participants in the scene, i.e., FUEs, have no information about the network environment and available channel quality. Each participant is selfish and hopes to obtain the maximum long-term reward by selecting the best channel and power. At any time t, the reward of the participant is affected by the current state of the network environment and the behavior of other participants. At the next moment, it will be converted into a completely new random state, which is affected by the previous state and the selected action of all participants. Therefore, the optimization problem can be regarded as a competitive game, which is converted into a completely cooperative game by using the same reward for all agents to ensure the performance of the global network.

[0085] Specifically, let s k (t) represents the state observed by user k at time t, which is used to represent the UDN environment. Public information (action selection information of each FUE, etc.) can be obtained from the MBS at the beginning of each time slot. The user at t position can be represented as a two-dimensional vector (x k (t),y k (t)) according to the Pythagorean theorem The user position is converted into a one-dimensional vector as the state storage, which can effectively reduce the state space. The user k position state is denoted as d k (t).

[0086] Each agent in the network needs to know the information of the entire network to determine the action selection, and each agent needs to determine the optimal strategy according to its own position. Therefore, the state function of user k at time t is:

[0087] s k (t) = {C(t), P(t), d k (t)}

[0088] In the formula, C(t) is the set of all user channel selection, P(t) is the set of all user power selection, d k (t) is the position vector of base station k at time t, s k (t) is the state vector of base station k at time t.

[0089] Action space: The resource allocation action in the ultra-dense network should include channel selection and transmit power selection. Therefore, the action function of each agent is:

[0090]

[0091] In the formula, is the transmit power of base station at time t, c k (t) is the channel selection vector at time t.

[0092] With the increase of the number of users, the action space increases exponentially. Therefore, in order to reduce the action space, the transmit power is discretized in this paper:

[0093]

[0094] Reward function: The reward function of the ultra-dense network resource allocation algorithm should be comprehensive. Therefore, the reward function is expressed as the system throughput to encourage agents to take actions to optimize the system throughput. At the same time, all agents use the same reward, which is converted into a completely cooperative game to ensure the performance of the global network. It can be expressed as:

[0095]

[0096] In the formula, R(t) is the reward value at time t, is the signal-to-interference ratio of base station k selecting channel n at time t.

[0097] When DQN updates the Q value, both action selection and action evaluation are based on the same network, so the Q value function can be overestimated. The algorithm used in the present application can effectively reduce the risk of overestimating the Q value by using two different parameter neural networks in double DQN (DDQN) to determine the Q value of the best action and estimate the action. The target value can be calculated as follows:

[0098]

[0099] In the formula, is the Q value obtained by using DDQN to update, γ is the discount factor, and u k is the reward value, Qk is the Q value function, s is the user state, θ is the neural network parameter, and a k is the action selection.

[0100] At the same time, the duel neural network is introduced, and the original Q function is decomposed into the state value function V(s) and the advantage function A(s,a k )=Q k (s,a k )-V(s). The advantage function A(s,a k ) describes the advantage of the action a k compared to other possible actions. In the duel architecture, the last layer of DDQN is divided into two subnets to estimate V(s) and A(s,a k ) respectively. By combining V(s) and A(s,a k ), the action value function Q k (s,a k ) can be estimated. This architecture better distinguishes the value of the state and the advantages of the action, and can more accurately evaluate the strategy.

[0101] As Figure 4 shows the smoothing steps in the training process under different numbers of hidden layer neurons. From the chart data, it can be clearly observed that when the number of hidden layer neurons is set to 256, the training is prone to fall into a local optimal solution in the early stage, which can affect the global performance of the model. On the contrary, when the number of neurons is set to 64, the optimization effect of the training process performs relatively poorly. Based on this, 128 neurons are finally selected in the D3QN network structure for each hidden layer. This configuration not only avoids the problem of falling into a local optimal solution, but also shows more excellent optimization performance compared to other settings.

[0102] As Figure 5The training process under different learning rates is shown. It is observed that the total throughput of the system is generally low at the beginning of the training, but gradually increases with the number of iterations. In particular, when the learning rate is set to 0.001, the system is prone to fall into a local optimal solution due to the excessively high learning speed, resulting in fluctuations in the performance curve. On the contrary, when the learning rate is 0.000001, it is too conservative, so that the model cannot learn the optimal strategy within a suitable time frame, resulting in slow convergence speed. In addition, compared with the learning rate of 0.000005, when the learning rate is 0.00001, the system is more prone to fall into a local optimal solution at the beginning of the training, and the overall performance is poor. Considering various factors, the learning rate of 0.000005 is finally selected. This setting not only avoids the adverse effects caused by excessively high or low learning rate, but also shows relatively better convergence and system performance in the experiment.

[0103] As Figure 6 The total throughput of the system under three different algorithms is shown in detail. It is observed that at the beginning of the D3QN training, its total throughput is basically the same as that of the random allocation algorithm. However, as the number of training increases, the performance of D3QN gradually emerges, not only exceeding the greedy allocation algorithm, but also after about 2000 iterations, the total throughput tends to be stable, close to the optimal state.

[0104] As Figure 7 The comparison of system throughput under different base station numbers and different channel numbers is shown. It can be observed that when the number of available channels remains unchanged, if the number of available channels is more than the number of base stations, the total throughput of the system will gradually increase with the increase of the number of base stations. However, when the number of base stations exceeds the number of available channels, the system throughput decreases slightly. This phenomenon is mainly due to the fact that under limited available resources, the increase of the number of base stations will lead to the increase of network internal interference, thereby limiting the further improvement of the throughput. On the other hand, when the number of base stations is fixed and greater than the number of available channels, increasing the number of available channels will lead to the reduction of subchannel selection conflict and the decrease of system internal interference, so that the throughput is improved.

[0105] The simulation results prove that the optimization method proposed in the application has good learning performance in the dense base station environment, especially in the total throughput of the system, the effect of using the method proposed in the application is obviously better than that of the traditional DQN method and other traditional resource allocation strategies.

[0106] Embodiment two

[0107] An embodiment of the application provides a real-time spectrum resource optimization system based on deep reinforcement learning, referring to Figure 2 、 Figure 3shown, comprising:

[0108] a behavior simulation module 10 for constructing a typical ultra-dense network downlink scenario and simulating the behavior of dynamic user movement according to a random walk model.

[0109] In one embodiment of the present application, the behavior simulation module 10 is configured to simulate the behavior of dynamic user movement according to a random walk model by the following steps:

[0110] Step S11: the user movement node randomly selects a direction and speed to move from the current position to a new position;

[0111] Step S12: the user movement node moves at a constant time interval or constant travel distance each time, and after the end, the new direction and speed of the user movement node are calculated;

[0112] Step S13: if the user movement node reaches the simulation boundary, it is bounced back from the simulation boundary, the bounce-back angle is determined by the incident direction, and the movement continues along this path, and steps S11-S13 are repeated until the end.

[0113] an optimization model generation module 20 for performing system modeling analysis according to the ultra-dense network downlink scenario, generating a joint optimization model with the goal of maximizing the total system throughput.

[0114] In one embodiment of the present application, a typical ultra-dense network downlink scenario is considered, including one macro base station (macrocell base station, MBS) and F femto base stations (Femto base station, FBS), and the macro base station serves as the information center of the entire network and can collect information of the entire network. Orthogonal spectrum resources are used between the MBS and the FBS, so there is no interlayer interference. Assuming that each FBS serves only one user at the same time, and both the FBS and the user are configured with a single antenna, the same set U = {0, 1, 2, 3…F-1} can be used to represent the kth base station and user, respectively, and fk and k are used to represent the kth base station and user. The number of available channels in the network is N, and the nth channel is represented by n∈B = {0, 1, 2…N-1}.

[0115] The user (Femto User Equipment, FUE) served by each femto base station moves randomly within the coverage range of the serving base station at time t. For the kth FUE, the binary channel allocation vector is represented by if the kth FUE uses channel n at time t, then otherwise For simplicity of discussion, it is assumed in this paper that each UE can only select one channel at any time t, i.e.:

[0116]

[0117] The FBSs in a scenario share the same spectrum resource, so the users will be interfered by the co-channel users, thus the signal-to-interference-and-noise ratio of the kth FUE on the nth sub-channel at time t can be expressed as:

[0118]

[0119] wherein, is the transmission power of the kth user served by the femto base station on the nth sub-channel, σ 2 is the Gaussian white noise, denotes the path loss from the base station f k to its served user k at time t; and α denotes the path loss exponent.

[0120]

[0121] wherein, L is the modified path loss constant; d k (t) denotes the path loss from the base station f k to its served user k at time t; and α denotes the path loss exponent. denotes the Rayleigh fading of the base station f k and its served user on the nth sub-channel at time t.

[0122] Thus, the total capacity of the kth user at time t can be obtained by the Shannon formula as follows:

[0123]

[0124] wherein, W is the bandwidth of the sub-channel. The total throughput of the system at time t can be expressed as:

[0125]

[0126] wherein, R total (t) denotes the total throughput of the system at time t, R k (t) denotes the throughput of the kth user at time t, t denotes time t, k denotes the user base station, and U denotes the total user set.

[0127] A joint optimization model is generated for the purpose of maximizing the total throughput of the system, and the joint optimization model is shown as follows:

[0128]

[0129]

[0130] wherein, represents the signal-to-interference-and-noise ratio of user k on subchannel n, C1 represents the linear constraint of power, Indicates the base station transmission power, P max Indicates the maximum power transmitted by the base station; C2 indicates that the transmission power cannot be less than 0; C3 indicates that the binary variable represents the relationship between user k and channel n; C4 indicates that each channel can only be allocated to one user. represents the user binary channel allocation vector.

[0131] Model solving module 30: used to solve the optimal solution of the joint optimization model through a deep reinforcement learning algorithm, and set the user's state function, the user's action function and the reward function to train the user to learn the optimal solution.

[0132] In one specific embodiment of the present invention, each user served by a femtocell is considered an agent, interacting with the unknown environment to gain experience and guide its own strategy. All participants in the scenario, or FUEs, have no information about the network environment or the quality of available channels. Each participant is selfish, aiming to maximize long-term rewards by selecting the optimal channel and power. At any time t, a participant's reward is influenced by the current state of the network environment and the actions of other participants. At the next time, the network transitions to a completely new, random state, influenced by the previous states and actions chosen by all participants. Therefore, the optimization problem can be viewed as a competitive game. By applying the same reward to all agents, it can be transformed into a fully cooperative game to ensure global network performance. The key elements of the design of spectrum resource allocation based on multi-agent RL are described in detail below.

[0133] Specifically, let s k (t) represents the state observed by user k at time slot t, which is used to characterize the UDN environment. Public information (such as the action selection information of each FUE) can be obtained from the MBS at the beginning of each time slot. The user at position t can be represented as a two-dimensional vector (x k (t),y k (t)), according to the Pythagorean theorem Converting the user position into a one-dimensional vector as state storage can effectively reduce the state space. The position state of user k is d k (t).

[0134] Each agent in the network needs to know the information of the entire network to decide the action to take, and each agent needs to determine the optimal strategy based on its own position. Therefore, the state function of user k at time t is:

[0135] s k (t)={C(t),P(t),d k (t)}

[0136] Where C(t) is the set of all user channel selections, P(t) is the set of all user power selections, and d k (t) is the position vector of base station k at time t, s k (t) is the state vector of base station k at time t.

[0137] Action space: Resource allocation actions in ultra-dense networks should include channel selection and transmit power selection. Therefore, the action function of each agent is:

[0138]

[0139] Where, is the base station transmission power at time t, c k (t) is the channel selection vector at time t.

[0140] As the number of users increases, the action space increases exponentially. Therefore, in order to reduce the action space, this paper discretizes the transmit power:

[0141]

[0142] Reward function: The reward function for an ultra-dense network resource allocation algorithm should be comprehensive. Therefore, the reward function is expressed as system throughput, encouraging agents to take actions that optimize the system's throughput. At the same time, all agents use the same reward, transforming it into a fully cooperative game to ensure global network performance. It can be expressed as:

[0143]

[0144] Where R(t) is the reward value at time t, Select the signal-to-interference-and-noise ratio of channel n for base station k at time t.

[0145] When DQN updates Q values, action selection and action evaluation are both based on the same network, so the Q value function may be overestimated. However, the algorithm used in this paper uses two neural networks with different parameters in the dual DQN (DDQN) to determine the best action and estimate the Q value of the action, which can effectively reduce the risk of overestimating the Q value. Its target value can be calculated as follows:

[0146]

[0147] Where, is the Q value update obtained by DDQN, γ is the discount factor, u k is the reward value, Qk is the Q value function, s is the state, θ is the neural network parameter, a k Select for action.

[0148] Meanwhile, Dueling Neural Networks are introduced to decompose the original Q function into a state value function V(s) and an advantage function A(s, a k ) = Q k (s, a k ) - V(s). The advantage function A(s, a k ) describes the advantage of action a k over other possible actions. In the Dueling architecture, the last layer of DDQN is divided into two subnets to estimate V(s) and A(s, a k ) respectively. By combining V(s) and A(s, a k ), the action value function Q k (s, a k ) can be estimated. This architecture better distinguishes the value of the state and the advantage of the action, which can more accurately evaluate the policy.

[0149] The above merely provides a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which shall be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

[0150] Each of the embodiments in the present specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts of each embodiment can be understood by referring to each other. The embodiments of the present application are described with reference to flowcharts and / or block diagrams of methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the computer or other programmable data processing terminal device generate a means for implementing the functions specified in one or more flows and / or blocks in the flowcharts and / or block diagrams. These computer program instructions can also be stored in a computer-readable memory that can guide the computer or other programmable data processing terminal device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a product including instruction means, which implements the functions specified in one or more flows and / or blocks in the flowcharts and / or block diagrams. Figure 1 The apparatuses for implementing the functions specified in one or more flows and / or blocks in the flowcharts and / or block diagrams. These computer program instructions can also be stored in a computer-readable memory that can guide the computer or other programmable data processing terminal device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a product including instruction means, which implements the functions specified in one or more flows and / or blocks in the flowcharts and / or block diagrams. Figure 1 The apparatuses for implementing the functions specified in one or more flows and / or blocks in the flowcharts and / or block diagrams. These computer program instructions can also be stored in a computer-readable memory that can guide the computer or other programmable data processing terminal device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a product including instruction means, which implements the functions specified in one or more flows and / or blocks in the flowcharts and / or block diagrams. Figure 1 The apparatuses for implementing the functions specified in one or more flows and / or blocks in the flowcharts and / or block diagrams. These computer program instructions can also be stored in a computer-readable memory that can guide the computer or other programmable data processing terminal device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a product including instruction means, which implements the functions specified in one or more flows and / or blocks in the flowcharts and / or block diagrams. Figure 1the functions specified in the block or blocks. Such computer program instructions can also be loaded into a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 the flowchart or flowchart Figure 1 the steps of the functions specified in the block or blocks. Although preferred embodiments of the application have been described, those skilled in the art will recognize that the application can be practiced with modification and alteration within the spirit and scope of the appended claims. Therefore, all such modifications and alterations should be considered within the scope of the application. Finally, the terms "comprise", "comprising", or any variation thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. The terms "include", "including", or any variation thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0151] The above detailed description has set forth various embodiments of the methods and devices provided by the application. The precise details of the application, including the specific examples, are intended to be illustrative only and not limiting to the scope of the application, which is set forth in the claims that follow. The description of the application set forth herein is intended to be illustrative only and is not intended to limit the scope of the application as set forth in the claims. Those skilled in the art will recognize that numerous variations on the methods and devices of the present application can be made without departing from the scope of the present application.

[0152] In the description of the present application, reference has been made to descriptive terms such as "one embodiment", "some embodiments", "an example", "a specific example" or "some examples" meant to

[0153] Finally, it should be noted that the above examples are only used to illustrate the technical solutions of the present application, and are not intended to limit the same; although the present application has been described in detail with reference to the foregoing examples, those of ordinary skill in the art will understand that they can still modify the technical solutions described in the foregoing examples, or make equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A real-time spectrum resource optimization method based on deep reinforcement learning, characterized in that: include: Step S1: Construct a typical ultra-dense network downlink scenario and simulate the user's dynamic movement behavior based on the random walk model; Step S2: performing system modeling analysis based on the ultra-dense network downlink scenario, and generating a joint optimization model with the goal of maximizing the total system throughput; Step S3: solving the optimal solution of the joint optimization model by a deep reinforcement learning algorithm, and setting the state function, action function and reward function to train the base station to learn the optimal solution; The step S3 includes: the state function is: s k (t)={C(t),P(t),d k (t)} Where C(t) is the set of all user channel selections, P(t) is the set of all user power selections, and d k (t) is the position vector of base station k at time t, s k (t) is the state vector of base station k at time t; The step S3 includes: the action function is: Where, is the base station transmission power at time t, c k (t) is the channel selection vector at time t; The step S3 includes: the reward function is: Where R(t) is the reward value at time t, The signal-to-interference-and-noise ratio of channel n selected by base station k at time t; The step S1 comprises: Step S11: The user mobile node randomly selects a direction and speed to move from the current location to the new location; Step S12: The user mobile node moves at a constant time interval or a constant travel distance each time, and after completion, a new direction and speed of the user mobile node are calculated; Step S13: If the user mobile node reaches the simulation boundary, it bounces back from the simulation boundary, the rebound angle is determined by the incident direction, and continues to move along this path, repeating the above steps until the end; The step S2 includes: generating a joint optimization model with the goal of maximizing the total system throughput, and the joint optimization model is as follows: Where, represents the signal-to-interference-and-noise ratio of user k on subchannel n, C1 represents the linear constraint of power, Indicates the base station transmission power, P max Indicates the maximum power transmitted by the base station; C2 indicates that the transmission power cannot be less than 0; C3 indicates that the binary variable represents the relationship between user k and channel n; C4 indicates that each channel can only be allocated to one user. represents the user binary channel allocation vector.

2. The real-time spectrum resource optimization method based on deep reinforcement learning according to claim 1 is characterized in that: The total throughput of the system is expressed as: Where R total (t) represents the total system throughput at time t, R k (t) represents the throughput of base station k at time t, t represents time t, k represents the user base station, and U represents the total user set.

3. A real-time spectrum resource optimization system based on deep reinforcement learning, characterized in that: include: Behavior simulation module: used to build a typical ultra-dense network downlink scenario and simulate user dynamic movement behavior based on the random walk model; An optimization model generation module is configured to perform system modeling analysis based on the ultra-dense network downlink scenario, and generate a joint optimization model with the goal of maximizing the total system throughput; Model solving module: used to solve the optimal solution of the joint optimization model through deep reinforcement learning algorithm, and set the state function, action function and reward function to train the base station to learn the optimal solution; The behavior simulation module is used to simulate the user's dynamic movement behavior according to the random walk model through the following steps: Step S11: The user mobile node randomly selects a direction and speed to move from the current location to the new location; Step S12: The user mobile node moves at a constant time interval or a constant travel distance each time, and after completion, a new direction and speed of the user mobile node are calculated; Step S13: If the user mobile node reaches the simulation boundary, it bounces back from the simulation boundary, the rebound angle is determined by the incident direction, and continues to move along this path, repeating steps S11-S13 until the end; The optimization model generation module is used to generate a joint optimization model with the goal of maximizing the total system throughput. The joint optimization model is as follows: Where, represents the signal-to-interference-and-noise ratio of user k on subchannel n, C1 represents the linear constraint of power, Indicates the base station transmission power, P max Indicates the maximum power transmitted by the base station; C2 indicates that the transmission power cannot be less than 0; C3 indicates that the binary variable represents the relationship between user k and channel n; C4 indicates that each channel can only be allocated to one user. represents the user binary channel allocation vector; The model solving module includes: the state function is: s k (t)={C(t),P(t),d k (t)} Where C(t) is the set of all user channel selections, P(t) is the set of all user power selections, and d k (t) is the position vector of base station k at time t, s k (t) is the state vector of base station k at time t; The model solving module includes: the action function is: Where, is the base station transmission power at time t, c k (t) is the channel selection vector at time t; The model solving module includes: the reward function is: Where R(t) is the reward value at time t, Select the signal-to-interference-and-noise ratio of channel n for base station k at time t.

Images