A d2d user resource allocation method based on a deep reinforcement learning algorithm and a storage medium

Abstract

This invention discloses a D2D user resource allocation method and storage medium based on a deep reinforcement learning algorithm, relating to the field of wireless communication technology. The method includes: constructing a wireless network model and discretizing the D2D transmit power; constructing a user signal-to-noise ratio calculation model with maximizing the communication system throughput as the optimization objective; setting a prediction policy network π, a prediction value network Q, a target policy network π′, and a target value network Q'; modeling the D2D communication environment as a Markov decision process, treating the D2D transmitter as an agent, iteratively loading the parameters of the target policy network to generate a policy that interacts with the environment, determining the state space, action space, and reward function; using the MAAC algorithm to optimize the policy for each D2D user; using a soft update method to iteratively update the parameters of the target policy network and the target value network until training is complete; and having the D2D user download the trained parameters of the target policy network and improve their policy.

Description

Technical Field

[0001] This invention relates to the field of wireless communication technology, and more specifically, to a D2D user resource allocation method and storage medium based on a deep reinforcement learning algorithm. Background Technology

[0002] In today's rapidly developing technological era, wireless communication technology has become inseparable from people's daily lives. People's demand for mobile communication is increasing rapidly, and their requirements are becoming increasingly stringent. From the initial need for mobile communication devices to simply make calls, to the later requirement for basic internet searches, and now to watching videos and listening to music, with ever-increasing demands for video clarity and audio quality. However, in environments with dense user numbers and significant communication interference, the problem of scarce spectrum resources is particularly prominent. Therefore, we have proposed many methods to address this issue.

[0003] Device-to-device (D2D) technology is one such example, referring to the technology that allows direct information exchange between neighboring devices in a communication network. Compared to traditional cellular communication, D2D communication does not require base stations as relays. Therefore, it can communicate in locations far from or even without base stations, effectively reducing the transmission load on base stations. Furthermore, D2D communication technology can share the spectrum resources of cellular users, significantly improving spectrum utilization, increasing system throughput, and enhancing the overall performance of the communication system.

[0004] In D2D communication technology, reasonable power allocation and resource block allocation for D2D User Equipment (DUE) are crucial. DUEs primarily reuse spectrum resources occupied by Cellular User Equipment (CUE), leading to mutual interference among DUEs, CUEs, and base stations (BS). To effectively avoid this interference and improve the Quality of Service (QoS) for D2D users, many solutions have been proposed. For example, machine learning, which has become increasingly popular in recent years, is used to handle channel allocation and power control. These solutions largely consider an ideal model where all user information is deterministic. However, in real-world environments, both DUEs and CUEs are mostly dynamic, containing information such as location and channel gain. The sheer volume of information and rapid changes in the environment result in high computational complexity, rendering traditional optimization methods inapplicable. Summary of the Invention

[0005] To overcome the shortcomings of the prior art in adapting to dynamic environments, the present invention provides a D2D user resource allocation method and storage medium based on a deep reinforcement learning algorithm.

[0006] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0007] Firstly, a D2D user resource allocation method based on a deep reinforcement learning algorithm includes:

[0008] A wireless network model is constructed, and the D2D transmit power is discretized to generate K power levels. The wireless network model includes a macro base station, L cellular users within the coverage area of ​​the macro base station network, N pairs of D2D user pairs, and M orthogonal spectrum resource blocks. The parameters configured in the wireless network model include user locations.

[0009] A user signal-to-noise ratio (SNR) calculation model is constructed to calculate the SNR information of D2D users and cellular users, and to set the QoS requirements for communication between D2D users and cellular users. The wireless network model is optimized with the goal of maximizing the throughput of the communication system composed of D2D users and cellular users. The user SNR includes the SNR of the D2D user receiver and the SNR of the cellular user.

[0010] The macro base station sets up a prediction policy network π, a prediction value network Q, a target policy network π′, and a target value network Q′ for each agent;

[0011] The D2D communication environment is modeled as a Markov decision process, with the D2D transmitter considered as an agent. After iteratively loading parameters from the target policy network, a policy is generated and interacts with the environment to determine the state space, action space, and reward function. Under the premise of meeting QoS requirements, each agent selects the communication mode to use at time t, executes an action based on the currently observed state, obtains a reward, and transitions to the next state. Experience groups are uploaded to an experience pool for centralized training. The communication modes include dedicated mode, multiplexing mode, and waiting mode. The states include the location information and signal-to-noise ratio information of D2D and cellular users. The actions include selecting the power value and resource block used for communication.

[0012] The MAAC algorithm is used to optimize the policy for each D2D user. Small-batch random sampling is performed from the experience pool for centralized training. The TD algorithm is used to update the prediction value network, and the gradient descent method is used to update the parameters of the prediction value network. The cumulative reward is calculated based on the rewards obtained by the agent performing actions. The policy gradient is set according to the cumulative reward, and the parameters of the prediction policy network are iteratively updated based on the policy gradient using the gradient ascent method. The learning objective of the MAAC algorithm is to learn a policy for each agent to obtain the maximum cumulative reward.

[0013] Based on the parameters of the prediction policy network and the prediction value network, the parameters of the target policy network and the target value network are updated cyclically using a soft update method until the learning and training are completed.

[0014] D2D users download the parameters of the trained target policy network, improve the policy, and select the communication mode, resource blocks, and / or communication power based on the observed current environment.

[0015] In a second aspect, a computer storage medium storing instructions that, when executed on a computer, cause the computer to perform the method described in the first aspect.

[0016] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0017] (1) Traditional algorithms are very difficult to handle communication scenarios with a large number of users and rapidly changing scenarios. The model-free reinforcement learning algorithm adopted in this invention can effectively solve decision-making problems in uncertain environments.

[0018] (2) The MAAC algorithm framework adopted in this invention coordinates the strategies among multiple agents, effectively overcomes the non-stationarity of the multi-agent environment, achieves the optimal energy efficiency of the communication system, and is suitable for complex and ever-changing communication scenarios.

[0019] (3) The present invention adopts a centralized training and distributed execution approach. D2D users upload useful information about their interaction with the environment to the experience pool, thereby transferring the complex training process to the base station and making the training process of the agent more efficient.

[0020] (4) The D2D user pair in this invention can operate in two modes: dedicated mode and multiplexing mode. The D2D user will prioritize the use of idle channels for communication. Before entering the multiplexing mode, it will first check whether the cellular user and the D2D user still meet the QoS requirements after the D2D user reuses the cellular user spectrum resources. Only if the requirements are met can multiplexing be performed. This improves the bandwidth utilization and greatly reduces the failure rate of cellular user data transmission, thus ensuring the reliability of data transmission.

[0021] (5) Each D2D user in this invention can autonomously select the transmission power based on the proposed algorithm while ensuring QoS quality, thus avoiding the D2D user always working at the highest transmission power for data transmission and reducing the power consumption of the system. Attached Figure Description

[0022] Figure 1 A flowchart of a D2D user resource allocation method based on a deep reinforcement learning algorithm;

[0023] Figure 2 This is a schematic diagram of the wireless network model in Example 1;

[0024] Figure 3 This is a schematic diagram of the interaction process between the intelligent agent and the environment in Example 1;

[0025] Figure 4 This is a schematic diagram of the network update process in Example 1;

[0026] Figure 5 This is a schematic diagram of information sharing between adjacent intelligent agents in Example 1;

[0027] Figure 6 This is a schematic diagram of the training process in Example 2. Detailed Implementation

[0028] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent.

[0029] To better illustrate this embodiment, some parts in the accompanying drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions;

[0030] It will be understood by those skilled in the art that certain well-known structures and their descriptions may be omitted in the accompanying drawings.

[0031] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0032] Example 1

[0033] This embodiment provides a D2D user resource allocation method based on a deep reinforcement learning algorithm. (See attached document.) Figure 1 ,include:

[0034] Building a wireless network model, such as Figure 2 As shown, the D2D transmit power is discretized to generate K power levels; the wireless network model includes a macro base station, L cellular users within the macro base station network coverage area and N pairs of D2D user pairs, and M orthogonal spectrum resource blocks. The parameters configured in the wireless network model include user locations.

[0035] A user signal-to-noise ratio (SNR) calculation model is constructed to calculate the SNR information of D2D users and cellular users, and to set the QoS requirements for communication between D2D users and cellular users. The wireless network model is optimized with the goal of maximizing the throughput of the communication system composed of D2D users and cellular users. The user SNR includes the SNR of the D2D user receiver and the SNR of the cellular user.

[0036] The macro base station sets up a prediction policy network π, a prediction value network Q, a target policy network π′, and a target value network Q′ for each agent;

[0037] The D2D communication environment is modeled as a Markov decision process, with the D2D transmitter treated as an intelligent agent. After iteratively loading the parameters of the target policy network π′, a policy is generated and interacts with the environment to determine the state space, action space, and reward function. (See also...) Figure 3 Under the premise of meeting QoS requirements, each agent selects the communication mode to be used at time t, performs action a according to the currently observed state s, obtains reward r and transitions to the next state s', and uploads the experience group (s,,',r) to the experience pool for centralized training; wherein, the communication mode includes dedicated mode, multiplexing mode and waiting mode, the state includes the location information and signal-to-noise ratio information of D2D users and cellular users, and the action includes selecting the power value and resource block for communication;

[0038] The MAAC algorithm is used to optimize the policy for each D2D user. Small-batch random sampling is performed from the experience pool for centralized training. The TD algorithm is used to update the prediction value network, and the gradient descent method is used to update the parameters of the prediction value network. The cumulative reward is calculated based on the rewards obtained by the agent performing actions. The policy gradient is set according to the cumulative reward, and the parameters of the prediction policy network are iteratively updated based on the policy gradient using the gradient ascent method. The learning objective of the MAAC algorithm is to learn a policy for each agent to obtain the maximum cumulative reward.

[0039] Based on the parameters of the prediction policy network and the prediction value network, the parameters of the target policy network and the target value network are updated cyclically using a soft update method until the learning and training are completed.

[0040] D2D users download the parameters of the trained target policy network, improve the policy, and select the communication mode, resource blocks, and / or communication power based on the observed current environment.

[0041] In this embodiment, the Markov decision process is a model-free reinforcement learning model. Each D2D transmitter is regarded as an agent. In an unknown environment, the agent (i.e., D2D user) improves the performance of the entire system through self-decision. Based on the interaction between multiple agents and the environment, continuous trial and error is carried out to achieve adaptive learning. It is particularly suitable for solving the problem of large amount of channel state information and rapid scene changes leading to high computational complexity in the field of wireless communication.

[0042] This embodiment also employs the MAAC (Multi-Agent Actor-Critic) algorithm, which can be divided into centralized training and distributed execution. Centralized training involves transferring the complex multi-agent training process to the base station, which can easily deploy hardware such as GPUs to accelerate computation. Distributed execution treats each D2D transmitter as an agent, interacting with the environment and sampling, reducing the base station's signaling overhead and computational load. This embodiment uses the MAAC algorithm to find the optimal strategy for each D2D user, thereby finding a strategy to maximize the overall system's energy efficiency and addressing the instability issues present during training.

[0043] Exemplary, the parameters configured for the wireless network model include user location, and also include, but are not limited to, network coverage radius, base station location, channel gain, and / or the number of resource blocks.

[0044] As an example, the QoS requirement is set based on the user's minimum signal-to-noise ratio.

[0045] It is understandable that the experience pool stores historical experiences generated by the interaction between the agent and the environment. Using small-batch random sampling of data from the experience pool for training can reduce the correlation between samples and avoid wasting experience.

[0046] As an example, the experience pool is a finite-sized area, and when the experience pool is full, the oldest experience will be discarded.

[0047] In one specific implementation, the experience pool size was set to 4026, and the batch size for each sampling was 128.

[0048] In a preferred embodiment, the user signal-to-noise ratio calculation model includes the SINR of the m-th D2D user receiver and the SINR of the l-th cellular user;

[0049] The SINR of the m-th D2D user receiver is expressed as:

[0050]

[0051] In the formula, This indicates the transmit power of the D2D transmitter; This represents the channel gain between the D2D transmitter and the D2D receiver; This represents the cellular resource sharing coefficient, used to distinguish D2D communication modes. When the m-th D2D user uses an idle channel for communication, i.e., the cellular user spectrum resource block is not reused, there is no interference from cellular users. When the spectrum resource blocks of cellular users are reused, then Indicates the transmit power of cellular users; This represents the channel gain from cellular user to D2D. This represents the D2D resource sharing coefficient. If another D2D user (nth user) is sharing the same resource block as another D2D user (mth user), otherwise Indicates the transmit power of other D2D users; This represents the channel gain used by other D2D users for this D2D; σ 2 Indicates Gaussian white noise;

[0052] The SINR of the l-th cellular user is expressed as follows:

[0053]

[0054] In the formula, Indicates the transmit power of cellular users; This represents the channel gain from the macro base station to the cellular user; This represents the resource block reuse factor. This indicates that a D2D user is reusing a cellular user resource block; otherwise, it is... This represents the transmit power of the nth D2D; σ represents the channel gain from D2D user n to cellular user l; 2 Indicates Gaussian white noise;

[0055] The system throughput Tp is expressed as follows:

[0056]

[0057] In the formula, Represents the bandwidth between cellular users and macro base stations. Tp represents the bandwidth between the D2D transmitter and the D2D receiver. C Tp represents the throughput on the cellular user side. D This represents the throughput on the D2D user side;

[0058] The QoS requirements for communication between D2D user pairs and cellular users are set, and the wireless network model is optimized with the goal of maximizing the throughput of the communication system composed of D2D users and cellular users. This is described by the following expression:

[0059] maxTp(3a)

[0060]

[0061]

[0062]

[0063] p C=C (3e)

[0064] Equation (3a) represents the optimization objective of maximizing system throughput, Equations (3b) and (3c) represent the SINR requirements of the D2D receiver and cellular users, and Equations (3d) and (3e) represent the constraints on the transmit power of the D2D transmitter and cellular users. This indicates the minimum signal-to-noise ratio requirement for D2D. This indicates the minimum signal-to-noise ratio requirement for cellular users; Indicates the minimum transmission power for D2D; Indicates the maximum transmission power of D2D; p represents the transmit power of the nth D2D pair; C This represents the transmit power of cellular users; C is a constant, indicating that the transmit power of all cellular users in the environment is a fixed value.

[0065] In an optional embodiment, the D2D communication environment is modeled as a Markov decision process, the D2D transmitter is regarded as an agent, the parameters of the target policy network π′ are loaded cyclically to generate a policy and interact with the environment, the state space, action space and reward function are determined, and under the premise of meeting QoS requirements, each agent selects the communication mode to be adopted at time t, executes action a according to the currently observed state s, obtains reward r and transitions to the next state s', and uploads the experience group (s,a,s',r) to the experience pool for centralized training. Specifically:

[0066] The D2D communication environment is modeled as a Markov decision process, and the D2D transmitter is regarded as an intelligent agent.

[0067] After the agent cyclically loads the parameters of the target policy network π′, it generates a policy and interacts with the environment. At time t, it selects the communication mode to be used, executes action a based on the observed state s at time t, obtains reward r, and transitions to the next state s'. The actions executed by the agent are all performed under the constraints of QoS requirements.

[0068] Let the state space of the m-th D2D user at time t be defined as... in, This represents the basic information of the D2D user at time t, including the D2D user's location information. And user signal-to-noise ratio information Right now This indicates basic information about cellular users, including their location information. And user signal-to-noise ratio information Right now

[0069] Let the action space of the m-th D2D user at time t be defined as... in, This indicates that the D2D user selects the x-th resource block, which has M dimensions. This indicates that the z-th power level is selected for communication, and there are K choices in total.

[0070] The reward obtained by the m-th user for performing an action at time t is defined as:

[0071]

[0072] in, It is a constant less than 0; This represents the signal-to-noise ratio at time t for the m-th D2D user. Indicates D2D user bandwidth;

[0073] The environment s before the transformation, the action a, the environment s' after the transformation, and the reward r are uploaded to the experience pool in the form of experience groups (s, a, s', r).

[0074] As an example, when the m-th user receives a reward for the agent, the corresponding experience group upload operation is performed.

[0075] As an example, when the m-th user receives a non-negative reward for the agent, the upload operation for the corresponding experience group is performed; otherwise, the upload operation is not performed.

[0076] In a preferred embodiment, each agent selects the communication mode to be used at time t, including:

[0077] Determine if there is an idle channel in the system: if so, use dedicated mode for communication;

[0078] Otherwise, determine whether the QoS requirements of D2D users and cellular users are met after reusing the resource block: if yes, D2D users enter dedicated mode and share cellular user resources for communication; otherwise, enter waiting mode and do not communicate until an idle channel is available in the system to initiate a communication request again.

[0079] In a preferred embodiment, the cumulative reward expression is:

[0080]

[0081] In the formula, γ n This represents the discount factor, with a value in the range [0,1]. Indicates expected reward; represents the immediate reward; n represents the power of the discount on the reward over several future steps.

[0082] In an optional embodiment, see Figure 4The method employs the MAAC algorithm to optimize the policy for each D2D user, performs centralized training by randomly sampling small batches from the experience pool, updates the prediction value network using the TD algorithm, updates the parameters of the prediction value network using the gradient descent method, calculates the cumulative reward based on the reward obtained by the agent performing actions, sets the policy gradient based on the cumulative reward, and iteratively updates the parameters of the prediction policy network using the gradient ascent method based on the policy gradient, including:

[0083] In a multi-agent environment, the prediction policy network of all agents is π = {π1, π2, ..., π}. N} and the predictive value network Q = {Q1, Q2…Q} N The parameters of} are defined as follows: and The target policy network of all agents is π′=(π′1,π′2......π′). N And the target value network Q′=(Q′1,Q′2......Q′) N The parameters of ) are defined as follows: and

[0084] Determine whether the number of experience groups stored in the experience pool meets the preset threshold: if yes, perform centralized training; otherwise, do nothing.

[0085] The centralized training includes:

[0086] A small batch of random samples is taken from the experience pool to create the training dataset for the current round.

[0087] The prediction policy network of the i-th agent is in state s. t As input, an ε-greedy strategy is used to generate a selection action a. t Under strategy A, the agent performs action a. t The state transitions to s' and the reward r is received. t The ε-greed strategy expression is as follows:

[0088]

[0089] In the formula, A represents the agent's action strategy; the value of ε decreases continuously during the learning process.

[0090] The action value function is approximated using a predictive value network. The TD algorithm is used to update the predictive value network, and the Bellman equation is used to learn the Q function, i.e., the action value function. The prediction value network of the i-th agent is based on the agent's state s. t and action a t As input, output action value function The target value network takes the transitioned state s' and the next action a' as input, and outputs the action value function for the next moment.

[0091] Based on the outputs of the predicted value network and the target value network, a function approximation method is used to update the predicted value network by minimizing the loss function; wherein the expression of the loss function is as follows:

[0092]

[0093] In the formula, y i The target value is generated by the target value network, y i =r i +γQ i (s′,a'|θ Q ), where γ represents the discount factor, which takes values ​​in the range [0,1]. The smaller γ is, the less concerned about future returns. When γ equals 0, it means that only immediate returns are considered. As γ gets closer to 1, it means that future returns are given more and more importance. The predicted value is output by the prediction value network;

[0094] Define TD-error as The gradient descent method is used to update the parameters θ of the prediction value network. Q This reduces TD-error, thereby reducing prediction error;

[0095] Based on the cumulative reward of the i-th agent, the policy gradient is defined as follows:

[0096]

[0097] In the formula, This represents the gradient of the Q-function obtained in the predictive value network; denoted by , which represents the deterministic policy gradient of the predictive policy network; D represents the experience pool.

[0098] Based on the policy gradient, the gradient ascent method is used to update the parameters of the prediction policy network.

[0099] In this optional embodiment, an ε-greedy strategy is used to generate the selection action a. t Strategy A involves randomly selecting an action with a probability ε and choosing the action with a probability of 1-ε that maximizes the value function of the next action. By allowing the value of ε to continuously decay during the learning process, the agent ensures that in the initial stage of learning, it uses more exploratory strategies to fully explore the entire state space and find all possible states. As learning progresses, the learning strategy becomes more and more mature, and therefore it uses more greedy strategies to select the action with the highest value at the moment.

[0100] As an example, the next action a' is generated by the prediction policy network.

[0101] Furthermore, a neighbor user mechanism is introduced into the input of the predicted value network, specifically as follows:

[0102] Set the distance constraint value Z o ;

[0103] The distance Z from the i-th agent will be... i～j Less than the constraint value Z o The j-th agent is placed into the neighbor set O. i ={D2D j |Z i～j ≤Z o In the sequence {j∈N}, the i-th agent and the j-th agent are neighboring users; the distance between different agents is the distance between D2D transmitters, calculated using the Euclidean distance formula; for position Z... i =(X i ,Y i The i-th agent and its position are Z. j =(X j ,Y j The expression for the distance between the j-th agent and other agents is:

[0104]

[0105] The input to the prediction value network for the i-th agent includes the state and action of the i-th agent, and also includes set O. i Given the states and actions of the agents, output the action value function of the i-th agent.

[0106] Because signal attenuation is widespread when D2D user pairs are far apart, interference is mainly related to nearby D2D users sharing the same spectrum. Even when distant D2D users share the same spectrum resources, interference between them is negligible. By introducing a neighbor user mechanism, the base station determines whether information sharing is necessary based on the distance between D2D user pairs, thus avoiding conflicts between closely located D2D pairs due to selecting the same spectrum. The prediction value network will then determine whether to share information based on set O. i Add state and action information from other nearby agents to evaluate the quality of the operation. See also Figure 5 This describes the process of information sharing between neighboring agents. Compared to the method where the base station needs to obtain global D2D user information for coordination, this method of selecting only a subset of D2D user pairs to share information significantly reduces the base station's computational overhead and improves system performance.

[0107] In a preferred embodiment, the parameters of the prediction policy network and the prediction value network... and θ Q The update process introduced an eligibility tracking mechanism, specifically:

[0108] θ π ←θ π +α π δz π

[0109] θ Q ←θ Q +α Q δz Q

[0110] Where δ represents TD-error, This represents the action value function that predicts the output of the value network. The λ reward, representing the n-step time difference error, is expressed as:

[0111]

[0112] In the formula, T represents the final time; λ is the decay rate parameter, which takes values ​​in the interval [0,1]. When λ = 0, the return is G. t：t+1 This refers to single-step reward. In this case, the update algorithm for λ reward is the single-step time-difference error algorithm. When λ = 1, the λ reward is G. t The update algorithm for the λ reward is the Monte Carlo algorithm.

[0113] z π z represents the qualification trace of the predictor policy network. Q The qualification trace representing the predictive value network is updated as follows:

[0114]

[0115]

[0116] Where λ is the attenuation rate parameter, λ∈[0,1]; γ is the discount factor; This represents the gradient of the prediction policy network; The gradient of the predictive value network is represented; the qualification trace accumulates a gradient value at each step and decays with γλ, tracking the components of the weight vector that have made a positive or negative contribution to the most recent state evaluation.

[0117] The qualification trace is a vector with the same dimension as the weight vector, serving as short-term memory. In this preferred embodiment, the qualification trace is used to assist the learning process, influencing the weight vector, which in turn determines the estimated value. The introduction of the qualification trace mechanism makes the training process of the agent more efficient. Furthermore, compared to the traditional method that only uses the λ reward of a single-step temporal difference error, this preferred embodiment uses the λ reward of n-step temporal difference errors, which can significantly improve the accuracy of prediction.

[0118] In a preferred embodiment, the parameter soft update process for the target policy network and the target value network is as follows:

[0119]

[0120] Where, θ π′ τ represents the parameters of the target policy network; τ represents the parameter update coefficients, taking values ​​in the interval [0,1]; θ π θ represents the parameters of the prediction policy network. Q′ θ represents the parameters of the target value network. Q The parameters represent the target value network.

[0121] In a specific implementation, τ is preset to 0.01, which makes the parameter updates of the target network slow and improves the stability of learning.

[0122] Example 2

[0123] This embodiment presents a simulation experiment of the method proposed in Embodiment 1, considering the uplink of the cellular network in a single cell, and sets the simulation environment as follows:

[0124] Initialize a communication environment, setting the base station coverage radius to 500m, the base station located in the center of the cell, and the base station height to 25m. Within the base station's coverage area, 8 cellular users and 16 D2D users are randomly distributed, moving at a speed of 4-8km / h. Set the distance constraint value Z for D2D users to share information. o =50m, allocating one resource block to each cellular user, with a resource block bandwidth of 180KHz. D2D users communicate within this area by reusing the resource blocks of cellular users.

[0125] The initial transmit power for cellular users is set to 46dBm, the transmit power for D2D users is set to [0,30], the minimum signal-to-noise ratio requirement for cellular users is 1dB, the Gaussian white noise is -114dBm, the path loss is 128.1+37.6log(R(km)), the bandwidth is 4MHz, and the carrier frequency is 2GHz.

[0126] Initialize the prediction policy network π, the prediction value network Q, the target policy network π′, and the target value network Q′. The prediction policy network π and the target policy network π′ are both fully connected neural networks with two hidden layers, with 256 and 128 neurons respectively, and a learning rate of 0.0001. The prediction value network Q and the target value network Q′ are both fully connected neural networks with three hidden layers, with 256, 128, and 64 neurons respectively, and a learning rate of 0.001. Optimize these networks using the ADMA optimizer, setting the initial qualification trace of each network to z. -1 =0.

[0127] Set negative reward value Set the discount factor γ = 0.95 to -1.

[0128] The experience pool D is 4026 in size, and the batch size for each sampling is 128.

[0129] Construct a wireless network model, including 1 base station, 8 cellular users, and 8 D2D user pairs.

[0130] A user signal-to-noise ratio calculation model is constructed, and the wireless network model is optimized with the goal of maximizing the throughput of the communication system composed of D2D users and cellular users.

[0131] The D2D communication environment is modeled as a Markov decision process, and the D2D transmitter is regarded as an agent. After cyclically loading the parameters of the target policy network, the policy is generated and interacts with the environment to determine the state space, action space and reward function. Under the premise of meeting QoS requirements, each agent selects the communication mode to be adopted at time t, executes action a according to the currently observed state s, obtains reward r and transitions to the next state s', and uploads the experience group to the experience pool for centralized training.

[0132] The MAAC algorithm is used to optimize the policy for each D2D user, the TD algorithm is used to update the prediction value network, the gradient descent method is used to update the parameters of the prediction value network, the cumulative reward is calculated based on the reward obtained by the agent to perform actions, the policy gradient is set according to the cumulative reward, a qualification trace mechanism is introduced, and the parameters of the prediction policy network are updated cyclically based on the policy gradient using the gradient ascent method.

[0133] Based on the parameters of the prediction policy network and the prediction value network, the parameters of the target policy network and the target value network are updated cyclically using a soft update method.

[0134] See Figure 6By training an agent in the environment for 10,000 rounds, with 100 steps per round and network parameters updated every 50 steps, and recording the reward situation according to the training process, the agent continuously optimizes its own strategy in the direction of maximizing the reward, and finally obtains a resource allocation scheme that maximizes energy efficiency.

[0135] Example 3

[0136] This example provides a computer storage medium storing instructions that, when executed on a computer, cause the computer to perform the method described in Example 1.

[0137] The same or similar labels correspond to the same or similar parts;

[0138] The terms used to describe positional relationships in the accompanying drawings are for illustrative purposes only and should not be construed as limiting this patent.

[0139] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A D2D user resource allocation method based on a deep reinforcement learning algorithm, characterized in that, include: Construct a wireless network model and discretize the D2D transmit power to generate K power levels; The wireless network model includes a macro base station, L cellular users within the macro base station network coverage area, N pairs of D2D user pairs, and M orthogonal spectrum resource blocks. The parameters configured in the wireless network model include user locations. A user signal-to-noise ratio (SNR) calculation model is constructed to calculate the SNR information of D2D users and cellular users, and to set the QoS requirements for communication between D2D users and cellular users. The wireless network model is optimized with the goal of maximizing the throughput of the communication system composed of D2D users and cellular users. The user SNR includes the SNR of the D2D user receiver and the SNR of the cellular user. Macro base stations set up prediction policy networks for each agent. Predictive value network Q, target policy network and the target value network Q'; The D2D communication environment is modeled as a Markov decision process, and the D2D transmitter is regarded as an intelligent agent, with a target policy network being cyclically loaded. After obtaining the parameters, the agent generates a policy and interacts with the environment to determine the state space, action space, and reward function; under the premise of meeting QoS requirements, each agent... t The communication mode to be used is selected at any time, based on the currently observed state. Execute action Receive rewards And transition to the next state. experience group Uploaded to the experience pool for centralized training; wherein, the communication mode includes dedicated mode, multiplexing mode and waiting mode, the status includes location information and signal-to-noise ratio information of D2D users and cellular users, and the action includes selecting power value and resource block for communication; The MAAC algorithm is used to optimize the policy for each D2D user. Small-batch random sampling is performed from the experience pool for centralized training. The TD algorithm is used to update the prediction value network, and the gradient descent method is used to update the parameters of the prediction value network. The cumulative reward is calculated based on the rewards obtained by the agent performing actions. The policy gradient is set according to the cumulative reward, and the parameters of the prediction policy network are iteratively updated based on the policy gradient using the gradient ascent method. The learning objective of the MAAC algorithm is to learn a policy for each agent to obtain the maximum cumulative reward. Based on the parameters of the prediction policy network and the prediction value network, the parameters of the target policy network and the target value network are updated cyclically using a soft update method until the learning and training are completed. D2D users download the parameters of the trained target policy network, improve the policy, and select the communication mode, resource blocks, and / or communication power based on the observed current environment.

2. The D2D user resource allocation method based on deep reinforcement learning algorithm according to claim 1, characterized in that, The user signal-to-noise ratio calculation model includes the SINR of the m-th D2D user receiver and the SINR of the l-th cellular user; The SINR of the m-th D2D user receiver is expressed as: In the formula, This indicates the transmit power of the D2D transmitter; This represents the channel gain between the D2D transmitter and the D2D receiver; This represents the cellular resource sharing coefficient, used to distinguish D2D communication modes. When the m-th D2D user uses an idle channel for communication, i.e., the cellular user spectrum resource block is not reused, there is no interference from cellular users. =0, when the spectrum resource blocks of cellular users are reused, then =1; Indicates the transmit power of cellular users; This represents the channel gain from cellular user to D2D. This represents the D2D resource sharing coefficient. If another D2D user (nth user) is sharing the same resource block as another D2D user (mth user), =1, otherwise =0; Indicates the transmit power of other D2D users; This indicates the channel gain used by other D2D users for this D2D user; Indicates Gaussian white noise; No. l The SINR of a cellular user is expressed as follows: In the formula, Indicates the transmit power of cellular users; This represents the channel gain from the macro base station to the cellular user; Represents the resource block reuse factor, if =1 indicates that a D2D user is reusing a cellular user resource block; otherwise, it is... =0; This represents the transmit power of the nth D2D; This represents D2D user n to cellular user n. l Channel gain; Indicates Gaussian white noise; The system throughput Tp The expression is: In the formula, Represents the bandwidth between cellular users and macro base stations. This represents the bandwidth between the D2D transmitter and the D2D receiver; This represents the throughput on the cellular user side; This represents the throughput on the D2D user side; The QoS requirements for communication between D2D user pairs and cellular users are set, and the wireless network model is optimized with the goal of maximizing the throughput of the communication system composed of D2D users and cellular users. This is described by the following expression: p (3a) (3b) (3c) (3d) (3e) Equation (3a) represents the optimization objective of maximizing system throughput, Equations (3b) and (3c) represent the SINR requirements of the D2D receiver and cellular users, and Equations (3d) and (3e) represent the constraints on the transmit power of the D2D transmitter and cellular users. This indicates the minimum signal-to-noise ratio requirement for D2D. This indicates the minimum signal-to-noise ratio requirement for cellular users; Indicates the minimum transmission power for D2D; Indicates the maximum D2D transmission power; This represents the transmit power of the nth D2D pair; Indicates the transmit power of cellular users; The constant represents the fixed transmit power of all cellular users in the environment.

3. The D2D user resource allocation method based on deep reinforcement learning algorithm according to claim 2, characterized in that, The D2D communication environment is modeled as a Markov decision process, and the D2D transmitter is regarded as an intelligent agent, with a target policy network being cyclically loaded. After obtaining the parameters, the agent generates a policy and interacts with the environment to determine the state space, action space, and reward function. Under the premise of meeting QoS requirements, each agent selects the communication mode to use at time t based on the currently observed state. Execute action Receive rewards And transition to the next state. experience group Uploaded to the experience pool for centralized training, specifically: The D2D communication environment is modeled as a Markov decision process, and the D2D transmitter is regarded as an intelligent agent. Agents cyclically load target policy networks After obtaining the parameters, a strategy is generated and interacts with the environment. At time t, the desired communication mode is selected based on the observed state at time t. Execute action Receive rewards And transition to the next state. The actions performed by the intelligent agent are all constrained by QoS requirements. Let the state space of the m-th D2D user at time t be defined as... };in, This represents the basic information of the D2D user at time t, including the D2D user's location information. And user signal-to-noise ratio information ,Right now ; This indicates basic information about cellular users, including their location information. And user signal-to-noise ratio information ,Right now ; Let the action space of the m-th D2D user at time t be defined as... ;in, This indicates that the D2D user selected the first... There are M resource blocks, with a total of M dimensions; Indicates the selection of the first There are K power levels for communication, with a total of K options; The first The reward a user receives for performing an action at time t is defined as: in, It is a constant less than 0; = , Indicates the first Signal-to-noise ratio of a D2D user at time t, Indicates D2D user bandwidth; The environment before the conversion Actions performed The transformed environment and rewards With experience group Upload it to the experience pool in the form of [format].

4. The D2D user resource allocation method based on deep reinforcement learning algorithm according to claim 1, characterized in that, Each agent selects the communication mode to use at time t, including: Determine if there is an idle channel in the system: if so, use dedicated mode for communication; Otherwise, determine whether the QoS requirements of D2D users and cellular users are met after reusing the resource block: if yes, D2D users enter dedicated mode and share cellular user resources for communication; otherwise, enter waiting mode and do not communicate until an idle channel is available in the system to initiate a communication request again.

5. The D2D user resource allocation method based on deep reinforcement learning algorithm according to claim 1, characterized in that, The cumulative reward expression is: In the formula, This represents the discount factor, with a value in the range [0,1]. Indicates expected reward; This indicates an immediate reward.

6. The D2D user resource allocation method based on deep reinforcement learning algorithm according to claim 5, characterized in that, The process involves using the MAAC algorithm to optimize the policy for each D2D user, performing centralized training by randomly sampling small batches from the experience pool, updating the prediction value network using the TD algorithm, updating the parameters of the prediction value network using the gradient descent method, calculating the cumulative reward based on the rewards obtained by the agent performing actions, setting the policy gradient based on the cumulative reward, and iteratively updating the parameters of the prediction policy network using the gradient ascent method based on the policy gradient. This includes: In a multi-agent environment, a prediction policy network is constructed for all agents. and predictive value networks The parameters are defined as follows: and The target policy network of all agents and target value network The parameters are defined as follows: and ; Determine whether the number of experience groups stored in the experience pool meets the preset threshold: if yes, perform centralized training; otherwise, do nothing. The centralized training includes: A small batch of random samples is taken from the experience pool to create the training dataset for the current round. No. The prediction policy network of individual agents is based on the state. For input, use Strategy generates action selection Strategy A, the agent performs the action State transition to And receive a reward ;in, The policy expression is: In the formula, A represents the action strategy of the agent; The value of decreases continuously during the learning process; The action value function is approximated using a predictive value network. The TD algorithm is used to update the predictive value network, and the Bellman equation is used to learn the Q function, i.e., the action value function. , No. The predictive value network of an agent is based on the agent's state. and actions As input, output action value function The target value network is in the transformed state. And the next moment's action Given the input, output the action value function for the next moment. ; Based on the outputs of the predicted value network and the target value network, a function approximation method is used to update the predicted value network by minimizing the loss function; wherein the expression of the loss function is as follows: In the formula, The target value is generated by the target value network. = , This represents the discount factor, with values ​​in the range [0,1]. The smaller the value, the less concerned about future returns. A value of 0 indicates that only immediate gains are considered, and as... The value increasingly approaching 1 indicates a growing emphasis on future returns; The predicted value is output by the prediction value network. definition for The gradient descent method is used to update the parameters of the prediction value network. , making This reduces the error in prediction. According to the The cumulative reward of each agent is defined by the policy gradient, which is expressed as follows: In the formula, This represents the gradient of the Q-function obtained in the predictive value network; denoted by , which represents the deterministic policy gradient of the predictive policy network; D represents the experience pool. Based on the policy gradient, the gradient ascent method is used to update the parameters of the prediction policy network. .

7. The D2D user resource allocation method based on deep reinforcement learning algorithm according to claim 6, characterized in that, The prediction value network incorporates a neighbor user mechanism into its input, specifically: Set distance constraint values ; The distance to the i-th agent Less than the constraint value The j Each agent is placed into a neighbor set. ={ | In}, the first i The first agent and the second j Each agent is a neighboring user; the distance between different agents is the distance between D2D transmitters, calculated using the Euclidean distance formula; for positions... The i-th agent and its position are The expression for the distance between the j-th agents is: ； No. i The input to the predictive value network of each agent includes the first... i The state and actions of each agent, including sets Given the states and actions of the agents, output the action value function of the i-th agent. .

8. The D2D user resource allocation method based on deep reinforcement learning algorithm according to claim 1, characterized in that, The parameters of the prediction policy network and the prediction value network and The update process introduced an eligibility tracking mechanism, specifically: in, Indicates TD-error, ; This represents the action value function that predicts the output of the value network; Represents the n-step timing difference error The return, expressed as: In the formula, T represents the final moment; The attenuation rate parameter takes values ​​within the interval [0,1]. hour, The return is That is, single-step return, at this time The update algorithm for the reward is the single-step time-series difference error algorithm, when hour, The return is ,Right now The algorithm for updating the reward is the Monte Carlo algorithm; The qualification trace of the prediction policy network is represented. The qualification trace representing the predictive value network is updated as follows: in, For the attenuation rate parameter, ; This is the discount factor; This represents the gradient of the prediction policy network; The gradient of the predictive value network is represented; the qualification trace accumulates a gradient value at each step and is expressed as follows: Decay tracks the components of the weight vector that have made a positive or negative contribution to the most recent state evaluation.

9. A D2D user resource allocation method based on a deep reinforcement learning algorithm according to claim 1, characterized in that, The parameter soft update process for the target policy network and the target value network is as follows: in, These represent the parameters of the target policy network; This represents the parameter update coefficient, with values ​​ranging from [0,1]. These represent the parameters of the prediction policy network; Parameters representing the target value network; The parameters represent the target value network.

10. A computer storage medium, characterized in that, The computer storage medium stores instructions that, when executed on the computer, cause the computer to perform the method as described in any one of claims 1-9.

Images