D2D communication resource allocation method based on joint optimization framework under dense user conditions

Abstract

The present invention belongs to the field of communications technology, and in particular, is a method for allocating D2D communication resources in dense user environments based on a joint optimization framework. The method comprises the following steps: S1: D2D communication resource allocation data collection and establishment of an MDP decision-making process for D2D communication resource allocation that combines D3QN and DDPG; S2: designing a progressive reward mechanism; and S3: constructing an action selection mechanism for combined D3QN and DDPG optimization. By employing a D3QN and DDPG-based D2D communication resource allocation and transmit power control method, the present invention enables each D2D user transmitter to adaptively control power and independently select resources based on D2D latency requirements. Furthermore, the use of a progressive reward mechanism allows the algorithm to gradually approach the optimal solution during the optimization process, addressing initial exploration difficulties and slow strategy convergence, and promptly balancing high user latency with low throughput. The present invention not only ensures normal communication for cellular users but also maximizes system throughput while achieving a balance between latency and throughput.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a D2D communication resource allocation method based on a joint optimization framework under dense user conditions. Background Art

[0002] The rapid development of mobile communication network technology has brought higher speeds, lower latency, larger connections, and greater spectrum efficiency. This has also heightened user expectations and triggered a growing demand for wireless networks. The interplay of these two factors has given rise to a series of innovative communication models.

[0003] Device-to-device (D2D) communication, as an emerging communication method, has been widely researched and applied. It is widely used in fields such as the Internet of Things, social networks, and emergency communications. Although D2D communication brings many potential benefits, in high-density D2D communication environments, communication quality and transmission time can be severely affected due to mutual interference between devices. This also greatly increases the difficulty of managing network interference within the system. Therefore, further research is still needed to improve the performance of D2D and cellular communications in DRL-based resource allocation and power control, especially in situations where latency requirements are very low. Summary of the Invention

[0004] (1) Technical problems solved

[0005] In view of the shortcomings of the existing technology, the present invention provides a D2D communication resource allocation method based on a joint optimization framework under dense user conditions, which solves the problems raised in the above background technology.

[0006] (2) Technical solution

[0007] In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions:

[0008] A D2D communication resource allocation method based on a joint optimization framework under dense user conditions includes the following steps:

[0009] S1: D2D communication resource allocation data collection part and establishment of the MDP decision process for joint D3QN and DDPG D2D communication resource allocation;

[0010] S2: Design a progressive reward mechanism;

[0011] S3: Build an action selection mechanism that combines D3QN and DDPG optimization;

[0012] S4: Design a deep reinforcement learning experience pool that includes positive and negative sample classification and priority experience replay to conduct data training for D2D communication resource allocation;

[0013] S5: Based on the designed D2D communication system model in a single-cell dense user scenario, the agent interacts with the communication environment using a joint D3QN and DDPG optimization method. 10,000 iterations of learning are performed to output the optimal result. This is followed by the data interaction part of D2D communication resource allocation and transmit power control.

[0014] S6: Output the optimal strategy after iterative learning, namely, user channel selection and transmit power allocation in D2D communication in a single-cell dense user scenario.

[0015] Furthermore, in the D2D communication resource allocation data collection part in S1, all D2D pairs in the environment are regarded as one intelligent agent. Each intelligent agent completes resource allocation once according to the current strategy. When all intelligent agents complete resource allocation or reach the set interaction limit, the first data collection is completed. Next, the cellular D2D communication network environment is initialized and interaction with the environment is continued until the intelligent agent strategy converges or the predetermined number of training times is reached, and then data collection is stopped.

[0016] In the above S1, in the MDP decision process for establishing joint D3QN and DDPG D2D communication resource allocation, it is necessary to establish an MDP model for the D2D communication network scenario in a cellular network with dense user scenarios in a single cell, specifically including the following contents:

[0017] State space: CUE's instantaneous channel information Ht, DUE's instantaneous channel information Gt, neighbor user's subchannel selection Nt-1 in the previous time slot, interference situation It-1 of each subchannel in the previous time slot, transmission time Tt that meets the delay constraint, and the remaining load Lt sent by the DUE;

[0018] Action space definition:

[0019] in represents the resource block allocation, the number of resource blocks is equal to the number of cellular users in the environment, represents all possible emission powers that the agent can take;

[0020] Reward function: system throughput, delay constraints, and interference of cellular users.

[0021] Furthermore, the progressive reward function in S2 is set as follows:

[0022]

[0023] where R c [m] and R d [n] represents the throughput of cellular user m and D2D user n, respectively. cellis the total amount of interference of cellular users in the current state, λ is the adjustment parameter, T t is the transmission time, and β is the weight parameter used to balance the relationship between system throughput and delay.

[0024] Furthermore, S3 needs to initialize the experience pool in the joint optimization network, set up a sample buffer area to store the current state, current action, current reward and next state obtained by the interaction between the agent and the environment; initialize the joint action selection optimization model, including the network parameters of D3QN and DDPG; and specifically include the following:

[0025] Based on the dual action selection mechanism setting in the joint optimization network:

[0026] Improved greedy strategy definition:

[0027]

[0028] Where T now represents the current round number, ε final represents the exploration rate after stabilization, T final Indicates the number of rounds when the exploration rate is stable;

[0029] Action definition in the dual action selection mechanism:

[0030]

[0031] in represents the optimal subchannel selection, represents the optimal transmit power under the current delay constraint;

[0032] D3QN is used to select the optimal user sub-channel allocation scheme Using DDPG to select the optimal transmit power

[0033] Furthermore, the deep reinforcement learning experience pool in S4 includes positive and negative sample classification and priority experience replay. The specific content of the positive and negative sample classification is: during the interaction between the intelligent agent and the environment, the successful D2D reuse is used as a positive sample, and the failed reuse is used as a negative sample. Then, a certain proportion of non-uniform sampling is used to form a "minimum" sample, and finally, the repeated "minimum" samples in the experience pool are removed.

[0034] The specific content of prioritized experience replay is as follows: For the experience replay part of DRL training, random sampling is used to improve traditional experience replay and introduce priority, which is called random prioritized experience replay. It gives higher priority experiences a higher sampling probability. Therefore, the time difference error is used to measure the priority of the sample, which is expressed as follows:

[0035]

[0036] Therefore, the number of samples for DRL training is defined as I, and the empirical priority of the i-th sample is:

[0037]

[0038] Rank(i) refers to the position of the i-th sample in the experience pool. During sampling, in order to avoid directly selecting the highest-ranked sample, a random factor η is introduced, that is, a truncated Gaussian distribution is used to generate random numbers. This can ensure that low-priority experiences also have a certain learning opportunity while maintaining a higher sampling probability for high-priority experiences, thereby increasing the diversity of training, avoiding overfitting, and improving training effects.

[0039] Furthermore, in the data interaction part of S5, in each unit time "t", the agent obtains the state from the simulation environment and transmits it to the joint optimization network. The joint optimization network calculates the Q value function and outputs the corresponding action selection. This action is the optimal action obtained after processing by the dual network model, namely the optimal subchannel selection and optimal power selection of D2D.

[0040] The specific process in S5 is as follows:

[0041] S5.1: Initialize D2D communication method parameters and network parameters, including the following steps:

[0042] S5.1.1: Initialize the sample capacity of the experience pool and store the state (user channel gain), action (subchannel allocation and transmit power control), reward (system throughput, interference, and delay constraints), and next-time state (user channel gain) obtained from the interaction of intelligent agents in a single-cell D2D communication environment into the experience pool.

[0043] S5.1.2: Initialize the joint action selection optimization model of the intelligent agents, including the network model and parameters of the subchannel selection unit (D3QN); the policy network and Q network model and parameters of the transmit power control unit (DDPG);

[0044] S5.1.3: Initialize the system model for a single-cell dense user scenario, assuming the base station is located at the center of the cell and cellular users and D2D users are randomly distributed in the cell.

[0045] S5.2: Run the combined D3QN and DDPG method to iteratively learn and output a D2D communication resource allocation strategy for a single cell; including the following steps:

[0046] S5.2.1: Use the state (s), action (a), and reward (r) obtained by the agent's interaction with the environment to construct an MDP decision process, calculate the link gain when users are reused within the system, and the user's signal-to-interference-and-noise ratio. Apply Shannon's theorem and function constraints to output the system throughput.

[0047] S5.2.2: Input the current action selection, reward value, and updated state into the experience pool as a sample;

[0048] S5.2.3: Input the small batch samples in the experience pool into the DDPG network and the D3QN network respectively, calculate and update the policy network parameters and target network parameters of DDPG, calculate and update the state value function and action value function parameters of D3QN, and output the user's subchannel selection and transmit power.

[0049] (3) Beneficial effects

[0050] Compared with the prior art, the present invention provides a D2D communication resource allocation method based on a joint optimization framework under dense user conditions, which has the following beneficial effects:

[0051] This invention utilizes D3QN and DDPG to jointly implement D2D communication resource allocation and transmit power control methods. Each D2D user transmitter can adaptively control power and independently select resources based on D2D latency requirements. A progressive reward mechanism is employed to gradually approach the optimal solution during the optimization process, addressing initial exploration difficulties and slow strategy convergence, and promptly balancing high user latency with low throughput. This invention not only ensures normal communication for cellular users but also maximizes system throughput while achieving a balance between latency and throughput. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 Flowchart for the implementation of the present invention;

[0053] Figure 2 It is a system model diagram of the present invention;

[0054] Figure 3 This is a network structure diagram of the joint optimization method of the present invention. DETAILED DESCRIPTION

[0055] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0056] Example

[0057] like Figure 1-3 As shown, an embodiment of the present invention proposes a D2D communication resource allocation method based on a joint optimization framework under dense user conditions, comprising the following steps:

[0058] S1: D2D communication resource allocation data collection part and establishment of the MDP decision process for joint D3QN and DDPG D2D communication resource allocation;

[0059] S2: Design a progressive reward mechanism;

[0060] S3: Build an action selection mechanism that combines D3QN and DDPG optimization;

[0061] S4: Design a deep reinforcement learning experience pool that includes positive and negative sample classification and priority experience replay to conduct data training for D2D communication resource allocation;

[0062] S5: Based on the designed D2D communication system model in a single-cell dense user scenario, the agent interacts with the communication environment using a joint D3QN and DDPG optimization method. 10,000 iterations of learning are performed to output the optimal result. This is followed by the data interaction part of D2D communication resource allocation and transmit power control.

[0063] S6: Output the optimal strategy after iterative learning, namely, user channel selection and transmit power allocation in D2D communication in a single-cell dense user scenario.

[0064] In some embodiments, during the D2D communication resource allocation data collection portion in S1, all D2D pairs in the environment are considered as one agent. Each agent completes a resource allocation according to the current strategy. When all agents complete resource allocation or reach the set interaction limit, the first data collection is completed. Next, the cellular D2D communication network environment is initialized and interactions with the environment are repeated until the agent's strategy converges or a predetermined number of training times is reached, at which point data collection is stopped.

[0065] In S1, during the MDP decision-making process for joint D3QN and DDPG D2D communication resource allocation, an MDP model for the D2D communication network scenario in a cellular network with dense user density in a single cell must be established. Specifically, the following contents must be included:

[0066] State space: CUE's instantaneous channel information Ht, DUE's instantaneous channel information Gt, neighbor user's subchannel selection Nt-1 in the previous time slot, interference situation It-1 of each subchannel in the previous time slot, transmission time Tt that meets the delay constraint, and the remaining load Lt sent by the DUE;

[0067] Action space definition:

[0068] in represents the resource block allocation, the number of resource blocks is equal to the number of cellular users in the environment, represents all possible emission powers that the agent can take;

[0069] Reward function: system throughput, delay constraints, and interference of cellular users.

[0070] In some embodiments, the progressive reward function in S2 is set as follows:

[0071]

[0072] where R c [m] and R d [n] represents the throughput of cellular user m and D2D user n, respectively. cell is the total amount of interference of cellular users in the current state, λ is the adjustment parameter, T t is the transmission time, and β is the weight parameter used to balance the relationship between system throughput and delay.

[0073] In some embodiments, S3 needs to initialize the experience pool in the joint optimization network, set a sample buffer area to store the current state, current action, current reward, and next state obtained by the interaction between the agent and the environment; initialize the joint action selection optimization model, including the network parameters of D3QN and DDPG; and specifically include the following:

[0074] Based on the dual action selection mechanism setting in the joint optimization network:

[0075] Improved greedy strategy definition:

[0076]

[0077] Where T now represents the current round number, ε final represents the exploration rate after stabilization, T final Indicates the number of rounds when the exploration rate is stable;

[0078] Action definition in the dual action selection mechanism:

[0079]

[0080] in represents the optimal subchannel selection, represents the optimal transmit power under the current delay constraint;

[0081] D3QN is used to select the optimal user sub-channel allocation scheme Using DDPG to select the optimal transmit power

[0082] In some embodiments, the deep reinforcement learning experience pool in S4 includes positive and negative sample classification and priority experience replay, wherein the specific content of the positive and negative sample classification is: during the interaction between the intelligent agent and the environment, the successful D2D reuse is used as a positive sample, and the failed reuse is used as a negative sample, and then a certain proportion of non-uniform sampling is used to form a "minimum" sample, and finally the repeated "minimum" samples in the experience pool are removed;

[0083] The specific content of prioritized experience replay is as follows: For the experience replay part of DRL training, random sampling is used to improve traditional experience replay and introduce priority, which is called random prioritized experience replay. It gives higher priority experiences a higher sampling probability. Therefore, the time difference error is used to measure the priority of the sample, which is expressed as follows:

[0084]

[0085] Therefore, the number of samples for DRL training is defined as I, and the empirical priority of the i-th sample is:

[0086]

[0087] Rank(i) refers to the position of the i-th sample in the experience pool. During sampling, in order to avoid directly selecting the highest-ranked sample, a random factor η is introduced, that is, a truncated Gaussian distribution is used to generate random numbers. This can ensure that low-priority experiences also have a certain learning opportunity while maintaining a higher sampling probability for high-priority experiences, thereby increasing the diversity of training, avoiding overfitting, and improving training effects.

[0088] In some embodiments, in the data interaction part of S5, in each unit time "t", the agent obtains the state from the simulation environment and transmits it to the joint optimization network, which calculates the Q-value function and outputs the corresponding action selection. The action is the optimal action obtained after processing by the dual network model, that is, the optimal subchannel selection and optimal power selection of D2D;

[0089] The specific process in S5 is as follows:

[0090] S5.1: Initialize D2D communication method parameters and network parameters, including the following steps:

[0091] S5.1.1: Initialize the sample capacity of the experience pool and store the state (user channel gain), action (subchannel allocation and transmit power control), reward (system throughput, interference, and delay constraints), and next-time state (user channel gain) obtained from the interaction of intelligent agents in a single-cell D2D communication environment into the experience pool.

[0092] S5.1.2: Initialize the joint action selection optimization model of the intelligent agents, including the network model and parameters of the subchannel selection unit (D3QN); the policy network and Q network model and parameters of the transmit power control unit (DDPG);

[0093] S5.1.3: Initialize the system model for a single-cell dense user scenario, assuming the base station is located at the center of the cell and cellular users and D2D users are randomly distributed in the cell.

[0094] S5.2: Run the combined D3QN and DDPG method to iteratively learn and output a D2D communication resource allocation strategy for a single cell; including the following steps:

[0095] S5.2.1: Use the state (s), action (a), and reward (r) obtained by the agent's interaction with the environment to construct an MDP decision process, calculate the link gain when users are reused within the system, and the user's signal-to-interference-and-noise ratio. Apply Shannon's theorem and function constraints to output the system throughput.

[0096] S5.2.2: Input the current action selection, reward value, and updated state into the experience pool as a sample;

[0097] S5.2.3: Input the small batch samples in the experience pool into the DDPG network and the D3QN network respectively, calculate and update the policy network parameters and target network parameters of DDPG, calculate and update the state value function and action value function parameters of D3QN, and output the user's subchannel selection and transmit power.

[0098] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art will be able to modify the technical solutions described in the aforementioned embodiments or substitute equivalents for some of the technical features. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A D2D communication resource allocation method based on a joint optimization framework under dense user conditions, characterized by: The following steps are involved: S1: D2D communication resource allocation data collection part and the MDP decision process of D2D communication resource allocation based on the joint dueling dual deep Q network D3QN and deep deterministic policy gradient DDPG; S2: Design a progressive reward mechanism; S3: Build an action selection mechanism that combines D3QN and DDPG optimization; S4: Design a deep reinforcement learning experience pool that includes positive and negative sample classification and priority experience replay to conduct data training for D2D communication resource allocation; S5: Based on the designed D2D communication system model in a single-cell dense user scenario, the agent interacts with the communication environment using a joint D3QN and DDPG optimization method. 10,000 iterations of learning are performed to output the optimal result. This is followed by the data interaction part of D2D communication resource allocation and transmit power control. S6: Output the optimal strategy after iterative learning, namely user channel selection and transmit power allocation in D2D communication in a single-cell dense user scenario; In S3, the experience pool in the joint optimization network needs to be initialized, and a sample buffer area needs to be set to store the current state, current action, current reward, and next-moment state obtained by the interaction between the agent and the environment; the joint action selection optimization model needs to be initialized, including the network parameters of D3QN and DDPG; and specifically includes the following: Based on the dual action selection mechanism setting in the joint optimization network: Improved greedy strategy definition: Where T now represents the current round number, ε final represents the exploration rate after stabilization, T final Indicates the number of rounds when the exploration rate is stable; Action definition in the dual action selection mechanism: in represents the optimal subchannel selection, represents the optimal transmit power under the current delay constraint; D3QN is used to select the optimal user sub-channel allocation scheme Using DDPG to select the optimal transmit power 2. The D2D communication resource allocation method based on a joint optimization framework under dense user conditions according to claim 1 is characterized by: In the D2D communication resource allocation data collection part in S1, all D2D pairs in the environment are regarded as one intelligent agent. Each intelligent agent completes a resource allocation according to the current strategy. When all intelligent agents complete resource allocation or reach the set interaction limit, the first data collection is completed. Next, the cellular D2D communication network environment is initialized and interacts with the environment again until the intelligent agent's strategy converges or the predetermined number of training times is reached, and then data collection stops. In the above S1, in the MDP decision process for establishing joint D3QN and DDPG D2D communication resource allocation, it is necessary to establish an MDP model for the D2D communication network scenario in a cellular network with dense user scenarios in a single cell, specifically including the following contents: State space: CUE's instantaneous channel information Ht, DUE's instantaneous channel information Gt, neighbor user's subchannel selection Nt-1 in the previous time slot, interference situation It-1 of each subchannel in the previous time slot, transmission time Tt that meets the delay constraint, and the remaining load Lt sent by the DUE; Action space definition: in represents the resource block allocation, the number of resource blocks is equal to the number of cellular users in the environment, represents all possible transmission powers taken by the agent; Reward function: system throughput, delay constraints, and interference of cellular users.

3. The D2D communication resource allocation method based on a joint optimization framework under dense user conditions according to claim 2, characterized in that: The progressive reward function in S2 is set as follows: where R c [m] and R d [n] represents the throughput of cellular user m and D2D user n, respectively. cell is the total amount of interference of cellular users in the current state, λ is the adjustment parameter, T t is the transmission time, and β is the weight parameter used to balance the relationship between system throughput and delay.

4. The D2D communication resource allocation method based on a joint optimization framework under dense user conditions according to claim 3 is characterized by: The deep reinforcement learning experience pool in S4 includes positive and negative sample classification and priority experience replay. The specific content of the positive and negative sample classification is: during the interaction between the intelligent agent and the environment, the successful D2D reuse is used as a positive sample, and the failed reuse is used as a negative sample. Then, a certain proportion of non-uniform sampling is used to form a minimum sample, and finally, the repeated minimum samples in the experience pool are removed. The specific content of prioritized experience replay is as follows: For the experience replay part of DRL training, random sampling is used to improve traditional experience replay and introduce priority, which is called random prioritized experience replay. It gives higher priority experiences a higher sampling probability. Therefore, the time difference error is used to measure the priority of the sample, which is expressed as follows: Therefore, the number of samples for DRL training is defined as I, and the empirical priority of the i-th sample is: Rank(i) refers to the position of the i-th sample in the experience pool. When sampling, a random factor η is introduced, that is, a truncated Gaussian distribution is used to generate random numbers.

5. The D2D communication resource allocation method based on a joint optimization framework under dense user conditions according to claim 4 is characterized by: In the data interaction part of S5, in each unit time t, the agent obtains the state from the simulation environment and transmits it to the joint optimization network. The joint optimization network calculates the Q value function and outputs the corresponding action selection. This action is the optimal action obtained after processing by the dual network model, namely the optimal subchannel selection and optimal power selection of D2D. The specific process in S5 is as follows: S5.1: Initialize D2D communication method parameters and network parameters, including the following steps: S5.1.1: Initialize the sample capacity of the experience pool and store the state, action, reward, and next-moment state obtained from the interaction of intelligent agents in a single-cell D2D communication environment into the experience pool. The state is the user's channel gain, the action is subchannel allocation and transmit power control, the reward is the system throughput, interference, and delay constraints, and the next-moment state is the user's channel gain. S5.1.2: Initialize the joint action selection optimization model of the intelligent agents, including the network model and parameters of the subchannel selection unit (D3QN); the policy network and Q network model and parameters of the transmit power control unit (DDPG); S5.1.3: Initialize the system model for a single-cell dense user scenario, assuming the base station is located at the center of the cell and cellular users and D2D users are randomly distributed in the cell. S5.2: Run the combined D3QN and DDPG method to iteratively learn and output a D2D communication resource allocation strategy for a single cell; including the following steps: S5.2.1: Use the state (s), action (a), and reward (r) obtained by the agent's interaction with the environment to construct an MDP decision process, calculate the link gain when users are reused within the system, and the user's signal-to-interference-and-noise ratio. Apply Shannon's theorem and function constraints to output the system throughput. S5.2.2: Input the current action selection, reward value, and updated state into the experience pool as a sample; S5.2.3: Input the small batch samples in the experience pool into the DDPG network and the D3QN network respectively, calculate and update the policy network parameters and target network parameters of DDPG, calculate and update the state value function and action value function parameters of D3QN, and output the user's subchannel selection and transmit power.

Images