A resource allocation method for IRS-assisted D2D system based on multi-agent DQN

Abstract

The present invention discloses a scenario of multiplexing uplink spectrum resources in intelligent reflecting surface (IRS)-assisted device-to-device (D2D) communication, and proposes a resource allocation method for IRS-assisted D2D system based on multi-agent DQN to improve the maximum transmission rate of cellular users. The present invention jointly optimizes the D2D user transmission power and IRS phase shift by satisfying the constraints of the minimum transmission rate of D2D users and the minimum energy collection of energy receivers, and constructs a model that maximizes the information transmission rate of cellular users. A multi-agent DQN resource allocation algorithm is proposed, in which each DQN network is only responsible for the learning of one agent and outputs the D2D power and IRS phase shift. Simulation results show that compared with other methods, the proposed resource allocation algorithm can learn from the environment and continuously improve behavior, significantly improve the maximum transmission rate of cellular users at a lower complexity, and have a good convergence effect.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a scenario in which uplink spectrum resources are reused in an intelligent reflecting surface (IRS)-assisted device-to-device (D2D) communication. A multi-agent DQN resource allocation method based on deep reinforcement learning is proposed. Background Art

[0002] Device-to-device (D2D) communication is considered one of the promising technologies in the fifth generation of wireless communications (5G). D2D communication technology enables D2D users to reuse the spectrum resources of cellular users, thereby achieving information transmission between D2D users and improving the spectrum utilization of the system. However, D2D communication also introduces new interference sources to the system, increasing interference to cellular users, seriously affecting communication quality and even preventing normal communication.

[0003] Intelligent Reflecting Surface (IRS) is considered as one of the potential technologies for the sixth generation of wireless communication (6G), which can achieve efficient use of energy and spectrum. IRS is an intelligent plane composed of a large number of low-cost passive reflective elements, each of which can adjust the amplitude and phase of the incident signal and reflect it. Therefore, by deploying IRS in wireless communication networks, a controllable communication link can be reconstructed between the transmitter and the receiver to enhance the desired signal and suppress interference.

[0004] At present, some scholars have conducted relevant research on IRS-assisted D2D communication. Among them, the research teams of Harbin Institute of Technology and Shenzhen University have proposed an optimization problem for IRS-assisted D2D communication system: maximizing the D2D user transmission rate under the premise of meeting the minimum transmission rate of cellular users. Then the block coordinate descent method is used to decompose the original problem into four sub-problems, and finally the beamforming, power allocation and phase shift matrices are solved, verifying the effectiveness of IRS in improving the D2D user transmission rate. The literature "Mao S, Chu X, Wu Q, et al. Intelligent reflecting surface enhanced D2D cooperative computing [J]. IEEE Wireless Communications Letters, 2021, 10 (7): 1419-1423" studied a D2D cooperative communication system in which users can use IRS to offload computing tasks to nearby idle users. The computing delay is minimized by jointly optimizing the computing task allocation, transmission power, frequency allocation and IRS phase shift, and an alternating optimization algorithm is proposed to solve this problem. Compared with the traditional D2D collaborative computing strategy, the IRS-assisted D2D collaborative computing solution can achieve lower computing latency.

[0005] As an important branch in the field of machine learning, reinforcement learning has achieved remarkable results in certain scenarios, such as Go, games, and unmanned driving. The basic idea of ​​reinforcement learning is to let the agent interact with the surrounding environment, take corresponding actions according to the current state and obtain rewards, and obtain a control strategy or rule through continuous learning to solve specific problems. Most of the existing research on IRS-assisted D2D is based on the alternating optimization method, which uses complex mathematical formulas and is difficult to truly solve the problem of multi-variable real-time joint optimization. In response to the above difficulties, the present invention proposes a multi-agent DQN algorithm based on deep reinforcement learning, which jointly optimizes IRS phase shift and DT transmission power, has lower complexity and better convergence effect. Summary of the invention

[0006] The present invention discloses a scenario of multiplexing uplink spectrum resources in device-to-device (D2D) communication assisted by an intelligent reflecting surface (IRS), and proposes a resource allocation method based on deep reinforcement learning. Under the constraints of ensuring the minimum transmission rate of D2D users and the minimum energy collection of energy receivers, the maximum transmission rate of cellular users is improved by jointly designing IRS phase shift and DT transmission power.

[0007] To achieve the above objectives, the technical method of the present invention comprises the following specific steps:

[0008] Step 1: Set the positions of the base station, IRS, cellular users, K pairs of D2D users and energy receivers; model the channels from cellular users to BS, IRS, D2D Receiver (DR), EHR, the channels from D2D Transmitter (DT) to DR, IRS, BS, EHR, and the channels from IRS to BS, DR, EHR, and obtain the channel gains;

[0009] Step 2: construct the maximum transmission rate problem of cellular users based on the channel gain in step 1;

[0010] Step 2.1, the maximum transmission rate of the signal sent by the cellular user to the base station can be expressed as R c ;

[0011] Step 2.2, the system goal is to optimize the IRS phase shift matrix and DT transmission power under the constraints of the minimum transmission rate of D2D users and the minimum energy collection of EHR, and construct the maximum transmission rate problem of cellular users;

[0012] Step 3: Based on the IRS phase shift matrix, DT transmission power allocation, D2D user minimum transmission rate, and EHR energy harvesting in the system, the three major elements of deep reinforcement learning: state, action, and reward are defined and designed respectively;

[0013] Step 4: Use the multi-agent DQN algorithm to optimize the reinforcement learning network model, and obtain the maximum transmission rate of cellular users based on the optimized network model.

[0014] Specifically, the channel gain in the communication system in step 1 specifically includes:

[0015] A quasi-static flat fading channel model is used, and all channel gains are in the ideal channel state information (CSI) mode, where the channel gains from cellular users to BS, IRS, DR, and EHR are The channel gains from DT to DR, IRS, BS, and EHR are The channel gains from IRS to BS, DR, and EHR are

[0016] Specifically, the cellular user transmission rate problem described in step 2.1 specifically includes:

[0017] In the uplink, the signals received by the base station are the transmission signals of cellular users, the reflected signals of IRS, and the co-channel interference signals generated by D2D users that reuse the cellular user spectrum. Therefore, the transmission rate of cellular users can be expressed as:

[0018]

[0019] Among them, P C , are the transmit powers of the cellular user and the i-th DT respectively, Θ is the IRS phase shift matrix, satisfying Main diagonal θ n =(0,2π) represents the phase shift of the nth (1≤n≤N) reflection element, σ 2 Represents noise.

[0020] Specifically, the problem of constructing the maximum transmission rate of cellular users in step 2.2 specifically includes:

[0021] The information transmission rate of each D2D user can be expressed as:

[0022]

[0023] The energy collected by EHR is expressed as:

[0024]

[0025] Where η is the energy conversion efficiency.

[0026] Therefore, the problem of maximizing the transmission rate of cellular users can be modeled as:

[0027]

[0028] C3:E≥E min

[0029]

[0030] in, C1 is the D2D user transmission rate constraint, R min is the minimum transmission rate for D2D users to communicate normally; C2 is the IRS constant modulus constraint; C3 is the energy harvesting constraint, E min is the minimum energy collection limit of EHR; C4 is the DT transmission power constraint, P max is the maximum transmit power of DT.

[0031] Specifically, the definition and design of the states, actions, and rewards described in step 3 specifically include:

[0032] State: The DQN network needs to select the best action based on the current state to obtain more rewards, so as to maximize the transmission rate of cellular users while ensuring that the minimum transmission rate of D2D users is met. For this goal, the state space S is defined as the SINR of the D2D user. Therefore, the state of the agent at time t is:

[0033]

[0034] Among them, SINR i (t) is the signal-to-interference-noise ratio of the ith agent at time t.

[0035] Action: In the DQN algorithm, each time a state is input, the DQN network will select an action according to the ε-greedy strategy. The agent performs this action to obtain a reward value and reach the next state. The ε-greedy strategy is expressed as:

[0036]

[0037] Among them, 0<ε<1.

[0038] For the IRS-assisted D2D communication system proposed in this chapter, the action space A should include all power selections and IRS phase shift selections. Therefore, the action a at time t is t Defined as:

[0039]

[0040] in, A 1 and A 2They represent the D2D user power allocation space and the IRS reflection element phase shift selection space respectively.

[0041] Reward function: This model uses the rate of cellular users as the instantaneous reward, considering the minimum transmission rate constraint of D2D users, and the reward value r t Defined as:

[0042]

[0043] The long-term accumulated reward value is:

[0044]

[0045] Among them, γ∈[0,1] is a discount factor, which indicates the degree of attention the agent pays to future rewards. When γ=0, the agent only pays attention to the reward value at the current moment. The larger the γ is, the more attention the agent pays to future rewards. This farsightedness usually helps the agent choose actions more wisely.

[0046] According to the IRS-assisted D2D system resource allocation method based on multi-agent DQN according to claim 1, it is characterized in that: the optimization of the reinforcement learning network model by using the multi-agent DQN algorithm in step 4 specifically includes:

[0047] The entire algorithm consists of two parts: online decision making and offline training:

[0048] In the online decision-making stage, first initialize the current environment and obtain the initial state s t , select an action according to the ε-greedy (0<ε<1) strategy, then calculate the reward value and reach the next state s t+1 , then (s t ,a t ,r t ,s t+1 ) is stored in the experience pool as a data sample.

[0049] During the offline training phase, a batch of data is randomly extracted from the experience pool, and the true Q value is calculated according to the following definition:

[0050] y t =r t +γmaxQ(s t+1 ,a t+1 ;θ)

[0051] Among them, θ is a parameter in the network. By training the current value network and updating the parameter θ, the Q value predicted by the network is not close to the actual Q value. Therefore, the loss function is expressed as:

[0052] ψ(t)=(y t -Q(st ,a t ;θ)) 2

[0053] In this algorithm, the Adam optimizer is used to minimize the loss function, and then back propagation is used to update the parameters in the network. After training for a period of time, the weight parameters in the current value network are copied to the target value network. It can be seen from the above technical solution that the method based on IRS-assisted D2D system communication provided by the present invention solves the problem of maximizing the information transmission rate of cellular users under the constraints of ensuring the minimum transmission rate of D2D users and the minimum energy collection of energy receivers by jointly designing IRS phase shift and DT transmission power.

[0054] The present invention has the following advantages:

[0055] 1. A scenario of reusing uplink spectrum resources in IRS-assisted D2D communication is disclosed, which maximizes the information transmission rate of cellular users under the constraints of ensuring the minimum transmission rate of D2D users and the minimum energy collection of energy receivers. Compared with the traditional D2D collaborative computing strategy, the IRS-assisted D2D collaborative computing solution can achieve lower computing latency.

[0056] 2. This invention introduces a multi-agent DQN network framework for the first time in IRS-assisted D2D communication. Compared with traditional mathematical optimization algorithms, it solves the problem that multiple parameters are difficult to jointly optimize in real time. In the proposed multi-agent DQN algorithm, a single agent is only responsible for learning the power allocation strategy for a pair of D2D users. They can learn in a distributed manner and interact with each other. This will significantly reduce the action space of a single agent, make the network easier to converge, and improve the robustness of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0058] Figure 1 This is a schematic diagram of the IRS-assisted D2D communication system model;

[0059] Figure 2 It is a multi-agent DQN network model;

[0060] Figure 3 It is the flow chart of the multi-agent DQN algorithm;

[0061] Figure 4 is the relationship between energy harvesting and the number of IRS reflective elements N;

[0062] Figure 5 is the relationship between energy harvesting and the number of D2D users K; DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] The specific steps are as follows:

[0065] Step 1: Set the positions of the base station, IRS, cellular users, K pairs of D2D users and energy receivers; model the channels from cellular users to BS, IRS, D2D Receiver (DR), EHR, the channels from D2D Transmitter (DT) to DR, IRS, BS, EHR, and the channels from IRS to BS, DR, EHR, and obtain the channel gains;

[0066] Step 2: construct the maximum transmission rate problem of cellular users based on the channel gain in step 1;

[0067] Step 2.1, the maximum transmission rate of the signal sent by the cellular user to the base station can be expressed as R c ;

[0068] Step 2.2, the system goal is to optimize the IRS phase shift matrix and DT transmission power under the constraints of the minimum transmission rate of D2D users and the minimum energy collection of EHR, and construct the maximum transmission rate problem of cellular users;

[0069] Step 3: Based on the IRS phase shift matrix, DT transmission power allocation, D2D user minimum transmission rate, and EHR energy harvesting in the system, the three major elements of deep reinforcement learning: state, action, and reward are defined and designed respectively;

[0070] Step 4: Use the multi-agent DQN algorithm to optimize the reinforcement learning network model, and obtain the maximum transmission rate of cellular users based on the optimized network model.

[0071] In this embodiment, the specific process of step 1 is as follows:

[0072] like Figure 1As shown in the figure, the IRS-assisted D2D communication system model includes: a base station (BS), surrounded by a cellular user, a pair of D2D users and an EHR, where each pair of D2D users includes a transmitting user and a receiving user. At the same time, an IRS composed of a large number of built-in programmable components is deployed in the cell, which plays a role in enhancing beneficial signals and suppressing interference by reflecting directional beams to the target receiver. During the communication process, the receiving end can not only receive the signal transmitted by the transmitter through a direct link, but also receive the reflected signal through the IRS reflection link. Considering the actual cost and feasibility, the reflection phase of the IRS is taken as a discrete value, where the IRS contains N reflection units. It is assumed that the phase shift of each reflection unit is discrete 4-bit, the phase shift range is [0,2π], and the amplitude of the IRS transmitting unit is 1, that is, there is no energy loss in the signal reflected by the IRS. In the Saleh-Valenzula theoretical channel, all channel gains are ideal channel state information (CSI) modes, where the channel gains from cellular users to BS, IRS, DR, and EHR are respectively The channel gains from DT to DR, IRS, BS, and EHR are The channel gains from IRS to BS, DR, and EHR are

[0073] In this embodiment, the specific process of step 2 is as follows:

[0074] Specifically, the cellular user transmission rate problem in step 2.1 includes:

[0075] In the uplink, the signals received by the base station are the transmission signals of cellular users, the reflected signals of IRS, and the co-channel interference signals generated by D2D users that reuse the cellular user spectrum. Therefore, the transmission rate of cellular users can be expressed as:

[0076]

[0077] Among them, P C , are the transmit powers of the cellular user and the i-th DT respectively, Θ is the IRS phase shift matrix, satisfying Main diagonal θ n =(0,2π) represents the phase shift of the nth (1≤n≤N) reflection element, σ 2 Represents noise.

[0078] Specifically, the problem of constructing the maximum transmission rate of cellular users in step 2.2 includes:

[0079] The information transmission rate of each D2D user can be expressed as:

[0080]

[0081] The energy collected by EHR is expressed as:

[0082]

[0083] Where η is the energy conversion efficiency.

[0084] The problem of maximizing the transmission rate of cellular users can be modeled as:

[0085]

[0086] C3:E≥E min

[0087]

[0088] in, C1 is the D2D user transmission rate constraint, R min is the minimum transmission rate for D2D users to communicate normally; C2 is the IRS constant modulus constraint; C3 is the energy harvesting constraint, E min is the minimum energy collection limit of EHR; C4 is the DT transmission power constraint, P max is the maximum transmit power of DT.

[0089] In this embodiment, the specific process of step 3 is as follows:

[0090] like Figure 2 As shown in the figure, the Markov decision process of the multi-agent DQN network is given. The specific three elements: state, action, and reward are described as follows:

[0091] State: The DQN network needs to select the best action based on the current state to obtain more rewards, so as to maximize the transmission rate of cellular users while ensuring that the minimum transmission rate of D2D users is met. For this goal, the state space S is defined as the SINR of the D2D user. Therefore, the state of the agent at time t is:

[0092]

[0093] Among them, SINR i (t) is the signal-to-interference-noise ratio of the ith agent at time t.

[0094] Action: In the DQN algorithm, each time a state is input, the DQN network will select an action according to the ε-greedy strategy. The agent performs this action to obtain a reward value and reach the next state. The ε-greedy strategy is expressed as:

[0095]

[0096] Among them, 0<ε<1.

[0097] For the IRS-assisted D2D communication system proposed in this chapter, the action space A should include all power selections and IRS phase shift selections. Therefore, the action a at time t is t Defined as:

[0098]

[0099] in, A 1 and A 2 They represent the D2D user power allocation space and the IRS reflection element phase shift selection space respectively.

[0100] Reward function: This model uses the rate of cellular users as the instantaneous reward, considering the minimum transmission rate constraint of D2D users, and the reward value r t Defined as:

[0101]

[0102] The long-term accumulated reward value is:

[0103]

[0104] Among them, γ∈[0,1] is a discount factor, which indicates the degree of attention the agent pays to future rewards. When γ=0, the agent only pays attention to the reward value at the current moment. The larger the γ is, the more attention the agent pays to future rewards. This farsightedness usually helps the agent choose actions more wisely.

[0105] In this embodiment, the specific process of step 4 is as follows:

[0106] like Figure 3 As shown in the figure, the algorithm flow chart of the multi-agent DQN algorithm is given. The entire algorithm consists of two parts: online decision-making and offline training:

[0107] In the online decision-making stage, first initialize the current environment and obtain the initial state s t , select an action according to the ε-greedy (0<ε<1) strategy, then calculate the reward value and reach the next state s t+1 , then (s t ,a t ,r t ,s t+1 ) is stored in the experience pool as a data sample.

[0108] During the offline training phase, a batch of data is randomly extracted from the experience pool, and the true Q value is calculated according to the following definition:

[0109] y t =r t +γmaxQ(s t+1 ,a t+1 ;θ)

[0110] Among them, θ is a parameter in the network. By training the current value network and updating the parameter θ, the Q value predicted by the network is not close to the actual Q value. Therefore, the loss function is expressed as:

[0111] ψ(t)=(y t -Q(s t ,a t ;θ)) 2

[0112] In this algorithm, the Adam optimizer is used to minimize the loss function, and then back-propagation is used to update the parameters in the network. After training for a period of time, the weight parameters in the current value network are copied to the target value network.

[0113] A comparison between the energy harvesting solution of the present invention and other existing solutions is given below to make the advantages and features of the present invention more obvious.

[0114] Figure 4 The relationship between the maximum transmission rate of cellular users and the number of IRS reflection elements is shown in the figure. The number of D2D users is K = 6. It can be seen from the figure that the multi-agent DQN algorithm proposed in the present invention has the best performance. The performance of the IRS-assisted communication scheme is significantly better than that of the scheme without IRS, and as the number of IRS reflection elements increases, the maximum transmission rate of cellular users increases, which proves the effectiveness of the proposed algorithm and the advantages of IRS in D2D communication systems.

[0115] Figure 5 The relationship between the maximum transmission rate of cellular users and the number of D2D users, where the number of IRS reflection elements N = 40, as the number of D2D users K increases, the maximum transmission rate of cellular users decreases. The reason is that as the number of D2D users that reuse cellular user spectrum resources increases, the interference to cellular users will also increase, thus affecting the transmission rate of cellular users. However, the proposed reinforcement learning algorithm is still better than the other three benchmark schemes.

[0116] It can be seen that the IRS-assisted D2D system resource allocation method based on multi-agent DQN proposed in this embodiment can maximize the transmission rate of cellular users by jointly designing the IRS phase shift and DT transmission power under the constraints of ensuring the minimum transmission rate of D2D users and the minimum energy collection of energy receivers, without the need for complex mathematical formula derivation.

[0117] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A multi-agent DQN-based IRS-assisted D2D system resource allocation method, characterized in that: Here are the steps: Step 1: Set the positions of the base station, IRS, cellular users, K pairs of D2D users and energy receivers (Energy Harvesting Receivers, EHR), model the channels from cellular users to BS, IRS, receiving users (D2D Receivers, DR), EHR, the channels from transmitting users (D2D Transmitters, DT) to DR, IRS, BS, EHR, and the channels from IRS to BS, DR, EHR, and obtain the channel gains; Step 2: construct the maximum transmission rate problem of cellular users based on the channel gain in step 1; Step 2.1, the maximum transmission rate of the signal sent by the cellular user to the base station is expressed as R c ; Step 2.2, the system goal is to optimize the IRS phase shift matrix and DT transmission power under the constraints of the minimum transmission rate of D2D users and the minimum energy collection of EHR, and construct the maximum transmission rate problem of cellular users; Step 3: Based on the IRS phase shift matrix, DT transmission power allocation, D2D user minimum transmission rate, and EHR energy harvesting in the system, the three major elements of deep reinforcement learning: state, action, and reward are defined and designed respectively, wherein the definition and design of the state, action, and reward specifically include: State: The DQN network needs to select the best action according to the current state to obtain more rewards, so as to maximize the transmission rate of cellular users while ensuring the minimum transmission rate of D2D users. For this goal, the state space S is defined as the SINR of D2D users. Therefore, the state of the agent at time t is: Among them, SINR i (t) is the signal-to-interference-noise ratio of the i-th agent at time t, that is, the transmission rate; Action: In the DQN algorithm, each time a state is input, the DQN network will select an action according to the ε-greedy strategy. The agent performs this action to obtain a reward value and reach the next state. The ε-greedy strategy is expressed as: Among them, 0<ε<1, for the proposed IRS-assisted D2D communication system, the action space A should include all power selections and IRS phase shift selections, so the action a at time t t Defined as: in, A1 and A2 represent the D2D user power allocation space and the IRS reflection element phase shift selection space, respectively; Reward function: This model uses the rate of cellular users as the instantaneous reward, considering the minimum transmission rate constraint of D2D users, and the reward value r t Defined as: in is the instantaneous signal-to-interference-and-noise ratio of the cellular user, is the instantaneous signal-to-interference-noise ratio of the D2D user, R min is the minimum transmission rate threshold, then the long-term accumulated reward value is: Among them, γ∈[0,1] is a discount factor, which indicates the degree of attention the agent pays to future rewards. When γ=0, the agent only pays attention to the reward value at the current moment. The larger the γ is, the more attention the agent pays to future rewards, which helps the agent choose actions more wisely. Step 4: Use the multi-agent DQN algorithm to optimize the reinforcement learning network model, and obtain the maximum transmission rate of cellular users based on the optimized network model. The algorithm used to optimize the reinforcement learning network model specifically includes: The entire algorithm consists of two parts: online decision making and offline training: In the online decision-making stage, first initialize the current environment and obtain the initial state s t , select an action according to the ε-greedy (0<ε<1) strategy, then calculate the reward value and reach the next state s t+1 , then (s t ,a t ,r t ,s t+1 ) is stored in the experience pool as a data sample, where s t is the SINR state of the D2D user at time t, a t is the action at time t; During the offline training phase, a batch of data is randomly extracted from the experience pool, and the true Q value is calculated according to the following definition: y t =r t +γmaxQ(s t+1 ,a t+1 (i) Among them, θ is a parameter in the network. By training the current value network and updating the parameter θ, the Q value predicted by the network is not close to the actual Q value. Therefore, the loss function is expressed as: ψ(t)=(y t -Q(s t ,a t ;i)) 2 Subsequently, the Adam optimizer is used to minimize the loss function, and then back-propagation is used to update the parameters in the network. After training for a period of time, the weight parameters in the current value network are copied to the target value network.

2. According to claim 1, a multi-agent DQN-based IRS-assisted D2D system resource allocation method is characterized by: The channel gain in step 1 specifically includes: A quasi-static flat fading channel model is used, and all channel gains are in the ideal channel state information (CSI) mode, where the channel gains from cellular users to BS, IRS, DR, and EHR are The channel gains from DT to DR, IRS, BS, and EHR are The channel gains from IRS to BS, DR, and EHR are 3. According to claim 2, a multi-agent DQN-based IRS-assisted D2D system resource allocation method is characterized by: The cellular user transmission rate problem described in step 2.1 specifically includes: In the uplink, the signals received by the base station are the transmission signals of the cellular users, the reflected signals of the IRS, and the co-channel interference signals generated by the D2D users that reuse the cellular user spectrum. Therefore, the transmission rate of the cellular user is expressed as: Among them, P C , are the transmit powers of the cellular user and the i-th DT respectively, Θ is the IRS phase shift matrix, satisfying Main diagonal θ n =(0,2π) represents the phase shift of the nth (1≤n≤N) reflection element, σ 2 Represents noise.

4. The IRS-assisted D2D system resource allocation method based on multi-agent DQN according to claim 2 is characterized by: The problem of constructing the maximum transmission rate of cellular users described in step 2.2 specifically includes: The information transmission rate of each D2D user is expressed as: The energy collected by EHR is expressed as: Among them, η is the energy conversion efficiency. Therefore, the problem of maximizing the transmission rate of cellular users is modeled as: in, C1 is the D2D user transmission rate constraint, R min is the minimum transmission rate for normal communication of D2D users; C2 is the IRS constant modulus constraint; C3 is the energy harvesting constraint, E min is the minimum energy collection limit of EHR; C4 is the DT transmission power constraint, P max is the maximum transmit power of DT.

Images