A method for allocating D2D communication resources

Abstract

The application provides a D2D communication resource allocation method. It relates to the technical field of communication. The method comprises the following steps: constructing a resource allocation model with power allocation parameters, reconfigurable intelligent surface phase parameters and channel allocation parameters as variables, aiming to maximize the total D2D rate; and solving the resource allocation model based on a reinforcement learning model integrating an A3C agent and a TD3 agent. The A3C agent generates multiple state transition samples based on multiple policy networks and stores them in an experience pool, providing space for the TD3 agent to explore; the TD3 agent samples from the experience pool, performs fine policy updating through a second policy network, and generates optimized actions. The method is used to solve the defect of inefficient state-action space exploration in the high-dimensional joint optimization scenario, and realizes the joint and collaborative optimization of more optimal power allocation, RIS phase control and channel allocation.

Description

Technical Field

[0001] This invention relates to the field of communication technology, and in particular to a D2D communication resource allocation method. Background Technology

[0002] In dense 5G / 6G networks, D2D (Device-to-Device) communication improves capacity through spectrum reuse, but also exacerbates intra-cell interference. To address this, Cooperative Non-Orthogonal Multiple Access (CNOMA) technology has been introduced. Utilizing superposition coding and successive interference cancellation mechanisms, it achieves power domain reuse and cooperative interference cancellation for users under different channel conditions, significantly improving user access capabilities in high-density scenarios. The emergence of Reconfigurable Intelligent Surface (RIS) provides a new approach to channel environment control. RIS can optimize channel quality by dynamically adjusting the reflection phase, further enhancing the system performance of CNOMA. However, the introduction of RIS phase control also expands the resource allocation problem from the traditional two-dimensional power-channel optimization to a multi-dimensional and complex optimization problem that includes phase control.

[0003] In the face of complex communication scenarios that deeply integrate D2D, CNOMA and RIS, traditional reinforcement learning-based resource allocation methods suffer from low exploration efficiency due to the explosion of state and action space dimensions. They cannot achieve joint optimal decision-making for power allocation, RIS phase modulation and channel allocation, and are unable to support the quality of service requirements of massive connections and ultra-low latency. Summary of the Invention

[0004] Based on this, it is necessary to provide a D2D communication resource allocation method to address the above-mentioned technical problems, so as to solve the defects of existing reinforcement learning algorithms in the inefficiency of state-action space exploration and insufficient optimization capability in high-dimensional joint optimization scenarios, and realize the joint and coordinated optimization of power allocation, RIS phase modulation and channel allocation.

[0005] The following technical solution is adopted in this specification: This specification provides a method for allocating D2D communication resources, including: With the goal of maximizing the total communication rate of all D2D user groups in the target communication system, and with the power allocation parameters, reconfigurable smart metasurface phase parameters, and channel allocation parameters of the target communication system as decision variables, a resource allocation model is constructed. The resource allocation model is solved using a pre-built reinforcement learning model to obtain the optimal decision variables, and communication resources are allocated based on these optimal decision variables. The reinforcement learning model includes: The A3C agent generates multiple state transition samples based on multiple configured first policy networks. The state transition samples include the first action representing the decision variable, the reward value representing the total communication rate, the current state representing the current resource allocation status of the target communication system, and the next state representing the resource allocation status of the target communication system after the action is executed. The experience pool is used to store multiple state transition samples generated by the A3C agent; The TD3 agent is equipped with a second policy network, which generates a second action based on state transition samples obtained from the experience pool.

[0006] Furthermore, the A3C agent is also configured with multiple first evaluation networks, which correspond to multiple first policy networks; the TD3 agent is also configured with two second evaluation networks; the step of solving the resource allocation model through a pre-built reinforcement learning model specifically includes: The optimization objective of the resource allocation model is equivalently transformed into the cumulative reward maximization problem of a multi-agent Markov decision process; Maximizing cumulative reward is used as the training objective of the reinforcement learning model. An asynchronous training and alternating update mechanism is employed to train the A3C and TD3 agents in the reinforcement learning model, including: I. Asynchronous training of multiple first policy networks and multiple first evaluation networks in the A3C agent; II. Store multiple state transition samples generated during the asynchronous training of the A3C agent into the experience pool; III. Sample the state transition samples in the experience pool, and alternately update the second policy network and two second evaluation networks in the TD3 agent based on the sampled state transition samples, including: fixing the parameters of the second policy network and updating the parameters of the two second evaluation networks; fixing the parameters of the two second evaluation networks and updating the parameters of the second policy network. Repeat steps I to III until the cumulative reward converges. The second action generated by the second policy network when the cumulative reward converges is taken as the optimal decision variable.

[0007] Furthermore, the multiple first policy networks and multiple first evaluation networks in the asynchronously trained A3C agent specifically include: Input the current state into each first policy network, and output multiple first actions accordingly; Apply each first action to the target communication system to obtain the reward value and next step status for each first action; Construct the objective function of the first policy network and the state function of the first evaluation network based on the reward value: ; ; ; in, The objective function of the first policy network; This is the first evaluation network state function; The dominant function; This is the current state; The first action; This is the reward value; Discount factor; The next step is now in progress; and Both are state value functions. Representing the next state The expected long-term cumulative reward that the A3C agent will obtain in the future; Representation in the current state The expected long-term cumulative reward that the A3C agent will obtain in the future; These are the parameters for the first policy network; To generate the first action The probability of; The parameters of the first policy network and the first evaluation network are updated with the training objectives of minimizing the objective function of the first policy network and minimizing the state function of the first evaluation network, respectively.

[0008] Furthermore, the process of fixing the parameters of the second policy network and updating the parameters of the two second evaluation networks specifically includes: The next state from the sampled state transition sample is input into the second policy network in the TD3 agent, and the next initial action is output. Gaussian noise is added to the next initial action to obtain the next action. The next state and the next action are respectively input into the two second evaluation networks in the TD3 agent to obtain the first action value prediction value and the second action value prediction value respectively. The minimum value between the first action value prediction and the second action value prediction is used as the target action value prediction. After discounting the target action value prediction based on the discount factor, it is added to the reward value in the sampled state transition sample to obtain the target value. Two loss functions for the second evaluation network are constructed based on the difference between the predicted value of the first action and the target value, and the difference between the predicted value of the second action and the target value, respectively. The parameters of the two second evaluation networks are updated with the goal of minimizing the loss function of the two second evaluation networks.

[0009] Furthermore, the step of fixing the parameters of the two second evaluation networks and updating the parameters of the second policy network specifically includes: The current state from the sampled state transition samples is input into the second policy network in the TD3 agent, and the second action is output. Input the current state and the second action into any second evaluation network in the TD3 agent to obtain the predicted value of the second action. The loss function of the second policy network is constructed based on the predicted action value of the second action; The parameters of the second policy network are updated with the goal of maximizing the loss function of the second policy network.

[0010] Furthermore, the second evaluation network updates its parameters at each training step, and the second policy network performs a delayed parameter update every two training steps; the delayed update suppresses training oscillations of the second policy network by reducing the update frequency of the second policy network.

[0011] Furthermore, the target communication system includes: At least one base station is used to transmit downlink signals; At least one cellular user is used to receive downlink signals; Multiple D2D user groups are configured, each D2D user group comprising a D2D transmitting user and a D2D receiving user pair, each D2D receiving user pair comprising a D2D strong receiving user and a D2D weak receiving user; wherein, the D2D transmitting user multiplexes the downlink spectrum resources of the cellular user to transmit D2D communication signals to the D2D strong receiving user and the D2D weak receiving user; the D2D strong receiving user receives and decodes the D2D communication signals, obtains decoded data, and forwards the decoded data to the D2D weak receiving user; At least one reconfigurable smart metasurface includes multiple programmable reflective elements. By adjusting the phase and amplitude of each programmable reflective element, the channel of the downlink from the base station to the cellular user, the D2D link from the D2D transmitting user to the D2D strong receiving user, the D2D link from the D2D transmitting user to the D2D weak receiving user, and the cooperative forwarding link from the D2D strong receiving user to the D2D weak receiving user is regulated.

[0012] Furthermore, the method for obtaining the D2D receiving user pair includes: Construct a candidate pairing matrix, wherein the candidate pairing matrix uses the receiving user as the row index and column index, and each matrix cell in the candidate pairing matrix represents a candidate D2D receiving user pair; Based on the phase alignment, interference, and power difference between the two receivers in each candidate D2D receiver pair, the joint pairing weight of each candidate D2D receiver pair is determined. In the candidate pairing matrix, filter and remove the candidate D2D receiver pair with the largest joint pairing weight. Repeat the filtering and removal steps based on a greedy strategy until all receivers are paired or the remaining receivers cannot be paired, thus obtaining multiple D2D receiver pairs.

[0013] Furthermore, the determination of the joint pairing weight for each candidate D2D receiver pair based on the phase alignment, interference, and power difference between the two receivers in each candidate D2D receiver pair specifically includes: Based on candidate D2D receiving user pairs ( ) receives users and receiving users The mean channel phase difference at each reflective element of the reconfigurable smart metasurface is used to obtain candidate D2D receiver pairs. Phase alignment factor : ; in, and Representing the receiving user and receiving users In the reconfigurable smart metasurface Channel phase of a programmable reflection unit This represents the total number of programmable reflection units; Based on candidate D2D receiving user pairs ( ) receives users and receiving users The orthogonality of equivalent channel vectors is used to obtain candidate D2D receiver pairs. Interference suppression ability factor : ; in, and These are the receiving users and receiving users The equivalent channel vector; yes The conjugate transpose of ; Based on candidate D2D receiving user pairs ( ) receives users and receiving users The power demand difference is used to obtain the power difference factor. : ; in, and These are the receiving users and receiving users Power requirements; Represents the vector norm; Indicates the modulus length; Phase alignment factor Interference inhibition ability factor and power difference factor The product of the two is used as a candidate D2D receiving user pair ( The joint pairing weights.

[0014] Furthermore, the optimization objective is to maximize the total communication rate of all D2D user groups in the target communication system. The resource allocation model is constructed using the power allocation parameters, reconfigurable smart metasurface phase parameters, and channel allocation parameters of the target communication system as decision variables. Specifically, this includes: The optimization objective of the resource allocation model is to maximize the total communication rate of all D2D user groups in the target communication system. ; in, For the first In the D2D user group, the D2D sending user and the first Channel multiplexing factor for each cellular user For the first Transmit power of D2D transmitting users in a D2D user group; For the first Cooperative transmit power of D2D strong receiver users in a D2D user group; For the reflection matrix of a reconfigurable smart metasurface, For the first reconfigurable smart metasurface The phase of each programmable reflective unit; For the first The main link power allocation factor in each D2D user group; For the first Secondary link power allocation coefficients in D2D user groups For the first Transmit power of each cellular user; The total number of D2D user groups. For the first Communication rate of D2D user groups; For the first The main link signal-to-interference-plus-noise ratio of each D2D user group; For the first The signal-to-interference-plus-noise ratio of the secondary links of each D2D user group; wherein, the primary link is the direct transmission path from the D2D sending user to the D2D receiving user, and the secondary link is the cooperative forwarding path assisted by the D2D cooperative receiving user; Limiting the minimum signal-to-interference-plus-noise ratio for cellular users Minimum signal-to-interference-plus-noise ratio (SIR) limits for D2D strong receivers and D2D weak receivers Channel multiplexing factor limitation Upper limits on transmit power during the D2D user group direct communication phase and cooperative power during the cooperative relay phase. Base station transmit power limits for each cellular user Limitations on the relationship between primary link power allocation coefficient and secondary link power allocation coefficient for D2D user groups and phase confinement of programmable reflective units in reconfigurable smart metasurfaces. As a constraint on the resource allocation model; St ; in, For the first Signal-to-interference-to-noise ratio per cellular user; The minimum signal-to-interference-plus-noise ratio (SIR) threshold for cellular users. Total number of cellular users; The minimum signal-to-interference-plus-noise ratio (SINORR) threshold for D2D links; The maximum transmit power threshold for D2D transmitting users. This represents the maximum cooperative transmit power for D2D strong receiver users; The maximum transmit power threshold for cellular users; For the first reconfigurable smart metasurface Phase adjustment angle of each programmable reflector unit This represents the total number of programmable reflective units in a reconfigurable smart metasurface.

[0015] The above-mentioned technical solutions adopted in this specification can achieve the following beneficial effects: This invention leverages the parallelism of A3C agents, enabling multiple first-policy networks to simultaneously interact with the environment and generate diverse state transition samples. This expands the exploration range of high-dimensional state and action spaces, mitigating the spatial dimension explosion problem caused by the coupling of multiple variables in power allocation, RIS phase modulation, and channel allocation. It also avoids the shortcomings of traditional single-threaded reinforcement learning, which is prone to getting trapped in local optima and has slow convergence. Furthermore, a TD3 agent is introduced to refine the learning of samples from the experience pool. This collaborative mechanism of broad exploration based on A3C and fine optimization based on TD3 not only achieves efficient joint optimization of multi-dimensional decision variables, ensuring the maximization of the system's total communication rate, but also dynamically adapts to complex and ever-changing channel environments, thereby supporting the stringent quality of service requirements for massive connections and ultra-low latency. Attached Figure Description

[0016] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This document provides a flowchart illustrating a D2D communication resource allocation method. Figure 2 This is a schematic diagram of a signal transmission link for a target communication system provided in this specification; Figure 3 This is a schematic diagram of a pairing strategy provided in this specification; Figure 4 This document provides a schematic diagram of an A3TD training process. Figure 5 This document provides a schematic diagram illustrating the relationship between the number of D2D pairs and the convergence time. Figure 6 This is a schematic diagram illustrating the relationship between the spectral efficiency and convergence speed of various algorithms provided in this specification. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments in this specification without creative effort are within the scope of protection of this application.

[0018] The technical solution provided by this invention can be applied to D2D communication systems based on RIS-assisted CNOMA. This invention proposes a user pairing strategy that integrates phase matching, interference suppression, and power control, introducing three key indicators: phase alignment, interference suppression capability, and power difference factor. This strategy utilizes the phase modulation capability of RIS to quantify the coherence between user channels, improving the success rate of SIC decoding by enhancing the phase alignment of signal superposition; simultaneously, the power difference factor ensures that paired users meet the decoding principles of NOMA, optimizing power allocation differences and reducing interference. This overcomes the neglect of phase alignment in traditional strategies, achieving adaptive pairing and improving decoding reliability and system performance. However, the optimization of the user pairing strategy requires extensive searching of joint power control, RIS phase configuration, and channel allocation, making it a high-dimensional non-convex problem. Furthermore, this invention designs a deep reinforcement learning framework—A3TD—that integrates Asynchronous Advantageous Actor-Commentator (A3C) and Dual-Delay Deep Deterministic Policy Gradient (TD3). This framework accelerates policy search and improves sample diversity through multi-threaded parallel exploration of A3C; it leverages the efficient optimization capabilities of TD3 in continuous action space, reduces value estimation bias through a dual-critic network, and enhances training stability by combining a delayed update strategy. A3TD achieves joint allocation of power, RIS phase, and channel resources, significantly improving convergence speed and system performance, effectively solving the challenges of fast convergence and stability in high-dimensional resource allocation problems. Simulation results demonstrate that the proposed scheme outperforms traditional methods in terms of spectral efficiency and training efficiency, providing an efficient resource management solution for RIS-assisted CNOMA systems.

[0019] The method of the present invention will now be described in detail with reference to the accompanying drawings.

[0020] Figure 1 This is a flowchart illustrating a D2D communication resource allocation method provided in this specification, such as... Figure 1 As shown, the method includes: S1. With the goal of maximizing the total communication rate of all D2D user groups in the target communication system, and with the power allocation parameters, reconfigurable smart metasurface phase parameters, and channel allocation parameters of the target communication system as decision variables, a resource allocation model is constructed.

[0021] For example, the target communication system is a RIS-assisted CNOMA-D2D communication system. The core of constructing this resource allocation model lies in clearly defining two key elements: the optimization objective and the decision variables. First, the optimization objective is clearly defined as maximizing the total communication rate of all D2D user groups in the target communication system, i.e., achieving the optimal cumulative communication rate of all D2D user groups within the system. Second, the decision variables are the core parameters in the model that need to be adjusted and optimized, specifically including: power allocation parameters, used to regulate the transmission power of each user and each communication link in the target communication system; RIS phase parameters, used to regulate the signal reflection phase of the RIS to optimize channel transmission conditions; and channel allocation parameters, used to rationally allocate the spectrum channel resources of the target communication system.

[0022] S2. Solve the resource allocation model using a pre-built reinforcement learning model to obtain the optimal decision variables, and allocate communication resources based on the optimal decision variables. The reinforcement learning model includes: an A3C agent, which generates multiple state transition samples based on multiple configured first policy networks; the state transition samples include a first action representing the decision variables, a reward value representing the total communication rate, the current state representing the current resource allocation status of the target communication system, and the next state representing the resource allocation status of the target communication system after executing the action; an experience pool for storing the multiple state transition samples generated by the A3C agent; and a TD3 (Twin Delayed Deep Deterministic Policy Gradient) agent, which is configured with a second policy network that generates a second action based on the state transition samples sampled from the experience pool.

[0023] For example, the pre-built reinforcement learning model is adapted to a RIS-assisted CNOMA-based D2D communication system. The specific implementation of each component is as follows: The A3C agent is configured with multiple parallel first policy networks. Each first policy network independently explores the resource allocation state of the target communication system and generates different first actions. Each first action corresponds to a specific set of power allocation parameters, reconfigurable smart metasurface phase parameters, and channel allocation parameters. After each first action is executed, the total communication rate of all D2D user groups in the target communication system is calculated. This total communication rate is used as the reward value for the corresponding first action. Simultaneously, the resource allocation state of the target communication system before executing the first action (i.e., the current state) and the resource allocation state of the target communication system after executing the first action (i.e., the next state) are recorded. These four values ​​together constitute a state transition sample. The experience pool continuously stores all state transition samples generated by the A3C agent, providing a stable source of samples for the TD3 agent. The second policy network configured by the TD3 agent randomly selects a certain number of state transition samples from the experience pool according to a preset sampling frequency, extracts the current state from the samples, performs decision calculations based on the current state, and generates the corresponding second action. The second action also corresponds to a set of specific power allocation parameters, reconfigurable smart metasurface phase parameters, and channel allocation parameters. Subsequently, through continuous iterative optimization, the optimal decision variables are finally output to complete the allocation of communication resources.

[0024] For example, based on the above Figure 1 In the embodiment shown, the structure of the target communication system in S1 described above can be found in [reference needed]. Figure 2 As shown, Figure 2 This is a schematic diagram of a signal transmission link for a target communication system provided in this specification. Its composition, interference type, channel information and signal transmission are shown in S111~S114.

[0025] S111, Composition of the target communication system.

[0026] A base station (BS) is configured with a single antenna and acts as a downlink signal transmitter to provide downlink communication services to cellular users.

[0027] K cellular users (CUs) As the downlink signal receiver of the base station, it receives the downlink signals transmitted by the base station.

[0028] M D2D user groups Each D2D user group consists of one D2D transmitting user and one D2D receiving user pair. Each D2D receiving user pair comprises one strong D2D receiving user and one weak D2D receiving user, with the strong receiving user possessing full-duplex communication capability. The D2D transmitting user performs direct D2D communication by multiplexing the downlink spectrum resources of cellular users, sending D2D communication signals to both the strong and weak receiving users. After receiving and decoding the signals, the strong receiving user forwards them to the weak receiving user via the D2D link, forming a cooperative NOMA transmission mechanism. The number of cellular users and the number of D2D user groups satisfy M>K.

[0029] RIS: Equipped with N×N programmable reflection units, it can regulate the channels of four types of core links in the target communication system by adjusting the phase and amplitude of each programmable reflection unit: downlink from base station to cellular user, D2D link from D2D transmitting user to D2D strong receiving user, D2D link from D2D transmitting user to D2D weak receiving user, and cooperative forwarding link from D2D strong receiving user to D2D weak receiving user.

[0030] S112, Interference type of the target communication system.

[0031] Because D2D users reuse cellular user spectrum resources and multiple D2D users communicate in parallel within the system, the interference in the system is mainly divided into three categories: (1) Interference between cellular users and D2D users: When D2D users reuse cellular user spectrum resources, the downlink signal sent by the base station to the cellular user will cause signal interference to the D2D receiver (strong and weak receiver users). (2) Interference between D2D pairs: Mutual interference between the received signals of strong and weak receiver users in the same D2D receiver pair. (3) Interference between D2D pairs: Mutual signal interference between different D2D receiver pairs that reuse the same cellular user spectrum resources.

[0032] S113, Channel information of the target communication system.

[0033] S1131, Channel information from base station to cellular user.

[0034] Setting up base stations to cellular users The direct link channel gain is The channel vector from the base station to the RIS is: R represents the real number field. It refers to the number of programmable reflection units; RIS to cellular users. The channel vector is Base station to cellular user Total composite channel for: ; in, Here is the reflection matrix of RIS. , ; For the first Adjustable phase offset of a programmable reflective unit.

[0035] S1132, D2D sends channel information from the user to the D2D receiving user.

[0036] Suppose D2D sending user To D2D receiving user (strong / weak) The direct link channel gain is D2D sending user The channel vector to RIS is RIS to D2D receiving user (strong / weak) The channel vector is Then D2D sends to the user. Composite channel to D2D receiving user (strong / weak) for: .

[0037] S114, Signal transmission of the target communication system.

[0038] S1141, Signal transmission and rate for cellular users.

[0039] Setting up base stations to cellular users The signal sent is The transmission power is cellular users The received signal is: ; in, It is from base station to cellular user The total composite channel; It is a D2D sending user and cellular users The channel multiplexing coefficient, where 1 indicates multiplexing and 0 indicates no multiplexing; It is a D2D sending user To cellular users Composite channels; It is a D2D sending user The transmitted signal, It follows a mean of 0 and a variance of . Additive white Gaussian noise.

[0040] Cellular users Signal-to-Interference plus Noise Ratio (SINR) and rate They are respectively: ; ; in, It is a D2D sending user The transmission power.

[0041] CNOMA two-stage signal transmission and rate for S1142 and D2D user groups.

[0042] The cooperative CNOMA transmission process of D2D user groups is divided into two phases: the first phase is the signal superposition transmission phase, and the second phase is the strong user cooperative forwarding phase.

[0043] In the first phase, D2D sends to the user Send superimposed signal : ; in, and It is the power allocation factor, which satisfies ,and . and These are the target signals sent to D2D strong receiving users and D2D weak receiving users, respectively.

[0044] D2D strong receiver user Received signal for: ; in, It is a D2D sending user To D2D strong receiver user Composite channels; It is the first Whether a D2D sending user reuses a cellular user Channel multiplexing factor of the spectrum; It is the first A D2D sending user to a D2D strong receiving user Composite channels; It is the first Each D2D transmitter sends an overlay signal sent by the user; It is a base station to D2D strong receiving user Composite channels; It is a D2D strong receiver user Additive white Gaussian noise.

[0045] D2D weak receiver user Received signal for: ; in, It is a D2D sending user To D2D weak receiver users Composite channels; It is the first A D2D sending user to a D2D weak receiving user Composite channels; It is a base station to D2D weak receiving user Composite channels; It is a D2D weak receiver user Additive white Gaussian noise.

[0046] In the second phase, D2D strong receiver users Successfully decoded the target signal Subsequently, it serves as a decoding and forwarding relay to D2D weak receiving users. Retransmit the signal. D2D strong receiver user. Sending signal for: ; in, For D2D strong receiver users The cooperative transmission power.

[0047] D2D weak receiver user Signal received in the second stage for: ; in, It is a D2D strong receiver user To D2D weak receiver users The D2D cooperative channel includes direct links and reflection links. In the collaboration phase A D2D sending user to a D2D weak receiving user Composite channel gain. It is a D2D weak receiver user Additive white Gaussian noise during the collaboration phase.

[0048] For D2D strong receiver users The SIC (Successive Interference Cancellation) technique is used to first decode the weak received user signal target signal. Then decode its own target signal Decoding weak received user signals and target signals. The SINR at that time was: .

[0049] in, To prevent D2D pair interference at the receiving user side; This is interference from cellular users to D2D strong receiving users.

[0050] The SINR when decoding its own signal is: .

[0051] For D2D weak receiver users It uses maximum ratio combining technology to combine the signals received in the two stages, and the effective SINR after combining is: .

[0052] Among them, D2D inter-pair interference on the strong receiving user side D2D inter-pair interference at the weak receiving user side Interference from cellular users to D2D strong receiving users Interference from cellular users to D2D weak receiver users And interference during the collaboration phase The expressions are as follows: ; ; ; ; ; in, For the first The transmit power of a D2D transmitting user during the signal superposition transmission phase; For the first A D2D sending user to a D2D strong receiving user Composite channels; For the first A D2D sending user to a D2D weak receiving user Composite channels; For the first The transmit power of each D2D transmitting user during the collaboration phase.

[0053] Then the first The rate of D2D user groups It can be represented as: .

[0054] For example, based on any of the above embodiments, in the target communication system, D2D receiving user pairs are dynamically obtained based on a greedy strategy pairing algorithm. The greedy strategy-based user pairing method is an efficient and intuitive approach. It progressively selects the optimal user pair under the current conditions, thereby achieving a near-optimal solution for global performance. Figure 3 This is a schematic diagram of a pairing strategy provided in this specification, such as... Figure 3 As shown, the pairing steps are as described in steps 1 to 9.

[0055] Step 1: Initialize parameters: Input the set of receiving users, system power parameters, RIS phase parameters, and channel information.

[0056] Step 2, Candidate Pairing Matrix Initialization: Calculate the composite channel consisting of the direct and reflected channels from each receiver user to the transmitter to construct a candidate pairing matrix. Using the receiver user as the row and column index, generate all possible candidate D2D receiver user pairs. Construct a candidate pairing matrix, where each matrix element represents a candidate D2D receiver pair. Initialize the pairing results. .

[0057] Step 3: Calculate the pairing weights: For each candidate D2D receiver pair in the candidate pairing matrix ( ), calculate the phase alignment factor sequentially. Interference inhibition ability factor Power difference factor The joint pairing weight of the user pair is determined by the product of the three factors. : ; Among them, candidate D2D receives user pairs ( Phase alignment factor Based on candidate D2D receiving user pairs ( ) receives users and receiving users The mean channel phase difference at each reflective unit of the reconfigurable smart metasurface is obtained to measure the similarity in phase between the equivalent channels of two users. and Representing the receiving user and receiving users In the reconfigurable smart metasurface Channel phase of a programmable reflection unit This represents the total number of programmable reflection units.

[0058] Candidate D2D receiving user pair ( Interference suppression ability factor Based on candidate D2D receiving user pairs ( ) receives users and receiving users The orthogonality of equivalent channel vectors is obtained, which measures the channel correlation and interference level between two users; and These are the receiving users and receiving users The equivalent channel vector represents the distance between the transmitter and the receiver. and receiving users The effective channel vector obtained by combining direct and reflected links; yes The conjugate transpose of ; Represents the vector norm; This represents the modulus. The interference level is then normalized. This is converted into interference suppression capability, where, It is all The minimum value in, It is all The maximum value in, A larger value indicates less interference.

[0059] Power difference factor Based on candidate D2D receiving user pairs ( ) receives users and receiving users The power demand difference is obtained to measure the degree of difference between the two users in terms of power demand; and These are the receiving users and receiving users The power requirements. In cooperative NOMA, the greater the difference in user power, the more effectively the characteristics of non-orthogonal resource allocation can be utilized to achieve clear signal separation and reduce decoding interference. The larger the value, the greater the difference in power requirements between the two users, and the more suitable it is for pairing strong and weak users in CNOMA.

[0060] In RIS-assisted CNOMA-D2D communication systems, user pairing is a crucial step in achieving efficient resource allocation. Traditional user pairing strategies are primarily based on channel gain, while this embodiment comprehensively considers three key factors: phase alignment, interference level, and power difference factor. The product of these factors is used to measure the pairing compatibility of two users in a RIS-assisted CNOMA system.

[0061] Step 4: Select user pairs to be evaluated: Select a set of candidate D2D receiver user pairs to be evaluated from the candidate pairing matrix. ).

[0062] Step 5: Determine the joint pairing weights Is it the maximum value: If the joint pairing weight of the current user pair is... If the result is not the maximum value in the matrix, return to step 4 to select a new user pair. If the result is the maximum value, perform the subsequent pairing operation.

[0063] Step 6: Store the optimal user pair: Store the candidate D2D receiver pair with the highest joint pairing weight ( Add it to the final pairing result set Pairs.

[0064] Step 7: Remove paired users: Remove paired receiving users from the candidate pairing matrix. and receiving users To avoid duplicate pairings.

[0065] Step 8: Determine the number of remaining users: If the number of remaining receiving users in the candidate pairing matrix is ​​greater than or equal to 2, return to Step 2, reinitialize the candidate pairing matrix based on the remaining users, and iterate the process. If the number of remaining receiving users is less than 2, execute the result output step.

[0066] Step 9: Output pairing results: Output the final pairing result set Pairs, obtaining multiple D2D receiver user pairs that meet the requirements.

[0067] Traditional NOMA user pairing methods primarily rely on channel gain and interference characteristics for user selection. This invention proposes a user pairing strategy that integrates phase matching, interference suppression, and power control. This strategy introduces a user channel phase matching evaluation mechanism, optimizing user pairing selection by quantifying phase alignment, effectively improving decoding success rate. Furthermore, this invention designs a power difference factor based on user power demand differences, ensuring paired users meet CNOMA decoding order constraints while further optimizing the power allocation strategy and improving system performance. Through methods such as... Figure 3The pairing strategy shown can achieve accurate and efficient dynamic matching of D2D receiving user pairs, so that the paired user pairs achieve the best overall performance in terms of phase adaptability, interference suppression capability and power demand matching. This lays the foundation for the joint optimization of subsequent system power allocation, RIS phase modulation and channel allocation, and improves the overall communication performance of the RIS-assisted CNOMA-D2D communication system.

[0068] For example, based on any of the above embodiments, the mathematical expression of the resource allocation model for the target communication system constructed in S1 above is as follows.

[0069] The optimization objective of the resource allocation model is to maximize the total communication rate of all D2D user groups in the target communication system. ; in, For the first In the D2D user group, the D2D sending user and the first Channel multiplexing factor for each cellular user For the first Transmit power of D2D transmitting users in a D2D user group; For the first Cooperative transmit power of D2D strong receiver users in a D2D user group; For the reflection matrix of a reconfigurable smart metasurface, For the first reconfigurable smart metasurface The phase of each programmable reflective unit; For the first The main link power allocation factor in each D2D user group; For the first Secondary link power allocation coefficients in D2D user groups For the first Transmit power of each cellular user; The total number of D2D user groups. For the first Communication rate of D2D user groups; For the first The main link signal-to-interference-plus-noise ratio of each D2D user group; For the first The signal-to-interference-plus-noise ratio of the secondary links of each D2D user group; wherein, the primary link is the direct transmission path from the D2D sending user to the D2D receiving user, and the secondary link is the cooperative forwarding path assisted by the D2D cooperative receiving user; Limiting the minimum signal-to-interference-plus-noise ratio for cellular users Minimum signal-to-interference-plus-noise ratio (SIR) limits for D2D strong receivers and D2D weak receivers Channel multiplexing factor limitation Upper limits on transmit power during the D2D user group direct communication phase and cooperative power during the cooperative relay phase. Base station transmit power limits for each cellular user Limitations on the relationship between primary link power allocation coefficient and secondary link power allocation coefficient for D2D user groups and phase confinement of programmable reflective units in reconfigurable smart metasurfaces. As a constraint on the resource allocation model: St ; in, For the first Signal-to-interference-to-noise ratio per cellular user; The minimum signal-to-interference-plus-noise ratio (SIR) threshold for cellular users. Total number of cellular users; The minimum signal-to-interference-plus-noise ratio (SINORR) threshold for D2D links; The maximum transmit power threshold for D2D transmitting users. This represents the maximum cooperative transmit power for D2D strong receiver users; The maximum transmit power threshold for cellular users; For the first reconfigurable smart metasurface Phase adjustment angle of each programmable reflector unit This represents the total number of programmable reflective units in a reconfigurable smart metasurface.

[0070] For example, based on any of the above embodiments, in S2 above, the resource allocation model is solved by a pre-built reinforcement learning model to obtain the optimal decision variables, including steps S210 to S230.

[0071] S210, Pre-built reinforcement learning model.

[0072] To maximize spectral efficiency by jointly optimizing power allocation, phase setting, and channel allocation under complex channel conditions, an A3TD reinforcement learning model combining A3C and TD3 agents is constructed. This model leverages the multi-threaded parallel capabilities of the A3C agent to explore complex environments and combines this with the efficient optimization capabilities of the TD3 agent in continuous action space to achieve rapid iterative and stable optimization of resource allocation. The A3C agent is configured with multiple first policy networks and multiple first evaluation networks, with the first evaluation networks corresponding to the multiple first policy networks. The TD3 agent is configured with one second policy network and two second evaluation networks.

[0073] The primary task of the A3C agent is to accelerate the learning process through multi-threaded parallel training. Each thread updates its local Actor and Critic network parameters and then synchronizes these updates to the global network. This asynchronous update approach allows the A3C agent to accelerate model training through the collaboration of multiple threads and avoid getting trapped in local optima. The parallelization strategy of the A3C agent not only improves training efficiency but also increases sample diversity, which helps enhance the model's generalization ability.

[0074] TD3 agent is an algorithm for continuous action optimization problems. It improves the stability and efficiency of optimization through a dual Critic network and a delayed update strategy, making it particularly suitable for resource allocation problems involving high-dimensional continuous action spaces. Its task is to optimize resource allocation decisions in a continuous action space. Based on sample data generated by the A3C agent, the TD3 agent's Critic network evaluates the value of each action and calculates the Q-value corresponding to the current action. The TD3 agent's Actor network generates corresponding actions based on feedback from the Critic network. These actions guide the resource allocation process, including power allocation, RIS phase configuration, and channel allocation. By adjusting these resources, the TD3 agent aims to maximize the overall spectral efficiency of the system.

[0075] During training, the A3C agent and the TD3 agent work collaboratively and complement each other. The multi-threaded parallel training of the A3C agent provides the TD3 agent with rich training samples, which help the TD3 agent better learn the characteristics of the environment. The global policy of the A3C agent helps guide the TD3 agent to optimize resource allocation more efficiently, thereby improving spectral efficiency. The Critic network of the TD3 agent guides the A3C agent to update its policy network by evaluating the value of each action, ensuring that the choice of each action produces higher system performance. Through this collaboration, the A3C agent and the TD3 agent can complement each other at different stages, making the overall system training process more efficient and stable, and ultimately achieving the optimization goal.

[0076] S220. The optimization objective of the resource allocation model is equivalently transformed into the cumulative reward maximization problem of a multi-agent Markov decision process.

[0077] In RIS-assisted CNOMA-D2D communication systems, the resource allocation problem involves multi-dimensional joint optimization, including user pairing, power control, RIS phase adjustment, and channel allocation. Traditional optimization methods face limitations such as high computational complexity and slow convergence speed in high-dimensional complex environments. To efficiently solve this problem, this invention transforms the resource allocation problem into a sequential decision-making process in deep reinforcement learning. By modeling it as a multi-agent Markov decision process, it utilizes the parallel exploration and policy optimization capabilities of deep reinforcement learning to achieve distributed resource allocation. D2D users communicate by reusing the spectrum resources of cellular users and optimize signal transmission paths through RIS reflection. The optimization objective is to maximize spectral efficiency, i.e., to maximize the total transmission rate of the system, defined as: ; in, Indicates the first The transmission rate of each D2D user group is determined by power allocation. RIS phase setting and channel allocation Decide, This is the set of all D2D user groups.

[0078] The reward function is directly set to the total spectral efficiency at the current moment, encouraging the system to increase transmission rates through resource allocation. (The above...) In the Each time step can be characterized as Then time step / training step Reward value This represents the current total transmission rate. By setting the reward as a value of spectral efficiency, the agent can learn strategies to improve spectral efficiency.

[0079] Current status This includes information describing the current network environment: ; in, Indicates receiving user and receiving users The channel gain matrix, This indicates the current reflection phase configuration of the RIS. It is the receiving user The power allocation value, It is the receiving user The channel allocation state. The state space contains all the key factors affecting the transmission rate and serves as the input to the reinforcement learning model's decision-making.

[0080] Action space for: ; in, The adjustment amount for power distribution. It is the RIS phase adjustment amount. This is the adjustment amount for channel allocation. The action space is a continuous value, covering the adjustment range of three types of resources, making it suitable for optimization using TD3.

[0081] S230. Maximizing the cumulative reward is taken as the training objective of the reinforcement learning model. An asynchronous training and alternating update training mechanism is used to train the A3C agent and TD3 agent in the reinforcement learning model.

[0082] Figure 4 This document provides a schematic diagram of an A3TD training process, such as... Figure 4 As shown, the left side illustrates the training process of the A3C agent; the upper right side illustrates the training process of the second policy network in the TD3 agent. The second policy network directly outputs resource allocation actions (power adjustment, RIS phase adjustment, and channel allocation adjustment) for environmental interaction. No noise is added during this training phase to ensure the determinism of the action output and the stability of gradient descent; the lower right side illustrates the training process of the second evaluation network in the TD3 agent. To avoid overestimating the value and enhance the robustness of the algorithm, truncated Gaussian noise is added to the actions output by the second policy network. This noise smooths the update trajectory of the target value function, effectively preventing the training process from getting stuck in local optima or oscillating. The specific training process is shown in steps I and IV.

[0083] I. Asynchronously train multiple first policy networks and multiple first evaluation networks in the A3C agent.

[0084] Current state Input each first policy network, and output multiple first actions accordingly. ; each first action It operates on the target communication system to obtain the reward value for each first action. and the next step status The A3C agent is responsible for generating diverse sampled data. Its first policy network optimizes the probability distribution of action selection by maximizing the objective function. ; ; in, The objective function of the first policy network; To generate the first action The probability of; These are the parameters for the first policy network; The advantage function measures the degree of improvement of the current action relative to the average behavior of the policy. This is a discount factor used to balance current and future rewards; and Both are state value functions. To proceed to the next state Starting from the current strategy, the expected sum of all future discount rewards is determined by the current action, i.e., the state in the next step. The expected long-term cumulative reward that the A3C agent will obtain in the future; To start from the current state Starting from the current strategy, the expected sum of all future discount rewards is calculated in the current state. The expected long-term cumulative reward that the A3C agent will obtain in the future.

[0085] Minimize the objective function of the first strategy network To achieve the training objective of increasing the probability of actions with higher advantage functions being selected, the parameters of the first policy network and the first evaluation network are updated as follows: ; in, The learning rate of the first-policy network. For the first strategy network loss function with respect to parameters The gradient.

[0086] Each first evaluation network obtains its state function by minimizing the mean square error: The first evaluation network assesses the long-term benefits of the current state, serving as an optimization reference for the first policy network. By minimizing prediction error, the first evaluation network can provide a more accurate value assessment; its parameter update formula is as follows: ; in, The parameters are those of the first evaluation network; The learning rate is used to evaluate the network first. The first evaluation network loss function is used to evaluate the parameters. The gradient is calculated. Multiple threads asynchronously update the global network parameters, improving the model's generalization ability through diverse samples.

[0087] II. Store multiple state transition samples generated during the asynchronous training of the A3C agent into the experience pool. : .

[0088] III. Sample the state transition samples in the experience pool, and then use the sampled state transition samples as a basis for... , The total number of samples, For the first The current time in each sample state, For the first In each sample, in state The action to be performed For the first In the samples, the action was performed The instant reward obtained afterward For the first Actions are performed in each sample. The next moment The state of the agent. The second policy network and two second evaluation networks in the TD3 agent are updated alternately. This includes: (1) Fix the parameters of the second strategy network and update the parameters of the two second evaluation networks.

[0089] Transition the sampled state to the next state in the sample. Input the second policy network in the TD3 agent and output the next initial action. In the next initial action Add Gaussian noise To obtain the next action The next state will be determined separately. and the next step Inputting the two second evaluation networks in the TD3 agent yields the corresponding predicted values ​​of the first action. Second action value prediction .

[0090] The minimum of the first action value prediction and the second action value prediction. As the predicted value of the target action, based on the discount factor After discounting the predicted value of the target action, it is added to the reward value in the sampled state transition samples to obtain the target value. : .

[0091] Two loss functions for the second evaluation network are constructed based on the differences between the predicted value of the first action and the target value, and the differences between the predicted value of the second action and the target value, respectively: ; ; in, and These are two independent evaluation networks in the TD3 agent. The TD3 agent uses two evaluation networks to solve the overestimation problem: the two networks are trained independently to estimate the value of the same state-action pair, and then the minimum value predicted by the two networks is taken as the target, so as to avoid the optimistic estimate of a single network being amplified, thereby improving the robustness of value estimation. and Two second evaluation networks respectively and The parameters.

[0092] The dual-evaluation network improves the robustness of the value function by selecting the minimum target value to avoid overestimation. The training objective of the evaluation network is to minimize the error between the predicted and target values, and gradient descent is used to update the parameters of the two second evaluation networks. and : ; ; in, The learning rate is used for the second evaluation network; The loss function of the second evaluation network is applied to the parameters. gradient, =1,2.

[0093] (2) Fix the parameters of the two second evaluation networks and update the parameters of the second policy network.

[0094] The current state in the sampled state transition sample. Input the second policy network into the TD3 agent and output the second action. ,in, In TD3 Network parameters. Current state. Second action Input any second evaluation network in the TD3 agent to obtain the predicted action value of the second action. The loss function of the second policy network is constructed based on the predicted action value of the second action: .

[0095] The training objective is to maximize the loss function of the second policy network, learn to generate optimal actions, and maximize the overall spectral efficiency of the system. This is achieved by using gradient ascent to optimize the parameters of the second policy network. Update: .

[0096] The TD3 agent improves training stability through delayed updates. The second evaluation network updates every step, and the second policy network updates every two steps. ; ; in, The parameters are those of the local policy network, which shares the same parameters as the second policy network. During training, the local policy network does not directly participate in gradient calculation, but only iterates using the parameters of the second policy network. For the first The parameters of the local evaluation network are updated in the same way as the local policy network. This is a soft update factor used to control the network's update rate. The delayed update strategy reduces training oscillations caused by unstable estimations from the second evaluation network in the second policy network.

[0097] IV. Repeat steps I to III until the cumulative reward converges. Use the second action generated by the second policy network when the cumulative reward converges as the optimal decision variable.

[0098] To address the high-dimensional non-convexity of the multi-dimensional resource joint optimization problem of phase matching, channel allocation, and power control in the CNOMA-RIS-D2D system, this invention proposes the A3TD reinforcement learning algorithm. Combining the multi-threaded asynchronous parallel exploration capability of A3C with the stable optimization characteristics of TD3 in the continuous action space, this algorithm improves sample diversity and exploration efficiency through parallel exploration and alleviates the overestimation problem of value through a dual Critic network, thereby achieving rapid convergence and stable optimization of D2D channel allocation, power control, and RIS phase configuration.

[0099] Furthermore, this specification also provides simulation results and performance analysis of the above methods.

[0100] The performance of the proposed algorithm was verified through system simulation experiments. The simulation platform was developed using Python, and the system model was built and trained using the deep learning platform PyTorch. The performance of the proposed A3TD algorithm in the RIS-assisted CNOMA-D2D communication system was comprehensively evaluated, and it was compared and analyzed with three benchmark algorithms: TD3, DDPG, and AC. This invention selected three baseline algorithms for comparison: (1) The AC baseline algorithm adopts an Actor-Critic architecture, uses an online experience pool for sample collection, and each agent learns independently. (2) The DDPG baseline algorithm adopts an experience replay and target network mechanism, uses a soft update strategy to update network parameters, and is suitable for resource allocation optimization in continuous action space. (3) The TD3 baseline algorithm introduces a dual-Critic network and a delayed policy update mechanism on the basis of DDPG. The dual-Critic network alleviates the value overestimation problem and improves training stability.

[0101] This invention uses sum rate, energy efficiency, and convergence time as performance evaluation metrics. Sum rate represents the total transmission rate of the system. Energy efficiency represents the transmission rate per unit of energy consumption, and the calculation formula is: ; in, This represents the total power consumption of the system. Convergence time is defined as the number of training rounds required for the algorithm to reach stable performance. It is calculated using the moving average method, which is the moving average of the sum and the rate. The algorithm is considered to have converged when the moving average changes less than a threshold within 200 consecutive rounds.

[0102] In the system settings, D2D user locations are randomly generated, and the RIS is deployed at the cell center. When calculating the SINR of the D2D communication link, the effects of strong-power NOMA and weak-power NOMA components are considered. Specific simulation parameter settings are shown in Table 1.

[0103] Table 1 Simulation Parameters

[0104] Figure 5 This document provides a schematic diagram illustrating the relationship between the number of D2D pairs and the convergence time, as shown below. Figure 5 As shown, from the perspective of convergence time, all algorithms exhibit a trend of significantly slower convergence speed as the number of D2D pairs increases, but the magnitude of the decrease varies significantly among different algorithms. The A3TD algorithm consistently maintains the shortest convergence time and optimal system performance across the entire range of D2D pairs, and its advantage over the baseline algorithm gradually expands as the number of D2D pairs increases. In contrast, the AC algorithm has lower learning efficiency in high-density scenarios, DDPG is unstable in multi-agent environments, and although TD3 improves upon DDPG, its exploration efficiency in large-scale scenarios remains insufficient. In dense network scenarios with a large number of D2D pairs, A3TD's convergence speed advantage over the baseline algorithm can shorten the training time by approximately 30%-45%. As the number of D2D pairs increases, the environmental complexity faced by each agent increases. The asynchronous parallel training mechanism of the A3TD algorithm allows multiple agents to explore the environment in parallel, breaking the temporal correlation between samples and quickly covering the state space. The dual Critic network effectively alleviates the overestimation problem of the value function by taking the minimum of the two Q-value estimates, making the policy update more stable and conservative, and avoiding policy oscillations caused by overestimation.

[0105] Figure 6 This is a schematic diagram illustrating the relationship between the spectral efficiency and convergence speed of various algorithms provided in this specification. Figure 6The relationship between the spectral efficiency and convergence speed of each algorithm is demonstrated. The training curves clearly show significant differences in convergence speed, training stability, and final performance among the four algorithms. A3TD exhibits the fastest convergence speed and best stability. The AC algorithm, due to its high variance, shows drastic fluctuations in the early stages of training and a significant performance decline after convergence, a typical manifestation of policy drift. DDPG and TD3 significantly improve training stability through experience replay and target networks, while TD3's dual-Critic architecture further enhances its final performance. The performance gap between A3TD and TD3 is mainly attributed to the contribution of asynchronous parallel exploration: traditional serial training, limited by a single exploration trajectory, is prone to getting trapped in local optima; while asynchronous parallel training, through the independent exploration of multiple agents, can more comprehensively cover the state-action space and discover better global policies. This performance ladder verifies the effectiveness and superiority of the A3TD hybrid architecture in the optimization of complex communication systems.

[0106] In summary, this invention proposes a joint resource optimization scheme based on deep reinforcement learning for RIS-assisted CNOMA-D2D communication systems, aiming to maximize system performance and rate. By designing the A3TD hybrid reinforcement learning algorithm, end-to-end joint optimization of multi-dimensional resources such as D2D channel allocation, power control, and RIS phase configuration is achieved. This scheme transforms the complex non-convex mixed-integer optimization problem into a Markov decision process, leveraging the powerful fitting ability of deep neural networks and the adaptive optimization characteristics of reinforcement learning, thus avoiding the strict assumptions about problem structure and high computational complexity limitations of traditional optimization methods. Simulation experiments comprehensively evaluate the superiority of the proposed A3TD algorithm.

[0107] The simulation experiments above demonstrate that the A3TD algorithm significantly outperforms baseline algorithms such as AC, DDPG, and TD3 in terms of convergence speed, system performance, and training stability. It achieves joint optimization of multi-dimensional resources such as D2D channel allocation, power control, and RIS phase configuration, thus verifying the feasibility of the proposed algorithm.

[0108] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0109] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A D2D communication resource allocation method, characterized in that, include: With the goal of maximizing the total communication rate of all D2D user groups in the target communication system, and with the power allocation parameters, reconfigurable smart metasurface phase parameters, and channel allocation parameters of the target communication system as decision variables, a resource allocation model is constructed. The resource allocation model is solved using a pre-built reinforcement learning model to obtain the optimal decision variables, and communication resources are allocated based on these optimal decision variables. The reinforcement learning model includes: The A3C agent generates multiple state transition samples based on multiple configured first policy networks. The state transition samples include the first action representing the decision variable, the reward value representing the total communication rate, the current state representing the current resource allocation status of the target communication system, and the next state representing the resource allocation status of the target communication system after the action is executed. The experience pool is used to store multiple state transition samples generated by the A3C agent; The TD3 agent is equipped with a second policy network, which generates a second action based on state transition samples obtained from the experience pool. The A3C agent is also configured with multiple first evaluation networks, which correspond to multiple first policy networks; the TD3 agent is also configured with two second evaluation networks; the process of solving the resource allocation model using a pre-built reinforcement learning model specifically includes: The optimization objective of the resource allocation model is equivalently transformed into the cumulative reward maximization problem of a multi-agent Markov decision process; Maximizing cumulative reward is used as the training objective of the reinforcement learning model. An asynchronous training and alternating update mechanism is employed to train the A3C and TD3 agents in the reinforcement learning model, including: I. Asynchronous training of multiple first policy networks and multiple first evaluation networks in the A3C agent; II. Store multiple state transition samples generated during the asynchronous training of the A3C agent into the experience pool; III. Sample the state transition samples in the experience pool, and alternately update the second policy network and two second evaluation networks in the TD3 agent based on the sampled state transition samples, including: fixing the parameters of the second policy network and updating the parameters of the two second evaluation networks; fixing the parameters of the two second evaluation networks and updating the parameters of the second policy network. Repeat steps I to III until the cumulative reward converges. The second action generated by the second policy network when the cumulative reward converges is taken as the optimal decision variable.

2. The D2D communication resource allocation method according to claim 1, characterized in that, The multiple first policy networks and multiple first evaluation networks in the asynchronously trained A3C agent specifically include: Input the current state into each first policy network, and output multiple first actions accordingly; Apply each first action to the target communication system to obtain the reward value and next step status for each first action; Construct the objective function of the first policy network and the state function of the first evaluation network based on the reward value: ； ； ； in, The objective function of the first policy network; This is the first evaluation network state function; The dominant function; This is the current state; The first action; This is the reward value; Discount factor; The next step; and Both are state value functions. Representing the next state The expected long-term cumulative reward that the A3C agent will obtain in the future; Representation in the current state The expected long-term cumulative reward that the A3C agent will obtain in the future; These are the parameters for the first policy network; To generate the first action The probability of; The parameters of the first policy network and the first evaluation network are updated with the training objectives of minimizing the objective function of the first policy network and minimizing the state function of the first evaluation network, respectively.

3. The D2D communication resource allocation method according to claim 1, characterized in that, The parameters of the fixed second policy network are used to update the parameters of the two second evaluation networks, specifically including: The next state from the sampled state transition sample is input into the second policy network in the TD3 agent, and the next initial action is output. Gaussian noise is added to the next initial action to obtain the next action. The next state and the next action are respectively input into the two second evaluation networks in the TD3 agent to obtain the first action value prediction value and the second action value prediction value respectively. The minimum value between the first action value prediction and the second action value prediction is used as the target action value prediction. After discounting the target action value prediction based on the discount factor, it is added to the reward value in the sampled state transition sample to obtain the target value. Two loss functions for the second evaluation network are constructed based on the difference between the predicted value of the first action and the target value, and the difference between the predicted value of the second action and the target value, respectively. The parameters of the two second evaluation networks are updated with the goal of minimizing the loss function of the two second evaluation networks.

4. The D2D communication resource allocation method according to claim 1, characterized in that, The step of fixing the parameters of the two second evaluation networks and updating the parameters of the second policy network specifically includes: The current state from the sampled state transition samples is input into the second policy network in the TD3 agent, and the second action is output. Input the current state and the second action into any second evaluation network in the TD3 agent to obtain the predicted value of the second action. The loss function of the second policy network is constructed based on the predicted action value of the second action; The parameters of the second policy network are updated with the goal of maximizing the loss function of the second policy network.

5. The D2D communication resource allocation method according to claim 1, characterized in that, The second evaluation network updates its parameters at each training step, and the second policy network performs a delayed parameter update every two training steps; the delayed update suppresses training oscillations of the second policy network by reducing the update frequency of the second policy network.

6. The D2D communication resource allocation method according to claim 1, characterized in that, The target communication system includes: At least one base station is used to transmit downlink signals; At least one cellular user is used to receive downlink signals; Multiple D2D user groups are configured, each D2D user group comprising a D2D transmitting user and a D2D receiving user pair, and each D2D receiving user pair comprising a D2D strong receiving user and a D2D weak receiving user. The D2D transmitting user reuses the downlink spectrum resources of the cellular user to transmit D2D communication signals to both the D2D strong receiving user and the D2D weak receiving user. The D2D strong receiving user receives and decodes the D2D communication signals, obtains the decoded data, and forwards the decoded data to the D2D weak receiving user. At least one reconfigurable smart metasurface includes multiple programmable reflective elements. By adjusting the phase and amplitude of each programmable reflective element, the channel of the downlink from the base station to the cellular user, the D2D link from the D2D transmitting user to the D2D strong receiving user, the D2D link from the D2D transmitting user to the D2D weak receiving user, and the cooperative forwarding link from the D2D strong receiving user to the D2D weak receiving user is regulated.

7. The D2D communication resource allocation method according to claim 6, characterized in that, The methods for obtaining the D2D receiving user pairs include: Construct a candidate pairing matrix, wherein the candidate pairing matrix uses the receiving user as the row index and column index, and each matrix cell in the candidate pairing matrix represents a candidate D2D receiving user pair; Based on the phase alignment, interference, and power difference between the two receivers in each candidate D2D receiver pair, the joint pairing weight of each candidate D2D receiver pair is determined. In the candidate pairing matrix, filter and remove the candidate D2D receiver pair with the largest joint pairing weight. Repeat the filtering and removal steps based on a greedy strategy until all receivers are paired or the remaining receivers cannot be paired, thus obtaining multiple D2D receiver pairs.

8. The D2D communication resource allocation method according to claim 7, characterized in that, The determination of the joint pairing weight for each candidate D2D receiver pair based on the phase alignment, interference, and power difference between the two receivers in each candidate D2D receiver pair specifically includes: Based on candidate D2D receiving user pairs ( ) receives users and receiving users The mean channel phase difference at each reflective element of the reconfigurable smart metasurface is used to obtain candidate D2D receiver pairs. Phase alignment factor : ； in, and Representing the receiving user and receiving users In the reconfigurable smart metasurface Channel phase of a programmable reflection unit This represents the total number of programmable reflection units; Based on candidate D2D receiving user pairs ( ) receives users and receiving users The orthogonality of equivalent channel vectors is used to obtain candidate D2D receiver pairs. Interference suppression ability factor : ； in, and These are the receiving users and receiving users The equivalent channel vector; yes The conjugate transpose of ; Represents the vector norm; Indicates the modulus length; Based on candidate D2D receiving user pairs ( ) receives users and receiving users The power demand difference is used to obtain the power difference factor. : ； in, and These are the receiving users and receiving users Power requirements; Phase alignment factor Interference inhibition ability factor and power difference factor The product of the two is used as a candidate D2D receiving user pair ( The joint pairing weights.

9. The D2D communication resource allocation method according to claim 1, characterized in that, The optimization objective is to maximize the total communication rate of all D2D user groups in the target communication system. A resource allocation model is constructed using the power allocation parameters, reconfigurable smart metasurface phase parameters, and channel allocation parameters of the target communication system as decision variables. Specifically, this includes: The optimization objective of the resource allocation model is to maximize the total communication rate of all D2D user groups in the target communication system. ； in, For the first In the D2D user group, the D2D sending user and the first Channel multiplexing factor for each cellular user For the first Transmit power of D2D transmitting users in a D2D user group; For the first Cooperative transmit power of D2D strong receiver users in a D2D user group; For the reflection matrix of a reconfigurable smart metasurface, For the first reconfigurable smart metasurface The phase of a programmable reflective unit; For the first The main link power allocation factor in each D2D user group; For the first Secondary link power allocation coefficients in D2D user groups For the first Transmit power of each cellular user; The total number of D2D user groups. For the first Communication rate of D2D user groups; For the first The main link signal-to-interference-plus-noise ratio of each D2D user group; For the first The signal-to-interference-plus-noise ratio of the secondary links of each D2D user group; wherein, the primary link is the direct transmission path from the D2D sending user to the D2D receiving user, and the secondary link is the cooperative forwarding path assisted by the D2D cooperative receiving user; Limiting the minimum signal-to-interference-plus-noise ratio for cellular users Minimum signal-to-interference-plus-noise ratio (SINORR) limits for D2D strong receivers and D2D weak receivers Channel multiplexing factor limitation Upper limits on transmit power during the D2D user group direct communication phase and cooperative power during the cooperative relay phase. Base station transmit power limits for each cellular user Limitations on the relationship between primary link power allocation coefficient and secondary link power allocation coefficient for D2D user groups and phase confinement of programmable reflective units in reconfigurable smart metasurfaces. As a constraint on the resource allocation model; S.t. ; in, For the first Signal-to-interference-to-noise ratio per cellular user; The minimum signal-to-interference-plus-noise ratio (SIR) threshold for cellular users. Total number of cellular users; The minimum signal-to-interference-plus-noise ratio (SINORR) threshold for D2D links; The maximum transmit power threshold for D2D transmitting users. This represents the maximum cooperative transmit power for D2D strong receiver users; The maximum transmit power threshold for cellular users; For the first reconfigurable smart metasurface Phase adjustment angle of each programmable reflector unit This represents the total number of programmable reflective units in a reconfigurable smart metasurface.

Images