Intelligent metasurface assisted unmanned aerial vehicle communication method, system and device based on deep reinforcement learning, and storage medium

Abstract

The invention discloses an intelligent metasurface assisted multi-unmanned aerial vehicle communication method, system and device based on deep reinforcement learning, and a storage medium. Calculating a signal value, an interference value, a signal-to-noise ratio, system throughput and energy efficiency obtained by the ground user equipment; the method comprises the following steps: constructing a deep reinforcement learning network model in an RIS-assisted multi-unmanned aerial vehicle communication system; the model interacts with the RIS-assisted multi-unmanned aerial vehicle communication system through channel information data to obtain an experience tuple, and the experience tuple is used to train the model; and deploying and implementing the trained model strategy to obtain a final experiment simulation effect. According to the method, the non-orthogonal multiple access communication technology is utilized, channel interference is reduced, deep reinforcement learning is combined, the optimization difficulty is effectively simplified, and the algorithm efficiency is improved; and meanwhile, more practical and more characteristic indexes are considered, and more comprehensive performance evaluation is obtained.

Description

Technical Field

[0001] The present invention belongs to the field of communication technology, and specifically, relates to an intelligent metasurface-assisted drone communication method, system, device and storage medium based on deep reinforcement learning. Background Art

[0002] With the rapid development of 5G networks, unmanned aerial vehicles (UAVs) are increasingly used as aerial base stations, especially in emergency communications, geographic exploration, and disaster relief scenarios, showing important value [1] (see Zeng Y, Zhang R. Energy-efficient UAV communication with trajectory optimization. IEEE Transactions on wireless communications, 2017, 16 (6): 3747-3760.). However, in multi-UAV communication scenarios, the line-of-sight obstruction problem in the air-to-ground channel significantly affects the communication quality. To solve this problem, researchers proposed the reconfigurable intelligent surface (RIS) technology [2] (see Nguyen KK, Khosravirad SR, DaCosta DB, et al. Reconfigurable intelligent surface-assisted multi-UAV networks: Efficient resource allocation with deep reinforcement learning. IEEE Journal of Selected Topics in Signal Processing, 2021, 16 (3): 358-368.). RIS is a programmable structure composed of a large number of passive reflection units that can optimize the signal transmission path by reflecting the signal without the need for an amplifier, thereby improving communication performance.

[0003] Although RIS technology has the potential to improve signal transmission, due to the delay introduced in the RIS reflection process, the existing technology still has certain limitations in meeting the diverse quality of service (QoS) requirements [3] (see Budhiraja I, Vishnoi V, Kumar N, et al. Energy-efficient optimization scheme for RIS-assisted communication underlaying UAV with NOMA, ICC 2022-IEEE International Conference on Communications. IEEE, 2022: 1-6.). In order to further improve the utilization efficiency of communication resources, non-orthogonal multiple access (NOMA) technology came into being [4] (see Nasir AA, Tuan HD, Duong TQ, et al. UAV-enabled communication using NOMA. IEEE Transactions on Communications, 2019, 67(7): 5126-5138.). NOMA effectively reduces interference between users by multiplexing power domains, making the allocation of communication resources more flexible and efficient, especially in multi-user scenarios [5] (see Zhao N, Pang X, Li Z, et al. Joint trajectory and precoding optimization for UAV-assisted NOMA networks. IEEE Transactions on Communications, 2019, 67(5): 3723-3735.).

[0004] At present, NOMA and RIS technologies have become the focus of many studies due to their advantages in improving the performance of communication systems [6] (see Khaleel A, Basar EA novel NOMA solution with RISpartitioning. IEEE Journal of Selected Topics in Signal Processing, 2021, 16(1): 70-81.) and have been widely verified. However, most existing studies focus on idealized scenarios, such as assuming that the ground equipment is stationary [7] (see Wen Z, Na Z, Zhang Y. Energy Efficiency Optimization for RIS-Assisted Uplink-NOMA UAV Network. IEEE Wireless Communications Letters, 2024.), requiring accurate channel state information (CSI), and single UAV assisted communication [8] (see Nguyen KK, Masaracchia A, Sharma V, et al. RIS-assisted UAV communications for IoT with wireless power transfer using deep reinforcement learning. IEEE Journal of Selected Topics in Signal Processing, 2022, 16(5): 1086-1096.). These studies often face the problem of insufficient optimization when dealing with more complex practical application scenarios. Summary of the invention

[0005] Purpose of the invention: One purpose of the present invention is to provide a method for intelligent metasurface-assisted UAV communication based on deep reinforcement learning in a UAV communication network by combining a reshapeable intelligent metasurface with non-orthogonal multiple access technology.

[0006] Another object of the present invention is to provide a system corresponding to the above method.

[0007] Technical solution: The intelligent metasurface-assisted UAV communication method based on deep reinforcement learning of the present invention comprises the following steps:

[0008] S1. Build a RIS-assisted multi-UAV communication system under NOMA technology, define the relevant channel parameters between UAVs, smart metasurfaces and ground user devices, calculate the signal value obtained by the ground user device, and then calculate the inter-cluster interference value and intra-cluster interference value; then calculate the signal-to-noise ratio, system throughput and energy efficiency;

[0009] S2. In the RIS-assisted multi-UAV communication system under NOMA technology, a Markov decision process of reinforcement learning is constructed. Dueling DQN is used to optimize power allocation, UAV path and RIS phase shift strategy to obtain the optimal action strategy, and a reward function is constructed to finally obtain a deep reinforcement learning network model.

[0010] S3. The deep reinforcement learning network model interacts with the RIS-assisted multi-UAV communication system through channel parameter data to obtain the reward value of each time step and generate the state of the next moment. The corresponding reinforcement learning Markov decision process is stored in the experience pool as an experience tuple, and a small batch of samples are randomly taken from the experience pool to optimize the loss function and the state value parameters of the behavior network and the target network and optimize the network parameters, thereby generating the best strategy and realizing the training of the deep reinforcement learning network model.

[0011] S4. Deploy and implement the trained deep reinforcement learning network model strategy to obtain the final experimental simulation effect.

[0012] Furthermore, the relevant channel parameters between the UAV, the intelligent metasurface and the ground user equipment in step S1 include:

[0013] UAV transmission power range: 0≤P u ≤P max , where P u is the transmission power of the u-th UAV, P max is the maximum transmission power of the UAV; the phase shift change matrix of RIS is expressed as in, represents the complex signal after phase modulation on the mth reflection unit at time step t, m∈[0,M], M is the total number of elements in the RIS phase shift change matrix; the channel gain between the UAV and RIS Channel gain between RIS and ground user equipment and the channel gain between the drone and the ground user equipment

[0014] Furthermore, the channel gain between the drone and RIS Channel gain between RIS and ground user equipment and the channel gain between the drone and the ground user equipment The calculation methods are:

[0015]

[0016] Among them, α 0 and β 1 represent the path loss at the reference distance and the path loss index of the UAV and RIS link, respectively. Indicates the distance from the drone to the RIS, Expressed as the deterministic component of the line of sight, the expression is: is the angle of arrival from the u-th UAV to the RIS, x and λ are the antenna spacing and carrier wavelength, respectively, and T is the normalized length of the time step; represents the distance from RI to ground user equipment k, α 1 represents the Rician factor, β 2 represents the path loss exponent of the link between RIS and ground users, Expressed as the deterministic component of non-line-of-sight, is the deterministic component of the line of sight; represents the transmission angle from RIS to the kth ground user; represents the distance from the UAV to the ground user device k, β 3 represents the path loss exponent of the link between the UAV and the ground user, represents small-scale fading designed by a complex Gaussian distribution with zero mean and unit variance.

[0017] Furthermore, the calculation formula of the signal value obtained by the ground user equipment is:

[0018]

[0019] It is represented as the signal value received by the kth ground user equipment from the uth UAV at time step t, s represents the signal sent by the uth UAV, and P u′ is the transmission power of other drones, is the channel gain parameter between other UAVs and ground user equipment except the UAV providing communication services, is the channel gain parameter between other UAVs and RIS except the UAV providing communication services, u′ is other UAVs, k′ is other ground user equipment, P k′ represents the transmission power of the kth ground user equipment, δ is the power noise signal;

[0020] Interference value within cluster Expressed as

[0021] Inter-cluster interference value It is expressed as:

[0022] The signal-to-noise ratio is: in, is the signal-to-noise ratio of the kth ground user equipment in time step t;

[0023] The system throughput is: in, is the throughput of the kth ground user equipment in the uth cluster, and B is the bandwidth;

[0024] Energy efficiency EE is: in, is the throughput of the entire communication system, U is the total number of drones, K u The number of ground user devices in the cluster serving the u-th UAV, is the total energy consumption of the UAV, P M is the energy consumption of RIS, P ckt It is the propulsion energy consumption of the UAV during flight.

[0025] Furthermore, step S2 is further as follows:

[0026] S21. Constructing a Markov decision process for reinforcement learning in the RIS-assisted UAV communication system (S t , A t , R t , S t+1 ), where S t is the state space, A t is the action space, R t is the reward function, S t+1 is the state space at the next moment;

[0027] The channel gains between the UAV and RIS, RIS and ground user equipment, and UAV and ground user equipment, as well as the corresponding signal interference and energy consumption of the UAV are taken as state inputs to obtain the state space: in, is the channel gain vector between the UAV and RIS, is the channel gain vector between RIS and ground user equipment, is the channel gain vector between the UAV and the ground user equipment, The vector of intra-cluster interference values, is the inter-cluster interference value vector, is the total energy consumption of the UAV;

[0028] S22. Use Dueling DQN to optimize the power allocation, drone path, and RIS phase shift strategy to obtain the optimal action strategy A.

[0029] The action space is represented as: A t =(L u , P u , ψ M ), where L u is the flight direction of the drone, specifically (South, East, North, West, Hover), P u represents the transmission power of the u-th UAV, expressed as (P 1 , P 2 , ..., P n ), where P i It is represented by the transmission power selected by the UAV at time step t, ψ M is the phase shift strategy change of RIS, expressed as (ψ 1 , ψ 2 , ..., ψ n ), where ψ i It is represented as the phase shift angle selected by RIS in time step t;

[0030] S23. Construct a reward function based on power allocation, path selection, and RIS phase shift strategy. Among them, α and β represent reward factors, It is expressed as the throughput generated at time step t, EE t It is expressed as the energy efficiency of the system at time step t, and η represents the penalty factor;

[0031] The constructed deep reinforcement learning network model is:

[0032]

[0033] C5:R Qos ≤R k (t),k∈K u ,

[0034] Where L, P, ψ represent the position coordinate constraint of the UAV, the transmission power constraint of the UAV and the phase shift change constraint of the RIS respectively; h u (t) represents the vertical height of the UAV, h min ,h max Respectively represent the minimum and maximum altitude of the drone, x u (t) and y u (t) represent the x-coordinate and y-coordinate of the drone in horizontal space, respectively. min , x max ,ymin ,y max Respectively represent the boundary values ​​of the drone in the horizontal space, Φ m represents the angle of RIS phase shift change, Indicates the threshold value of signal interference, R k (t) represents the signal rate received by the kth ground user equipment at time step t, R Qos It is the service quality threshold of receiving rate.

[0035] Furthermore, step S3 is further as follows:

[0036] S31. Channel gain, signal interference, and energy consumption are used as state space inputs into the deep reinforcement learning network model, where the state space is specifically expressed as

[0037] S32, after obtaining the state space, interact with the RIS-assisted UAV communication system and execute the action space At=(L u , P u , ψ M ), where L u is the flight direction of the drone, specifically (South, East, North, West, Hover), P u represents the transmission power of the u-th UAV, expressed as (P 1 , P 2 , ..., Pn), ψ M is the phase shift change of RIS, expressed as (ψ 1 , ψ 2 , ..., ψ n ) and then obtain the reward generated at the current time step t and the state information at the next moment;

[0038] S33, the state space, action space, reward function and the state space at the next moment (S t , A t , R t , S t+1 ) as the Markov model experience tuple of reinforcement learning, stored in the experience pool, and repeating steps S31 to S33 until the experience pool is full;

[0039] S34, randomly extracting an experience tuple e from the experience pool, and then updating the parameters of the Dueling DQN network based on the extracted experience tuple using the gradient descent method;

[0040] S35. Based on the extracted experience tuples, construct the loss function of the Dueling DQN network, and update the parameters ψ′ of the Dueling DQN network by minimizing the loss function of the Dueling DQN network.q and ψ′ w , specifically expressed as

[0041] ψ q ′←τψ q +(1-τ)ψ q ′,

[0042] ψ w ′←τψ w +(1-τ)ψ w ′;

[0043] Among them, ψ′ q is the state value parameter of the target network, τ is the hyperparameter of soft update, ψ q is the state value parameter of the behavior network, ψ′ w is the dominant network parameter of the target network, ψ w is the dominant network parameter of the behavioral network;

[0044] S36. Based on the Dueling DQN network with updated parameters, a trained deep reinforcement learning network model is obtained;

[0045] S37. Use the trained deep reinforcement learning network model to obtain the optimal power allocation, path planning and phase shift strategy.

[0046] Furthermore, step S4 is further as follows:

[0047] S41. Deploy the trained model on a computer equipped with an i9 processor and an RTX 3060 graphics card.

[0048] S42. Export the trained reinforcement learning policy into a deployable format and ensure that the relevant model parameters are frozen to prevent accidental modifications during deployment;

[0049] S43. Deploy the reinforcement learning strategy to the simulation platform and set the corresponding reinforcement learning hyperparameters to perform actual simulation.

[0050] The system corresponding to the method includes:

[0051] The parameter calculation unit is used to build the RIS-assisted multi-UAV communication system under the NOMA technology, define the relevant channel parameters between the UAV, the intelligent metasurface and the ground user equipment, and calculate the signal value obtained by the ground user equipment, and then calculate the inter-cluster interference value and the intra-cluster interference value; and then calculate the signal-to-noise ratio, system throughput and energy efficiency;

[0052] The deep reinforcement learning network model building unit is used to build a Markov decision process of reinforcement learning in the RIS-assisted multi-UAV communication system under NOMA technology, use Dueling DQN to optimize power allocation, UAV path and RIS phase shift strategy, obtain the optimal action strategy, and build a reward function to finally obtain a deep reinforcement learning network model;

[0053] Model training unit: The deep reinforcement learning network model interacts with the RIS-assisted multi-UAV communication system through channel parameter data to obtain the reward value of each time step and generate the state of the next moment. The corresponding reinforcement learning Markov decision process is stored in the experience pool as an experience tuple, and small batch samples are randomly taken from the experience pool to optimize the loss function and the state value parameters of the behavior network and the target network, and optimize the network parameters, so as to generate the best strategy and realize the training of the deep reinforcement learning network model.

[0054] The experimental simulation unit is used to deploy and implement the trained deep reinforcement learning network model strategy to obtain the final experimental simulation effect.

[0055] An electronic device for storing and executing the method, comprising:

[0056] A memory storing executable program code;

[0057] a processor coupled to the memory;

[0058] The processor calls the executable program code stored in the memory to execute the steps of the RIS-assisted drone communication method based on deep reinforcement learning.

[0059] A computer-readable storage medium for storing and executing the method, wherein the computer-readable storage medium stores computer instructions, and when the computer instructions are called, they are used to execute the steps of the RIS-assisted drone communication method based on deep reinforcement learning.

[0060] Beneficial effects: Compared with the prior art, the present invention has the following significant technical effects: (1) Compared with the ideal situation considered by most UAV-assisted communication models, the scheme considered by the present invention has a time-varying complex environment. At each time step t, the position of the UAV and the array of RIS will be optimized in real time according to the reward value given by the environment at the current time step t; (2) Compared with the problems of traditional methods (such as sequential convex approximation) in terms of calculation and complexity, the present invention proposes to use the deep reinforcement learning (DRL) method to solve the joint trajectory-phase shift problem of RIS-assisted UAVs, define the trajectory optimization of UAVs as a discrete learning problem, and use multi-agent Dueling to solve the joint trajectory-phase shift problem of RIS-assisted UAVs. The DQN algorithm enables the algorithm to converge quickly and be stable, and obtain the optimal solution to the joint problem; (3) Most of the existing reinforcement learning UAV-assisted communication methods consider optimizing a single problem, such as simply optimizing throughput or energy efficiency. The present invention considers two optimization objectives at the same time, and optimizes the throughput and energy efficiency of the communication system as reward values ​​at the same time, which is more in line with reality; (4) When implementing RIS passive phase shift, the existing technology does not fully consider the real-time position change of the three-dimensional trajectory of the online learning UAV; the present invention optimizes the RIS passive phase shift in the DRL process, so that the RIS can align different signals according to the three-dimensional trajectory of the online learning of the UAV and the position of the ground user, thereby maximizing the data transmission rate; compared with the existing technology, the RIS passive phase shift of the present invention more effectively realizes signal redirection and significantly improves the performance of air-to-ground wireless communication; (5) The RIS-assisted UAV communication system based on deep reinforcement learning proposed by the present invention is highly adaptable and easy to expand. It can form a UAV swarm by appropriately increasing the number of UAVs to form a UAV-assisted communication network, which has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 is a flow chart of the method of the present invention;

[0062] Figure 2 Schematic diagram of signal transmission of RIS-assisted UAV communication system in an embodiment of the present invention;

[0063] Figure 3 It is a schematic diagram of position changes of multiple UAVs during real-time communication in the simulation experiment of the present invention;

[0064] Figure 4 is a schematic diagram of real-time position changes of drones using different reinforcement learning algorithms in the simulation experiment of the present invention;

[0065] Figure 5 is a throughput performance result diagram of a communication system in a simulation experiment of the present invention;

[0066] Figure 6 is a graph showing the data rate results of a communication system in a simulation experiment of the present invention;

[0067] Figure 7 It is a schematic diagram of the energy efficiency performance of the communication system in the simulation experiment of the present invention. DETAILED DESCRIPTION

[0068] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0069] In the specific implementation process, the various parameters of the RIS-assisted multi-UAV communication system under NOMA technology are first defined, including the channel gain parameters between UAVs and ground user equipment, RIS and ground user equipment, and UAVs and RIS, as well as the phase shift strategy of RIS, etc. These definitions have an important impact on the subsequent signal transmission processing and resource allocation. Then, a reinforcement learning (DRL) algorithm model based on dueling network (Dueling DQN) is designed. The model can more effectively evaluate the resource allocation strategy by estimating the state value and advantage function respectively. Under different channel environments, the model can dynamically learn the resource allocation strategy to adapt to the changing flight state of the UAV and the needs of the ground user equipment. Through the interaction of state information between the reinforcement learning algorithm model based on Dueling DQN and the RIS-assisted multi-UAV communication system under NOMA technology, experience tuples (including state, action, reward and next state) are obtained. Finally, these experience tuples are used to train the reinforcement learning algorithm model based on Dueling DQN so that it converges to the optimal action execution strategy, thereby optimizing the overall performance of the UAV communication system.

[0070] NOMA is a technology that supports multi-user sharing of communication resources, allowing multiple drones to transmit data under the same time and frequency resources. By building a multi-drone communication system assisted by RIS, resource utilization and spectrum efficiency can be further improved. Since the reinforcement learning algorithm model based on Dueling DQN combines deep neural networks and reinforcement learning algorithms, it can perform more efficient resource allocation optimization in complex environments. In this embodiment, the Dueling DQN reinforcement learning algorithm model calculates the state value and advantage function respectively to more accurately evaluate the action strategy under each state and optimize the system performance. The experience tuple includes the action strategy and corresponding performance feedback obtained from the system. These data will be used to further train the Dueling DQN reinforcement learning algorithm model to generate the best action strategy.

[0071] By combining NOMA technology, RIS technology and multi-UAV collaborative communication technology, the present invention can more effectively utilize communication spectrum resources and improve overall spectrum efficiency. The reinforcement learning algorithm model based on Dueling DQN can adaptively allocate resources according to the real-time channel status to maximize system performance. By continuously training the reinforcement learning algorithm model based on Dueling DQN, it is possible to learn a better resource allocation strategy, improve communication quality and reduce latency. The RIS-assisted multi-UAV communication system based on deep reinforcement learning not only improves the overall performance but also reduces resource waste, providing new ideas for the development of future UAV communication technology.

[0072] like Figure 1 As shown, the intelligent metasurface-assisted UAV communication method based on deep reinforcement learning of the present invention comprises the following steps:

[0073] S1. Construct a RIS-assisted multi-UAV communication system based on NOMA technology and define the relevant channel parameters between UAVs, smart metasurfaces, and ground user equipment; including:

[0074] S11. Construct RIS-assisted multi-UAV communication system and define the parameters of RIS-assisted multi-UAV communication system, including the transmission power range of UAV: ​​0≤P u ≤P max , where P u is the transmission power of the u-th UAV, P max is the maximum transmission power of the UAV; the phase shift change matrix of RIS is expressed as in, represents the complex signal after phase modulation on the mth reflection unit at time step t, is the phase at the mth reflection unit at time step t, m∈[0,M], M is the total number of elements in the RIS phase shift change matrix; the channel gain between the UAV and RIS Channel gain between RIS and ground user equipment and the channel gain between the drone and the ground user equipment

[0075] Assume a RIS-assisted communication network model for multiple UAVs. The model consists of a group of UAVs, denoted as U = {1, 2, ..., U}, and a group of ground user devices in each cluster, denoted as K = {1, 2, ..., K u}. The drone provides services to users in the cluster through a non-orthogonal multiple access scheme. Each drone is assigned to a cluster, and each drone is equipped with an antenna to provide services to users in the cluster. The users in each cluster are randomly deployed within the coverage range of the cluster center at the initial moment. However, in some practical scenarios, the ground user equipment is located in a dense environment, and there are various obstacles and objects, such as high-rise buildings, walls, and concrete, which block the channel between the drone and the ground user equipment. Therefore, in such a complex environment, the received signal of the ground user equipment suffers from significant attenuation and severe link loss. In order to improve the communication quality, RIS is installed on the wall of a high-rise building to improve the communication quality by reflecting the drone signal to the ground user equipment. In order to improve network performance, this embodiment deploys a RIS consisting of M elements. At each time step t, the three-dimensional coordinates of the drone in the u-th cluster are The location of the kth ground user equipment is The RIS component is fixed on a building or high place, and its coordinates are Therefore, at each time step t, the distance between UAV u and RIS element m is calculated as follows:

[0076]

[0077] At the same time, the distance between RIS and ground users is calculated as:

[0078]

[0079] Due to the presence of tall buildings, surrounding objects and obstacles, it is difficult to establish direct communication between drones and ground users. The reflected signal from the RIS panel can be used to build an indirect link between the drone and the ground user equipment. The link between the drone and RIS is regarded as an air-to-air channel (AAC), while the link between RIS and the ground user equipment can be modeled as an air-to-ground channel (AGC).

[0080] Channel gain between drone and RIS Channel gain between RIS and ground user equipment and the channel gain between the drone and the ground user equipment The calculation methods are:

[0081]

[0082] Among them, α 0 and β 1 Represent the path loss at the reference distance of 1m and the path loss index of the UAV and RIS link, Indicates the distance from the drone to the RIS, Expressed as the deterministic component of the line of sight, the expression is: is the angle of arrival from the u-th UAV to the RIS, x and λ are the antenna spacing and carrier wavelength, respectively, and T is the normalized length of the time step; represents the distance from RIS to ground user equipment k, α 1 represents the Rician factor, β 2 represents the path loss exponent of the link between RIS and ground users, Expressed as the deterministic component of non-line-of-sight, is the deterministic component of the line of sight; represents the transmission angle from RIS to the kth ground user; represents the distance from the UAV to the ground user device k, β 3 represents the path loss exponent of the link between the UAV and the ground user, represents small-scale fading designed by a complex Gaussian distribution with zero mean and unit variance.

[0083] S12. Calculate the signal value obtained by the ground user equipment based on the relevant parameter information of the RIS-assisted multi-UAV communication system Then calculate the inter-cluster interference value And the interference value within the cluster Then calculate the signal-to-noise ratio and system throughput and energy efficiency EE;

[0084] Under NOMA technology, since the RIS reflects the signal from the UAV, the signal received by the kth ground user device is the sum of the direct signal from the UAV to the ground user device and the signal reflected from the RIS. So at time step t, the signal value received by the kth ground user device from the uth UAV is It is expressed as:

[0085]

[0086] in, It is represented by the signal value received by the kth ground user equipment from the uth UAV at time step t, and represents the signal sent by the uth UAV, P u′ is the transmission power of other drones, is the channel gain parameter between other UAVs and ground user equipment except the UAV providing communication services, is the channel gain parameter between other UAVs and RIS except the UAV providing communication services, u′ is other UAVs, k′ is other ground user equipment, P k′ represents the transmission power of the kth ground user equipment, δ is the power noise signal, which obeys the complex normal distribution.

[0087] Assume that the signal of the u-th UAV is received by the k-th ground user equipment in the corresponding u-th cluster, and the signals from other UAVs to the k-th ground user equipment are considered as the interference value obtained by the ground user equipment, which is defined as the intra-cluster interference value It is expressed as: Due to the use of NOMA technology, the user interference caused by other ground user equipment to the ground user equipment k is defined as the inter-cluster interference value It is expressed as:

[0088] Then, at time step t, the signal-to-noise ratio of the kth ground user equipment is defined as:

[0089]

[0090] Therefore, the throughput of the kth ground user equipment in the uth cluster is: Where B is the bandwidth. At time step t, the throughput of the entire communication system needs to be determined by adding the throughput of all ground user devices in all clusters, which is specifically defined as Among them, U is the total number of drones, K u The number of ground user devices in the cluster serving the u-th UAV. Since the UAV will generate corresponding energy consumption during the flight process and the array adjustment changes of RIS, the specific calculation method is as follows: in, is the total energy consumption of the UAV, P M is the energy consumption of RIS, P ckt is the propulsion energy consumption of the drone during flight. Finally, the energy efficiency EE is calculated as:

[0091]

[0092] After having the above communication scheme, the reinforcement learning algorithm model based on Dueling DQN will be introduced for optimization.

[0093] S2. Construct a deep reinforcement learning network model in the RIS-assisted multi-UAV communication system under NOMA technology; including:

[0094] S21. Constructing a Markov decision process (MDP) for reinforcement learning in the RIS-assisted multi-UAV communication system (S t , A t , R t , S t+1 ), where S t is the state space, A t is the action space, R tis the reward function, S t+1 is the state space at the next moment;

[0095] S22. Use Dueling DQN to optimize the power allocation, drone path, and RIS phase shift strategy to obtain the optimal action strategy A.

[0096] S23. Construct a reward function based on power allocation, path selection, and RIS phase shift strategy. That is, the throughput of the RIS-assisted multi-UAV communication system at each moment and energy efficiency EE t . Among them, α, β are reward coefficients, and η is the penalty coefficient.

[0097] In a further embodiment, the channel gains between the UAV and RIS, RIS and ground user equipment, and the UAV and ground user equipment, as well as the corresponding signal interference and the energy consumption of the UAV are taken as state inputs to obtain the state space: in, is the channel gain vector between the UAV and RIS, is the channel gain vector between RIS and ground user equipment, is the channel gain vector between the UAV and the ground user equipment, The vector of intra-cluster interference values, is the inter-cluster interference value vector, is the total energy consumption of the UAV; Dueling DQN algorithm is responsible for optimizing the relevant action strategies of the UAV and RIS. The specific action space is expressed as: A t =(L u , P u , ψ M ). Where L u is the flight direction of the drone, specifically (South, East, North, West, Hover), P u represents the transmission power of the u-th UAV, expressed as (P 1 , P 2 , ..., P n ), where P t It is represented by the transmission power selected by the UAV at time step t, ψ M is the phase shift strategy change of RIS, expressed as (ψ 1 , ψ 2 , ..., ψ n ), where ψ t It is represented as the phase shift angle selected by RIS in time step t. After the drone and RIS perform actions according to the current state, they will receive the corresponding environmental reward R t, that is, the goal that the system needs to optimize: the optimization of the throughput and energy efficiency of the entire communication system, expressed as: Among them, α and β represent reward factors, It is expressed as the throughput generated at time step t, EE t It is expressed as the system energy efficiency at time step t, and η represents the penalty factor.

[0098] The constructed deep reinforcement learning network model is:

[0099]

[0100] C5:R Qos ≤R k , k∈K u ,

[0101] Where L, P, and ψ represent the position coordinate constraint of the UAV, the transmission power constraint of the UAV, and the phase shift change constraint of the RIS, respectively. u (t) represents the vertical height of the UAV, h min ,h max Respectively represent the minimum and maximum altitude of the drone, x u (t) and y u (t) represent the x-coordinate and y-coordinate of the drone in horizontal space, respectively. min , x max ,y min ,y max They represent the boundary values ​​of the drone in the horizontal space, Φ m represents the angle of RIS phase shift change, Indicates the threshold value of signal interference, R k (t) represents the signal rate received by the kth ground user equipment at time step t, R Qos It is the service quality threshold of receiving rate.

[0102] Among them, constraint C1 represents the transmission power range of the UAV. C2 represents the working area limit of the UAV. C3 describes the phase shift angle variation range of the mth element of RIS. C4 represents the interference threshold of the ground user. C5 represents the lower limit of the signal receiving rate of the ground equipment.

[0103] S3, the deep reinforcement learning network model interacts with the RIS-assisted UAV communication system through channel gain information to obtain experience tuples, and uses the experience tuples to train the deep reinforcement learning network model;

[0104] The agent continuously interacts with the RIS-assisted multi-UAV communication system to obtain the reward value at each time step t, and generates the state of the next moment, and converts the corresponding (S t , At , R t , S t+1 ) is stored in the experience pool as an experience tuple, and a small batch of samples is randomly taken from the experience pool to optimize the loss function and the state value parameters of the behavior network and the target network and optimize the network parameters to generate the best strategy.

[0105] Specifically include:

[0106] S31. Channel gain, signal interference, and energy consumption are used as state space inputs into the deep reinforcement learning network model, where the state space is specifically expressed as

[0107] S32, after obtaining the state space, interact with the RIS-assisted multi-UAV communication system and execute the action space A t =(L u , P u , ψ M ). Then we can get the reward generated at the current time step t and the state information at the next moment;

[0108] S33, the state space, action space, reward function and the state space at the next moment (S t , A t , R t , S t+1 ) as the Markov model experience tuple of reinforcement learning, stored in the experience pool, and repeating steps S31 to S33 until the experience pool is full;

[0109] S34, randomly extracting an experience tuple e from the experience pool, and then updating the parameters of the Dueling DQN network based on the extracted experience tuple using the gradient descent method;

[0110] S35. Based on the extracted experience tuples, construct the loss function of the Dueling DQN network, and update the parameters ψ′ of the Dueling DQN network by minimizing the loss function of the Dueling DQN network. q and ψ′ w , specifically expressed as

[0111] ψ q ′←τψ q +(1-τ)ψ q ′,

[0112] ψ w ′←τψ w +(1-τ)ψ w ′;

[0113] Among them, ψ′ qis the state value parameter of the target network, τ is the hyperparameter of soft update, ψ q is the state value parameter of the behavior network, ψ′ w is the dominant network parameter of the target network, ψ w is the dominant network parameter of the behavioral network.

[0114] S36. Based on the Dueling DQN network with updated parameters, a trained deep reinforcement learning network model is obtained;

[0115] S37. Use the trained deep reinforcement learning network model to obtain the optimal power allocation, path planning and phase shift strategy.

[0116] In a further embodiment, each round of interaction between the agent and the RIS-assisted multi-UAV communication system is performed through the state space And adopt ∈ greedy strategy to select action space At=(L u , P u , ψ M ) and executes. Then, the agent obtains immediate environmental reward R according to the system feedback. t This process completes a round of interaction with the system environment and converts the current experience tuple (S t , A t , R t , S t+1 ) is stored in the agent’s experience pool for subsequent training. Through the separation structure of the advantage network and value network of Dueling DQN, the agent can more effectively evaluate the strategies under different states, thereby achieving better strategy selection.

[0117] The Dueling DQN network structure separates the state value flow and the advantage flow to improve the efficiency of resource allocation and the accuracy of decision-making. In the Dueling DQN network, the Q value is represented by the combination of the advantage network A(s, a) and the state value V(s). The formula is as follows:

[0118]

[0119] Among them, Q(s i , a i ψ q ) is the Q value function calculated by the behavior network of Dueling DQN algorithm, V(s i ψ v ) is the state value function, which represents the global value in the current state, A(s i , a i ψ a ) is in the current state s i Next, take the current action ai The advantage function when is for all possible actions a i 'Calculate the sum of the advantage functions to represent the expected advantage of all actions in state s, is the size of the action space, i.e. the total number of possible actions, ψ q represents the Q-value function Q(s i , a i ψ q ), ψ v Represents the state value function V(s i ψ v ), ψ a Denotes the advantage function A(s i , a i ψ a ) parameter set. To optimize the state value network, the loss function is updated by minimizing the error between the actual reward and the estimated Q value. The loss function is defined as follows:

[0120]

[0121] Among them, s i , a i They represent the current state and action respectively, L represents the loss function, E represents the number of small batch samples randomly taken from the experience pool, and finally, the target value y i The calculation formula is:

[0122] y i =r i (s i , a i )+γQ′(s i+1 , a i+1 ψ q ′)|a i+1 =W(s i+1 ψ w ′),

[0123] Here, r i (s i , a i ) is the immediate reward, γ is the discount factor used to measure the importance of future rewards, Q′(s i+1 , a i+1 ψ q ′) is the Q value calculated by the target network, W(s i+1 ψ w ′) is the parameter ψ w ′, the action selection function under i+1 is the action of the next state, s i+1 is the next state, ψq ′,ψ w ′ represent the state value parameter of the target network and the dominant network parameter respectively.

[0124] Furthermore, the parameters of the behavior network are updated based on the gradient descent method of the value network. The parameters of the action network are updated by the following formula:

[0125]

[0126] in, represents the gradient operator, a i Represents the current action, It means Q(s i , a i ψ q ) for action a i The gradient of W(s i ψ w ) for the parameter ψ w The gradient of q , ψ w They represent the state value parameter and advantage network parameter of the behavior network calculation, W(s i ) represents the action selection function, W(s i ψ w ) indicates that the parameter ψ w This means that the action network updates its parameters by extracting information from the gradient of the value network, thereby selecting a better strategy.

[0127] In order to improve the stability of the model, a soft target update strategy is used to update the parameters of the target network. The formula for soft update is as follows:

[0128] ψ q ′←τψ q +(1-τ)ψ q ′,

[0129] ψ w ′←τψ w +(1-ατ)ψ w ′,

[0130] Among them, the value range of the soft update hyperparameter τ is 0 to 1. Through this update strategy, the target network can be more stably and gradually approach the main network, so that the network's strategy selection tends to the same goal, accelerating the convergence of the entire intelligent network.

[0131] S4. Deploy and implement the trained model strategy to obtain the final simulation state effect.

[0132] S41. Based on the trained model, deploy it on a computer equipped with an i9 processor and an RTX 3060 graphics card.

[0133] S42. Export the trained reinforcement learning policy into a deployable format and ensure that the relevant model parameters are frozen to prevent accidental modifications during deployment.

[0134] S43. Deploy the reinforcement learning strategy to the simulation platform and set the corresponding reinforcement learning hyperparameters to perform actual simulation.

[0135] The system corresponding to the method includes:

[0136] The parameter calculation unit is used to build the RIS-assisted multi-UAV communication system under the NOMA technology, define the relevant channel parameters between the UAV, the intelligent metasurface and the ground user equipment, and calculate the signal value obtained by the ground user equipment, and then calculate the inter-cluster interference value and the intra-cluster interference value; and then calculate the signal-to-noise ratio, system throughput and energy efficiency;

[0137] The deep reinforcement learning network model building unit is used to build a Markov decision process of reinforcement learning in the RIS-assisted multi-UAV communication system under NOMA technology, use Dueling DQN to optimize power allocation, UAV path and RIS phase shift strategy, obtain the optimal action strategy, and build a reward function to finally obtain a deep reinforcement learning network model;

[0138] Model training unit: The deep reinforcement learning network model interacts with the RIS-assisted multi-UAV communication system through channel parameter data to obtain the reward value of each time step and generate the state of the next moment. The corresponding reinforcement learning Markov decision process is stored in the experience pool as an experience tuple, and small batch samples are randomly taken from the experience pool to optimize the loss function and the state value parameters of the behavior network and the target network, and optimize the network parameters, so as to generate the best strategy and realize the training of the deep reinforcement learning network model.

[0139] The experimental simulation unit is used to deploy and implement the trained deep reinforcement learning network model strategy to obtain the final experimental simulation effect.

[0140] An electronic device for storing and executing the method, comprising:

[0141] A memory storing executable program code;

[0142] a processor coupled to the memory;

[0143] The processor calls the executable program code stored in the memory to execute the steps of the RIS-assisted drone communication method based on deep reinforcement learning.

[0144] A computer-readable storage medium for storing and executing the method, wherein the computer-readable storage medium stores computer instructions, and when the computer instructions are called, they are used to execute the steps of the RIS-assisted drone communication method based on deep reinforcement learning.

[0145] In the specific simulation environment, Intel i9 processors are used to ensure efficient computing performance to support complex reinforcement learning algorithm models based on Dueling DQN and equipped with NVIDIA RTX 3060 graphics cards to accelerate the reasoning process of the reinforcement learning algorithm model based on Dueling DQN, thereby improving simulation efficiency. The state value network and advantage network trained by the Dueling DQN algorithm are used to ensure that the model weights and structures are frozen after export to prevent accidental modifications during deployment. In terms of hyperparameter design, the model performance is ensured by keeping it consistent with the optimal learning rate parameter during training. The discount factor is set to the discount factor used during training to ensure that the cumulative calculation method of the reward is consistent. During deployment, the ∈ value in the ∈-greedy strategy is appropriately adjusted to balance exploration and utilization. In the early stage, a high exploration rate can be maintained and gradually reduced to improve decision stability. Finally, according to the real-time requirements of the simulation platform, a suitable batch size is set to ensure that the model can process environmental feedback in a timely manner.

[0146] This application takes the throughput and energy efficiency of the entire system as the optimization objectives, so the entire problem can be modeled as:

[0147]

[0148] C5:R Qos ≤R k , k∈K u ,

[0149] The constraint C1 represents the transmission power range of the UAV. C2 represents the working area limit of the UAV. C3 describes the phase shift angle variation range of the mth element of the RIS. C4 represents the interference threshold of the ground user. C5 represents the lower limit of the signal receiving rate of the ground device.

[0150] This application uses a deep reinforcement learning algorithm to solve the problems of drone path planning, RIS phase shift change, and drone power allocation in auxiliary communication systems, and proposes a deep reinforcement learning framework based on Dueling DQN. By dynamically training the state value network and advantage network of the Dueling DQN algorithm, the output strategy of the network is adjusted to maximize the energy efficiency and throughput of the system. Figure 2 The communication method between multiple drones, RIS and ground user equipment is demonstrated. Figure 3 (a) and (b) show the real-time 2D and 3D path changes of multiple drones when providing communication services to ground user equipment. The trajectory change diagrams of drones under different schemes are shown in Figure 1. Figure 4 As shown in (a) and (b), compared with the scheme with RIS but no optimization and the scheme without RIS, the path planning of the drone is more efficient when the DuelingDQN algorithm is used to optimize the drone path, RIS phase shift, and drone power. The throughput and energy efficiency diagrams of the entire communication system are shown in Figure 1. Figure 5 and Figure 7 As shown, it can be seen that the performance is better when DuelingDQN is used to optimize the RIS-assisted multi-UAV system. Figure 6 It shows the data rate received by the ground user equipment. It can be seen that the data rate is higher when the Dueling DQN algorithm is used. Therefore, from the above simulation experiments, it can be seen that the Dueling DQN algorithm proposed in the present invention can be used to optimize the RIS-assisted multi-UAV system, which effectively optimizes the path, power and phase shift strategy of the UAV. The satisfactory simulation results verify the solution designed by the present invention.

[0151] The present invention utilizes non-orthogonal multiple access communication technology to reduce channel interference, and combines deep reinforcement learning to effectively simplify the optimization difficulty and improve the efficiency of the algorithm; at the same time, it considers more practical and characteristic indicators to obtain a more comprehensive performance evaluation.

[0152] The preferred embodiments of the present invention are described in detail above; however, the present invention is not limited to the specific details in the above embodiments. Within the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all belong to the protection scope of the present invention.

Claims

1. An intelligent metasurface-assisted UAV communication method based on deep reinforcement learning, characterized in that: The following steps are involved: S1. Build a RIS-assisted multi-UAV communication system under NOMA technology, define the relevant channel parameters between UAVs, smart metasurfaces and ground user devices, calculate the signal value obtained by the ground user device, and then calculate the inter-cluster interference value and intra-cluster interference value; then calculate the signal-to-noise ratio, system throughput and energy efficiency; S2. In the RIS-assisted multi-UAV communication system under NOMA technology, a Markov decision process of reinforcement learning is constructed. Dueling DQN is used to optimize power allocation, UAV path and RIS phase shift strategy to obtain the optimal action strategy, and a reward function is constructed to finally obtain a deep reinforcement learning network model. S3. The deep reinforcement learning network model interacts with the RIS-assisted multi-UAV communication system through channel parameter data to obtain the reward value of each time step and generate the state of the next moment. The corresponding reinforcement learning Markov decision process is stored in the experience pool as an experience tuple, and a small batch of samples are randomly taken from the experience pool to optimize the loss function and the state value parameters of the behavior network and the target network and optimize the network parameters, thereby generating the best strategy and realizing the training of the deep reinforcement learning network model. S4. Deploy and implement the trained deep reinforcement learning network model strategy to obtain the final experimental simulation effect.

2. The RIS-assisted UAV communication method based on deep reinforcement learning according to claim 1 is characterized in that: The relevant channel parameters between the UAV, the intelligent metasurface and the ground user equipment in step S1 include: UAV transmission power range: 0≤P u ≤P max , where P u is the transmission power of the u-th UAV, P max is the maximum transmission power of the UAV; the phase shift change matrix of RIS is expressed as in, represents the complex signal after phase modulation on the mth reflection unit at time step t, m∈[0,M], M is the total number of elements in the RIS phase shift change matrix; the channel gain between the UAV and RIS Channel gain between RIS and ground user equipment and the channel gain between the drone and the ground user equipment 3. The RIS-assisted UAV communication method based on deep reinforcement learning according to claim 2 is characterized in that: Channel gain between drone and RIS Channel gain between RIS and ground user equipment and the channel gain between the drone and the ground user equipment The calculation methods are: Where α0 and β1 represent the path loss at the reference distance and the path loss index of the UAV and RIS link, respectively. Indicates the distance from the drone to the RIS, Expressed as the deterministic component of the line of sight, the expression is: is the angle of arrival from the u-th UAV to the RIS, x and λ are the antenna spacing and carrier wavelength, respectively, and T is the normalized length of the time step; represents the distance from RI to ground user equipment k, α1 represents the Rician factor, β2 represents the path loss index of the link between RIS and ground user, Expressed as the deterministic component of non-line-of-sight, is the deterministic component of the line of sight; represents the transmission angle from RIS to the kth ground user; represents the distance from the UAV to the ground user device k, β3 represents the path loss index of the link between the UAV and the ground user, represents small-scale fading designed by a complex Gaussian distribution with zero mean and unit variance.

4. The RIS-assisted UAV communication method based on deep reinforcement learning according to claim 1 is characterized in that: The calculation formula of the signal value obtained by the ground user equipment is: It is represented as the signal value received by the kth ground user equipment from the uth UAV at time step t, s represents the signal sent by the uth UAV, and P u′ is the transmission power of other drones, is the channel gain parameter between other UAVs and ground user equipment except the UAV providing communication services, is the channel gain parameter between other UAVs and RIS except the UAV providing communication services, u′ is other UAVs, k′ is other ground user equipment, P k′ represents the transmission power of the kth ground user equipment, δ is the power noise signal; Interference value within cluster Expressed as Inter-cluster interference value It is expressed as: The signal-to-noise ratio is: in, is the signal-to-noise ratio of the kth ground user equipment at time step t; The system throughput is: in, is the throughput of the kth terrestrial user equipment in the uth cluster, B is bandwidth; Energy efficiency EE is: in, is the throughput of the entire communication system, U is the total number of drones, K u The number of ground user devices in the cluster serving the u-th UAV, is the total energy consumption of the UAV, P M is the energy consumption of RIS, P ckt It is the propulsion energy consumption of the UAV during flight.

5. The RIS-assisted UAV communication method based on deep reinforcement learning according to claim 1 is characterized in that: Step S2 is further as follows: S21. Constructing a Markov decision process for reinforcement learning in the RIS-assisted UAV communication system (S t , A t , R t , S t+1 ), where S t is the state space, A t is the action space, R t is the reward function, S t+1 is the state space at the next moment; The channel gains between the UAV and RIS, RIS and ground user equipment, and UAV and ground user equipment, as well as the corresponding signal interference and energy consumption of the UAV are taken as state inputs to obtain the state space: in, is the channel gain vector between the UAV and RIS, is the channel gain vector between RIS and ground user equipment, is the channel gain vector between the UAV and the ground user equipment, The vector of intra-cluster interference values, is the inter-cluster interference value vector, is the total energy consumption of the UAV; S22. Use Dueling DQN to optimize the power allocation, drone path, and RIS phase shift strategy to obtain the optimal action strategy A. The action space is represented as: A t =(L u , P u , ψ M ), where L u is the flight direction of the drone, specifically (South, East, North, West, Hover), P u represents the transmission power of the u-th UAV, expressed as (P1, P2, ..., P n ), where P i It is represented by the transmission power selected by the UAV at time step t, ψ M is the phase shift strategy change of RIS, expressed as (ψ1, ψ2, ..., ψ n ), where ψ i It is represented as the phase shift angle selected by RIS in time step t; S23. Construct a reward function based on power allocation, path selection, and RIS phase shift strategy. Among them, α and β represent reward factors, It is expressed as the throughput generated at time step t, EE t It is expressed as the energy efficiency of the system at time step t, and η represents the penalty factor; The constructed deep reinforcement learning network model is: C5:R Qos ≤R k (t),k∈K u , Where L, P, ψ represent the position coordinate constraint of the UAV, the transmission power constraint of the UAV and the phase shift change constraint of the RIS respectively; h u (t) represents the vertical height of the UAV, h min ,h max Respectively represent the minimum and maximum altitude of the drone, x u (t) and y u (t) represent the x-coordinate and y-coordinate of the drone in horizontal space, respectively. min , x max ,y min ,y max Respectively represent the boundary values ​​of the drone in the horizontal space, Φ m represents the angle of RIS phase shift change, Indicates the threshold value of signal interference, R k (t) represents the signal rate received by the kth ground user equipment at time step t, R Qos It is the service quality threshold of receiving rate.

6. The RIS-assisted UAV communication method based on deep reinforcement learning according to claim 1 is characterized in that: Step S3 is further as follows: S31. Channel gain, signal interference, and energy consumption are used as state space inputs into the deep reinforcement learning network model, where the state space is specifically expressed as S32, after obtaining the state space, interact with the RIS-assisted UAV communication system and execute the action space A t =(L u , P u , ψ M ), where L u is the flight direction of the drone, specifically (South, East, North, West, Hover), P u represents the transmission power of the u-th UAV, expressed as (P1, P2, ..., Pn), ψ M is the phase shift change of RIS, expressed as (ψ1, ψ2, ..., ψ n ) and then obtain the reward generated at the current time step t and the state information at the next moment; S33, the state space, action space, reward function and the state space at the next moment (S t , A t , R t , S t+1 ) as the Markov model experience tuple of reinforcement learning, stored in the experience pool, and repeating steps S31 to S33 until the experience pool is full; S34, randomly extracting an experience tuple e from the experience pool, and then updating the parameters of the Dueling DQN network based on the extracted experience tuple using the gradient descent method; S35. Based on the extracted experience tuples, construct the loss function of the Dueling DQN network, and update the parameters ψ′ of the Dueling DQN network by minimizing the loss function of the Dueling DQN network. q and ψ′ w , specifically expressed as ψ q '←tψ q +(1-t)ψ q ′, ψ ω ′←tψ ω +(1-T)ψ ω '; Among them, ψ′ q is the state value parameter of the target network, τ is the hyperparameter of soft update, ψ q is the state value parameter of the behavior network, ψ′ ω is the dominant network parameter of the target network, ψ ω is the dominant network parameter of the behavioral network; S36. Based on the Dueling DQN network with updated parameters, a trained deep reinforcement learning network model is obtained; S37. Use the trained deep reinforcement learning network model to obtain the optimal power allocation, path planning and phase shift strategy.

7. The RIS-assisted UAV communication method based on deep reinforcement learning according to claim 1 is characterized in that: Step S4 is further as follows: S41. Deploy the trained model on a computer equipped with an i9 processor and an RTX 3060 graphics card. S42. Export the trained reinforcement learning policy into a deployable format and ensure that the relevant model parameters are frozen to prevent accidental modifications during deployment; S43. Deploy the reinforcement learning strategy to the simulation platform and set the corresponding reinforcement learning hyperparameters to perform actual simulation.

8. Intelligent metasurface-assisted UAV communication system based on deep reinforcement learning, characterized in that: include: The parameter calculation unit is used to build the RIS-assisted multi-UAV communication system under the NOMA technology, define the relevant channel parameters between the UAV, the intelligent metasurface and the ground user equipment, and calculate the signal value obtained by the ground user equipment, and then calculate the inter-cluster interference value and the intra-cluster interference value; and then calculate the signal-to-noise ratio, system throughput and energy efficiency; The deep reinforcement learning network model building unit is used to build a Markov decision process of reinforcement learning in the RIS-assisted multi-UAV communication system under NOMA technology, use Dueling DQN to optimize power allocation, UAV path and RIS phase shift strategy, obtain the optimal action strategy, and build a reward function to finally obtain a deep reinforcement learning network model; The model training unit is used for the deep reinforcement learning network model to interact with the RIS-assisted multi-UAV communication system through channel parameter data, obtain the reward value of each time step, and generate the state of the next moment. The corresponding reinforcement learning Markov decision process is stored in the experience pool as an experience tuple, and a small batch of samples are randomly taken from the experience pool to optimize the loss function and the state value parameters of the behavior network and the target network, and optimize the network parameters, so as to generate the best strategy and realize the training of the deep reinforcement learning network model; The experimental simulation unit is used to deploy and implement the trained deep reinforcement learning network model strategy to obtain the final experimental simulation effect.

9. An electronic device, characterized in that: The device comprises: A memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the steps of the RIS-assisted drone communication method based on deep reinforcement learning as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, which, when called, are used to execute the steps of the RIS-assisted drone communication method based on deep reinforcement learning as described in any one of claims 1 to 7.

Images