An RIS-UAV cooperative communication optimization method based on adversarial value calibration

Abstract

An optimization method for RIS-UAV cooperative communication based on adversarial value calibration includes the following steps: Step 1, constructing a cooperative communication model between a Smart Metasurface (RIS) and a Unmanned Aerial Vehicle (UAV) for a dynamic power Internet of Things (IoT) to improve network resource utilization efficiency; Step 2, modeling the optimization problem constructed in Step 1 as a Markov Decision Process (MDP) to provide data input for the subsequent DRL algorithm; Step 3, designing an adversarial value calibration DRL algorithm, which provides a global temporal difference (TD) target distribution through a target generator. The generator approximates this distribution under the dual supervision of TD error and adversarial loss, while the discriminator provides adversarial supervision signals to enhance the calibration effect, thereby significantly suppressing action value estimation fluctuations and improving algorithm stability. This invention optimizes the MDP model established in Step 2 using the algorithm proposed in Step 3 to optimize communication performance. This method provides a new solution and technical path for power grid auxiliary communication research.

Description

Technical Field

[0001] This invention relates to the fields of communication technology and computer technology, and in particular to a RIS-UAV cooperative communication optimization method based on adversarial value calibration. Background Technology

[0002] Unmanned Aerial Vehicles (UAVs), with their high mobility and rapid deployment capabilities, can serve as aerial relays to build flexible and efficient communication links. However, in complex urban environments, communication links between UAVs and ground power equipment are often obstructed by obstacles such as buildings and transmission towers, facing non-line-of-sight (NLS) transmission problems and resulting in a significant decrease in communication quality. Reconfigurable Intelligent Surface (RIS) technology offers a new approach to solving these problems. By adjusting the phase of the reflective unit in real time, it can precisely change the signal propagation direction, thereby establishing a reliable NLS transmission path for obstructed links. Therefore, the RIS-UAV collaborative communication system can provide a highly valuable solution for power emergency communication scenarios. For example, the paper "Performance Optimization of UAV RIS-Assisted Communication Network under No-Fly Zone Constraints" discloses a RIS-UAV cooperative communication technology. This scheme integrates RIS into the UAV platform, jointly optimizing UAV trajectory, RIS phase shift, and base station beamforming, and employs a soft actor-commentator deep reinforcement learning algorithm to ensure user communication while the UAV strictly avoids no-fly zones. However, the aforementioned existing scheme still has the following technical limitations: First, this scheme mainly optimizes communication for users in fixed locations, failing to incorporate the dynamic mobility of mobile power equipment into the joint optimization framework of trajectory planning and phase shift design. When mobile power equipment rapidly changes position, the pre-optimized RIS reflection link gain will quickly decay, making it difficult to guarantee continuous and stable communication quality. Second, this scheme uses a conventional deep reinforcement learning temporal differential update mechanism. In complex interactive environments such as UAV dynamic flight and no-fly zone avoidance, its action value estimation is prone to large fluctuations, leading to slow algorithm convergence, insufficient training stability, and limited improvement in communication rate, making it difficult to meet the dual requirements of real-time performance and reliability for power emergency communication.

[0003] To address the shortcomings of existing solutions in terms of adaptability to mobile power equipment services and stability of deep reinforcement learning algorithms, this invention proposes a RIS-UAV cooperative communication optimization method based on adversarial value calibration. Through a cooperative adversarial learning mechanism between the target generator and discriminator, it provides an overall temporal differential target distribution to constrain value estimation, effectively suppressing action value fluctuations and improving algorithm convergence speed and training stability. This results in superior communication performance in complex dynamic environments. Researching and solving these core challenges is of great significance and value for improving the performance of power emergency communication systems. Summary of the Invention

[0004] This invention addresses the problems of existing power emergency communication technologies, such as the susceptibility of signals to obstruction by complex urban obstacles leading to a sharp drop in communication quality, and the drastic consumption of limited power by simply increasing the transmission power of drones. This makes it difficult for existing cooperative communication systems that incorporate reflector-assisted signals to perform high-dimensional joint optimization of the drone's flight trajectory and the phase shift of the reflector. Furthermore, traditional reinforcement learning algorithms are prone to large fluctuations in action value estimation and difficulty in model stable convergence when dealing with such dynamic high-dimensional spaces, ultimately failing to balance system transmission rate and energy efficiency. Therefore, this invention is proposed to address these technical issues.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A RIS-UAV cooperative communication optimization method based on adversarial value calibration includes the following steps: Step 1: Construct a collaborative communication model between UAVs and reconfigurable smart surfaces for dynamic power Internet of Things, establish the relationship between UAV movement and energy consumption, the trajectory path of mobile power equipment, and the channel fading environment, and finally obtain the system optimization problem with the goal of maximizing the ratio of system transmission rate to UAV energy consumption; in order to avoid the limitations of traditional single optimization and achieve a balance between communication quality and physical energy consumption. Step 2: Transform the system optimization problem obtained in Step 1 into a Markov decision process, integrate position information and system transmission rate, set a joint control dimension of continuous motion and discrete phase shift, and construct a feedback mechanism that combines positive incentives and constraint penalties. Finally, output the state space, action space and reward function for reinforcement learning training; provide a reliable data input framework for intelligent decision-making in complex high-dimensional dynamic environments. Step 3: Using the state space, action space, and reward function output in Step 2, execute a deep reinforcement learning algorithm based on an adversarial value calibration mechanism. Through adversarial game and parameter iteration among multiple internal network nodes, obtain the optimal flight trajectory for controlling the UAV and the optimal phase shift matrix for controlling the reconfigurable smart surface. Schedule the UAV to the target spatial position according to the obtained optimal flight trajectory, and reconstruct the reflection phase of the smart surface in real time according to the obtained optimal phase shift matrix, thereby establishing a reliable non-line-of-sight communication link. This overcomes the technical problems of large evaluation fluctuations and difficult convergence in dynamic environments of traditional algorithms, and achieves full coverage and high-energy-efficiency communication optimization for the UAV and smart surface collaborative communication system.

[0006] Step 1 specifically includes the following sub-steps: Step 1-1) Establish a UAV motion and energy consumption model, calculate the two-dimensional horizontal displacement information of the UAV in different time slots, and calculate the power consumption information generated by the UAV during flight based on the propulsion force, so as to provide physical boundary support for subsequent energy efficiency trade-offs. Steps 1-2) Establish a trajectory model of the mobile power equipment. Generate a baseline straight path based on the starting and ending points. Superimpose a disturbance offset term composed of a sinusoidal signal on the baseline straight path to generate the position coordinate information of the mobile power equipment in different time slots. This is to describe the dynamic trajectory changes caused by uneven roads or external interference in the real urban environment and improve the environmental realism of the model. Steps 1-3) Establish a communication model, combine the two-dimensional horizontal displacement information of the UAV and the position coordinate information of the mobile power equipment, characterize the multipath fading environment formed by building reflection and scattering, calculate the total channel transmission rate information of the signal reaching the mobile power equipment after being reflected by the reconfigurable smart surface, and thus quantify the signal compensation gain of the smart surface in a complex occlusion environment. Steps 1-4) Establish a system optimization model, combine the total channel transmission rate information with the power consumption information, introduce service quality requirements, dynamic constraints and physical boundaries as constraints, construct the associated objectives aimed at improving the system transmission rate and reducing energy loss, and finally obtain the system optimization problem, ensuring that the subsequent optimization process strictly meets the communication baseline of power equipment and flight safety requirements.

[0007] Step 2 specifically includes the following sub-steps: Step 2-1) Perform state design by splicing and integrating the horizontal and vertical coordinates of the UAV, the horizontal and vertical coordinates of the mobile power equipment, and the system transmission rate to obtain a state space that can fully support perception and decision-making, enabling the intelligent agent to have instantaneous insight into the global spatial topology and communication status. Step 2-2) Perform motion design, extract the displacement increment parameters of the UAV and the phase shift angle parameters of multiple reflection units of the reconfigurable smart surface, and combine them to obtain the motion space, unifying continuous motion and discrete phase shift into the hybrid control dimension; Steps 2-3) Design a reward function, extract the ratio of transmission rate to energy consumption to form a positive incentive to guide performance improvement, and incorporate penalties for exceeding limits, speeding, exceeding total energy consumption limits, and failing to meet the rate target. Output the reward function for reinforcement learning training. By combining this positive incentive with multiple physical constraint penalties, the agent is driven to autonomously avoid high-risk and high-consumption strategies during the exploration process, accelerating the convergence of the algorithm to the high-energy-efficiency region.

[0008] Step 3 specifically includes the following sub-steps: Step 3-1) Execute the target generator network calculation and update, use the state and action concatenation vector mapping to calculate the target action value, combine the reward mechanism to generate the time difference target value, thereby providing the overall time difference target value distribution information, and finally iterate the target generator network parameters through soft update, providing the generator network with a global calibration benchmark with temporal consistency and distribution stability. Step 3-2) Perform discriminator network computation and update, receive action value and time difference target value as input, calculate and minimize the binary cross-entropy loss function during training, distinguish the two types of input to provide adversarial supervision signal, and prevent the generator from generating over-estimation bias in complex action space; Step 3-3) Execute generator network computation and update, map state and action to action value estimate, minimize the joint loss function of fusion time difference error and adversarial loss, make the generator output action value distribution approximate the time difference target value distribution, and suppress abnormal fluctuations in the action value estimation process; Steps 3-4) Perform actor network computation and update, receive the action value gradient provided by the generator network, calculate the gradient of the corresponding actor network policy parameters, and update the network parameters by decreasing based on the learning rate; Steps 3-5) Execute target actor network update, introduce soft update parameters and smoothly iterate the internal parameters of target actor network, and finally obtain the optimal flight trajectory for controlling UAV flight and the optimal phase shift matrix for controlling reconfigurable smart surface; send the obtained optimal flight trajectory to UAV flight control module to execute spatial movement, and send the obtained optimal phase shift matrix to reconfigurable smart surface controller to execute array phase deflection, thereby completing the global optimal physical configuration of channel transmission rate and system energy consumption.

[0009] In step 1-1, the two-dimensional horizontal displacement information of the UAV in different time slots is calculated using the following formula: (1); (2); In the formula, , Representing drones in time slots t The position at that time , Represents the drone in the previous time slot t The position at -1 , Representing drones x shaft and y Displacement in the axial direction; t Represents a time slot, and t Index for discrete time steps; Combining the above calculation formulas (1) and (2), the two-dimensional horizontal displacement information of the UAV is constrained by the maximum flight speed, and the constrained formula is expressed as: (3); In the formula, The maximum speed of the UAV. The duration of each time slot; Furthermore, the power consumption information generated during the flight of the UAV is calculated using the propulsion energy consumption formula, which is expressed as follows: (4); In the formula, , These represent the blade profile power and induced power during UAV hovering, respectively. For the flight speed of the UAV, For rotor tip speed, For rotor design parameters, , , , These are the fuselage drag coefficient, rotor solidity, air density, and rotor disk area, respectively. For drones in time slots t The dominant energy consumption within the system.

[0010] In steps 1-2, a baseline straight line path is generated based on the starting and ending points. The evolution formulas for the x and y coordinates are expressed as follows: (5); (6); In the formula, The starting point of the mobile power equipment. The endpoint of the mobile power equipment. T This represents the total number of time slots. , The first k A mobile power device in a time slot t The x and y coordinates of the baseline straight line path; Combining the baseline straight path obtained by formulas (5) and (6) above, and superimposing a disturbance offset term composed of a sinusoidal signal, the position coordinate information of the mobile power equipment in different time slots is generated. The corresponding calculation formula is as follows: (7); In the formula, For the first k A mobile power device in a time slot t The disturbed ordinate; , , Mobile power equipment k The p The amplitude, frequency, and phase of each sinusoidal perturbation component; For mobile power equipment k In the time slot t Location; P This represents the total number of sinusoidal signal components that constitute the disturbance.

[0011] In steps 1-3, the total channel transmission rate information of the signal reaching the mobile power device after reflection from the reconfigurable smart surface is calculated. First, the channel vectors from the base station to the reconfigurable smart surface and the UAV relay link are calculated using the following formula: (8); In the formula, ρ The reference distance is 1 Path loss at the location; The distance between the base station and the UAV. α BU The path loss index; For the line-of-sight component between the base station and the RIS-UAV; For the line-of-sight component in the above formula (8), the array response vector is used to give: (9); In the formula, λ The carrier wavelength; d For reconfigurable smart surface RIS cell spacing; M The number of RIS units in the reconfigurable smart surface; The cosine of the angle of arrival. j The imaginary unit; Next, the angle is converted for the cosine component of the arrival angle in the above formula (9), and the expression is as follows: (10); In the formula, and For UAV and base station x coordinate; Subsequently, the RIS-UAV and the first The channel vector of a mobile power device link is represented as follows: (11); In the formula, ρ The reference distance is 1 Path loss at the location, Let be the channel vector of the link between the reconfigurable smart surface and the k-th mobile power device; The distance between the reconfigurable smart surface and the k-th mobile power device. This corresponds to the path loss index. This refers to the small-scale fading components of the link; RIS is M It consists of passive reflective units, whose phase shift matrix is ​​in the time slot t It can be represented as: (12); In the formula, For the phase matrix, For the first Each unit in the time slot phase shift angle, Construct operators for diagonal matrices; Based on the aforementioned channel vector and phase shift matrix, after RIS reflection, the data from the base station to the power equipment... k In the time slot t The equivalent end-to-end composite channel is: (13); In the formula, For an equivalent end-to-end composite channel, This is the channel vector from the base station to the RIS-UAV. For the channel matrix, This refers to the channel vector from RIS-UAV to the power equipment. Then, according to Shannon's formula, the communication transmission rate of this link is modeled as follows: (14); In the formula, For communication transmission rate, The total system bandwidth, For transmission power, In time slot t Base station to power equipment k The channel vector, For noise power, It is the square of the complex modulus of the equivalent end-to-end composite channel; In steps 1-4), the total channel transmission rate information and power consumption information are combined to construct the system optimization problem, which is expressed as: (15); Where max represents the operator for maximizing the objective function. T The total number of gaps, K The number of mobile power devices; To ensure that the solution process remains within a safe range and conforms to physical laws, corresponding constraints are configured for the optimization problem of this system, as follows: (16); In the formula, constraints C1 to C6 form a joint constraint set; constraints Used to ensure the Quality of Service (QoS) of power equipment, among which Minimum speed requirement for electrical equipment; , These represent the drones in time slots. t +1 and time slot t The three-dimensional spatial position vector; The squared norm of a vector; The maximum physical displacement limit for a drone within a time slot; This represents the upper limit of the total energy consumption for drone flight. , flying respectively by drone x shaft and y Upper limit of axis physical boundary; constraint conditions Constraints are used to ensure that the speed of the UAV does not exceed its maximum speed. Phase offset limit representing RIS; constraints The first constraint represents the energy consumption constraint of the UAV; constraint conditions C5 and C6 represent the constraints on the boundary range of the UAV.

[0012] In step 2-1), the formula for calculating the state space is expressed as follows: (17); In the formula, For the current moment t The state space; For each UAV at the current moment t coordinates For electrical equipment at time t The x and y coordinates, For a moment tThe system transmission rate; this state space simultaneously covers two key types of information, spatial location and communication quality, which can fully support the agent's perception and decision-making in the dynamic environment; In step 2-2), to collaboratively optimize the UAV trajectory and RIS phase shift, the obtained motion space must simultaneously support joint control of the UAV's continuous motion and the metasurface's discrete phase shift; therefore, the motion vector is defined as: (18); In the formula, For action vectors, The UAV at time t along x shaft and y The displacement increment of the shaft, Indicates the first m Each reflective unit at time... t phase shift angle, M This represents the total number of reflection units; thus, the two-dimensional continuous movement of the UAV and the multi-unit discrete phase shift of the RIS can be unified into a hybrid action space, providing a complete decision dimension for joint optimization. In steps 2-3), the formula for calculating the reward function is expressed as follows: (19); In the formula, For the reward function; and They are time points t The transmission rate and energy consumption are compared, and the ratio between the two is used as a positive incentive to guide the improvement of system performance. These are penalties for restricting drones from exceeding boundaries, speeding, exceeding total energy consumption limits, and having system transmission rates below service quality requirements.

[0013] In step 3-1, the mapping formula used to calculate the value of the target action is: (twenty two); In the formula, For the value of the target action, This represents a feedforward network consisting of two hidden layers with ReLU activation functions and one linear output layer; This is a feature concatenation vector of state and action. The next-moment motion vector output by the network for the target actor; Based on formula (22), the update rule formula for calculating the time difference target value is as follows: (twenty three); In the formula, This is the time difference target value, i.e., the TD target value; This is a discount factor for future rewards, used to measure the weight of long-term returns; This is the state vector for the next time step; After calculating the time difference target value, the formula for calculating the parameters of the iterative target generator network is as follows: (twenty four); In the formula, For the target generator network parameters, For soft update parameters, This indicates a parameter assignment and update operation.

[0014] In step 3-2, the calculation process for the probability estimate of the discriminator network output between 0 and 1 is expressed as follows: (25); In the formula, Let x be the probability estimate output by the discriminator, and let x be the input variable of the discriminator, i.e., the scalar value to be judged. This represents a feature transformation function with two fully connected hidden layers; Use the Sigmoid activation function; To provide adversarial supervision signals, the formula for minimizing the binary cross-entropy loss function is as follows: (26); In the formula, Let the binary cross-entropy loss function of the discriminator be . Indicates the target value of TD y t Distribution and action value Q ( s t , a t The expected value of the distribution is obtained to ensure that the optimization of the discriminator is based on the overall distribution characteristics; In step 3-3, the formula for mapping states and actions to estimated action values ​​is expressed as follows: (27); In the formula, It is a feedforward network consisting of two ReLU hidden layers and one linear output layer; The joint loss function that combines temporal difference error and adversarial loss is expressed as: (28); In the formula, The joint loss function for the generator; In steps 3-4, the formula for calculating the gradient of the corresponding actor's network policy parameters is expressed as follows: (29); In the formula, J represents the expected return of the strategic objective; T This represents the total number of time slots. Output the gradient of action value to the generator, indicating the direction in which action value increases; The gradient of the strategy with respect to the parameters is calculated for the actor network. The action output by the actor network based on the current parameter strategy; The formula for calculating network parameter descent updates based on the learning rate is: (30); In the formula, For the parameters of the actor network, The learning rate of the actor network; In steps 3-5, the formula for smoothly changing the internal parameters of the target actor network is expressed as follows: (31); In the formula, Network parameters for the target actor.

[0015] Compared with the prior art, the present invention has the following technical effects: 1) This invention addresses the challenge of jointly optimizing the trajectory of a UAV and the phase shift of the reflecting surface. It innovatively models this problem as a Markov decision process (MDP) that maximizes the ratio of system transmission rate to UAV energy consumption. This integrates UAV spatial dynamics, phase shift discreteness, and equipment service quality requirements, providing a solution and data foundation for strategy solving in complex physical environments. 2) To address the technical problems of large fluctuations and difficulty in convergence of action value estimation in traditional deep reinforcement learning (DRL) in dynamic environments, this invention designs the DRL-AVC framework. By introducing a target generator and an adversarial game between the generator and the discriminator, the action value distribution of the generator is made to accurately approximate the distribution of the real target value, which can effectively suppress overestimation bias and improve the learning stability and convergence speed of the algorithm in the dynamic power Internet of Things environment. 3) Comparative experimental results show that, regardless of whether it is a single device or a complex scenario with multiple devices, the DDPG-AVC and TD3-AVC algorithms proposed in this invention can converge quickly and maintain extremely high average reward values. In particular, when the scale of power equipment increases dramatically and the environmental dynamics become stronger, the performance of traditional mainstream algorithms (such as SAC and PPO) deteriorates sharply, while the method of this invention still maintains stable high energy efficiency and high transmission rate, demonstrating excellent environmental adaptability. Attached Figure Description

[0016] The present invention will be further described below with reference to the accompanying drawings and embodiments: Figure 1 This is an overall flowchart of the present invention; Figure 2 This is a comparison chart of reward values ​​for different algorithms in the single-vehicle scenario in the embodiment; Figure 3 This is a comparison chart of reward values ​​for different algorithms in a dual-vehicle scenario in the embodiment; Figure 4 This is a comparison chart of the reward values ​​of different algorithms for the four vehicle scenarios in the embodiment; Figure 5 This is a comparison chart of different algorithms for average transmission rate in the embodiments; Figure 6 This is a trajectory diagram of the mobile device and the drone in the embodiment. Detailed Implementation

[0017] like Figure 1 As shown, a RIS-UAV cooperative communication optimization method based on adversarial value calibration includes the following steps: Step 1: Construct a collaborative communication model between RIS and UAV for dynamic vehicle networking, which specifically includes a UAV motion and energy consumption model, a mobile power equipment trajectory model, a communication model, and a system optimization model; Step 2: Model the overall system model from Step 1 into an MDP, specifically including state design, action design, and reward function design; Step 3: Design an adversarial value calibration DRL algorithm, which mainly includes a target generator network, a discriminator network, a generator network, an actor network, and a target actor network, as well as their updates.

[0018] Specifically, it includes: Step 1 includes the following steps: Step 1.1: Establish a UAV motion and energy consumption model; Step 1.2: Establish a trajectory model for the mobile power equipment; Step 1.3, establish the communication model; Step 1.4: Establish a system optimization model.

[0019] Step 1.1: UAV Motion and Energy Consumption Model; To adapt to the discrete decision-making framework, the total task duration is divided into... T There are 1 time slot, and the length of each time slot is 1. τ sl In the time slot t Inside, the UAV is at a fixed height H The position is updated by performing a two-dimensional horizontal action, and the update method is designed as follows: (1); (2); In the formula, Indicates UAV inx shaft and y Displacement in the axial direction; For UAV in time slots t The position at that time.

[0020] Meanwhile, the maneuverability of UAVs is limited by physical properties, and their maximum flight speed must meet the following requirements: (3); In the formula, The maximum speed of the UAV. The duration of each time slot.

[0021] Besides speed limitations, the endurance of UAVs is also constrained by onboard energy. Since the energy consumption for communication processing and RIS control is far lower than the energy consumption for UAV propulsion, considering only the dominant propulsion energy consumption, the calculation formula is as follows: (4); In the formula, , These represent the blade profile power and induced power during UAV hovering, respectively. For the flight speed of the UAV, For rotor tip speed, For rotor design parameters, , , , These are the fuselage drag coefficient, rotor solidity, air density, and rotor disk area, respectively.

[0022] Step 1.2: Trajectory model of mobile power equipment; In dynamic power Internet of Things (IoT) systems, the mobility of power equipment is a key external factor causing time-varying environmental changes. To generate motion trajectories that are both realistic and reproducible, a parametric deterministic model is employed. Let the mobile power equipment... k The trajectory is generated as follows, with the vehicle traveling from a preset starting point to the destination.

[0023] First, a baseline straight path is generated based on the starting and ending points, and the mobile power equipment moves within the time slot. t The x and y coordinates can be represented as: (5); (6); In the formula, The starting point of the mobile power equipment. The endpoint of the mobile power equipment. .

[0024] Secondly, to simulate trajectory disturbances caused by uneven road surfaces, movement fluctuations, or environmental interference during actual driving, an offset term composed of a multi-frequency sinusoidal signal is superimposed on the vertical axis. This signal is composed of... P Composed of 3 sinusoidal components, each component having amplitude, frequency, and phase, the expression for the perturbed longitudinal coordinate is: (7); In the formula, , , These are amplitude, frequency, and phase, respectively. For mobile power equipment k In the time slot t The location.

[0025] Step 1.3: Communication Model; Considering the lack of direct line-of-sight links between base stations and RIS-UAV relays in power emergency communication scenarios, a Rayleigh channel model is used to characterize the multipath fading environment formed by reflection and scattering from buildings. Based on the established UAV motion model and mobile power equipment trajectory model, a complete double-hop reflection communication link is constructed, and system energy efficiency is ultimately used as the optimization objective.

[0026] The channel vector from the base station to the RIS-UAV relay link can be expressed as: (8); In the formula, ρ The reference distance is 1 Path loss at the location; The distance between the base station and the UAV. α BU The path loss index; The line-of-sight components between the base station and the RIS-UAV are given by the array response vector: (9); In the formula, λ The carrier wavelength; d This refers to the spacing between RIS cells; M This represents the number of RIS units. The cosine of the angle of arrival is expressed as follows: (10); In the formula, and For UAV and base station x coordinate.

[0027] Similarly, RIS-UAV and the first The channel vector of a mobile power device link is represented as follows: (11); RIS is M It consists of passive reflective units, whose phase shift matrix is ​​in the time slot t It can be represented as: (12); In the formula, For the first Each unit in the time slot The phase shift angle.

[0028] After RIS reflection, the base station connects to the power equipment. k In the time slot The equivalent end-to-end composite channel is: (13); In the formula, This represents the channel vector from the base station to the RIS-UAV. It is the phase shift matrix; This is the channel vector from RIS-UAV to the power equipment.

[0029] According to Shannon's formula, the communication transmission rate of this link is modeled as follows: (14); In the formula, The total system bandwidth For transmission power, In time slot t Base station to power equipment k The channel vector, This represents noise power.

[0030] Step 1.4: System optimization model; The proposed RIS-UAV co-optimization problem for dynamic power IoT aims to maximize the long-term average system transmission rate to UAV energy consumption by co-optimizing the UAV flight trajectory and RIS phase shift matrix, while satisfying communication service quality constraints, UAV dynamics constraints, and energy limitations. This optimization problem can be formally expressed as follows: (15); (16); In the formula, the constraint conditions Used to ensure the Quality of Service (QoS) of power equipment, among which This refers to the minimum speed requirement for electrical equipment. (Constraints) Constraints are used to ensure that the speed of the UAV does not exceed its maximum speed. Represents the phase offset constraint of RIS. Constraints C5 and C6 represent the energy consumption constraints of the UAV. C5 and C6 represent the constraints on the boundary range of the UAV.

[0031] Step 2: Model the system optimization model from Step 1 into an MDP, which specifically includes state design, action design, and reward function design. Step 2.1, State Design; Step 2.2, Motion Design; Step 2.3, Reward Function Design; Step 2.1: State Design: To accurately characterize system dynamics and provide sufficient basis for agent decision-making, this paper designs a state space that integrates the spatial location of the UAV, the location information of the power equipment, and the system's transmission rate. Therefore, the state space is designed as follows: (17); In the formula, For each UAV at the current moment t coordinates For electrical equipment at time t The x and y coordinates, For a moment t The system transmission rate. This state space simultaneously encompasses two key types of information: spatial location and communication quality, enabling intelligent agents to fully perceive and make decisions about dynamic environments.

[0032] Step 2.2: Motion Design: To collaboratively optimize the UAV trajectory and RIS phase shift, the action space designed in this paper needs to simultaneously support joint control of the UAV's continuous motion and the discrete phase shift of the metasurface. Therefore, the action vector is defined as: (18); In the formula, The UAV at time t along x shaft and y The displacement increment of the shaft, Indicates the first m Each reflective unit at time... t phase shift angle, M This represents the total number of reflection units. This design unifies the two-dimensional continuous movement of the UAV with the multi-unit discrete phase shift of the RIS into a hybrid action space, providing a complete decision dimension for joint optimization.

[0033] Step 2.3: Reward Function Design: The design of the reward function directly determines the learning direction of the agent, and its core is to drive the system to maximize its overall performance while satisfying physical constraints. To this end, a reward function combining positive incentives and constraint penalties is constructed: (19); In the formula, and They are time points t The transmission rate and energy consumption are compared, and the ratio between the two is used as a positive incentive to guide the improvement of system performance. These constraints are designed to address situations such as UAVs exceeding boundaries, speeding, exceeding total energy consumption limits, and system transmission rates falling below quality of service requirements. This design enables the agent to adaptively satisfy various key constraints while exploring energy-efficient strategies.

[0034] Step 3 proposes an adversarial value calibration (DRL) algorithm to optimize the solution of the MDP established in Step 2, including: Step 3.1, Target Generator Network and its Update; Step 3.2, Discriminator Network and its Update; Step 3.3, Generator Network and its Update; Step 3.4, Actor Network and its Updates; Step 3.3, Current actor network and its updates; Step 3.1: Target Generator Network and its Updates: The target generator network is primarily responsible for two functions: first, outputting the target action value; and second, calculating the TD target value based on this value, thereby providing the generator with overall TD target value distribution information. Its network structure employs a three-layer fully connected network, integrating state... s t+1 With action a The concatenated vector of ' is mapped to the target action value: (twenty two); In the formula, This represents a feedforward network consisting of two hidden layers with ReLU activation functions and one linear output layer. Furthermore, the target generator calculates the TD target value according to the TD update rule: (twenty three); The TD target value is continuously generated during training, forming a temporally consistent and stably distributed set of TD target values. This provides the generator network with reliable overall TD target distribution information, thereby prompting the generator network to generate an action value distribution that closely approximates the TD target value distribution. This effectively suppresses fluctuations in action value estimation and improves the stability of the algorithm.

[0035] The target generator network depends on the next state. s t+1 To calculate the value of the target action and combine it with the current reward. r t To generate TD target values yt This provides the generator network with overall TD target distribution information. The target generator network parameters are updated via soft updates, specifically as follows: (28); In the formula, For the target generator network parameters, These are parameters for soft updates.

[0036] Step 3.2: Discriminator Network and its Updates: The discriminator is responsible for providing adversarial supervision signals to the generator. Its core task is to accurately distinguish the action value output by the generator from the target TD target value calculated by the target generator, and to provide adversarial supervision signals by maximizing the distribution difference between the two, thereby guiding the generator to perform calibration. The input of the discriminator is the scalar value to be discriminated. Q ( s t , a t )or y t The process involves a nonlinear transformation through two fully connected hidden layers, and finally, a probability estimate between 0 and 1 is output through the Sigmoid activation function for the TD target value. y t The discriminator learns to output close to 1; for Q ( s t , a t If the learning output is close to 0, then the learning output is close to 0. The specific calculation process is as follows: (twenty four); In the formula, This represents a feature transformation function with two fully connected hidden layers; The sigmoid activation function is used. The objective of the discriminator is to minimize the following binary cross-entropy loss function: (25); In the formula, E Indicates the target value of TD y t Distribution and action value Q ( s t , a t The expected value of the distribution is calculated to ensure that the optimization of the discriminator is based on the overall distribution characteristics. This loss function guides the discriminator to perform two tasks: first, to prompt the discriminator to distribute samples for the target generator, i.e., the TD target value. y tThe first is to output a probability close to 1; the second is to encourage the discriminator to distribute samples according to the generator's distribution, i.e., the action value. Q ( s t , a t The probability of the output being close to 0 is calculated. By optimizing this loss, the discriminator can effectively distinguish between two types of inputs: TD target value and action value, and provide the generator with a stable, adversarial distribution calibration signal.

[0037] Step 3.3: Generator Network and its Updates: The core function of the generator network is to fit the action value function and, through a dual supervision mechanism, make its output action value distribution approximate the TD target value distribution provided by the target generator. The generator adopts a three-layer fully connected structure consistent with the target generator, storing the state s... t With action a t The concatenated vector is mapped to action value estimation: (26); In the formula, It is a feedforward network consisting of two ReLU hidden layers and a linear output layer. The generator's overall loss function combines the TD error and the adversarial loss: (27); In the formula, E This represents the mathematical expectation from the corresponding state-action distribution samples. The first term on the right-hand side of the equation is the TD error, which prompts the generator to approximate the TD target value. The second term is the adversarial loss, whose design motivation stems from the principle of generative adversarial training: the discriminator is trained to output the generator's... Q ( s t , a t The result is determined to be false, that is... The generator, on the other hand, minimizes... Strive to make the discriminator output Q ( s t , a t ) misjudged as true, that is .

[0038] Step 3.4: Actor Network and its Updates: The actor network is updated based on the concept of deterministic policy gradients, where the gradients are derived from stable action value estimates provided by the generator. The specific update method is as follows: (29); In the formula, T This represents the total number of time slots. Output the gradient of action value to the generator, indicating the direction in which action value increases; The gradient of the policy with respect to the parameters is calculated for the actor network. The parameter updates are as follows: (30); In the formula, The learning rate of the actor network.

[0039] Step 3.5: Target Actor Network and its Updates: The target actor network updates its parameters via soft updates: (31); In the formula, Network parameters for the target actor.

[0040] Results Analysis: Comparative experiments were conducted with other mainstream DRL algorithms, DDPG-AVC and TD3-AVC, as presented in this invention. The results show that DDPG-AVC and TD3-AVC converge quickly and maintain a high and stable average reward value in all scenarios, significantly outperforming other algorithms. Furthermore, in all scenarios from single-device to four-device setups, DDPG-AVC and TD3-AVC achieved optimal results, with significantly improved average transmission rate and average energy efficiency compared to algorithms such as SAC and PPO. As the number of power devices increases and environmental dynamics intensify, the performance of traditional algorithms generally declines, while DDPG-AVC and TD3-AVC remain stable and efficient, demonstrating superior environmental adaptability and performance robustness.

[0041] Reward functions for different numbers of power devices, such as Figure 2-4 As shown, the horizontal axis represents the number of rounds, and the vertical axis represents the reward value.

[0042] Average transmission rate under different numbers of power devices, such as Figure 5 As shown in the figure, the horizontal axis represents the number of devices, and the vertical axis represents the average transmission rate.

[0043] The trajectories of mobile devices and drones are as follows Figure 6 As shown, the horizontal axis represents the horizontal direction, and the vertical axis represents the vertical direction.

[0044] The trajectories of DDPG-AVC and TD3-AVC, as described in this invention, exhibit clear vehicle-following characteristics. They can dynamically adjust their flight paths based on vehicle positions, maintaining optimal communication distance and thus ensuring link quality. Simultaneously, the smooth trajectories avoid unnecessary drastic maneuvers, achieving an effective balance between communication rate and energy consumption. In contrast, SAC's trajectory exhibits some deviation and fails to stably follow vehicle movement; PPO's trajectory fluctuates significantly, failing to maintain effective tracking behavior, resulting in transmission rates and energy efficiency far lower than those of this invention.

[0045] In summary, the method proposed in this invention first constructs a RIS-UAV cooperative communication model for the power Internet of Things (IoT), modeling the joint optimization problem of UAV trajectory and RIS phase shift as maximizing the ratio of system transmission rate to UAV energy consumption. Simultaneously, a novel adversarial value calibration mechanism is proposed, and based on this, a DRL-AVC framework is put forward. The AVC mechanism mainly consists of a target generator, a generator, and a discriminator. The target generator is responsible for outputting the target action value and calculating the TD target value, providing the generator with overall TD target value distribution information. The generator uses a dual supervision strategy that simultaneously minimizes TD error and adversarial loss, making the generated action value distribution continuously approach the TD target value distribution. The discriminator is responsible for distinguishing the action value output by the generator from the TD target value calculated by the target generator, providing adversarial supervision signals by maximizing the difference between the two. During the adversarial process, the generator ultimately outputs an action value distribution that highly approximates the TD target value distribution, thereby effectively suppressing action value estimation fluctuations and improving algorithm stability.

Claims

1. A RIS-UAV cooperative communication optimization method based on adversarial value calibration, characterized in that, Includes the following steps: Step 1: Construct a collaborative communication model between UAVs and reconfigurable smart surfaces for dynamic power Internet of Things, establish the relationship between UAV movement and energy consumption, the trajectory path of mobile power equipment, and the channel fading environment, and finally obtain the system optimization problem with the goal of maximizing the ratio of system transmission rate to UAV energy consumption. Step 2: Transform the system optimization problem obtained in Step 1 into a Markov decision process, integrate position information and system transmission rate, set a joint control dimension of continuous motion and discrete phase shift, and construct a feedback mechanism that combines positive incentives and constraint penalties. Finally, output the state space, action space and reward function for reinforcement learning training. Step 3: Using the state space, action space, and reward function output in Step 2, execute a deep reinforcement learning algorithm based on an adversarial value calibration mechanism. Through adversarial game and parameter iteration among multiple internal network nodes, obtain the optimal flight trajectory for controlling the UAV and the optimal phase shift matrix for controlling the reconfigurable smart surface. Schedule the UAV to the target spatial position according to the obtained optimal flight trajectory, and reconstruct the reflection phase of the smart surface in real time according to the obtained optimal phase shift matrix, thereby establishing a reliable non-line-of-sight communication link.

2. The method according to claim 1, characterized in that, Step 1 specifically includes the following sub-steps: Step 1-1) Establish a UAV motion and energy consumption model, calculate the two-dimensional horizontal displacement information of the UAV in different time slots, and calculate the power consumption information generated by the UAV during flight based on the propulsion force. Step 1-2) Establish a trajectory model of the mobile power equipment, generate a reference straight path based on the starting and ending points, and superimpose a disturbance offset term composed of a sine signal on the reference straight path to generate the position coordinate information of the mobile power equipment in different time slots; Steps 1-3) Establish a communication model, combine the two-dimensional horizontal displacement information of the UAV and the position coordinate information of the mobile power equipment, characterize the multipath fading environment formed by building reflection and scattering, and calculate the total channel transmission rate information of the signal reaching the mobile power equipment after reflection through the reconfigurable smart surface. Steps 1-4) Establish a system optimization model, combine the total channel transmission rate information with the power consumption information, introduce service quality requirements, dynamic constraints and physical boundaries as constraints, construct the associated objectives aimed at improving the system transmission rate and reducing energy loss, and finally obtain the system optimization problem.

3. The method according to claim 1 or 2, characterized in that, Step 2 specifically includes the following sub-steps: Step 2-1) Perform state design by splicing and integrating the horizontal and vertical coordinates of the UAV, the horizontal and vertical coordinates of the mobile power equipment, and the system transmission rate to obtain a state space that can fully support perception and decision-making. Step 2-2) Perform motion design, extract the displacement increment parameters of the UAV and the phase shift angle parameters of multiple reflection units of the reconfigurable smart surface, and combine them to obtain the motion space; Steps 2-3) Design a reward function, extract the ratio of transmission rate to energy consumption to form a positive incentive to guide performance improvement, and at the same time incorporate penalties for exceeding limits, speeding, exceeding total energy consumption limits, and failing to meet the rate target, and output the final reward function for reinforcement learning training.

4. The method according to claim 3, characterized in that, Step 3 specifically includes the following sub-steps: Step 3-1) Execute the target generator network calculation and update, use the state and action concatenation vector mapping to calculate the target action value, combine the reward mechanism to generate the time difference target value, thereby providing the overall time difference target value distribution information, and finally update the target generator network parameters through soft update. Step 3-2) Perform discriminator network computation and update, receive action value and temporal difference target value as input, and distinguish the two types of input by calculating and minimizing the binary cross-entropy loss function during training, thereby providing adversarial supervision signals; Step 3-3) Execute generator network computation and update, map state and action to action value estimate, and minimize the joint loss function of fusion temporal difference error and adversarial loss to make the generator output action value distribution approximate the temporal difference target value distribution; Steps 3-4) Perform actor network computation and update, receive the action value gradient provided by the generator network, calculate the gradient of the corresponding actor network policy parameters, and update the network parameters by decreasing based on the learning rate; Steps 3-5) Execute target actor network update, introduce soft update parameters and smoothly iterate the internal parameters of target actor network, and finally obtain the optimal flight trajectory for controlling UAV flight and the optimal phase shift matrix for controlling reconfigurable smart surface; send the obtained optimal flight trajectory to UAV flight control module to execute spatial movement, and send the obtained optimal phase shift matrix to reconfigurable smart surface controller to execute array phase deflection.

5. The method according to claim 2 or 4, characterized in that, In step 1-1, the two-dimensional horizontal displacement information of the UAV in different time slots is calculated using the following formula: （1）； （2）； In the formula, , Representing drones in time slots t The position at that time , Represents the drone in the previous time slot t The position at -1 , Representing drones x shaft and y Displacement in the axial direction; t Represents a time slot, and t Index for discrete time steps; Combining the above calculation formulas (1) and (2), the two-dimensional horizontal displacement information of the UAV is constrained by the maximum flight speed, and the constrained formula is expressed as: （3）； In the formula, The maximum speed of the UAV. The duration of each time slot; Furthermore, the power consumption information generated during the flight of the UAV is calculated using the propulsion energy consumption formula, which is expressed as follows: （4）； In the formula, , These represent the blade profile power and induced power during UAV hovering, respectively. For the flight speed of the UAV, For rotor tip speed, For rotor design parameters, , , , These are the fuselage drag coefficient, rotor solidity, air density, and rotor disk area, respectively. For drones in time slots t The dominant energy consumption within the system.

6. The method according to claim 5, characterized in that, In steps 1-2, a baseline straight line path is generated based on the starting and ending points. The evolution formulas for the x and y coordinates are expressed as follows: (5)； (6)； In the formula, The starting point of the mobile power equipment. The endpoint of the mobile power equipment. T This represents the total number of time slots. , The first k A mobile power device in a time slot t The x and y coordinates of the baseline straight line path; Combining the baseline straight path obtained by formulas (5) and (6) above, and superimposing a disturbance offset term composed of a sinusoidal signal, the position coordinate information of the mobile power equipment in different time slots is generated. The corresponding calculation formula is as follows: （7）； In the formula, For the first k A mobile power device in a time slot t The disturbed ordinate; , , Mobile power equipment k The p The amplitude, frequency, and phase of each sinusoidal perturbation component; For mobile power equipment k In the time slot t Location; P This represents the total number of sinusoidal signal components that constitute the disturbance.

7. The method according to claim 6, characterized in that, In steps 1-3, the total channel transmission rate information of the signal reaching the mobile power device after reflection from the reconfigurable smart surface is calculated. First, the channel vectors from the base station to the reconfigurable smart surface and the UAV relay link are calculated using the following formula: （8）； In the formula, ρ The reference distance is 1 Path loss at the location; The distance between the base station and the UAV. α BU The path loss index; For the line-of-sight component between the base station and the RIS-UAV; For the line-of-sight component in the above formula (8), the array response vector is used to give: （9）； In the formula, λ The carrier wavelength; d For reconfigurable smart surface RIS cell spacing; M The number of RIS units in the reconfigurable smart surface; The cosine of the angle of arrival. j The imaginary unit; Next, the angle is converted for the cosine component of the arrival angle in the above formula (9), and the expression is as follows: （10）； In the formula, and For UAV and base station x coordinate; Subsequently, the RIS-UAV and the first The channel vector of a mobile power device link is represented as follows: （11）； In the formula, ρ The reference distance is 1 Path loss at the location, Let be the channel vector of the link between the reconfigurable smart surface and the k-th mobile power device; The distance between the reconfigurable smart surface and the k-th mobile power device. This corresponds to the path loss index. This refers to the small-scale fading components of the link; RIS is M It consists of passive reflective units, whose phase shift matrix is ​​in the time slot t It can be represented as: （12）； In the formula, For the phase matrix, For the first Each unit in the time slot phase shift angle, Construct operators for diagonal matrices; Based on the aforementioned channel vector and phase shift matrix, after RIS reflection, the data from the base station to the power equipment... k In the time slot t The equivalent end-to-end composite channel is: （13）； In the formula, For an equivalent end-to-end composite channel, This is the channel vector from the base station to the RIS-UAV. For the channel matrix, This refers to the channel vector from RIS-UAV to the power equipment. Then, according to Shannon's formula, the communication transmission rate of this link is modeled as follows: （14）； In the formula, For communication transmission rate, The total system bandwidth, For transmission power, In time slot t Base station to power equipment k The channel vector, For noise power, It is the square of the complex modulus of the equivalent end-to-end composite channel; In steps 1-4), the total channel transmission rate information and power consumption information are combined to construct the system optimization problem, which is expressed as: （15）； Where max represents the operator for maximizing the objective function. T The total number of gaps, K The number of mobile power devices; To ensure that the solution process remains within a safe range and conforms to physical laws, corresponding constraints are configured for the optimization problem of this system, as follows: （16）； In the formula, constraints C1 to C6 form a joint constraint set; constraints Used to ensure the Quality of Service (QoS) of power equipment, among which Minimum speed requirement for electrical equipment; , These represent the drones in time slots. t +1 and time slot t The three-dimensional spatial position vector; The squared norm of a vector; The maximum physical displacement limit for a drone within a time slot; This represents the upper limit of the total energy consumption for drone flight. , flying respectively by drone x shaft and y Upper limit of axis physical boundary; constraint conditions Constraints are used to ensure that the speed of the UAV does not exceed its maximum speed. Phase offset limit representing RIS; constraints The first constraint represents the energy consumption constraint of the UAV; constraint conditions C5 and C6 represent the constraints on the boundary range of the UAV.

8. The method according to claim 3, 4, 6, or 7, characterized in that, In step 2-1), the formula for calculating the state space is expressed as follows: （17）； In the formula, For the current moment t The state space; For each UAV at the current moment t coordinates For electrical equipment at time t The x and y coordinates, For a moment t The system transmission rate; this state space simultaneously covers two key types of information, spatial location and communication quality, which can fully support the agent's perception and decision-making in the dynamic environment; In step 2-2), to collaboratively optimize the UAV trajectory and RIS phase shift, the obtained motion space must simultaneously support joint control of the UAV's continuous motion and the metasurface's discrete phase shift; therefore, the motion vector is defined as: （18）； In the formula, For action vectors, The UAV at time t along x shaft and y The displacement increment of the shaft, Indicates the first m Each reflective unit at time... t phase shift angle, M This represents the total number of reflection units; thus, the two-dimensional continuous movement of the UAV and the multi-unit discrete phase shift of the RIS can be unified into a hybrid action space, providing a complete decision dimension for joint optimization. In steps 2-3), the formula for calculating the reward function is expressed as follows: （19）； In the formula, For the reward function; and They are time points t The transmission rate and energy consumption are compared, and the ratio between the two is used as a positive incentive to guide the improvement of system performance. These are penalties for restricting drones from exceeding boundaries, speeding, exceeding total energy consumption limits, and having system transmission rates below service quality requirements.

9. The method according to claim 8, characterized in that, In step 3-1, the mapping formula used to calculate the value of the target action is: （22）； In the formula, For the value of the target action, This represents a feedforward network consisting of two hidden layers with ReLU activation functions and one linear output layer; This is a feature concatenation vector of state and action. The next-moment motion vector output by the network for the target actor; Based on formula (22), the update rule formula for calculating the time difference target value is as follows: （23）； In the formula, This is the time difference target value, i.e., the TD target value; This is a discount factor for future rewards, used to measure the weight of long-term returns; This is the state vector for the next time step; After calculating the time difference target value, the formula for calculating the parameters of the iterative target generator network is as follows: （24）； In the formula, For the target generator network parameters, For soft update parameters, This indicates a parameter assignment and update operation.

10. The method according to claim 9, characterized in that, In step 3-2, the calculation process for the probability estimate of the discriminator network output between 0 and 1 is expressed as follows: （25）； In the formula, Let x be the probability estimate output by the discriminator, and let x be the input variable of the discriminator, i.e., the scalar value to be judged. This represents a feature transformation function with two fully connected hidden layers; Use the Sigmoid activation function; To provide adversarial supervision signals, the formula for minimizing the binary cross-entropy loss function is as follows: （26）； In the formula, Let the binary cross-entropy loss function of the discriminator be . Indicates the target value of TD y t Distribution and action value Q ( s t , a t The expected value of the distribution is obtained to ensure that the optimization of the discriminator is based on the overall distribution characteristics; In step 3-3, the formula for mapping states and actions to estimated action values ​​is expressed as follows: （27）； In the formula, It is a feedforward network consisting of two ReLU hidden layers and one linear output layer; The joint loss function that combines temporal difference error and adversarial loss is expressed as: （28）； In the formula, The joint loss function for the generator; In steps 3-4, the formula for calculating the gradient of the corresponding actor's network policy parameters is expressed as follows: （29）； In the formula, J represents the expected return of the strategic objective; T This represents the total number of time slots. Output the gradient of action value to the generator, indicating the direction in which action value increases; The gradient of the strategy with respect to the parameters is calculated for the actor network. The action output by the actor network based on the current parameter strategy; The formula for calculating network parameter descent updates based on the learning rate is: （30）； In the formula, For the parameters of the actor network, The learning rate of the actor network; In steps 3-5, the formula for smoothly changing the internal parameters of the target actor network is expressed as follows: （31）； In the formula, Network parameters for the target actor.

Images