Hybrid energy laser-powered unmanned aerial vehicle communication method and system based on reinforcement learning

Abstract

This invention discloses a hybrid energy laser-powered UAV communication method and system based on reinforcement learning, belonging to the field of UAV energy management technology in 6G air-to-ground communication. Addressing the problem of rotorcraft UAVs struggling to maintain long-term service in hotspot scenarios due to limited onboard battery capacity, a photovoltaic-grid hybrid power supply laser transmitter is constructed to continuously wirelessly charge the UAV. The UAV uses non-orthogonal multiple access (NOMA) to serve ground users. With the goal of maximizing the total system throughput, under constraints such as UAV battery state of charge and user service quality, the UAV trajectory, laser emission power, and fractional-order power allocation exponent are jointly optimized. This non-convex temporally coupled problem is modeled as a Markov decision process, and an improved dual-delay deep deterministic policy gradient algorithm (I-TD3) integrating priority experience replay and n-step reward is proposed for solving it.

Description

Technical Field

[0001] This invention relates to the field of 6G air-to-ground communication for unmanned aerial vehicle (UAV) energy management technology, and in particular to a hybrid energy laser-powered UAV communication method and system based on reinforcement learning. Background Technology

[0002] With the development of sixth-generation (6G) mobile communication technology, using unmanned aerial vehicles (UAVs) as airborne base stations to provide temporary communication coverage for ground emergency hotspots (such as large-scale events and disaster relief) has become an important means of integrated air-space-ground networks. However, due to their huge propulsion power consumption and limited onboard battery capacity, rotary-wing UAVs face a serious "range anxiety" problem, which greatly limits their service endurance.

[0003] In recent years, using ground-based laser beam directives (LBDs) to wirelessly replenish the power of drones without interrupting communication missions has become a promising solution for extending drone endurance. However, existing laser-powered drone communication systems still suffer from the following key drawbacks:

[0004] The energy supply is extensive and highly dependent on the power grid: Existing laser power supply solutions usually adopt a fixed maximum power transmission mode, without considering the spatial transmission loss caused by the change of UAV position and the variable weather conditions, resulting in huge energy consumption of the ground power grid and a lack of effective introduction and dynamic scheduling mechanism for green and clean energy.

[0005] Joint optimization of trajectory and multidimensional resources is challenging: In non-orthogonal multiple access (NOMA) downlink communication, the UAV's three-dimensional flight trajectory, laser emission power control, and dynamic power allocation among users are highly coupled in the time domain, and are constrained by the UAV's battery state of charge (SoC) safety boundary and the minimum quality of service (QoS) constraints for ground users. This optimization problem is a highly complex non-convex temporally coupled optimization problem, and traditional convex optimization algorithms or heuristic algorithms are difficult to solve in real-time and efficiently in highly dynamic emergency scenarios with user mobility.

[0006] In summary, existing laser-powered drone communication systems face an irreconcilable contradiction between "green and low-carbon operation" and "dynamic, uninterrupted service." On the one hand, the crude, fixed-power laser emission mode not only causes severe energy waste but also significantly threatens the safe lifespan of the drone's onboard batteries. On the other hand, traditional optimization methods, due to their poor timeliness, cannot adaptively respond to the stringent requirements of ground emergency users' spatiotemporal movement for dynamic adjustments of multidimensional resources. Therefore, how to introduce a clean energy consumption mechanism to significantly reduce grid energy consumption while ensuring drone battery safety and high-continuity communication is a bottleneck problem that urgently needs to be solved for the practical application of laser-powered drone networks. Summary of the Invention

[0007] This invention provides a hybrid energy laser-powered drone communication method and system based on reinforcement learning. It introduces a clean energy consumption mechanism, which significantly reduces grid energy consumption while ensuring drone battery safety and high-continuity communication, and maximizes the total system throughput.

[0008] The technical solution adopted by the embodiments of the present invention to solve its technical problem is as follows:

[0009] The first aspect of this invention provides a communication method for hybrid energy laser-powered unmanned aerial vehicles based on reinforcement learning, comprising:

[0010] The system consists of photovoltaic panels, a power grid, laser beam guides, rotary-wing drones, and A downlink communication system model consisting of several ground users is described, in which the laser beam director is fixed to the top of a building, and its power input is supplied by a combination of solar energy converted from photovoltaic panels and electricity provided by the power grid; the laser beam director converts the input power into electro-optical energy with high efficiency. The laser beam is converted into a laser beam and continuously emitted towards the rotorcraft drone flying in the air. The rotorcraft drone is equipped with a photoelectric receiver and energy harvesting circuitry to receive laser energy to charge its onboard battery. At the same time, the rotorcraft drone acts as an aerial base station at a fixed altitude. Onboard flight, utilizing non-orthogonal multiple access methods with Downlink communication is conducted among ground users, who are sorted according to channel gain and signal detection is performed using continuous interference cancellation technology.

[0011] The entire mission cycle T is discretized into N equal-length time slots. Within each time slot, it is assumed that the UAV position, user position, channel gain, and all control variables remain constant. With the objective of maximizing the total throughput for all users within the mission cycle, a joint optimization of the UAV horizontal trajectory is established under constraints including UAV battery state-of-charge safety, minimum communication rate per user per time slot, maximum UAV flight speed, upper and lower limits of laser emission power, and the range of the fractional-order power allocation exponent for non-orthogonal multiple access. Laser beam guide laser emission power and fractional power distribution index The optimization problem of nonconvex temporal coupling;

[0012] The non-convex temporally coupled optimization problem is transformed into a Markov decision process, and a state space is defined. Action space and reward function Among them, state Based on the current battery power of the drone Horizontal position Flight speed V(t), laser emission power in the previous time slot Current power grid supply capacity The previous time slot fractional power allocation index and the current channel gain vectors of all K users. It is pieced together; action For the horizontal displacement increment of the UAV Laser emission power increment and fractional power distribution exponential increment The continuous vector is composed of the following: the reward function r(t) is composed of the weighted throughput reward, the weighted throughput increment reward, the net change in battery energy reward, the penalty for the battery falling below the safety limit, the penalty for the user not meeting the minimum rate requirement, and the penalty for the drone flying out of the service area boundary.

[0013] An improved dual-delay deep deterministic policy gradient algorithm, which integrates priority experience replay and n-step reward, is used to train the Markov decision process offline, learning the optimal policy that can output continuous actions based on the current state. During training, the sampling probability of key samples with high temporal difference error is increased through the priority experience replay mechanism, and the value estimation propagation is accelerated simultaneously using the n-step reward. The actor network and the dual critic network adopt delayed update and soft update strategies to stabilize the training process.

[0014] After training, in actual operation, based on the real-time status observed in each time slot, the trained actor network directly provides the action, and after amplitude limiting, dynamically adjusts the horizontal displacement of the rotor drone, the laser emission power of the laser beam guide, and the fractional power allocation index of the non-orthogonal multiple access, thereby maximizing throughput and sustainable management of drone battery energy.

[0015] Preferably, the electrical input power of the laser beam guide is... With laser emission power The relationship is ,and Solar power provided by photovoltaic panels and supplemental power provided by the power grid Common satisfaction, that is Among them, solar power The solar radiation intensity I(t) and photovoltaic conversion efficiency in the current time slot Photovoltaic panel area and cloud attenuation factor according to Calculated.

[0016] Preferably, the average power of the laser received by the rotary-wing UAV is... Based on the free-space light propagation loss and atmospheric attenuation model, the calculation is as follows:

[0017]

[0018] in, The electro-optical conversion efficiency of a photodiode. This is the turbulence and alignment loss factor, which takes a value of 1 under the assumptions of ideal alignment and clear sky. This is the optical efficiency factor. The effective area of ​​the photoelectric receiver carried by the drone. The combined optical efficiency of the transmitting and receiving system is given by denoted ... The real-time spatial distance between the laser beam guide and the UAV, where For the fixed height of the laser beam guide, The divergence angle of the laser beam. The atmospheric attenuation coefficient per unit distance. This represents the Euclidean norm.

[0019] Preferably, the energy dynamic equation of the onboard battery of the rotary-wing UAV is:

[0020]

[0021] in The battery energy storage at the start of time slot t, with each time slot having a length of . , This represents the total power consumption of the rotary-wing UAV in the current time slot, which consists of three parts: propulsion power consumption related to flight speed and aerodynamic parameters. Fixed power consumption of airborne circuits And communication power consumption for downlink non-orthogonal multiple access transmission. Battery energy is limited by its maximum capacity. and the safety lower limit to prevent over-discharge That is, it must always satisfy .

[0022] Preferably, the propulsion power consumption of the rotary-wing UAV is... Calculated using a classic rotor aerodynamic model, its speed is comparable to the horizontal flight speed of the UAV. The relationship is:

[0023]

[0024] in, This represents the blade profile power in a hovering state. This refers to the induced power during hovering. The rotor tip speed, The average rotor induced velocity during hovering. For the fuselage drag ratio, ρ is the air density, s is the rotor solidity, and A is the rotor disk area.

[0025] Preferably, the non-orthogonal multiple access fractional power allocation index Used to adaptively adjust the communication power allocation ratio of each user in each time slot according to the user channel gain; the communication power allocated to the k-th user. Represented as:

[0026]

[0027] in, For the first The channel power gain of user k, where j represents the user with a channel gain greater than k. For reference, the channel gain at a distance of 1 meter. The distance between the drone and the user is expressed in European terms. Let be the total communication power transmitted by the UAV; the range of values ​​for α(t) is... .

[0028] Preferably, the reward function r(t) is expressed as:

[0029]

[0030] in, Let B be the achievable rate of the k-th user in time slot t, which is determined by bandwidth B, allocated power, channel gain, and noise power. Determined based on the non-orthogonal multiple access rate formula; to The weighting coefficient is a positive value. This is an indicator function that takes the value 1 if the condition is met, and 0 otherwise.

[0031] and The length and width of the service area; This represents the minimum rate threshold.

[0032] Preferably, the training process of the improved dual-delay deep deterministic policy gradient algorithm that integrates priority experience replay and n-step reward includes:

[0033] Randomly initialize actor network and the dual critics network The weight parameters are set, and their respective target networks are initialized with the same weights. Establish a priority experience back slowdown storage area based on a summation tree structure. A first-in, first-out Step transition queue And set the n-step length and priority index. and importance sampling index ;

[0034] At the start of each training round, the environment is reset to obtain the initial state. For time slots To N, select actions based on the current actor network and explored noise. ,in Mean 0, standard deviation The Gaussian noise gradually decays as training progresses; the environment provides an immediate reward after the action is performed. Next state and termination mark ;

[0035] Transfer samples in one step Store in n-step queue The tail; when When the accumulated number of samples reaches the preset n-step length n, retrieve the earliest n consecutive samples and calculate the n-step cumulative return. ,in, for The time slot offset index within the step window ranges from 0 to... , As a discount factor, As an immediate reward for the corresponding time slot; the status ,action n-step return Final state and final termination mark Combined into an n-step transition, and stored in the priority experience replay cache with the highest priority in the current cache. middle;

[0036] When cache Once the number of samples reaches the preset batch size, a network update is performed every certain number of time slots: from The data is sampled in a priority distribution, containing J transitions, where the first... The sampling probability of the transition is ,in for The priority value of the transfer. The TD error is calculated; and the importance sampling weights for each sample are calculated. To correct for distribution offset;

[0037] For each sampled transition, the target action for the next state is calculated using the target actor network. ,in To smooth the target policy, Gaussian noise truncated to [-c, c] is used; then the target Q-value is calculated. In the formula For the transfer Step-by-step return, As a termination marker, For the next state, The discount factor is used; the smaller value in the dual-Q network is used to suppress overestimation.

[0038] By minimizing the weighted mean square error loss function Simultaneously update both critic networks; after the update is complete, adjust according to the new temporal difference error. Recalculate the priority of each transfer and update the corresponding priority value in the cache;

[0039] Every Step 1: Update the actor network: Maximize via gradient ascent The desired optimization of actor parameters is achieved; subsequently, a soft update is performed on all target networks: ,in As a soft update factor, Set the delay update interval for the actor network and the target network; repeat the above training process until the preset maximum number of training rounds is reached, and finally save the trained actor network parameters as a joint optimization strategy.

[0040] Preferably, in the calculation of the n-step cumulative return, the discount factor The value range is (0,1], and a value close to 1 is used to emphasize long-term returns. When the actual number of steps that can be collected in the n-step queue is less than the preset n due to the early termination of the round, the discounted return corresponding to the actual number of steps is used as the return value for the n-step transfer.

[0041] A second aspect of the present invention provides a hybrid energy laser-powered unmanned aerial vehicle (UAV) communication system based on reinforcement learning, characterized in that the system, for implementing the aforementioned method, comprises:

[0042] The model building unit is used to construct a model of a non-orthogonal multiple access downlink communication system for a laser-powered UAV with a hybrid photovoltaic-grid power supply. This model includes a UAV-to-ground channel model, a non-orthogonal multiple access rate model, a laser beam guide energy emission model, a photovoltaic power generation model, and a rotary-wing UAV energy consumption and battery dynamic model, as well as determining various constraints.

[0043] The problem transformation unit is used to construct the joint optimization problem of maximizing the total system throughput into a Markov decision process, defining the state space, action space and composite reward function;

[0044] The training unit is used to run an improved dual-delay deep deterministic policy gradient algorithm that integrates priority experience replay and n-step reward to train the Markov decision process offline and obtain a joint optimization policy.

[0045] The decision execution unit is used during the online operation phase to call upon the actor network to output actions based on the real-time status, and dynamically adjust the drone's horizontal displacement, laser emission power, and fractional-order power distribution index.

[0046] As can be seen from the above technical solution, the present invention provides a photovoltaic-grid hybrid laser-powered UAV NOMA joint optimization method and system based on improved TD3. It fully explores the replenishment potential of photovoltaic energy for laser charging. Through joint dynamic optimization of UAV trajectory, laser power and NOMA power allocation factor, the UAV maintains battery energy above a safe level while serving mobile / static users, and maximizes the total network throughput. The introduction of priority experience replay and n-step reward significantly improves the training convergence speed and final performance, reducing grid energy consumption to about 36.4% compared to the traditional fixed laser power pure grid power supply scheme, and realizing the synergy of efficient communication and green energy supply. Attached Figure Description

[0047] Figure 1 This is a scene diagram of the photovoltaic grid hybrid laser-powered drone non-orthogonal multiple access system of the present invention.

[0048] Figure 2 This is a schematic diagram of the three-dimensional trajectory of the UAV and the distribution of users in an embodiment of the present invention.

[0049] Figure 3 This is a schematic diagram illustrating the throughput variations of different algorithm systems in embodiments of the present invention.

[0050] Figure 4 This is a dynamic change diagram of the electrical power, solar power, and grid power at the laser beam guide in an embodiment of the present invention.

[0051] Figure 5 This is a comparison chart of the change in the state of charge of the drone battery over time under various schemes in the embodiments of the present invention. Detailed Implementation

[0052] The technical solution and effects of the present invention will be further described in detail below with reference to the accompanying drawings.

[0053] This invention proposes a hybrid energy laser-powered UAV communication method and system based on reinforcement learning, and uses an improved TD3 deep reinforcement learning algorithm to jointly optimize the UAV trajectory, laser emission power, and NOMA power allocation.

[0054] The first aspect of this invention provides a communication method for hybrid energy laser-powered unmanned aerial vehicles based on reinforcement learning, comprising:

[0055] The system consists of photovoltaic panels, a power grid, laser beam guides, rotary-wing drones, and A downlink communication system model consisting of several ground users has a laser beam director fixed to the top of a building. Its power input is supplied by a combination of solar energy converted from photovoltaic panels and electricity provided by the power grid. The laser beam director converts the input power into electro-optical energy with high efficiency. The laser beam is converted into a laser beam and continuously emitted towards the rotorcraft drone flying in the air. The rotorcraft drone is equipped with a photoelectric receiver and energy harvesting circuitry to receive laser energy to charge its onboard battery. At the same time, the rotorcraft drone acts as an aerial base station at a fixed altitude. Onboard flight, utilizing non-orthogonal multiple access methods with Downlink communication is conducted among ground users, who are sorted according to channel gain and signal detection is performed using continuous interference cancellation technology.

[0056] The entire mission cycle T is discretized into N equal-length time slots. Within each time slot, it is assumed that the UAV position, user position, channel gain, and all control variables remain constant. With the objective of maximizing the total throughput for all users within the mission cycle, a joint optimization of the UAV horizontal trajectory is established under constraints including UAV battery state-of-charge safety, minimum communication rate per user per time slot, maximum UAV flight speed, upper and lower limits of laser emission power, and the range of the fractional-order power allocation exponent for non-orthogonal multiple access. Laser beam guide laser emission power and fractional power distribution index The optimization problem of nonconvex temporal coupling;

[0057] The non-convex temporally coupled optimization problem is transformed into a Markov decision process, and the state space is defined. Action space and reward function Among them, state Based on the current battery power of the drone Horizontal position Flight speed V(t), laser emission power in the previous time slot Current power grid supply capacity The previous time slot fractional power allocation index and the current channel gain vectors of all K users. It is pieced together; action For the horizontal displacement increment of the UAV Laser emission power increment and fractional power distribution exponential increment The continuous vector is composed of the following: the reward function r(t) is composed of the weighted throughput reward, the weighted throughput increment reward, the net change in battery energy reward, the penalty for the battery falling below the safety limit, the penalty for the user not meeting the minimum rate requirement, and the penalty for the drone flying out of the service area boundary.

[0058] An improved dual-delay deep deterministic policy gradient algorithm, which integrates priority experience replay and n-step reward, is used to train the Markov decision process offline, learning the optimal policy that can output continuous actions based on the current state. During training, the sampling probability of key samples with high temporal difference error is increased by the priority experience replay mechanism, and the value estimation propagation is accelerated by using n-step reward. The actor network and the dual critic network adopt delayed update and soft update strategies to stabilize the training process.

[0059] After training, in actual operation, based on the real-time status observed in each time slot, the trained actor network directly provides the action, and after amplitude limiting, dynamically adjusts the horizontal displacement of the rotor drone, the laser emission power of the laser beam guide, and the fractional power allocation index of the non-orthogonal multiple access, thereby maximizing throughput and sustainable management of drone battery energy.

[0060] Electrical input power of laser beam guide With laser emission power The relationship is ,and Solar power provided by photovoltaic panels and supplemental power provided by the power grid Common satisfaction, that is Among them, solar power The solar radiation intensity I(t) and photovoltaic conversion efficiency in the current time slot Photovoltaic panel area and cloud attenuation factor according to Calculations show that when the solar power exceeds the electrical power required by the laser beam guide, the power supply from the grid can be reduced or even fed back to the grid, thereby achieving on-demand allocation of hybrid energy.

[0061] Average laser power received by rotary-wing drones Based on the free-space light propagation loss and atmospheric attenuation model, the calculation is as follows:

[0062]

[0063] in, The electro-optical conversion efficiency of a photodiode. This is the turbulence and alignment loss factor, which takes a value of 1 under the assumptions of ideal alignment and clear sky. This is the optical efficiency factor. The effective area of ​​the photoelectric receiver carried by the drone. The combined optical efficiency of the transmitting and receiving system is given by denoted ... The real-time spatial distance between the laser beam guide and the UAV, where For the fixed height of the laser beam guide, The divergence angle of the laser beam. The atmospheric attenuation coefficient per unit distance. This represents the Euclidean norm.

[0064] The energy dynamic equation for the onboard battery of a rotary-wing UAV is:

[0065]

[0066] in The battery energy storage at the start of time slot t, with each time slot having a length of . , This represents the total power consumption of the rotary-wing UAV in the current time slot, which consists of three parts: propulsion power consumption related to flight speed and aerodynamic parameters. Fixed power consumption of airborne circuits And communication power consumption for downlink non-orthogonal multiple access transmission. Battery energy is limited by its maximum capacity. and the safety lower limit to prevent over-discharge That is, it must always satisfy .

[0067] Rotary-wing drone propulsion power consumption Calculated using a classic rotor aerodynamic model, its speed is comparable to the horizontal flight speed of the UAV. The relationship is:

[0068]

[0069] in, This represents the blade profile power in a hovering state. This refers to the induced power during hovering. The rotor tip speed, The average rotor induced velocity during hovering. For the fuselage drag ratio, Let be the air density, s be the rotor solidity, and A be the rotor disk area. This model accurately reflects the propulsion energy consumption characteristics of a rotary-wing UAV in hovering, low-speed, and high-speed flight states, providing a basis for the accurate calculation of battery dynamic equations.

[0070] Non-orthogonal multiple access fractional power allocation index Used to adaptively adjust the communication power allocation ratio of each user in each time slot according to the user channel gain; the communication power allocated to the k-th user. Represented as:

[0071]

[0072] in, For the first The channel power gain of user k, where j represents the user with a channel gain greater than k. For reference, the channel gain at a distance of 1 meter. The distance between the drone and the user is expressed in European terms. Let be the total communication power transmitted by the UAV; the range of values ​​for α(t) is... .when As the distribution approaches the lower limit, it tends to become more equal. When the capacity is increased, more power is allocated to edge users with poor channel performance, thus achieving a flexible trade-off between throughput and user fairness.

[0073] The reward function r(t) is expressed as:

[0074]

[0075] in, Let B be the achievable rate of the k-th user in time slot t, which is determined by bandwidth B, allocated power, channel gain, and noise power. Determined based on the non-orthogonal multiple access rate formula; to These are positive weighting coefficients used to adjust the relative importance of various rewards; The reward function is set to 1 if the condition is met, and 0 otherwise. This reward function is designed so that while pursuing high throughput, the agent must maintain battery power within a safe range and ensure that each user's rate does not fall below a minimum threshold. And to prevent drones from flying out of the preset service boundaries.

[0076] and The length and width of the service area; This represents the minimum rate threshold.

[0077] The training process of the improved dual-delay deep deterministic policy gradient algorithm, which integrates priority experience replay and n-step reward, includes:

[0078] Randomly initialize actor network and the dual critics network The weight parameters are set, and their respective target networks are initialized with the same weights. Establish a priority experience back slowdown storage area based on a summation tree structure. A first-in, first-out Step transition queue And set the n-step length and priority index. and importance sampling index ;

[0079] At the start of each training round, the environment is reset to obtain the initial state. For time slots To N, select actions based on the current actor network and explored noise. ,in Mean 0, standard deviation The Gaussian noise gradually decays as training progresses; the environment provides an immediate reward after the action is performed. Next state and termination mark ;

[0080] Transfer samples in one step Store in n-step queue The tail; when When the accumulated number of samples reaches the preset n-step length n, retrieve the earliest n consecutive samples and calculate the n-step cumulative return. ,in, for The time slot offset index within the step window ranges from 0 to... , As a discount factor, As an immediate reward for the corresponding time slot; the status ,action n-step return Final state and final termination mark Combined into an n-step transition, and stored in the priority experience replay cache with the highest priority in the current cache. middle;

[0081] When cache Once the number of samples reaches the preset batch size, a network update is performed every certain number of time slots: from The data is sampled in a priority distribution, containing J transitions, where the first... The sampling probability of the transition is ,in for The priority value of the transfer. The TD error is calculated; and the importance sampling weights for each sample are calculated. To correct for distribution offset;

[0082] For each sampled transition, the target action for the next state is calculated using the target actor network. ,in To smooth the target policy, Gaussian noise truncated to [-c, c] is used; then the target Q-value is calculated. In the formula For the transfer Step-by-step return, As a termination marker, For the next state, The discount factor is used; the smaller value in the dual-Q network is used to suppress overestimation.

[0083] By minimizing the weighted mean square error loss function Simultaneously update both critic networks; after the update is complete, adjust according to the new temporal difference error. Recalculate the priority of each transfer and update the corresponding priority value in the cache;

[0084] Every Step 1: Update the actor network: Maximize via gradient ascent The desired optimization of actor parameters is achieved; subsequently, a soft update is performed on all target networks: ,in As a soft update factor, Set the delay update interval for the actor network and the target network; repeat the above training process until the preset maximum number of training rounds is reached, and finally save the trained actor network parameters as a joint optimization strategy.

[0085] In the calculation of n-step cumulative return, the discount factor The value range is (0,1], and a value close to 1 is taken to emphasize long-term rewards. When the actual number of steps that can be collected in the n-step queue is less than the preset n due to the early termination of the round, the discounted reward corresponding to the actual number of steps is used as the reward value for the n-step transition, thereby ensuring the stability of the training process.

[0086] A second aspect of the present invention provides a hybrid energy laser-powered unmanned aerial vehicle (UAV) communication system based on reinforcement learning, characterized in that the system, for implementing the aforementioned method, comprises:

[0087] The model building unit is used to construct a model of a non-orthogonal multiple access downlink communication system for a laser-powered UAV with a hybrid photovoltaic-grid power supply. This model includes a UAV-to-ground channel model, a non-orthogonal multiple access rate model, a laser beam guide energy emission model, a photovoltaic power generation model, and a rotary-wing UAV energy consumption and battery dynamic model, as well as determining various constraints.

[0088] The problem transformation unit is used to construct the joint optimization problem of maximizing the total system throughput into a Markov decision process, defining the state space, action space and composite reward function;

[0089] The training unit is used to run the improved dual-delay deep deterministic policy gradient algorithm that combines priority experience replay and n-step reward to train the Markov decision process offline and obtain the joint optimization policy.

[0090] The decision execution unit is used during the online operation phase to call upon the actor network to output actions based on the real-time status, and dynamically adjust the drone's horizontal displacement, laser emission power, and fractional-order power distribution index.

[0091] The following is a specific implementation example, which is used to illustrate the application of the solution and analyze its effects with specific parameters.

[0092] like Figure 1 As shown, the system consists of photovoltaic panels, a power grid, a laser beam director (LBD), a rotary-wing UAV, and K ground users. The LBD is fixed to the roof of a building, receiving hybrid power from the photovoltaic panels and the power grid, and after electro-optical conversion, transmitting a laser beam to the UAV. The UAV carries a photoelectric receiver and energy harvesting circuitry, converting the received laser energy into electrical energy to charge its onboard battery; simultaneously, the UAV acts as an airborne base station, using a non-orthogonal multiple access method for downlink communication with ground users. The total mission duration T is divided into N time slots, each time slot having a length of... .

[0093] The channel between the UAV and user k adopts a line-of-sight dominant model, and the channel power gain is:

[0094]

[0095] Users are arranged in descending order of channel gain. Perfect continuous interference cancellation is performed at the receiver. The achievable rate of user k is... Determined by total bandwidth B and allocated power and noise power Determined jointly. A fractional-order power allocation strategy is adopted, with the allocated power determined by the fractional-order power allocation index. Decide.

[0096] The distance between the LBD and the drone is The average received laser power is calculated using the formula described above, where each parameter is set according to the actual laser device, such as the photodiode efficiency. Atmospheric attenuation coefficient Electro-optical conversion efficiency Etc. Photovoltaic power generation The solar irradiance intensity I(t) is determined by real-time solar irradiance intensity I(t) (which can be obtained from actual meteorological data platforms) and photovoltaic panel area. Photovoltaic efficiency and cloud attenuation factor Calculated.

[0097] Total energy consumption of drones The expression is:

[0098]

[0099] The definitions and values ​​of each parameter are as follows:

[0100] (Circuit power consumption is a constant value);

[0101] (Communication power consumption, a constant value);

[0102] To reduce power consumption, the classic rotor power consumption model is adopted, and its expression is:

[0103]

[0104] In the formula, For blade profile power, For induced power, key parameters involved include: rotor tip speed. Hovering guidance speed Airframe drag ratio air density , rotor solidity s and rotor disk area A.

[0105] Drone batteries dynamically follow an energy balance equation, and their charge level needs to be maintained within a certain range. Inside, the battery's maximum capacity .

[0106] Markov decision process modeling:

[0107] The joint control problem for each time slot is modeled as an MDP. State vector Include drone location ,speed The previous moment Current power grid The previous moment and the channel gain of all users Action vectors Displacement increment of UAV Laser power increment and exponential increment The reward function r(t) is composed of several components, which are scaled and limited before being applied to the environment. The reward function r(t) is a weighted composite of the throughput term, throughput increment term, net battery change term, low battery voltage penalty, user rate failure penalty, and boundary violation penalty. The weighting coefficients are optimized through experiments.

[0108] Improved TD3 algorithm training:

[0109] An improved TD3 algorithm is adopted, introducing Priority Experience Replay (PER) and a 3-step reward. An experience replay cache is constructed, using the absolute value of the temporal difference error as the priority indicator. During each sampling, a small batch of samples is drawn according to priority probability, and bias is corrected using importance sampling weights. The n-step reward accumulates the discounted rewards of the next n consecutive steps, allowing value estimation to reflect future gains more quickly, which is particularly helpful for rapidly learning the consequences of battery critical states. Both the actor network and the dual critic network are fully connected neural networks. During training, the critic network calculates the target value using a truncated double Q-learning method. Only one update is needed for the actor network and target network, with a soft update coefficient. .

[0110] Simulation verification:

[0111] The simulation was implemented using Python 3.8.8 on a computer equipped with an Intel Core i5-11300H processor. (Refer to...) Figure 2 Within a 500m x 500m service area, 10 ground users are deployed, divided into two hotspot clusters. Some users move during the middle of the mission to simulate personnel flow in hotspot scenarios such as stadiums. The user movement follows a unidirectional continuous flow model from fluid dynamics. The UAV starts at (0,0,100m) and has a maximum speed of... Laser emission power The power output is adjustable between 200W and 1000W. In simulations, the method of this invention (I-TD3) was compared with standard TD3, DDPG, SAC, PPO, and PSO algorithms. The UAV possesses omnidirectional adaptive position adjustment capabilities, enabling it to perceive the distribution and movement of ground users in real time, dynamically adjusting its own two-dimensional coordinates to maintain efficient coverage of core user clusters. When user clusters experience aggregation or dispersion changes, the hovering position and flight path are updated synchronously to ensure stable laser energy transmission and communication links. For mobile users who shift during the mission, the UAV relies on user trajectory prediction and real-time position feedback for dynamic response, rapidly changing its flight direction and speed to avoid link interruptions and performance degradation caused by user movement. Simultaneously, the laser emission power... Under the multi-objective constraints of energy transmission efficiency and communication throughput, the system completes collaborative optimization decision-making with the help of reinforcement learning, autonomously outputs the optimal hovering height and horizontal coordinates, and takes into account both global coverage of ground user clusters and local precise services. The complete flight trajectory of the UAV from the initial position (0, 0, 100m) to the destination is smooth and continuous, strictly follows the maximum flight speed constraint, and continuously iterates and optimizes its own position based on the spatiotemporal distribution characteristics of users at different time steps represented by the color gradient.

[0112] The simulation parameter settings are shown in Table 1:

[0113] Table 1. Simulation Parameter Settings for I-TD3 Algorithm

[0114] Figure 3 The throughput variations of different algorithm systems are shown. I-TD3 converges to 94.8 Mbps in approximately 100 rounds, faster than all baselines, and also achieves the highest final throughput. The proposed I-TD3 algorithm converges quickly in the early stages of training, with an average throughput significantly higher than other comparative algorithms, and maintains the highest and most stable performance level throughout the entire training process. This demonstrates that the method of this invention can autonomously learn to achieve an efficient balance between maximizing throughput and energy sustainability.

[0115] Figure 4 Real-time variation curves of LBD-side power, solar power, and grid power are presented. During the complete mission cycle of 600 time slots, the solar power... Determined by solar irradiance and photovoltaic parameters, the power remains relatively constant at approximately 684W. (Laser emission power) The system automatically adjusts to the lowest setting when the drone approaches the LBD to avoid overcharging, and gradually increases the setting as the drone moves away to compensate for transmission losses; correspondingly, the grid provides supplemental power. It exhibits a characteristic of dynamically adjusting with the laser emission power. This contrasts with traditional methods that always drive at maximum laser power. =1000W, corresponding electrical power Compared to a pure grid power supply scheme (e.g., 2000W), the present invention reduces the average grid energy consumption to approximately 36.4%, significantly saving electricity costs and demonstrating the dual benefits of photovoltaic clean energy substitution and dynamic power management.

[0116] Figure 5 The dynamic changes in the UAV battery state of charge (SOC) under various algorithms throughout the mission cycle are shown. The I-TD3 algorithm proposed in this invention can maintain the battery SOC at a safe level of approximately 28.6% while ensuring high throughput. Initially, due to increased flight distance and power consumption, it slightly decreases to around 26.4%. Subsequently, through adaptive adjustment of laser receiver power and flight strategy, the battery level gradually recovers and stabilizes at 27.3%~28.6%, consistently remaining well above the safe lower limit of 20%. In contrast, the DDPG and PSO schemes show a continuous upward trend in battery SOC, reaching approximately 46% and 43.5% respectively, indicating a tendency to conservatively accumulate excess energy, resulting in energy waste. The SAC and PPO schemes exhibit larger fluctuations in battery SOC, initially rising and then falling. The standard TD3 scheme, however, gradually decreases to approximately 25%, closer to the safe lower limit. These results verify the I-TD3's ability to finely coordinate energy balance and communication performance.

[0117] In summary, this invention relates to multiple interdisciplinary technical fields such as wireless communication, UAV applications, laser wireless power transmission, and deep reinforcement learning. In particular, it relates to a method and system for joint optimization of trajectory, power, and resources of photovoltaic-grid hybrid laser-powered UAV nonorthogonal multiple access (NOMA) networks based on the improved dual-delay deep deterministic policy gradient (I-TD3) algorithm for hotspot capacity enhancement scenarios.

[0118] Specifically, at the communication system level, this invention relates to UAV-assisted non-orthogonal multiple access technology, which significantly increases system capacity and the number of users accessing the system under limited spectrum resources by achieving multi-user multiplexing in the power domain. At the energy supply level, this invention relates to laser wireless power transmission technology and a photovoltaic-grid hybrid power supply architecture, which uses a laser beam director to convert solar energy collected by photovoltaic panels and supplemental power from the grid into a highly directional laser beam to continuously charge rotorcraft UAVs in flight without contact, eliminating service interruptions caused by traditional return-to-base charging or battery swapping schemes, while significantly reducing dependence on grid power. At the decision optimization level, this invention relates to the application of deep reinforcement learning in the joint optimization of wireless communication and UAV trajectory, which models the multidimensional continuous variable joint optimization problem as a Markov decision process and introduces an improved TD3 algorithm that integrates priority experience replay and an n-step reward mechanism, to achieve real-time dynamic joint optimization of UAV trajectory, laser emission power, and NOMA power allocation index under multiple constraints such as battery safety, communication quality, and flight boundary. The technical solution of this invention spans multiple disciplines, including wireless communication network architecture, laser optical transmission, photovoltaic energy systems, rotorcraft flight dynamics, and artificial intelligence algorithms, and has significant characteristics of multi-field technology integration.

[0119] This invention solves the problem of coordinating energy efficiency and communication performance optimization in dynamic hotspot scenarios for laser-powered UAV networks by introducing photovoltaic clean energy, dynamically adjusting laser power, and jointly optimizing multi-dimensional resources, and by using an improved deep reinforcement learning algorithm. It has advantages such as strong robustness, fast convergence, and low grid dependence.

[0120] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A communication method for hybrid energy laser-powered unmanned aerial vehicles based on reinforcement learning, characterized in that, include: The system consists of photovoltaic panels, a power grid, laser beam guides, rotary-wing drones, and A downlink communication system model consisting of several ground users is described, in which the laser beam director is fixed to the top of a building, and its power input is supplied by a combination of solar energy converted from photovoltaic panels and electricity provided by the power grid; the laser beam director converts the input power into electro-optical energy with high efficiency. The laser beam is converted into a laser beam and continuously emitted towards the rotorcraft drone flying in the air. The rotorcraft drone is equipped with a photoelectric receiver and energy harvesting circuitry to receive laser energy to charge its onboard battery. At the same time, the rotorcraft drone acts as an aerial base station at a fixed altitude. Onboard flight, utilizing non-orthogonal multiple access methods with Downlink communication is conducted among ground users, who are sorted according to channel gain and signal detection is performed using continuous interference cancellation technology. The entire mission cycle T is discretized into N equal-length time slots. Within each time slot, it is assumed that the UAV position, user position, channel gain, and all control variables remain constant. With the objective of maximizing the total throughput for all users within the mission cycle, a joint optimization of the UAV horizontal trajectory is established under constraints including UAV battery state-of-charge safety, minimum communication rate per user per time slot, maximum UAV flight speed, upper and lower limits of laser emission power, and the range of the fractional-order power allocation exponent for non-orthogonal multiple access. Laser beam guide laser emission power and fractional power distribution index The optimization problem of nonconvex temporal coupling; The non-convex temporally coupled optimization problem is transformed into a Markov decision process, and a state space is defined. Action space and reward function Among them, state Based on the current battery power of the drone Horizontal position Flight speed V(t), laser emission power in the previous time slot Current power grid supply capacity The previous time slot fractional power allocation index and the current channel gain vectors of all K users. It is pieced together; action For the horizontal displacement increment of the UAV Laser emission power increment and fractional power distribution exponential increment The continuous vector is composed of the following: the reward function r(t) is composed of the weighted throughput reward, the weighted throughput increment reward, the net change in battery energy reward, the penalty for the battery falling below the safety limit, the penalty for the user not meeting the minimum rate requirement, and the penalty for the drone flying out of the service area boundary. An improved dual-delay deep deterministic policy gradient algorithm, which integrates priority experience replay and n-step reward, is used to train the Markov decision process offline, learning the optimal policy that can output continuous actions based on the current state. During training, the sampling probability of key samples with high temporal difference error is increased through the priority experience replay mechanism, and the value estimation propagation is accelerated simultaneously using the n-step reward. The actor network and the dual critic network adopt delayed update and soft update strategies to stabilize the training process. After training, in actual operation, based on the real-time status observed in each time slot, the trained actor network directly provides the action, and after amplitude limiting, dynamically adjusts the horizontal displacement of the rotor drone, the laser emission power of the laser beam guide, and the fractional power allocation index of the non-orthogonal multiple access, thereby maximizing throughput and sustainable management of drone battery energy.

2. The method according to claim 1, characterized in that, The electrical input power of the laser beam guide With laser emission power The relationship is ,and Solar power provided by photovoltaic panels and supplemental power provided by the power grid Common satisfaction, that is Among them, solar power The solar radiation intensity I(t) and photovoltaic conversion efficiency in the current time slot Photovoltaic panel area and cloud attenuation factor according to Calculated.

3. The method according to claim 1, characterized in that, The average power of the laser received by the rotary-wing UAV Based on the free-space light propagation loss and atmospheric attenuation model, the calculation is as follows: ； in, The electro-optical conversion efficiency of a photodiode. This is the turbulence and alignment loss factor, which takes a value of 1 under the assumptions of ideal alignment and clear sky. This is the optical efficiency factor. The effective area of ​​the photoelectric receiver carried by the drone. The combined optical efficiency of the transmitting and receiving system is given by denoted ... The real-time spatial distance between the laser beam guide and the UAV, where For the fixed height of the laser beam guide, The divergence angle of the laser beam. The atmospheric attenuation coefficient per unit distance. This represents the Euclidean norm.

4. The method according to claim 3, characterized in that, The energy dynamic equation for the onboard battery of the rotary-wing UAV is: ； in The battery energy storage at the start of time slot t, with each time slot having a length of . , This represents the total power consumption of the rotary-wing UAV in the current time slot, which consists of three parts: propulsion power consumption related to flight speed and aerodynamic parameters. Fixed power consumption of airborne circuits And communication power consumption for downlink non-orthogonal multiple access transmission. Battery energy is limited by its maximum capacity. and the safety lower limit to prevent over-discharge That is, it must always satisfy .

5. The method according to claim 4, characterized in that, The propulsion power consumption of the rotary-wing drone Calculated using a classic rotor aerodynamic model, its speed is comparable to the horizontal flight speed of the UAV. The relationship is: ； in, This represents the blade profile power in a hovering state. This refers to the induced power during hovering. The rotor tip speed, The average rotor induced velocity during hovering. For the fuselage drag ratio, ρ is the air density, s is the rotor solidity, and A is the rotor disk area.

6. The method according to claim 5, characterized in that, The non-orthogonal multiple access fractional power allocation index Used to adaptively adjust the communication power allocation ratio of each user in each time slot according to the user channel gain; the communication power allocated to the k-th user. Represented as: ； in, For the first The channel power gain of user k, where j represents the user with a channel gain greater than k. For reference, the channel gain at a distance of 1 meter. The distance between the drone and the user is expressed in European terms. Let be the total communication power transmitted by the UAV; the range of values ​​for α(t) is... .

7. The method according to claim 6, characterized in that, The reward function r(t) is expressed as: ； in, Let B be the achievable rate of the k-th user in time slot t, which is determined by bandwidth B, allocated power, channel gain, and noise power. Determined based on the non-orthogonal multiple access rate formula; to The weighting coefficient is a positive value. This is an indicator function that takes the value 1 if the condition is met, and 0 otherwise. and The length and width of the service area; This represents the minimum rate threshold.

8. The method according to claim 7, characterized in that, The training process of the improved dual-delay deep deterministic policy gradient algorithm that integrates priority experience replay and n-step reward includes: Randomly initialize actor network and the dual critics network The weight parameters are set, and their respective target networks are initialized with the same weights. Establish a priority experience back slowdown storage area based on a summation tree structure. A first-in, first-out Step transition queue And set the n-step length and priority index. and importance sampling index ; At the start of each training round, the environment is reset to obtain the initial state. For time slots To N, select actions based on the current actor network and explored noise. ,in Mean 0, standard deviation The Gaussian noise gradually decays as training progresses; the environment provides an immediate reward after the action is performed. Next state and termination mark ; Transfer samples in one step Store in n-step queue The tail; when When the accumulated number of samples reaches the preset n-step length n, retrieve the earliest n consecutive samples and calculate the n-step cumulative return. ,in, for The time slot offset index within the step window ranges from 0 to... , As a discount factor, As an immediate reward for the corresponding time slot; the status ,action n-step return Final state and final termination mark Combined into an n-step transition, and stored in the priority experience replay cache with the highest priority in the current cache. middle; When cache Once the number of samples reaches the preset batch size, a network update is performed every certain number of time slots: from The data is sampled in a priority distribution, containing J transitions, where the first... The sampling probability of the transition is ,in for The priority value of the transfer. The TD error is calculated; and the importance sampling weights for each sample are calculated. To correct for distribution offset; For each sampled transition, the target action for the next state is calculated using the target actor network. ,in To smooth the target policy, Gaussian noise truncated to [-c, c] is used; then the target Q-value is calculated. In the formula For the transfer Step-by-step return, As a termination marker, For the next state, The discount factor is used; the smaller value in the dual-Q network is used to suppress overestimation. By minimizing the weighted mean square error loss function Simultaneously update both critic networks; after the update is complete, adjust according to the new temporal difference error. Recalculate the priority of each transfer and update the corresponding priority value in the cache; Every Step 1: Update the actor network: Maximize via gradient ascent The desired optimization of actor parameters is achieved; subsequently, a soft update is performed on all target networks: ,in As a soft update factor, The delay interval between the actor network and the target network is set; the training process is repeated until the preset maximum number of training rounds is reached, and finally the trained actor network parameters are saved as a joint optimization strategy.

9. The method according to claim 8, characterized in that, In the calculation of the n-step cumulative return, the discount factor The value range is (0,1], and a value close to 1 is used to emphasize long-term returns. When the actual number of steps that can be collected in the n-step queue is less than the preset n due to the early termination of the round, the discounted return corresponding to the actual number of steps is used as the return value for the n-step transfer.

10. A hybrid energy laser-powered unmanned aerial vehicle (UAV) communication system based on reinforcement learning, characterized in that, The system for implementing the method according to any one of claims 1-9 comprises: The model building unit is used to construct a model of a non-orthogonal multiple access downlink communication system for a laser-powered UAV with a hybrid photovoltaic-grid power supply. This model includes a UAV-to-ground channel model, a non-orthogonal multiple access rate model, a laser beam guide energy emission model, a photovoltaic power generation model, and a rotary-wing UAV energy consumption and battery dynamic model, as well as determining various constraints. The problem transformation unit is used to construct the joint optimization problem of maximizing the total system throughput into a Markov decision process, defining the state space, action space and composite reward function; The training unit is used to run an improved dual-delay deep deterministic policy gradient algorithm that integrates priority experience replay and n-step reward to train the Markov decision process offline and obtain a joint optimization policy. The decision execution unit is used during the online operation phase to call upon the actor network to output actions based on the real-time status, and dynamically adjust the drone's horizontal displacement, laser emission power, and fractional-order power distribution index.

Images