An unmanned aerial vehicle group energy balance optimization method and system based on laser transmission

Abstract

This invention discloses a method and system for optimizing the energy balance of a drone swarm based on laser transmission. The system includes a cloud server and several drones, each equipped with an onboard communication module, an onboard computing module, an onboard battery, and a laser energy receiving module; several ground-based IoT devices for generating data to be collected; and several ground-based laser charging stations, each equipped with a laser transmitter. The cloud server, drones, ground-based IoT devices, and ground-based laser charging stations constitute a drone-assisted data acquisition system. The advantages of this invention are: by employing a federated multi-agent reinforcement learning framework, each drone only uploads local Actor and Critic network parameters to the cloud server, rather than the original observation trajectory, power consumption, or reward data, significantly reducing the amount of data transmitted, avoiding communication congestion, and fundamentally eliminating the risk of sensitive information leakage caused by centralized training.

Description

Technical Field

[0001] This invention relates to the field of collaborative control technology, specifically to a method and system for optimizing the energy balance of unmanned aerial vehicle (UAV) swarms based on laser transmission. Background Technology

[0002] Wide-area IoT data acquisition technology based on drones has been widely applied in fields such as environmental monitoring, precision agriculture, and disaster early warning. Drones, with their high mobility and direct-line-of-sight transmission links, have become an important aerial platform for data acquisition from ground-based sensing devices. However, the limited capacity of drone onboard batteries severely restricts their ability to perform long-term automated data acquisition tasks.

[0003] To overcome energy limitations, laser-based wireless power transfer technology has attracted widespread attention. Ground-based laser charging stations can power drones in real time using directional laser beams, enabling the system to achieve self-sufficient and sustainable operation. However, in multi-drone systems employing laser charging, the joint optimization of flight trajectory planning, charging scheduling, and data acquisition and communication resources is a high-dimensional non-convex optimization problem involving discrete and continuous variables.

[0004] Multi-agent reinforcement learning has become an effective method for solving distributed cooperative control problems, but it still faces three major challenges in large-scale UAV networks: First, centralized training frameworks rely on global state sharing, which leads to huge communication overhead between UAVs and the central controller, easily causing congestion and high latency; Second, the centralized uploading of sensitive information such as UAV flight trajectories and remaining battery power significantly increases the risk of data leakage; Third, as the cluster size increases, competition among multiple UAVs for limited charging and communication resources intensifies, and traditional algorithms struggle to achieve a balance between scalability, communication costs, and learning efficiency.

[0005] Federated learning, as a distributed paradigm that only shares model parameters without transmitting raw data, has been introduced into unmanned aerial vehicle (UAV) networks. However, existing research focuses on optimizing model training costs or allocating single resources. For multi-agent cooperative scheduling problems with continuous action spaces and highly coupled reward mechanisms, there is still a lack of effective solutions to achieve multi-agent joint policy evolution through federated aggregation.

[0006] Therefore, to address the aforementioned technical issues, it is necessary to provide a method and system for optimizing the energy balance of unmanned aerial vehicle (UAV) swarms based on laser transmission. Summary of the Invention

[0007] To address the shortcomings of the existing technologies, the present invention aims to provide a method and system for optimizing the energy balance of unmanned aerial vehicle (UAV) swarms based on laser transmission, thereby solving the problems mentioned in the background section.

[0008] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: This invention provides a method for optimizing the energy balance of a drone swarm based on laser transmission, comprising the following steps: Step S1: Build a system consisting of cloud servers, drones Taiwan ground IoT devices and A drone-assisted data acquisition system consisting of several ground-based laser charging stations divides the entire mission cycle into... There are 1 time slot, and the duration of each time slot is 1. ;in, All are positive integers; Step S2: Model the joint trajectory planning and data acquisition problem of multiple UAVs as a partially observable Markov decision process, for each UAV... Define the local observation space, action space, and reward function. ; Step S3: A federated multi-agent reinforcement learning framework is adopted. Each UAV independently maintains a local Actor network and a local Critic network. The local Actor network outputs flight control actions based on the local observation space, and the local Critic network evaluates the expected cumulative reward of the current state. Step S4: Each UAV collects local experience trajectory during its interaction with the environment, and independently updates its local Actor network and local Critic network based on the local experience trajectory using a near-end strategy optimization method. Step S5: Each drone completes... After local training, the parameters of the local Actor network and the parameters of the local Critic network are uploaded to the cloud server. It is a positive integer; Step S6: The cloud server performs global weighted aggregation on the parameters of the local Actor network of all received drones to obtain global Actor network parameters, and performs global weighted aggregation on the parameters of the local Critic network of all received drones to obtain global Critic network parameters. Step S7: The cloud server broadcasts the global Actor network parameters and the global Critic network parameters to all drones, and each drone updates its local Actor network and local Critic network with the received global Actor network parameters and global Critic network parameters. Step S8: In each decision time slot of the federated multi-agent reinforcement learning framework, the underlying resource allocation module uses a heuristic algorithm to allocate laser power and communication bandwidth based on the flight status and battery level of each UAV.

[0009] In one or more embodiments of the present invention, in step S1, the drone is at a fixed safe altitude. Flight, the flight status of each drone includes flight status and hovering status, the flight status being at a fixed speed When flying horizontally, the spatial coordinates of the UAV remain unchanged during the hovering state; The ground-based laser charging station provides wireless power replenishment to hovering drones within the effective charging radius using a directional laser beam. The ground-based IoT device continuously generates data to be collected, which is then collected by drones within the sensing range of the ground-based IoT device.

[0010] In one or more embodiments of the present invention, in step S2, the local observation space includes the current position of the UAV, the remaining battery power, the set of spatial positions of other UAVs within the sensing range, the amount of data to be collected and the spatial distribution of ground IoT devices within the sensing range, and the location and status information of nearby ground laser charging stations. The action space includes the horizontal flight direction angle. and flight status indicator variables ,in Indicates hovering. Indicates a constant speed Flight; the reward function is defined as:

[0011] in, Indicates drone In the time slot Instant rewards Indicates drone In the time slot Data acquisition rate, Indicates drone In the time slot The replenished energy obtained Indicates drone In the time slot Total energy consumption This indicates the maximum battery capacity of the drone.

[0012] In one or more embodiments of the present invention, in step S4, the proximal policy optimization method includes: calculating the advantage function using generalized advantage estimation. ,in Indicates drone In the time slot The dominant function value, Indicates the discount factor. The smoothing hyperparameters of the generalized dominance estimation are represented. Indicates timing difference error; The update of the local Actor network uses a truncated objective function: ,

[0013] in, Indicates drone Local Actor network parameters, This represents the probability ratio between the new strategy and the old strategy. Indicates the cutoff threshold; The local Critic network is updated using the mean squared error loss function. ,in Indicates drone Local Critic network parameters, This indicates the state of the local Critic network. Value prediction This represents the target value.

[0014] In one or more embodiments of the present invention, in step S6, the global weighted aggregation employs a federated average algorithm:

[0015] in, This represents the global Actor network parameters. This represents the global Critic network parameters. Indicates drone Local Actor network parameters, Indicates drone Local Critic network parameters, Indicates drone The aggregation weight coefficient, and .

[0016] In one or more embodiments of the present invention, in step S8, the laser power allocation includes: each ground laser charging station identifying a set of hovering drones within its effective charging radius; According to the formula Calculate the charging priority score for each hovering drone, where... Indicates drone For charging stations Charging priority rating, Indicates drone In the time slot The remaining battery power, Indicates drone With charging station The Euclidean distance between them; Each charging station selects the one with the highest charging priority score. Establish charging associations by using drones as charging service targets; Laser power is allocated to each charging station according to the charging priority score ratio:

[0017] in, Indicates charging station In the time slot To drones The power of the launch, Indicates charging station Select a collection of drones for the service. This indicates the maximum emission power of a single ground-based laser charging station.

[0018] In one or more embodiments of the present invention, step S8, communication bandwidth allocation includes: Each hovering drone identifies a collection of ground-based IoT devices within its sensing range; According to the formula Calculate the communication scheduling weight for each terrestrial IoT device, where Indicates device For drones Communication scheduling weight, Indicates equipment In the time slot Data queue backlog Indicates device The transmission power, Indicates device With drones Channel gain between Indicates background noise power; Each UAV selects the one with the highest communication scheduling weight. Establish communication links between ground-based IoT devices; Each drone is allocated communication bandwidth according to its communication scheduling weight ratio:

[0019] in, Indicates drone In the time slot Assigned to device Communication bandwidth, Show drone Select the collection of ground-based IoT devices for the service. This represents the available communication bandwidth budget for a single drone.

[0020] In one or more embodiments of the present invention, in step S8, the laser power allocation satisfies the constraint that a single UAV can only receive service from one ground laser charging station in the same time slot:

[0021] in, Indicates drone With charging station In the time slot Charging-related indicator variables; The laser power allocation also satisfies the constraint that the total power allocated to all UAVs by a single ground laser charging station in the same time slot does not exceed its maximum transmission power:

[0022] The communication bandwidth allocation satisfies the constraint that the total bandwidth allocated to all associated devices by a single UAV in the same time slot does not exceed its available communication bandwidth budget:

[0023] in, Indicates drone With equipment In the time slot Communication-related indicator variables, .

[0024] A laser-transmitted unmanned aerial vehicle (UAV) swarm energy equalization optimization system includes: Cloud server; Each drone is equipped with an onboard communication module, an onboard computing module, an onboard battery, and a laser energy receiving module. A ground-based IoT device is used to generate data to be collected; Each ground-based laser charging station is equipped with a laser emitter; The cloud server, drones Taiwan ground IoT devices and A ground-based laser charging station constitutes an unmanned aerial vehicle (UAV) assisted data acquisition system.

[0025] In one or more embodiments of the present invention, each of the UAVs is equipped with a local Actor network and a local Critic network, wherein the local Actor network is used to output flight control actions based on local observation states, and the local Critic network is used to evaluate the expected cumulative reward of the current state. The cloud server is equipped with a global parameter aggregation module, which is used to receive local Actor network parameters and local Critic network parameters uploaded by each UAV, perform global weighted aggregation, and broadcast the aggregated global Actor network parameters and global Critic network parameters to each UAV. The ground laser charging station has a built-in laser power allocation module, which is used to identify hovering drones within the effective charging radius, calculate the charging priority score, and allocate laser power according to the priority ratio. Each of the aforementioned drones is equipped with a communication bandwidth allocation module, which is used to identify ground IoT devices within the sensing range, calculate communication scheduling weights, and allocate communication bandwidth according to the weight ratio.

[0026] The beneficial effects of this invention are as follows: 1. By adopting a federated multi-agent reinforcement learning framework, each drone only uploads local Actor and Critic network parameters to the cloud server, instead of the original observation trajectory, battery level, or reward data, which greatly reduces the amount of data transmitted, avoids communication congestion, and fundamentally eliminates the risk of sensitive information leakage caused by centralized training. 2. The underlying heuristic resource allocation module dynamically allocates laser power and communication bandwidth proportionally based on charging priority scoring and communication scheduling weights, transforming multi-machine competition into ordered priority scheduling. It can maintain efficient and fair resource allocation even when the cluster size expands, and has excellent scalability. 3. A two-layer collaborative mechanism is adopted to decompose the high-dimensional mixed integer non-convex problem into solvable subproblems, taking into account the coupling between long-term strategy and short-term scheduling. The overall performance is significantly better than existing baseline methods. 4. Multiple constraints, including battery safety, charging station power limits, communication bandwidth limits, and collision avoidance, are incorporated to ensure energy neutrality and flight safety during long-term operation; 5. By introducing a low-level heuristic resource allocation module into the reinforcement learning framework, multi-machine resource competition is transformed into ordered priority scheduling based on charging priority scoring and communication scheduling weight. At the same time, the reinforcement learning action space only includes flight direction angle and hovering indication, and charging and bandwidth allocation are solved directly by the algorithm, avoiding additional exploration overhead in the resource competition dimension. It can still maintain high learning efficiency and excellent scalability in large-scale clusters. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a schematic diagram of an unmanned aerial vehicle (UAV) swarm energy balance optimization system in one embodiment of the present invention; Figure 2 This is a schematic diagram of the FMLD architecture of an unmanned aerial vehicle (UAV) swarm energy balance optimization system in one embodiment of the present invention; Figure 3 This is a flight trajectory diagram of a drone under different deployment scenarios in one embodiment of the present invention; Figure 4 This is a schematic diagram illustrating the relationship between normalized reward and training rounds in one embodiment of the present invention; Figure 5 This is a schematic diagram illustrating the system performance under different numbers of IoT devices in one embodiment of the present invention; Figure 6 This is a schematic diagram of system performance under different numbers of laser charging stations in one embodiment of the present invention; Figure 7 This is a schematic diagram of system performance under different numbers of drones in one embodiment of the present invention; Figure 8 This is a schematic diagram of the ablation result in one embodiment of the present invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] Example 1: like Figures 1 to 8 As shown, an embodiment of the present invention provides an energy balance optimization method for UAV swarms based on laser transmission, comprising the following steps: Step S1, construct a system consisting of cloud servers, drones, Taiwan ground IoT devices and A drone-assisted data acquisition system consisting of several ground-based laser charging stations divides the entire mission cycle into T time slots, with each time slot lasting for a duration of [duration missing]. ,in, All values ​​are positive integers. In a specific embodiment of the present invention, the cloud server, as the central node for global parameter aggregation and model distribution, possesses strong computing power and a stable backhaul communication link. Each UAV is equipped with an onboard communication module, an onboard computing module, an onboard battery, and a laser energy receiving module; ground-based IoT devices are used to generate data to be collected; each ground-based laser charging station is equipped with a laser transmitter, which provides wireless energy replenishment to UAVs within its coverage area via a directional laser beam. The simulation time domain is divided into multiple time slots, and the duration of each time slot is preferably set to... Second.

[0031] Step S2: Model the joint trajectory planning and data acquisition problem of multiple UAVs as a partially observable Markov decision process, for each UAV... Define the local observation space, action space, and reward function, where, Since UAVs are limited by their communication and perception ranges in real-world applications and cannot acquire accurate global information, it is reasonable and necessary to model the problem as a partially observable Markov decision process.

[0032] Step S3: A federated multi-agent reinforcement learning framework is adopted. Each UAV independently maintains a local Actor network and a local Critic network. The local Actor network outputs flight control actions based on the local observation space, that is, based on the current local observation state. Probability distribution of output flight control actions The local Critic network evaluates the expected cumulative reward of the current state, i.e., the state value function. .

[0033] Step S4: Each UAV collects local experience trajectories during its interaction with the environment, and generates complete experience tuples. Stored to local track buffer In addition, each local Actor network and local Critic network are independently updated using a proximal strategy optimization method based on local experience trajectories.

[0034] Step S5, each drone completes each... After a round of local training, the parameters of the local Actor network are... Parameters of the local Critic network Upload to cloud server It is a positive integer. The value of needs to strike a balance between communication overhead and model fusion frequency, preferably . It can be an integer between 5 and 10.

[0035] Step S6: The cloud server performs global weighted aggregation on the parameters of the local Actor networks of all received drones to obtain the global Actor network parameters. The parameters of the local Critic network of all received drones are then globally weighted and aggregated to obtain the global Critic network parameters. .

[0036] Step S7, the cloud server will send global Actor network parameters The global Critic network parameters Ψ are broadcast to all drones. Each drone updates its local Actor network and local Critic network based on the received global Actor network parameters and global Critic network parameters, i.e., ... , .

[0037] In step S8, at each decision slot of the federated multi-agent reinforcement learning framework, the underlying resource allocation module uses a heuristic algorithm to allocate laser power and communication bandwidth based on the flight status and battery level of each UAV. Through the above steps, this invention achieves a long-term optimal balance between energy replenishment and data acquisition and transmission for multiple UAVs, while reducing communication overhead and ensuring data privacy.

[0038] Furthermore, in step S1, the drone maintains a fixed safe altitude. Flight, the flight status of each drone includes flight status and hovering status. Flight status is at a fixed speed. When flying horizontally and hovering, the drone's spatial coordinates remain unchanged.

[0039] Specifically, drones In the time slot The three-dimensional spatial position is denoted as When the drone is in flight, its position is updated by the horizontal flight direction angle. It is determined that the state transition equation satisfies When a drone is hovering, its spatial coordinates remain unchanged. In this state, the drone can perform model calculations, data acquisition, or receive laser energy replenishment from a ground-based laser charging station. The ground-based laser charging station provides wireless energy replenishment to hovering drones within its effective charging radius via a directional laser beam. With charging station The Euclidean distance between them is ,in For charging stations Two-dimensional coordinates, only when At that time, drones Only then can charging stations be accepted. Laser energy replenishment, This indicates the effective charging radius. Ground-based IoT devices continuously generate data to be collected, which is then collected by drones within the sensing range of these devices. With equipment The communication association between them is indicated by the indicator variable. express, Indicates device In the time slot Interior and drones Establish a connection and transfer data. This indicates that no association has been established.

[0040] Furthermore, in step S2, the local observation space includes the current position of the UAV. Remaining battery power Spatial location set of other drones within the sensing range (For obstacle avoidance), the amount of data to be collected and the spatial distribution of ground-based IoT devices within the sensing range. and the location and status information of nearby ground-based laser charging stations. drones In the time slot Local observation status .

[0041] The maneuver space includes the horizontal flight direction angle. and flight status indicator variables ,in Indicates hovering to collect data or receive laser charging. Indicates a constant speed Flight, i.e., drones In the time slot Action definition The reward function is defined as:

[0042] in, Indicates drone In the time slot Instant rewards Indicates drone In the time slot The data acquisition rate, i.e., the drone In the time slot The internal data rate is the rate at which its locally associated devices collect data. Indicates drone In the time slot The replenished energy obtained Indicates drone In the time slot Total energy consumption This represents the drone's maximum battery capacity. The reward function is designed to ensure that the local reward of a single drone aligns with the overall utility of the system. It represents the energy balance state, with positive values ​​indicating an energy surplus and negative values ​​indicating an energy deficit, and the data acquisition rate. The product of energy balance factor guides the UAV to maximize data acquisition rate while ensuring energy sustainability, thus achieving an organic unity between local decision-making and global optimization.

[0043] In step S4, the near-end policy optimization method includes calculating the advantage function using generalized advantage estimation. First, the temporal difference error at the current time step is calculated using a Critic network. ,in Indicates the discount factor. Indicates the local Critic network state The value prediction is then performed. Then, based on the time-series difference error, the generalized advantage estimate is calculated:

[0044] in, Indicates drone In the time slot The dominant function value, This represents the smoothing hyperparameter of the generalized dominance estimate, used to achieve a balance between the variance and bias of the dominance estimate.

[0045] The local Actor network update uses a truncated objective function: ,

[0046] in, Indicates drone Local Actor network parameters, This represents the probability ratio between the new strategy and the old strategy. , Indicates the cutoff threshold; the clip function limits the probability ratio to a certain value. Within the range, this avoids excessive policy updates that could lead to training divergence.

[0047] Local Actor Network Parameters Update using gradient ascent:

[0048] in, This represents the learning rate of the Actor network. The local Critic network is updated using the mean squared error loss function. ,in Indicates drone Local Critic network parameters, Indicates the local Critic network state Value prediction This represents the target value. It is obtained by adding the advantage function to the current state value, i.e. Critic network parameters Update using gradient descent:

[0049] in This represents the learning rate of the Critic network. During the local training phase, each UAV utilizes its own collected environmental experience trajectories to... Local iterative updates are performed, and the local trajectory buffer is cleared after the update is complete. This is to prevent old data from causing the strategy to diverge.

[0050] Furthermore, in step S6, the global weighted aggregation employs a federated average algorithm: , ,in Represents the global Actor network parameters. This represents the global Critic network parameters. Indicates drone Local Actor network parameters, Indicates drone Local Critic network parameters, Indicates drone The aggregation weight coefficient, and .

[0051] In one embodiment of the present invention, the aggregation weight coefficients are set with equal weights, that is... This ensures fair integration of the local training results from each drone. In heterogeneous scenarios, where there are significant differences in the amount of local training data or computing power among drones, the weighting coefficients can also be differentiated based on the number of local empirical trajectories or network performance indicators for each drone. For example, it can be set to... ,in Indicates drone The increased number of experience tuples in the local trajectory buffer allows drones with richer training data and more experience to contribute more significantly to global model aggregation. Through this cloud-based federated parameter aggregation mechanism, each drone only needs to interact with the cloud server regarding neural network model weights, without sharing raw local observation data or interaction reward information. This enables cross-drone experience sharing and collaborative strategy evolution, while significantly reducing communication overhead and effectively protecting data privacy. After completing global aggregation, the cloud server will allocate global Actor network parameters... and global Critic network parameters Broadcast to all drones, each drone orders , This enables synchronization between the local model and the global model.

[0052] Furthermore, in step S8, the specific process of laser power allocation is as follows: First, each ground laser charging station identifies its effective charging radius. The collection of hovering drones within each charging station Identify and satisfy and (Hovering) drone , forming a set Drones With charging station The Euclidean distance between them is Then, according to the formula Calculate the charging priority score for each hovering drone, where Indicates drone For charging stations Charging priority rating, Indicates drone In the time slot The remaining battery power, Indicates drone With charging station The Euclidean distance between them. This charging priority scoring system is designed to take into account both the drone's energy urgency ( The larger the value, the lower the battery level and the more urgent the charging need (and spatial distance factors). The smaller the value, the less energy transmission loss and the higher the charging efficiency, ensuring good numerical stability of the score within the physically feasible domain and avoiding score anomalies when the distance is too large.

[0053] Each charging station selects the one with the highest charging priority score. By treating drones as charging service recipients, a charging association is established, that is, the corresponding charging association indicator variables are set. ,in A positive integer, representing the maximum number of drones that a single charging station can serve in a single time slot. Its value is determined by the number of laser emitters and the energy distribution capability of the charging station. In the preferred embodiment of the present invention... Typical values ​​range from 2 to 4. Laser power is allocated to each charging station according to its charging priority score. ,in Indicates charging station In the time slot To drones The power of the launch, Indicates charging station Select a collection of drones for the service. This indicates the maximum emission power of a single ground-based laser charging station. Through this proportional allocation method, drones with larger energy deficits and closer distances receive more laser power, achieving efficient utilization of charging resources.

[0054] drones In the time slot The actual charging power received is:

[0055] in, For photoelectric conversion efficiency, The effective area of ​​the receiver. Using the atmospheric attenuation coefficient, this model accurately characterizes the power attenuation characteristics of laser propagation in the atmosphere. (UAV) In the time slot The energy obtained is .

[0056] Furthermore, the specific process of communication bandwidth allocation is as follows: First, each hovering drone identifies the set of ground-based IoT devices within its perception range; that is, for each hovering drone... (satisfy ), identify terrestrial IoT devices within its sensing range. , forming a set .

[0057] Then, according to the formula Calculate the communication scheduling weight for each terrestrial IoT device, where Indicates equipment For drones Communication scheduling weight, Indicates equipment In the time slot Data queue backlog Indicates equipment The transmission power, Indicates equipment With drones Channel gain between This represents the background noise power. The design of this communication scheduling weight also considers the data backlog status of the devices. The larger the value, the more urgent the data acquisition needs of the device) and the uplink channel quality ( Reflects the equipment To drones The theoretical transmission rate limit is determined by the system's communication scheduling algorithm. This prioritizes services to devices with severe data backlogs and good channel conditions, while ensuring fairness, thereby maximizing the system's data acquisition throughput. Each UAV selects the one with the highest communication scheduling weight. Establish a communication association between terrestrial IoT devices, that is, set the corresponding communication association indicator variable. ,in A positive integer, representing the maximum number of devices that a single UAV can serve simultaneously within a single time slot. Its value is determined by the UAV's communication processing capabilities and bandwidth resources. In the preferred embodiment of this invention... Typical values ​​range from 3 to 5. Communication bandwidth is allocated to each UAV according to its communication scheduling weight.

[0058] in, Indicates drone In the time slot Assigned to device Communication bandwidth, Indicates drone Select the collection of terrestrial IoT devices for the service. This represents the available communication bandwidth budget for a single drone. Through this proportional allocation method, devices with more severe data backlogs and better channel quality receive more bandwidth allocation, achieving efficient utilization of communication resources.

[0059] Ground IoT devices To drones The uplink transmission rate is Drones In the time slot The total amount of data collected internally is Data acquisition rate .

[0060] Furthermore, the underlying resource allocation needs to satisfy a series of physical and system operational constraints. Laser power allocation must satisfy the constraint that a single UAV can only receive service from one ground-based laser charging station within the same time slot:

[0061] in, Indicates drone With charging station In the time slot The charging-related indicator variable ensures that the drone will not be simultaneously exposed to lasers from multiple charging stations, avoiding energy reception chaos and potential equipment safety issues.

[0062] Laser power allocation also satisfies the constraint that the total power allocated to all drones by a single ground laser charging station in the same time slot does not exceed its maximum emission power: This constraint ensures that the transmission power of each charging station does not exceed its physical maximum transmission capacity.

[0063] Communication bandwidth allocation satisfies the constraint that the total bandwidth allocated to all associated devices by a single UAV in the same time slot does not exceed its available communication bandwidth budget:

[0064] in, Indicates drone With equipment In the time slot Communication-related indicator variables, This constraint ensures that the communication bandwidth allocated to each drone does not exceed the upper limit of its hardware capabilities.

[0065] In addition, the system must also meet battery safety constraints:

[0066] in, This represents the lower limit threshold for battery safety, which is the threshold at which a certain agent's policy leads to... In this case, the reinforcement learning environment will apply significant negative rewards, guiding the drone to proactively terminate its data collection mission and fly to the laser charging station's charging range to recharge, preventing the drone from crashing due to running out of power.

[0067] Furthermore, the system must also meet the collision avoidance safety constraints between drones:

[0068] in, This represents the minimum safe distance between drones, a constraint that ensures no mid-air collisions occur during collaborative operations. These constraints collectively constitute the complete feasible region of the optimization problem, ensuring the system's safety, reliability, and rational resource allocation during long-term operation.

[0069] Example 2: like Figure 1 Figure 8As shown, the present invention also provides a laser-transmission-based UAV swarm energy balance optimization system for executing the aforementioned laser-transmission-based UAV swarm energy balance optimization method. This system includes a cloud server, drones Taiwan ground IoT devices and One ground-based laser charging station, among which , , All values ​​are positive integers. Each drone is equipped with an onboard communication module, an onboard computing module, an onboard battery, and a laser energy receiving module. The onboard communication module is used for data communication with ground-based IoT devices and for uploading local network parameters to the cloud server. The onboard computing module is used for inference and training of the local Actor network and Critic network. The onboard battery is used to store electrical energy. The laser energy receiving module is used to receive laser beams emitted by ground-based laser charging stations and convert them into electrical energy. A set of ground-based IoT devices is used to generate data to be collected. Each device continuously generates data and stores it in a local queue to await collection by drones. Each ground-based laser charging station is equipped with a laser transmitter, which provides wireless power replenishment to hovering drones within its effective charging radius via a directional laser beam. (Cloud server, ...) drones, Taiwan ground IoT devices and The ground-based laser charging stations together constitute a drone-assisted data acquisition system. This system realizes collaborative trajectory planning, charging scheduling, and data acquisition for multiple drones under a federated multi-agent reinforcement learning framework.

[0070] Specifically, each drone is equipped with a local Actor network and a local Critic network. The local Actor network has parameters. Used to determine the local observation status Output flight control actions That is, output horizontal flight direction angle and flight status indicator variables The decision, among which , The local Critic network has parameters. Used to assess the current state The expected cumulative return, i.e., the state value function .

[0071] The cloud server is equipped with a global parameter aggregation module, which is used to receive local Actor network parameters uploaded by each drone. and local Critic network parameters Perform global weighted aggregation and broadcast the aggregated global Actor network parameters Φ and global Critic network parameters Ψ to each UAV. The specific aggregation method is as follows:

[0072] Ground-based laser charging stations have a built-in laser power distribution module, which is used to identify hovering drones within the effective charging radius and calculate charging priority scores. And allocate laser power according to priority ratio. Each drone has a built-in communication bandwidth allocation module, which is used to identify ground-based IoT devices within the sensing range and calculate communication scheduling weights. And allocate communication bandwidth according to weight ratio. The modules work together, with the high-level federated multi-agent reinforcement learning module responsible for UAV flight trajectory planning and state control, and the low-level resource allocation module responsible for real-time scheduling of charging and communication resources, jointly achieving efficient data acquisition for multiple UAVs under the premise of sustainable energy.

[0073] Furthermore, the complete training process for the federated multi-agent reinforcement learning framework is as follows: First, initialization is performed; the cloud server initializes the global Actor parameters. and global Critic parameters Each drone sets local parameters , And clear the local trajectory buffer. Then, for each training round, the environment is initialized with the positions of each drone. and battery power In each time slot Inside, each drone Observe the current local state According to the current strategy Sampling action .

[0074] The system calls the underlying resource allocation algorithm to calculate scheduling decisions, including charging association. Communication association Laser power and communication bandwidth Perform joint operations, update drone status, calculate energy consumption and collect data, and obtain immediate rewards. and new status Each drone will have an experience tuple. Store to local track buffer middle.

[0075] After completing the environmental interaction, the local training phase begins: each drone utilizes the buffer zone. Calculate the timing difference error at all time steps. and GAE Advantage Function Then proceed Round-robin local updates, by maximizing the truncation of the target function Update local Actor network parameters By minimizing the mean square error loss Update local Critic network parameters After the update is complete, clear the local track buffer. .

[0076] When the number of training rounds reaches When the value is an integer multiple of the network parameters, the process enters the federated aggregation phase: all drones will update their local network parameters. and Uploaded to the cloud server; the cloud server performs global federated average aggregation to obtain... and The cloud server broadcasts global parameters to all drones. and Each UAV synchronizes its global model parameters, making , Repeat the above process until the global network converges.

[0077] In a specific embodiment of the present invention, the drone Onboard battery capacity Update using the following formula:

[0078] in, Indicates drone In the time slot Total energy consumption To reduce energy consumption, the hovering power is At a constant speed Power during flight ; Energy consumption for data acquisition and communication; For local calculation of energy consumption, among which This indicates that the local training trigger indicator variable, The effective switching capacitor coefficient; Upload energy consumption for federal parameters, including This indicates a federal synchronization indicator variable.

[0079] The overall system utility is evaluated using three core performance indicators: Data collection rate ; Service fairness ; Energy gain rate ; System overall utility .

[0080] The optimization objective of this invention is to maximize the long-term cumulative comprehensive utility. Extensive simulation experiments demonstrate that the FMLD framework of this invention outperforms classical multi-agent reinforcement learning methods and baseline methods in terms of data acquisition ratio, service fairness, energy benefits, comprehensive indicators, and convergence. It also exhibits significant advantages in communication efficiency, privacy protection, data throughput, and fairness, making it highly suitable for large-scale UAV swarm collaborative scheduling scenarios.

[0081] In a preferred embodiment of the present invention, the horizontal flight direction angle The action space is discretized and uniformly quantized into... A discrete direction Typical values ​​for are 8, 16, or 36, corresponding to directional resolutions of 45 degrees, 22.5 degrees, or 10 degrees, respectively, to reduce the output dimensionality of the reinforcement learning policy network and accelerate convergence. The hyperparameter settings for the PPO algorithm are as follows: discount factor Set the smoothing parameter to 0.98 for generalized dominance estimation. Set the threshold to 0.95. The learning rate for both the Actor and Critic networks is set to 0.1. and All values ​​are set to 0.001, the hidden layer dimension is set to 128, the activation function is ReLU, and the optimizer is Adam. Local training epochs. The typical value is 10 rounds, and the federated aggregation cycle is... The typical value is 5 training rounds.

[0082] In resource allocation algorithms, The typical value is 3. A typical value is 4. The maximum emission power of a ground-based laser charging station. The typical value is 100 watts, and the photoelectric conversion efficiency is... The typical value is 0.2, atmospheric attenuation coefficient. The typical value is 0.01 / meter, and the effective area of ​​the receiver is... The typical value is 0.01 square meters, and the effective charging radius is... The typical value is 100 meters.

[0083] In the communication configuration, the total available communication bandwidth The typical value is 20MHz, and the background noise power is... A typical value is -100 dBm. Drone flight altitude. The typical value is 100 meters, cruising speed The typical value is 10 m / s, hovering power The typical value is 280.5 watts, flight power. The typical value is 363 watts, and the maximum battery capacity is... A typical value is 500 watt-hours, the lower limit of battery safety. The typical value is 15% of the minimum safe distance between drones The typical value is 5 meters. The above parameter values ​​are all based on actual engineering scenarios and mature configurations in existing literature, ensuring the feasibility and effectiveness of the technical solution of this invention in engineering practice.

[0084] Obviously, the above-described embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention also intends to include these modifications and variations.

Claims

1. A method for optimizing the energy balance of unmanned aerial vehicle (UAV) swarms based on laser transmission, characterized in that, Includes the following steps: Step S1: Build a system consisting of cloud servers, drones, Taiwan ground IoT devices and A drone-assisted data acquisition system consisting of several ground-based laser charging stations divides the entire mission cycle into... There are 1 time slot, and the duration of each time slot is 1. ,in, All are positive integers; Step S2: Model the joint trajectory planning and data acquisition problem of multiple UAVs as a partially observable Markov decision process, for each UAV... Define the local observation space, action space, and reward function. ; Step S3: A federated multi-agent reinforcement learning framework is adopted. Each UAV independently maintains a local Actor network and a local Critic network. The local Actor network outputs flight control actions based on the local observation space, and the local Critic network evaluates the expected cumulative reward of the current state. Step S4: Each UAV collects local experience trajectory during its interaction with the environment, and independently updates its local Actor network and local Critic network based on the local experience trajectory using a near-end strategy optimization method. Step S5: Each drone completes... After a round of local training, the parameters of the local Actor network and the parameters of the local Critic network are uploaded to the cloud server. It is a positive integer; Step S6: The cloud server performs global weighted aggregation on the parameters of the local Actor network of all received drones to obtain global Actor network parameters, and performs global weighted aggregation on the parameters of the local Critic network of all received drones to obtain global Critic network parameters. Step S7: The cloud server broadcasts the global Actor network parameters and the global Critic network parameters to all drones, and each drone updates its local Actor network and local Critic network with the received global Actor network parameters and global Critic network parameters. Step S8: In each decision time slot of the federated multi-agent reinforcement learning framework, the underlying resource allocation module uses a heuristic algorithm to allocate laser power and communication bandwidth based on the flight status and battery level of each UAV.

2. The method for optimizing the energy balance of UAV swarms based on laser transmission as described in claim 1, characterized in that, In step S1, the drone maintains a fixed safe altitude. Flight, the flight status of each drone includes flight status and hovering status, the flight status being at a fixed speed. When flying horizontally, the spatial coordinates of the UAV remain unchanged during the hovering state; The ground-based laser charging station provides wireless power replenishment to hovering drones within the effective charging radius using a directional laser beam. The ground-based IoT device continuously generates data to be collected, which is then collected by a drone within the sensing range of the ground-based IoT device.

3. The method for energy balance optimization of UAV swarms based on laser transmission as described in claim 1, characterized in that, In step S2, the local observation space includes the current position of the UAV, the remaining battery power, the spatial position set of other UAVs within the sensing range, the amount of data to be collected and the spatial distribution of ground IoT devices within the sensing range, and the location and status information of nearby ground laser charging stations. The action space includes the horizontal flight direction angle. and flight status indicator variables ,in Indicates hovering. Indicates a constant speed Flight; the reward function is defined as: in, Indicates drone In the time slot Instant rewards Indicates drone In the time slot Data acquisition rate, Indicates drone In the time slot The replenished energy obtained Indicates drone In the time slot Total energy consumption This indicates the maximum battery capacity of the drone.

4. The method for optimizing the energy balance of UAV swarms based on laser transmission as described in claim 1, characterized in that, In step S4, the near-end strategy optimization method includes: calculating the advantage function using generalized advantage estimation. ,in Indicates drone In the time slot The dominant function value, Indicates the discount factor. The smoothing hyperparameters of the generalized dominance estimation are represented. Indicates timing difference error; The update of the local Actor network uses a truncated objective function: , in, Indicates drone Local Actor network parameters, This represents the probability ratio between the new strategy and the old strategy. Indicates the cutoff threshold; The local Critic network is updated using the mean squared error loss function. ,in Indicates drone Local Critic network parameters, This indicates the state of the local Critic network. Value prediction This represents the target value.

5. The method for optimizing the energy balance of unmanned aerial vehicle swarms based on laser transmission as described in claim 1, characterized in that, In step S6, the global weighted aggregation uses a federated average algorithm: in, This represents the global Actor network parameters. This represents the global Critic network parameters. Indicates drone Local Actor network parameters, Indicates drone Local Critic network parameters, Indicates drone The aggregation weight coefficient, and .

6. The method for optimizing the energy balance of unmanned aerial vehicle swarms based on laser transmission as described in claim 1, characterized in that, In step S8, the laser power allocation includes: each ground laser charging station identifying a set of hovering drones within its effective charging radius; According to the formula Calculate the charging priority score for each hovering drone, where... Indicates drone For charging stations Charging priority rating, Indicates drone In the time slot The remaining battery power, Indicates drone With charging station The Euclidean distance between them; Each charging station selects the one with the highest charging priority score. Establish charging associations by using drones as charging service targets; Laser power is allocated to each charging station according to the charging priority score ratio: in, Indicates charging station In the time slot To drones The power of the launch, Indicates charging station Select a collection of drones for the service. This indicates the maximum emission power of a single ground-based laser charging station.

7. The method for optimizing the energy balance of unmanned aerial vehicle swarms based on laser transmission as described in claim 1, characterized in that, In step S8, the communication bandwidth allocation includes: Each hovering drone identifies a collection of ground-based IoT devices within its sensing range; According to the formula Calculate the communication scheduling weight for each terrestrial IoT device, where Indicates device For drones Communication scheduling weight, Indicates device In the time slot Data queue backlog Indicates device The transmission power, Indicates device With drones Channel gain between Indicates background noise power; Each UAV selects the one with the highest communication scheduling weight. Establish communication links between ground-based IoT devices; Each drone is allocated communication bandwidth according to its communication scheduling weight ratio: in, Indicates drone In the time slot Assigned to device Communication bandwidth, Show drone Select the collection of terrestrial IoT devices for the service. This represents the available communication bandwidth budget for a single drone.

8. The method for energy balance optimization of UAV swarms based on laser transmission as described in claim 1, characterized in that, In step S8, the laser power allocation satisfies the constraint that a single UAV can only receive service from one ground laser charging station in the same time slot: , , in, Indicates drone With charging station In the time slot Charging-related indicator variables; The laser power allocation also satisfies the constraint that the total power allocated to all UAVs by a single ground laser charging station in the same time slot does not exceed its maximum transmission power: The communication bandwidth allocation satisfies the constraint that the total bandwidth allocated to all associated devices by a single UAV in the same time slot does not exceed its available communication bandwidth budget: in, Indicates drone With equipment In the time slot Communication-related indicator variables, .

9. A laser-transmission-based UAV swarm energy balance optimization system, used to execute the laser-transmission-based UAV swarm energy balance optimization method as described in any one of claims 1 to 8, characterized in that, include: Cloud server; Each drone is equipped with an onboard communication module, an onboard computing module, an onboard battery, and a laser energy receiving module. A ground-based IoT device is used to generate data to be collected; Each ground-based laser charging station is equipped with a laser emitter; The cloud server, drones, Taiwan ground IoT devices and A ground-based laser charging station constitutes an unmanned aerial vehicle (UAV) assisted data acquisition system.

10. The laser-transmission-based unmanned aerial vehicle (UAV) swarm energy equalization optimization system as described in claim 9, characterized in that, Each of the aforementioned UAVs is equipped with a local Actor network and a local Critic network. The local Actor network is used to output flight control actions based on local observation states, and the local Critic network is used to evaluate the expected cumulative reward of the current state. The cloud server is equipped with a global parameter aggregation module, which is used to receive local Actor network parameters and local Critic network parameters uploaded by each UAV, perform global weighted aggregation, and broadcast the aggregated global Actor network parameters and global Critic network parameters to each UAV. The ground laser charging station has a built-in laser power allocation module, which is used to identify hovering drones within the effective charging radius, calculate the charging priority score, and allocate laser power according to the priority ratio. Each of the aforementioned drones is equipped with a communication bandwidth allocation module, which is used to identify ground IoT devices within the sensing range, calculate communication scheduling weights, and allocate communication bandwidth according to the weight ratio.

Images