Multi-uav cooperative perception communication and computing offloading method, system and device

Abstract

The present application relates to the field of intelligent transportation systems and edge computing technology, and particularly relates to a multi-unmanned aerial vehicle cooperative sensing communication and computing offloading method, system and device, the method comprising obtaining the position parameters of unmanned aerial vehicles and vehicles in a target area and the residual computing power proportion of an edge server to construct a global environment state vector; an unmanned aerial vehicle sensing offloading model analyzes the global environment state vector to output an offloading decision; action data of the unmanned aerial vehicle when executing sensing and offloading tasks according to the offloading decision is received, the global environment state vector is updated according to the action data and a weighted reward is calculated; a value evaluation model obtains a global state estimate value according to the global environment state vector, the unmanned aerial vehicle sensing offloading model and the value evaluation model are updated based on the global state estimate value and the weighted reward, whether a preset update cycle interval is reached is determined, and if so, the offloading decision is updated. The present scheme solves the technical problems of spectrum interference and computing power squeeze in multi-unmanned aerial vehicle cooperative sensing communication and computing offloading.

Description

Technical Field

[0001] This invention relates to the fields of intelligent transportation systems and edge computing technology, specifically to a method, system, and device for multi-UAV collaborative perception, communication, and computation offloading. Background Technology

[0002] In high-speed vehicle traffic scenarios, achieving comprehensive, real-time monitoring of complex intersections and wide road sections is crucial. Unmanned aerial vehicles (UAVs) deployed in the air, acting as flexible mobile nodes, can utilize integrated sensing, perception, and computing (ISAC) technology to perceive the dynamic position and trajectory information of ground vehicles in real time. Due to extremely strict physical constraints on UAVs' size, weight, and power consumption, this computationally perceived data must be offloaded in real-time to powerful multi-access edge computing (MEC) servers via wireless communication links for data processing. Multiple UAVs share the limited computing power of these ground-based edge servers. Without efficient global coordination and scheduling, multiple UAVs are prone to simultaneously offloading data to the same edge server. This not only causes severe wireless channel interference but also leads to significant queuing delays on the edge server due to computing power strain.

[0003] In addressing the problem of collaborative resource scheduling among multiple drones, existing technologies often employ traditional multi-agent reinforcement learning algorithms (such as Multi-Agent Deep Deterministic Policy Gradient (MADDPG) and Multi-Agent Proximal Policy Optimization (MAPPO)) for allocation. However, in the highly dynamic and resource-contested environment of the Internet of Vehicles (IoV), these traditional algorithms are showing performance bottlenecks. For example, independent reinforcement learning agents may perceive the actions of other drones as environmental noise, leading to severe non-stationarity issues. Directly applying traditional federated learning mechanisms (such as the FedAvg algorithm) to forcibly average the network parameters of all drones can easily obliterate their individual decision-making capabilities based on their geographical location, leading to a homogenization trap. Furthermore, directly overriding parameters can cause the momentum of local deep learning optimizers (such as Adam) to momentarily fail, resulting in drastic oscillations in policy evaluation. Therefore, there is an urgent need for in-depth optimization of the algorithm architecture specifically tailored to the resource competition characteristics of the IoV's integrated sensing and computing scenarios. Summary of the Invention

[0004] The purpose of this invention is to provide a method, system, and apparatus for multi-UAV collaborative sensing, communication, and computational offloading, in order to solve the technical problems of spectrum interference and computing power squeezing in existing multi-UAV collaborative sensing, communication, and computational offloading methods.

[0005] In a first aspect, the present invention provides a method for multi-UAV cooperative sensing, communication, and computational offloading, comprising the following steps: S1. Obtain the location parameters of one or more UAVs and one or more vehicles in the target area, as well as the remaining computing power ratio of the edge server. Calculate the amount of UAV sensing data based on the UAV location parameters and vehicle location parameters, and construct a global environment state vector based on the UAV location parameters, vehicle location parameters, UAV sensing data, and the remaining computing power ratio of the edge server. S2. Construct a UAV perception and unloading model. Input the global environment state vector into the UAV perception and unloading model. The UAV perception and unloading model parses the global environment state vector and outputs the unloading decision. S3. Receive action data of the UAV when it performs perception and unloading tasks based on the unloading decision, update the global environment state vector based on the action data, and calculate the weighted reward based on the action data. S4. Construct a value assessment model. The value assessment model obtains a global state prediction value based on the global environment state vector. The network parameters of the value assessment model are updated based on the global state prediction value and sent to the UAV to control the UAV perception unloading model to update the policy gradient. S5. Determine whether the time since the last unloading decision update has reached the preset update cycle interval. If yes, return to step S2 and update the unloading decision based on the updated UAV perception unloading model; otherwise, return to step S3 and receive the action data of the UAV when performing perception and unloading tasks according to the current unloading decision.

[0006] The significant advantages of this invention are as follows: Addressing the issues of spectrum interference and computing power constraints, this solution constructs an unloading decision based on the global environment state vector to determine the unique unloading target for each UAV, and guides group collaboration to avoid conflicts through weighted reward calculation. Furthermore, this solution combines the amount of UAV perception data with the proportion of remaining computing power on edge servers to achieve secure and efficient system communication, making it more adaptable to the heterogeneous environment and computing power allocation characteristics of UAVs. Moreover, by updating the policy gradients of the UAV perception unloading model and value assessment model, it effectively ensures real-time unloading scheduling of multiple UAVs under stringent latency constraints, significantly reducing the overall overhead of multi-UAV collaborative perception communication and computational unloading, and greatly improving energy efficiency in the multi-UAV collaborative perception communication and computational unloading process.

[0007] Furthermore, the steps of the UAV perception-based unloading model to obtain unloading decisions based on the global environment state vector include: Local observation states are obtained from the global environment state vector; The local observation state is input into the UAV perception and unloading model, and the UAV perception and unloading model obtains the action mean based on the local observation state. The average value of activated actions is obtained from the average value of the actions, and the local calculation frequency of the drone is calculated based on the average value of the activated actions. Generate a willingness score vector for the drone to unload its tasks to each target edge server; The continuous action vector of the UAV is constructed based on the UAV's local computing frequency and intention score vector, and the intention score vector is mapped to the unloading decision matrix.

[0008] By extracting local observation states from the global environment state vector, each UAV makes decentralized decisions based on its own position, the amount of sensing data, and the remaining computing power of the associated server, thus avoiding the dimensionality explosion problem caused by centralized decision-making. By mapping discrete unloading decisions to continuous values ​​(Softmax + argmax), the problem of the mixed integer action space being difficult to optimize directly is solved, making reinforcement learning policy gradient methods highly applicable.

[0009] Furthermore, the step of calculating the weighted reward includes: Calculate the air-to-ground communication transmission rate between the drone and the edge server, the drone's local preprocessing latency, and the drone's local energy consumption; The unloading transmission delay of the UAV is calculated based on the air-to-ground communication transmission rate, and the transmission energy consumption of the UAV is calculated based on the unloading transmission delay. The amount of data received by the edge server is calculated based on the offloading decision, and the edge server's computing latency and computing energy consumption are also calculated. The latency from the drone to the edge server is calculated based on the drone's local preprocessing latency, edge server computing latency, and offloading transmission latency. The energy consumption from the drone to the edge server is calculated based on the drone's local energy consumption, edge server computing energy consumption, and transmission energy consumption. Based on the energy consumption and latency of drone-to-edge server computing system, the total overhead of the computing system is calculated. A reward function is constructed based on the total system overhead, and a weighted reward is calculated based on the reward function.

[0010] By accurately modeling the latency and energy consumption of the three stages of UAV local preprocessing, offloading and transmission, and edge server computing, the system achieves full-link quantification of the overall system overhead, providing accurate optimization signals for the reward function and guiding the UAV swarm to collaboratively minimize the total system overhead in a resource-competitive environment.

[0011] Furthermore, the reward function consists of a baseline reward constant, total system overhead, a constraint violation indicator function, and a minimum penalty cutoff value. The constraint violation indicator function is selected based on the following conditions: If either the radar perception constraint or the time delay constraint is violated, the constraint violation indication function takes the value of 1; otherwise, the constraint violation indication function takes the value of 0.

[0012] The reward function unifies the modeling of baseline reward, system overall overhead penalty, and constraint violation penalty. The constraint violation indicator function triggers penalties when the radar perception mutual information is below a threshold or the processing delay exceeds the time slot length. This drives the agent to strictly meet perception quality constraints while optimizing energy consumption and latency, avoiding the problem of simply pursuing efficiency while ignoring perception performance.

[0013] Furthermore, the latency constraint is that the latency from the UAV to the edge server is less than or equal to the gap length, and the radar perception constraint is that the radar mutual information is greater than or equal to the minimum mutual information. The calculation steps for the radar mutual information include: Calculate the three-dimensional physical distance between the UAV and the vehicle based on the UAV's position parameters and the vehicle's position parameters; The beneficial channel gain of the two-way radar echo in radar perception between the UAV and the vehicle is calculated based on the three-dimensional spatial physical distance between the UAV and the vehicle. Calculate radar mutual information based on the beneficial channel gain of the two-way radar echo.

[0014] By introducing radar mutual information as a unified metric for perception quality, perception quality constraints are transformed into calculable mathematical expressions. Combined with the dynamic calculation of the two-way radar echo channel gain based on the three-dimensional spatial distance between the UAV and the vehicle, real-time online assessment of perception quality is achieved, providing a precise basis for constraining and penalizing violations.

[0015] Furthermore, the steps of updating the UAV perception offloading model and value assessment model based on the global state prediction value and weighted reward include: The optimization objective function of the value assessment model is constructed based on the value estimated by Monte Carlo return and global state, and the policy gradient of the value assessment model is updated based on the optimization objective function of the value assessment model. The ratio of new to old strategies is calculated based on the continuous motion vectors and value assessment model of drones. Calculate dynamic pruning parameters and dynamic information entropy coefficients based on the current training round; An optimization objective function for the UAV perception-unloading model is constructed based on the action advantage function, the ratio of new to old policies, dynamic pruning parameters, and dynamic information entropy coefficients. The policy gradient of the UAV perception-unloading model is then updated based on the optimization objective function of the UAV perception-unloading model.

[0016] By introducing dynamically decaying pruning parameters and information entropy coefficients, the policy is encouraged to explore fully in the early stages of training, while the pruning range is narrowed in the later stages to ensure smooth policy convergence. This effectively prevents gradient explosion and policy oscillation caused by concurrent updates from multiple drones. The standardization of Monte Carlo rewards and advantage functions further reduces training variance and improves sample efficiency.

[0017] Furthermore, the UAV perception unloading model and the value assessment model are trained based on a federated learning framework, and the training steps include: The system obtains the drone position parameters, vehicle position parameters, drone perception data volume, and the remaining computing power ratio of the edge server for the sample, and constructs a global environmental state vector.

[0018] The UAV perception-based unloading model obtains unloading decisions based on the global environment state vector. Based on the unloading decision, the system executes the perception and unloading task, and calculates the weighted reward during the task execution process; The value assessment model obtains the global state prediction value based on the global environment state vector, and updates the policy gradients of the UAV perception offloading model and the value assessment model based on the global state prediction value and the weighted reward. Determine whether the time since the last update of the UAV perception-unloading model and value assessment model has reached the preset update cycle interval. If so, update the parameters of the UAV perception-unloading model and value assessment model based on the policy gradient; otherwise, continue to execute the perception and unloading tasks, and calculate the weighted reward and update the policy gradient during the task execution process.

[0019] Furthermore, when updating the parameters of the UAV perception offloading model and the value assessment model based on the policy gradient, the parameter update of the UAV perception offloading model includes: The edge server calculates the shallow feature layer parameters of the UAV perception unloading model; The UAV updates the parameters of the UAV perception unloading model by overwriting the local shallow hidden layer parameters with the shallow feature layer parameters.

[0020] By designing a decoupled, personalized federated aggregation mechanism, federated averaging is performed only on the shallow feature extraction layers of the UAV perception offloading model network to share a common environment understanding, while retaining the personalized parameters of the deep decision heads of each UAV; this avoids the homogenization trap and optimizer momentum failure problem. Secondly, this invention provides a multi-UAV cooperative sensing, communication, and computational offloading system for implementing the aforementioned multi-UAV cooperative sensing, communication, and computational offloading method, including: The state initialization module is used to obtain the drone's position parameters, vehicle's position parameters, and the remaining computing power ratio of the edge server, and to construct an initial global state vector. The distributed decision-making module is used to obtain the unloading decision based on the global environment state vector; The environment interaction module is used to execute perception and unloading tasks based on the unloading decision and to calculate weighted rewards; The policy gradient update module is used to obtain the global state prediction based on the global environment state vector, and update the policy gradient based on the global state prediction.

[0021] Thirdly, the present invention provides a multi-UAV cooperative sensing, communication and computation offloading device, including a memory, a processor and a computer program stored in the memory, wherein the processor executes the steps of the above-described multi-UAV cooperative sensing, communication and computation offloading method. Attached Figure Description

[0022] Figure 1 This is a flowchart of the multi-UAV collaborative sensing, communication, and computational offloading method in an embodiment of the present invention; Figure 2 This is a diagram of a multi-UAV collaborative perception, communication, and computation offloading model in an embodiment of the present invention; Figure 3 This is a graph showing the trend of average processing latency and average energy consumption during the average training process of the multi-UAV collaborative perception, communication and computation offloading method in this embodiment of the invention. Figure 4 This is a comparison chart of the average reward convergence of the proposed solution and different algorithms in this embodiment of the invention; Figure 5 This is a comparison chart of the energy efficiency of the proposed solution and different algorithms under different numbers of drones in this embodiment of the invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, a clear and complete description will be provided below in conjunction with the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the protection scope of the present invention.

[0024] See appendix Figure 1 and attached Figure 2 The multi-UAV cooperative sensing, communication, and computation offloading method shown includes the following steps: S1. Obtain the position parameters of one or more UAVs, one or more vehicles, and the remaining computing power ratio of the edge server. Calculate the amount of UAV sensing data based on the UAV and vehicle position parameters, and construct a global environment state vector based on the UAV position parameters, vehicle position parameters, UAV sensing data amount, and the remaining computing power ratio of the edge server. Specifically, when obtaining the position parameters of the UAVs and vehicles, the three-dimensional coordinates of each UAV and the dynamic coordinates of each vehicle are obtained in a Cartesian coordinate system within a time interval. The expression for the three-dimensional coordinates of the UAV is: In the formula, For the first The drone in Three-dimensional coordinates in each time slot For the first The drone in In the Cartesian coordinate system during each time interval Axis coordinates For the first The drone in In the Cartesian coordinate system during each time interval Axis coordinates For the first The drone in The height in each time slot This is a transpose operation.

[0025] And construct the drone position vector based on the three-dimensional coordinates of each drone: In the formula, Let the UAV's position vector be... This refers to the number of drones.

[0026] The expression for the vehicle's dynamic coordinates is: In the formula, For the first The car in Dynamic coordinates within each time slot For the first The vehicle in In the Cartesian coordinate system during each time interval Axis coordinates For the first The car in In the Cartesian coordinate system during each time interval Axis coordinates.

[0027] And construct a vehicle position vector based on the dynamic coordinates of each vehicle: In the formula, Let the vehicle position vector be... The number of vehicles.

[0028] When calculating the amount of UAV sensing data, the raw sensing data of the UAV is generated based on the dynamic changes in the number of vehicles covered by the UAV in the time slot. The calculation formula is as follows: In the formula, For the first The drone in The amount of raw sensing data per time slot, Based on the amount of basic environment data, Adding data to a single vehicle For the first The drone in The horizontal two-dimensional coordinates of each time interval For the first The car in The horizontal two-dimensional coordinates of each time interval For sensing radius, This is an indicator function. When the condition in the function is met, the value of the indicator function is 1; otherwise, the value of the indicator function is 0.

[0029] Construct a perception data vector for the UAV based on its raw perception data volume: In the formula, This is the vector of the perceived data volume of the UAV.

[0030] The UAV position vector, vehicle position vector, UAV perception data vector, and edge server remaining computing power ratio vector are flattened and concatenated in a fixed order. Each component is then min-max normalized to [0,1] to generate a global environment state vector. The specific details of concatenating the parameters to generate the global vector and the normalization process are existing technologies and will not be elaborated here. This global state includes the entire network topology and resource reserves, eliminating environmental non-stationary interference between multiple agents. The expression for the global environment state vector is: In the formula, This is the global environment state vector. This is a vector representing the proportion of remaining computing power of edge servers. For the first The edge server in the first The proportion of remaining computing power in each time slot. This refers to the number of edge servers.

[0031] S2. Construct a UAV perception and unloading model. Input the global environment state vector into the UAV perception and unloading model, and the UAV perception and unloading model parses the global environment state vector to output the unloading decision. Specifically, each UAV is treated as an agent, and each UAV maintains an independent UAV perception and unloading model. The UAV perception and unloading model is constructed by an Actor network, and the expression of the Actor network is as follows: In the formula, For the first The stochastic policy function of the UAV represents the local observation state. Below, based on network parameters, output actions. The probability density, For action, This is a partial observation of the UAV. These are the parameters of the Actor network.

[0032] This network takes local observation states as input and outputs the action mean, then samples from a Gaussian distribution to obtain the final action. The Actor network is designed to output Gaussian distributed parameters. The Actor network includes a first fully connected hidden layer, a second fully connected hidden layer, and an action mean output head connected sequentially. The first and second fully connected hidden layers are shallow feature extraction layers used for federated aggregation, while the action mean output head is a deep decision head. It also includes an independent trainable log-standard deviation parameter to control the balance between exploration and exploitation. Its specific steps include: A1. Each UAV obtains local observation status based on the global environment state vector. Specifically, each UAV extracts local information related to itself from the global environment state vector according to a preset index as local observation status. The local observation status includes the local 3D coordinates, vehicle coordinates within the coverage area, the amount of its own sensing data, and the proportion of remaining computing power of the associated edge server.

[0033] A2. The local observation state is input into the UAV perception unloading model. The first fully connected hidden layer in the UAV perception unloading model obtains intermediate features based on the local observation state, the second fully connected hidden layer obtains deep features based on the intermediate features, and the action mean output head obtains the action mean based on the generated features. The specific analysis steps of the first fully connected hidden layer, the second fully connected hidden layer, and the action mean output head are existing technologies and will not be elaborated here.

[0034] A3. The action mean is activated by an activation function to obtain the activated action mean. Specifically, in this embodiment, the action mean is activated by the Tanh hyperbolic tangent activation function so that the action mean can be limited to the range of [-1, 1]. A4. The average activation action value is converted into a physical quantity through linear mapping. Specifically, an airborne radar is mounted on the UAV, and ISAC (Integrated Communication and Sensing) technology is used to conduct radar detection on ground vehicles. The UAV transmits detection signals to the ground vehicles. After the signals are scattered by the vehicle targets, the echoes return to the UAV, forming a two-way air-to-ground radar loop. The purpose of radar sensing is to obtain the position and dynamic information of ground vehicles. The parameters of the UAV when sensing ground vehicles through radar include sensing bandwidth, sensing power, and local calculation frequency. When converting the average activation action value into a physical quantity through linear mapping, the sensing bandwidth ratio and sensing power ratio are mapped to [0,1], and the local calculation frequency is linearly mapped to a preset range. The specific expression for the local calculation frequency is: In the formula, For the first The drone in Local calculation frequency for each time slot, This is the minimum value of the locally calculated frequency. This is the maximum value of the locally calculated frequency.

[0035] A5. Generate the willingness score vector of the UAV to unload tasks to each target edge server and the computing frequency ratio allocated to the edge server; the specific willingness score vector is the component corresponding to each edge server in the output of the Actor network action mean output head of the UAV perception unloading model. After normalization by the Softmax function, the unloading probability distribution of each edge server is obtained, and then mapped to the one-hot unloading decision matrix by the argmax operation; the computing power ratio allocated to the edge server is also output by the Actor network of the UAV perception unloading model, and linearly mapped to the range of (0,1], representing the computing power ratio of the edge server requested by the UAV.

[0036] A6. Construct the continuous motion vectors of the UAV, and construct the UAV motion space based on the continuous motion vectors of the UAV, wherein the expression of the continuous motion vectors of the UAV is: In the formula, For the first The drone in A continuous action vector with time intervals, For the first The drone in The proportion of sensing bandwidth allocated to each time slot. For the first The drone in The power allocation ratio for each time slot For the first The drone in The proportion of local computing frequencies in each time slot, For each edge server, there is a score vector representing the willingness to unload. The computing power is allocated to the edge servers. The continuous motion vectors of the drone are obtained by sampling from a Gaussian distribution and are truncated before entering the environment.

[0037] When constructing the UAV action space, the continuous action vectors of all time slots of the UAV are combined to construct the UAV action space.

[0038] The unloading intention score vector is mapped to a probability distribution using the Softmax function, and then mapped to a discrete unloading decision matrix in one-hot format using the argmax operation to determine the unique task unloading target edge server. The expression for the unloading decision is as follows: In the formula, For unloading decisions, used to represent the first A drone in time interval Should the task be uninstalled to the first... The probability of the nth edge server, where 0 represents the probability of the nth edge server. A drone in time interval Do not uninstall the task to the first The first edge server, 1 indicates offloading the first edge server. A drone in time interval Unload the task to the An edge server.

[0039] S3. The UAV performs perception and unloading tasks based on the unloading decision. The system receives action data from the UAV during these tasks, updates the global environment state vector based on the action data, and calculates a weighted reward. Specifically, the weighted reward calculation includes the following steps: B1. Construct a communication model between the drone and the edge server, and calculate the air-to-ground communication transmission rate, drone local preprocessing latency, and drone local energy consumption in the communication model; the formula for calculating the air-to-ground communication transmission rate is: In the formula, For the first The drone in The time interval to the first The air-to-ground communication transmission rate for data transmission between edge servers For communication bandwidth, It is a logarithmic function. For communication noise power, For communication power, This is the gain for one-way communication channels.

[0040] The formula for calculating communication bandwidth is: In the formula, For the first The drone in The proportion of bandwidth allocated to the sensing task in each time slot. This represents the total available bandwidth of the system. The formula for calculating communication power is: In the formula, For the first The drone in The proportion of power allocated to the sensing task in each time slot. This represents the total power of the system.

[0041] The formula for calculating communication noise power is: In the formula, For the first The drone in Communication noise power per time slot, For thermal noise power spectral density, This represents the noise figure.

[0042] The formula for calculating the channel gain in one-way communication is: In the formula, For the first The drone in One-way air-to-ground communication channel gain per time slot, The channel gain constant at the reference distance. This is the path loss index. For the first The drone and the first The edge server in the first The three-dimensional spatial distance of each time slot For the first The drone in The horizontal two-dimensional coordinates of each time interval For the first The edge server in the first The horizontal two-dimensional coordinates of each time interval It is the Euclidean norm. This refers to the flight altitude of the drone.

[0043] The formula for calculating the local preprocessing latency of the drone is: In the formula, Due to the local preprocessing latency of the drone, The number of CPU cycles required to process 1 bit of data for a drone. Calculate the frequency locally.

[0044] The formula for calculating the local energy consumption of a drone is: In the formula, For the local power consumption of drones, This is the effective capacitance constant.

[0045] B2. Calculate the unloading transmission delay of the UAV based on the air-to-ground communication transmission rate, and calculate the transmission energy consumption of the UAV based on the unloading transmission delay; the formula for calculating the unloading transmission delay is: In the formula, To offload transmission delay, This represents the data compression ratio.

[0046] The formula for calculating transmission energy consumption is: In the formula, For transmission energy consumption.

[0047] B3. Calculate the amount of data received by the edge server based on the offloading decision, and calculate the edge server's computing latency and energy consumption; the formula for calculating the amount of data received by the edge server is: In the formula, For the first An edge server in a time slot The amount of data received This refers to the number of drones.

[0048] The formula for calculating edge server computing latency is: In the formula, For the first An edge server in a time slot The computational delay, For the first The computing frequency of an edge server The number of CPU cycles required for an edge server to process 1 bit of data.

[0049] The formula for calculating the computing energy consumption of an edge server is: In the formula, For the first An edge server in a time slot Computational energy consumption, This represents the effective capacitance constant of the edge server.

[0050] B4. Calculate the drone-to-edge server latency based on the drone's local preprocessing latency, edge server computation latency, and offloading transmission latency; calculate the drone-to-edge server energy consumption based on the drone's local energy consumption, edge server computation energy consumption, and transmission energy consumption; the formula for calculating the drone-to-edge server latency is: In the formula, For the latency from drone to edge server, It is a function with maximum value. This refers to the number of edge servers.

[0051] The formula for calculating the energy consumption from drone to edge server is: In the formula, Energy consumption for drones to edge servers.

[0052] B5. Calculate the total system overhead based on the energy consumption and latency of the drone-to-edge server connection; the total system overhead is defined as the weighted sum of the drone-to-edge server latency and energy consumption, and its calculation formula is: In the formula, The total system overhead, As a weight for the latency from the drone to the edge server, Weighting of energy consumption from drones to edge servers.

[0053] B6. Construct a reward function based on the total system cost, and calculate the weighted reward based on the reward function. The expression for the reward function is: In the formula, For the reward function, It is a function with maximum value. As the baseline reward constant, To constrain the penalty coefficient for violations, To constrain the violation indicator function, This is the minimum penalty cutoff value. The reward function enables interaction with the environment, the collection of experience, and synchronization across all drones, guiding the swarm strategy to iteratively evolve towards minimizing the total system overhead.

[0054] The expression for the constraint violation indicator function is: In the formula, For radar mutual information, To minimize radar mutual information, Constraints for radar perception. The gap length, For time delay constraints.

[0055] The calculation steps for radar mutual information include: C1. Calculate the three-dimensional physical distance between the UAV and the vehicle based on the UAV's position parameters and the vehicle's position parameters. Specifically, in this embodiment: In the formula, For the first The drone and the first The car in The three-dimensional spatial physical distance in each time slot.

[0056] C2. Calculate the beneficial channel gain of the two-way radar echo in radar sensing between the UAV and the vehicle based on the three-dimensional physical distance between the UAV and the vehicle. Specifically, in this embodiment, the UAV carries an airborne radar and uses ISAC (Integrated Information and Communication) technology to perform radar detection on ground vehicles. The UAV transmits detection signals to the ground vehicle, and the signals are scattered by the vehicle target and echo back to the UAV, forming a two-way air-to-ground radar loop. The purpose of radar sensing is to obtain the position and dynamic information of the ground vehicle. The formula for calculating the beneficial channel gain of the two-way radar echo is: In the formula, For the first The drone in Beneficial channel gain of bidirectional radar echoes in each time slot The channel gain constant at the reference distance. This is the path loss index.

[0057] C3. Calculate radar mutual information based on the beneficial channel gain of the two-way radar echo: In the formula, For the first The drone in Radar mutual information in each time slot, For the first The drone in The sensing bandwidth of airborne radar in each time slot It is a logarithmic function. For the first The drone in The sensing power of the airborne radar in each time slot The power of Gaussian white noise. For the first The drone in The proportion of bandwidth allocated to the sensing task in each time slot. This represents the total available bandwidth in radar sensing. For the first The drone in The proportion of power allocated to the sensing task in each time slot. This represents the total power perceived by the radar.

[0058] S4. Construct a value assessment model. The value assessment model obtains a global state prediction value based on the global environment state vector. The network parameters of the value assessment model are updated based on the global state prediction value and sent to the UAV to control the UAV perception and unloading model to update the policy gradient. The value assessment model is composed of a Critic network. That is, the UAV perception and unloading model and the value assessment model use the Actor network and Critic network in reinforcement learning, respectively. The state space of the Markov decision process in reinforcement learning is the global state space, the action space is the UAV action space, and the reward function is the establishment function constructed in step S3. The specific structure and implementation of the Actor network and Critic network in reinforcement learning are existing technologies and will not be described in detail here.

[0059] The specific value assessment network concatenates the states of all UAVs into a global joint state vector as input, where the expression for the global joint state vector is: In the formula, For the global joint state vector, For the first The drone in Local observation status in each time slot.

[0060] The value assessment network outputs a global state prediction value based on the global joint state vector to eliminate non-stationary interference caused by concurrent updates from multiple agents. The expression for the global state prediction value is: In the formula, Critic network with parameters For the global joint state Perform an evaluation of the state value of the output; For the first Critic network parameters configured for the drone The steps involved in updating the policy gradient based on the global state prediction and weighted reward include: D1. Calculate the Monte Carlo return based on weighted rewards, and construct the action advantage function based on the Monte Carlo return and the estimated value of the global state; the formula for calculating the Monte Carlo return is: In the formula, In return for Monte Carlo, As a discount factor, For the first Weighted rewards for each step.

[0061] The expression for the action advantage function is: In the formula, This represents the action advantage. In the embodiment, the advantage function of all samples within the same batch is further standardized to zero mean and unit variance, and its calculation formula is: In the formula, For the first The drone in The action dominance function values ​​for each time slot after zero-mean, unit-variance standardization. Action dominance function for all samples within the same batch The mean, The standard deviation of the dominance function. To prevent constants with a denominator of 0.

[0062] D2. Construct an optimization objective function for the value assessment model based on the Monte Carlo reward and the estimated value of the global state, and update the policy gradient of the value assessment model based on the optimization objective function of the value assessment model. The optimization objective of the value assessment model is to minimize the mean squared error, and its expression is: In the formula, For the first The optimization objective function value of the drone value assessment model (Critic network) is calculated. For the first Standardized Monte Carlo cumulative discount return for each time step.

[0063] D3. Calculate the ratio of new to old strategies based on the continuous motion vectors and value assessment model of UAVs: In the formula, The ratio of new to old strategies. Let be the probability density function of the current policy. For the first The drone in Actions taken during each time interval For the first The drone in Local observation status in each time slot, For the parameters of the valuation model, This is the probability density function of the old policy before the last update. These are the parameters for the old Actor network.

[0064] D4. Calculate the dynamic pruning parameters and dynamic information entropy coefficient based on the current training round, where the formula for calculating the dynamic pruning parameters is: In the formula, The dynamic pruning parameters for the current training round. To find the maximum value function, These are the initial clipping parameters. This represents the current training progress percentage. The minimum trimming parameter, To find the minimum value function, The maximum number of training rounds, This represents the current training round number.

[0065] The formula for calculating the dynamic information entropy coefficient is: In the formula, This represents the dynamic information entropy coefficient for the current training round. The initial information entropy coefficient, This is the minimum information entropy coefficient.

[0066] D5. Construct the optimization objective function for the UAV perception-unloading model based on the action advantage function, the ratio of new to old policies, dynamic pruning parameters, and dynamic information entropy coefficients. Update the policy gradient of the UAV perception-unloading model based on the optimization objective function. The optimization objective of the UAV perception-unloading model is to maximize the sum of the pruned alternative objective function and the information entropy, expressed as: In the formula, For the first The optimization objective function value of the drone perception offloading model (Actor network) is the expected value of the sum of the pruned alternative objective function and the information entropy. These are the current Actor network parameters. For the expected operator, The ratio of new to old strategies. For the clipping function, This represents the dynamic information entropy coefficient for the current training round. Shannon information entropy for the strategy.

[0067] When updating the policy gradient, the policy gradient is updated by combining local PPO training with a dynamic pruning mechanism. The specific details of the policy gradient update are existing technologies and will not be elaborated here.

[0068] The UAV perception offloading model and value assessment model are trained based on a federated learning framework. The training steps for the UAV perception offloading model and value assessment model include: E1. Obtain the UAV position parameters, vehicle position parameters, UAV perception data volume, and the remaining computing power ratio of the edge server for the sample, and construct a global environment state vector.

[0069] E2. The UAV perception-based unloading model obtains unloading decisions based on the global environment state vector. E3. Execute the perception and unloading tasks based on the unloading decision, and calculate the weighted reward during the task execution process; E4. The value assessment model obtains the global state prediction value based on the global environment state vector, and updates the policy gradient of the UAV perception offloading model and the value assessment model based on the global state prediction value. E5. Determine whether the update of the UAV perception offloading model and value assessment model has reached the preset update cycle interval. If so, update the parameters of the UAV perception offloading model and value assessment model based on the policy gradient; otherwise, return to step E3.

[0070] Specifically, when updating the UAV perception offloading model and value assessment model, the policy gradients obtained from local training on each UAV are acquired, and personalized federated updates of the UAV perception offloading model and value assessment model are performed on the edge server through federated aggregation. The preset update cycle is determined by whether the federated aggregation cycle has reached the preset cycle. The specific steps for updating the UAV perception offloading model and value assessment model include: When updating the parameters of the drone perception unloading model, a personalized decoupling federated aggregation mechanism is triggered once every preset training round (where the preset training round is an adjustable hyperparameter with a value range of 10 rounds).

[0071] In the UAV perception and unloading model, the Actor network is decoupled and federated into a shallow fully connected hidden layer, primarily responsible for extracting general physical features from the complex environment. The deeper action mean output head and exploration variance are responsible for formulating personalized resource allocation strategies highly tied to local geographic locations. The edge server, acting as the federated parameter aggregation center, performs a global federated average of the weights of the shallow fully connected hidden layers of the Actor network for each UAV to obtain the shallow feature layer parameters. In the formula, These are parameters for the shallow feature layer. For the number of drones, These are the weight parameters for the shallow feature extraction layer of the local Actor network for each UAV.

[0072] Subsequently, each UAV only uses shallow feature layer parameters to cover its local shallow hidden layer parameters, while strictly retaining its own deep motion mean output head and exploration variance parameters, which are not included in the fusion. This mechanism enables UAVs to share a global understanding of the physical characteristics of the environment while perfectly maintaining personalized decision-making capabilities for heterogeneous spatial locations, thus avoiding the homogenization trap.

[0073] When updating the parameters of the value assessment model, the traditional federated average direct coverage of network parameters in the Critic network anti-collapse soft update aggregation can cause the momentum state of the local Adam optimizer to instantly fail, leading to violent oscillations in the network value assessment. This solution employs a soft update mechanism for Critic aggregation, specifically by first calculating the globally averaged value network parameters: In the formula, This represents the global average of the Critic network parameters for all drones.

[0074] Then, the global experience is gently injected into the local Critic network using a soft update smoothing coefficient, the expression of which is: In the formula, This is an assignment operation used to update the variable on the left with the result of the calculation on the right. This is the smoothing coefficient for soft updates.

[0075] E6. Determine whether the training stopping condition has been met. If the training stopping condition has been met, output the completed UAV perception unloading model and value assessment model. Otherwise, return to step E2 to start the next round of training. The training stopping condition is that the number of training iterations reaches the maximum number of training iterations.

[0076] S5. Determine if the preset update cycle interval has been reached since the last unloading decision update. If yes, return to step S2 and update the unloading decision based on the updated UAV perception unloading model; otherwise, return to step S3, and the UAV continues to perform perception and unloading tasks according to the current unloading decision. Specifically, in this embodiment, the preset update cycle interval is 10 update rounds of the UAV perception unloading model and value assessment model. That is, after every 10 updates of the UAV perception unloading model and value assessment model, a federated aggregation and model parameter update are triggered. After the update is completed, return to S2 and re-output the unloading decision based on the new model; otherwise, the UAV continues to perform perception and unloading tasks according to the current strategy and accumulate experience.

[0077] By designing a personalized decoupled federated aggregation mechanism that is triggered at preset update intervals, global federated averaging is performed only on the shallow feature extraction layer of the Actor network, preserving the personalized parameters of each UAV's deep decision head; a soft update mechanism (weighted averaging instead of direct overlay) is adopted for the Critic network to protect the momentum continuity of the local Adam optimizer and avoid the violent oscillations in value assessment caused by direct parameter overlay, which significantly accelerates the convergence of the algorithm in heterogeneous environments.

[0078] The following graph shows the trend of average processing latency and average energy consumption during the average training process of the multi-UAV cooperative sensing, communication, and computation offloading method in this scheme. Figure 3 As shown, MAPPO is a multi-agent proximal policy optimization algorithm where all agents share policy parameters; MADDPG is a multi-agent deep deterministic policy gradient algorithm where each agent is equipped with a centralized Critic that can access the global state; and MAA2C ​​is a multi-agent dominant actor / critic algorithm that achieves multi-agent collaborative policy optimization by centrally training the Critic and distributing the execution of Actors. This scheme uses a dynamic decay mechanism to ensure training stability in the later stages of training by reducing the pruning boundary, thereby reducing average processing latency and average UAV energy consumption.

[0079] As attached Figure 4 As shown, all four algorithms start with a negative reward of approximately -100. Our proposed scheme converges the fastest, with a final convergence value of approximately 110, significantly higher than MAPPO's approximately 90, MADDPG's approximately 55, and MAA2C's approximately 55. This indicates that our scheme has superior policy convergence quality in multi-UAV cooperative resource allocation. While achieving value consensus among multiple agents, our scheme preserves the continuity of gradients in the deep neural network optimizer, significantly accelerating the convergence speed of the algorithm in heterogeneous environments.

[0080] The edge server distributes the aggregated shallow feature layer parameters and the soft-updated Critic parameters to each UAV, which then updates the network parameters of its UAV perception offloading model and value assessment model accordingly. This invention's decoupled federated aggregation and Critic soft-updating mechanism effectively breaks optimizer shock, achieving not only good reward convergence but also strictly satisfying the physical constraint of maximum tolerable latency.

[0081] The energy efficiency comparison of this solution with different algorithms under different numbers of drones is attached. Figure 5 As shown, the energy efficiency of all four algorithms decreases as the number of drones increases from 2 to 5, but our proposed solution consistently maintains its leading position across all scales. Taking 3 drones as an example, our solution achieves approximately 91 kbits / Joule, MAPPO approximately 85 kbits / Joule, MADDPG approximately 57 kbits / Joule, and MAA2C ​​approximately 79 kbits / Joule. With the increasing number of drones, the rate of energy efficiency decline in our solution is significantly lower than that of other algorithms, demonstrating the good scalability of the personalized federated aggregation mechanism in large-scale scenarios.

[0082] This solution addresses the issues of spectrum interference and computing power strain by incorporating the bandwidth / power allocation and unloading target selection of each UAV into a unified continuous action space. It uses Softmax mapping to determine a unique unloading target and designs a constraint and penalty mechanism to guide group collaboration and avoid conflicts. To address the multi-agent nonstationarity problem, a centralized training and distributed execution (CTDE) architecture is adopted. The Critic network uses the global joint state for value evaluation, explicitly incorporating the state changes of all UAVs into the value function, thereby fundamentally eliminating nonstationary disturbances. To address the homogenization trap and optimizer momentum failure, a decoupled personalized federated aggregation mechanism is designed. Federated averaging is performed only on the shallow feature extraction layers of the Actor network to share a common understanding of the environment, while preserving the personalized parameters of the deep decision heads of each network. At the same time, a soft update mechanism (weighted averaging instead of direct overlay) is adopted for the Critic network to protect the continuity of the optimizer gradient.

[0083] Furthermore, this solution integrates the characteristics of communication, sensing, and computing to achieve secure and efficient system communication. Compared with existing technologies, this solution is more adaptable to the heterogeneous environments and computing power allocation characteristics of multiple UAVs. Through a network-layer decoupled federated aggregation mechanism, it perfectly preserves the personalized division of labor capability for local states, avoiding the homogenization trap; through a soft update mechanism, it maintains the momentum stability of the optimizer, significantly improving the system's convergence speed; and ultimately, it effectively ensures real-time offloading and scheduling of multiple UAVs under strict latency constraints, significantly reducing overall system overhead and greatly improving system energy efficiency.

[0084] This invention also aims to provide a multi-UAV cooperative sensing, communication, and computation offloading system for implementing the aforementioned multi-UAV cooperative sensing, communication, and computation offloading method, comprising: The state initialization module is used to obtain the drone's position parameters, vehicle's position parameters, and the remaining computing power ratio of the edge server, and to construct an initial global state vector. The distributed decision-making module is used to obtain the unloading decision based on the global environment state vector; The environment interaction module is used to execute perception and unloading tasks based on the unloading decision and to calculate weighted rewards; The policy gradient update module is used to obtain the global state prediction based on the global environment state vector, and update the policy gradient based on the global state prediction.

[0085] The present invention also aims to provide a multi-UAV cooperative sensing, communication and computation offloading device, including a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described multi-UAV cooperative sensing, communication and computation offloading method.

[0086] The present invention also aims to provide a computer-readable storage medium containing a computer program, wherein the computer program is stored thereon, and when the computer program is executed by one or more processors, implements the steps of the above-described multi-UAV cooperative perception communication and computation offloading method.

[0087] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the scope of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for multi-UAV cooperative sensing, communication, and computational offloading, characterized in that, Includes the following steps: S1. Obtain the location parameters of one or more UAVs and one or more vehicles in the target area, as well as the remaining computing power ratio of the edge server. Calculate the amount of UAV sensing data based on the UAV location parameters and vehicle location parameters, and construct a global environment state vector based on the UAV location parameters, vehicle location parameters, UAV sensing data, and the remaining computing power ratio of the edge server. S2. Construct a UAV perception and unloading model. Input the global environment state vector into the UAV perception and unloading model. The UAV perception and unloading model parses the global environment state vector and outputs the unloading decision. S3. Receive action data of the UAV when it performs perception and unloading tasks based on the unloading decision, update the global environment state vector based on the action data, and calculate the weighted reward based on the action data. S4. Construct a value assessment model. The value assessment model obtains a global state prediction value based on the global environment state vector. The network parameters of the value assessment model are updated based on the global state prediction value and sent to the UAV to control the UAV perception unloading model to update the policy gradient. S5. Determine whether the time since the last unloading decision update has reached the preset update cycle interval. If so, return to step S2 and update the unloading decision based on the updated UAV perception unloading model. Otherwise, return to step S3 to receive the action data of the UAV when performing the perception and unloading tasks based on the current unloading decision.

2. The multi-UAV cooperative sensing, communication, and computational offloading method according to claim 1, characterized in that, The steps of the UAV perception-based unloading model to obtain unloading decisions based on the global environment state vector include: Local observation states are obtained from the global environment state vector; The local observation state is input into the UAV perception and unloading model, and the UAV perception and unloading model obtains the action mean based on the local observation state. The average value of activated actions is obtained from the average value of the actions, and the local calculation frequency of the drone is calculated based on the average value of the activated actions. Generate a willingness score vector for the drone to unload its tasks to each target edge server; The continuous action vector of the UAV is constructed based on the UAV's local computing frequency and intention score vector, and the intention score vector is mapped to the unloading decision matrix.

3. The multi-UAV cooperative sensing, communication, and computational offloading method according to claim 1, characterized in that, The steps for calculating the weighted reward include: Calculate the air-to-ground communication transmission rate between the drone and the edge server, the drone's local preprocessing latency, and the drone's local energy consumption; The unloading transmission delay of the UAV is calculated based on the air-to-ground communication transmission rate, and the transmission energy consumption of the UAV is calculated based on the unloading transmission delay. The amount of data received by the edge server is calculated based on the offloading decision, and the edge server's computing latency and computing energy consumption are also calculated. The latency from the drone to the edge server is calculated based on the drone's local preprocessing latency, edge server computing latency, and offloading transmission latency. The energy consumption from the drone to the edge server is calculated based on the drone's local energy consumption, edge server computing energy consumption, and transmission energy consumption. Based on the energy consumption and latency of drone-to-edge server computing system, the total overhead of the computing system is calculated. A reward function is constructed based on the total system overhead, and a weighted reward is calculated based on the reward function.

4. The multi-UAV cooperative sensing, communication, and computational offloading method according to claim 3, characterized in that, The reward function consists of a baseline reward constant, total system overhead, a constraint violation indicator function, and a minimum penalty cutoff value. The constraint violation indicator function is selected based on the following conditions: If either the radar perception constraint or the time delay constraint is violated, the constraint violation indication function takes the value of 1; otherwise, the constraint violation indication function takes the value of 0.

5. The multi-UAV cooperative sensing, communication, and computational offloading method according to claim 4, characterized in that, The latency constraint is that the latency from the UAV to the edge server is less than or equal to the gap length; the radar perception constraint is that the radar mutual information is greater than or equal to the minimum mutual information; the calculation steps for the radar mutual information include: Calculate the three-dimensional physical distance between the UAV and the vehicle based on the UAV's position parameters and the vehicle's position parameters; The beneficial channel gain of the two-way radar echo in radar perception between the UAV and the vehicle is calculated based on the three-dimensional spatial physical distance between the UAV and the vehicle. Calculate radar mutual information based on the beneficial channel gain of the two-way radar echo.

6. The multi-UAV cooperative sensing, communication, and computational offloading method according to claim 1, characterized in that, The step of updating the network parameters of the value assessment model based on the global state prediction and sending them to the UAV to control the UAV perception unloading model to update the policy gradient includes: Monte Carlo returns are calculated based on weighted rewards, and an action advantage function is constructed based on Monte Carlo returns and global state prediction values. The optimization objective function of the value assessment model is constructed based on the value estimated by Monte Carlo return and global state, and the policy gradient of the value assessment model is updated based on the optimization objective function of the value assessment model. The ratio of new to old strategies is calculated based on the continuous motion vectors and value assessment model of drones. Calculate dynamic pruning parameters and dynamic information entropy coefficients based on the current training round; An optimization objective function for the UAV perception-unloading model is constructed based on the action advantage function, the ratio of new to old policies, dynamic pruning parameters, and dynamic information entropy coefficients. The policy gradient of the UAV perception-unloading model is then updated based on the optimization objective function of the UAV perception-unloading model.

7. The multi-UAV cooperative sensing, communication, and computational offloading method according to claim 1, characterized in that, The UAV perception unloading model and the value assessment model are trained based on a federated learning framework. The training steps include: The system obtains the drone position parameters, vehicle position parameters, drone perception data volume, and the remaining computing power ratio of the edge server for the sample, and constructs a global environmental state vector. The UAV perception-based unloading model obtains unloading decisions based on the global environment state vector. Based on the unloading decision, the system executes the perception and unloading task, and calculates the weighted reward during the task execution process; The value assessment model obtains the global state prediction value based on the global environment state vector, and updates the policy gradients of the UAV perception offloading model and the value assessment model based on the global state prediction value. Determine whether the time since the last update of the UAV perception-unloading model and value assessment model has reached the preset update cycle interval. If so, update the parameters of the UAV perception-unloading model and value assessment model based on the policy gradient; otherwise, continue to execute the perception and unloading tasks, and calculate the weighted reward and update the policy gradient during the task execution process.

8. The multi-UAV cooperative sensing, communication, and computational offloading method according to claim 7, characterized in that, When updating the parameters of the UAV perception offloading model and the value assessment model based on the policy gradient, the parameter update of the UAV perception offloading model includes: The edge server calculates the shallow feature layer parameters of the UAV perception unloading model; The UAV updates the parameters of the UAV perception unloading model by overwriting the local shallow hidden layer parameters with the shallow feature layer parameters.

9. A multi-UAV cooperative sensing, communication, and computational offloading system, used to implement the multi-UAV cooperative sensing, communication, and computational offloading method as described in any one of claims 1-8, characterized in that, include: The state initialization module is used to obtain the drone's position parameters, vehicle's position parameters, and the remaining computing power ratio of the edge server, and to construct an initial global state vector. The distributed decision-making module is used to obtain the unloading decision based on the global environment state vector; The environment interaction module is used to execute perception and unloading tasks based on the unloading decision and to calculate weighted rewards; The policy gradient update module is used to obtain the global state prediction based on the global environment state vector, and update the policy gradient based on the global state prediction and weighted reward.

10. A multi-UAV collaborative sensing, communication, and computational offloading device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the multi-UAV cooperative perception, communication and computation offloading method according to any one of claims 1-8.

Images