An unmanned aerial vehicle (UAV) trajectory design and resource allocation integrated optimization method

Abstract

This invention discloses a joint optimization method for trajectory design and resource allocation for UAVs with integrated sensing, computing, and communication capabilities, relating to the fields of low-altitude economics and UAV integrated sensing, computing, and communication technologies. The method includes: constructing a low-altitude surveillance system model comprising a monitoring UAV, a ground base station, and dynamic aerial targets, and initializing parameters; constructing a system mathematical model including communication, sensing, computing, and energy consumption models; constructing an optimization problem based on the model with the objective of minimizing the weighted sum of total system processing latency and total energy consumption, and decomposing it into a joint optimization sub-problem of trajectory and beamforming, and a computational offloading and resource allocation sub-problem; solving the optimization problem based on a hybrid decision-making mechanism and deep reinforcement learning to obtain the optimal trajectory design and resource allocation strategy for the monitoring UAV. This invention solves the problems of difficulty in joint optimization of multi-dimensional resource coupling and insufficient target tracking and prediction capabilities in existing technologies, significantly reducing system energy consumption and processing latency.

Description

Technical Field

[0001] This invention relates to the field of low-altitude economy and UAV integrated sensing and computing technology, and in particular to a joint optimization method for trajectory design and resource allocation for UAV integrated sensing and computing, applicable to low-altitude wireless network application scenarios such as low-altitude airspace supervision, urban aerial dynamic target surveillance, and emergency reconnaissance. Background Technology

[0002] With the rapid development of the low-altitude economy and low-altitude networks, the flight density in low-altitude airspace has increased dramatically, placing higher demands on airspace supervision. Traditional ground monitoring base stations are limited by static topology deployment and the obstruction effect caused by complex geographical environments, making it difficult to achieve comprehensive and efficient aerial surveillance. Unmanned Aerial Vehicles (UAVs), with their high mobility and on-demand deployment advantages, can serve as an effective supplement to ground networks, filling perception blind spots and building a seamless air-ground collaborative surveillance system.

[0003] Modern surveillance missions often involve the real-time processing and transmission of massive amounts of high-definition video streams or radar point cloud data. If UAVs directly transmit all raw sensing data back to the ground, it will consume a large amount of wireless bandwidth resources and introduce significant transmission latency, affecting the system's response speed. To address this, Integrated Sensing, Communication and Computation (ISCC) technology is applied at the edge of UAVs. By integrating sensing, communication, and computing functions and sharing spectrum and hardware resources, it completes the processing and feature extraction of sensing data locally, transmitting only key information back to the ground, thereby improving the system's real-time performance and energy efficiency under resource-constrained conditions.

[0004] In integrated sensing, communication, and computing systems for unmanned aerial vehicles (UAVs), there are complex coupling and competitive relationships between sensing accuracy, communication rate, and computation latency. The UAV's flight trajectory directly affects the channel quality of sensing and communication, while beamforming and computation offloading strategies affect energy consumption and processing latency. Therefore, how to jointly optimize the UAV's 3D flight trajectory, beamforming, and computation offloading ratio to minimize system energy consumption and processing latency has become a key problem that urgently needs to be solved in this field.

[0005] To address the aforementioned issues, existing technologies mainly fall into two categories: those based on traditional convex optimization and those based on deep reinforcement learning, but both have certain limitations. Traditional methods typically utilize alternating optimization or Karush-Kuhn-Tucker (KKT) conditions to decompose non-convex joint optimization problems into multiple sub-problems for iterative solving. These methods are theoretically rigorous, but often rely on idealized static environment assumptions, and their computational complexity increases exponentially with the dimensionality of variables, making it difficult to meet the needs of online real-time decision-making for UAVs. To cope with dynamic environments, some studies employ end-to-end deep reinforcement learning (DRL) algorithms such as deep deterministic policy gradient or soft actor-critic (SAC), treating all decision variables as neural network outputs. However, the synesthetic computing integrated system has a high-dimensional mixed action space, and pure end-to-end learning faces the problem of excessively high dimensionality, leading to low agent exploration efficiency and a tendency to get trapped in local optima or fail to converge.

[0006] Furthermore, in low-altitude surveillance scenarios, existing target tracking and control strategies mostly rely on instantaneous state snapshots for decision-making, ignoring the target's motion trends implicit in historical state sequences, such as spatiotemporal characteristics like direction, velocity, and acceleration. This control method, lacking predictive capabilities, struggles to overcome decision lags caused by system processing and transmission delays, easily leading to beam alignment deviations, reduced radar echo signal-to-noise ratio and sensing accuracy, and forcing the system to adopt aggressive high-power transmission strategies to passively compensate for link losses, significantly increasing energy consumption.

[0007] In summary, existing technologies suffer from the following shortcomings: First, traditional optimization algorithms exhibit low computational efficiency and poor convergence of end-to-end reinforcement learning algorithms when facing high-dimensional, strongly coupled synesthetic computational resource optimization problems. Second, the decision-making mechanism lacks effective extraction of historical spatiotemporal features, making it difficult to achieve accurate prediction and robust beam alignment of highly dynamic targets. Third, resource allocation does not fully utilize the advantages of mathematical analytical solutions, resulting in an excessively large learning space for reinforcement learning algorithms, leading to insufficient efficiency and robustness. Therefore, there is an urgent need for a joint optimization method for UAVs that effectively integrates analytical optimization and reinforcement learning, and possesses spatiotemporal feature extraction capabilities. Summary of the Invention

[0008] To address this, this invention provides a joint optimization method for trajectory design and resource allocation in UAV integrated sensing and computing, which solves the technical problems of poor tracking performance and high energy consumption in existing UAV integrated sensing and computing dynamic monitoring due to the strong coupling of multi-dimensional resources.

[0009] To address the aforementioned technical problems, this invention provides a joint optimization method for trajectory design and resource allocation for unmanned aerial vehicles (UAVs) integrating sensing and computation. This method includes the following steps:

[0010] Step S1: Construct a low-altitude surveillance system model that includes monitoring drones, ground base stations, and dynamic aerial targets, and initialize system parameters;

[0011] Step S2: Construct a mathematical model of the system, including a communication model, a sensing model, a computing model, and an energy consumption model;

[0012] Step S3: Based on the mathematical model of the system, construct an optimization problem with the goal of minimizing the weighted sum of the total system processing delay and total energy consumption, and decompose the optimization problem into a joint optimization sub-problem of trajectory and beamforming and a sub-problem of computational offloading resource allocation;

[0013] Step S4: Solve the optimization problem based on a hybrid decision-making mechanism and deep reinforcement learning to obtain the optimal trajectory design and resource allocation strategy for the monitoring drone.

[0014] Preferably, in step S1, the construction of a low-altitude surveillance system model including monitoring drones, ground base stations, and dynamic aerial targets, and the initialization of system parameters specifically include:

[0015] Construct a three-dimensional Cartesian coordinate system to describe the spatial structure of the low-altitude surveillance area, and define a set of ground base stations. With aerial dynamic target set ,in and These represent the total number of ground base stations and aerial dynamic targets, respectively.

[0016] The monitoring drone antenna system is configured as a uniform planar array, consisting of... The antenna array consists of three antennas, with the spacing between the antenna elements set to half a wavelength. This represents the number of rows in the antenna array. This represents the number of columns in the antenna array;

[0017] Divide the total task time into There are 1 time slot, and the duration of each time slot is 1. ;

[0018] Set the maximum flight altitude for monitoring drones Maximum flight speed Minimum safe collision avoidance distance with dynamic aerial targets Maximum total transmission power and the maximum computing frequency of the onboard processor ;

[0019] Configure the computing frequency of the base station edge server .

[0020] Preferably, in step S2, constructing the system mathematical model specifically includes:

[0021] Construct a model for the uplink communication rate between the monitoring drone and the ground base station:

[0022] ;

[0023] in, For uplink communication rate, To monitor the index of drones, For the index of ground base stations, To represent the time slot index; For system bandwidth; This is the conjugate transpose of the channel gain vector. Represents the conjugate transpose operation; For communication beamforming vectors; Indicates service to other base stations Communication beamforming vector; To sense beamforming vectors, An index for dynamic aerial targets; This represents the ambient background noise power.

[0024] Construct a sensing rate model for monitoring UAVs of dynamic aerial targets:

[0025] ;

[0026] in, For sensing rate; This is a two-way radar channel matrix that includes the radar cross-section characteristics of the target; This is the conjugate transpose of the two-way radar channel matrix; Indicates targeting other objectives Perceived beamforming vector;

[0027] Construct a processing latency model for sensing data:

[0028] ;

[0029] in, To detect the processing latency of the data; To calculate the decision variables for the unloading ratio; To perceive the amount of task data; To calculate density, it represents the number of CPU clock cycles required to process each bit of data; To monitor the computing frequency of the drone's onboard processor, this indicates the monitoring of the drone's local CPU's processing speed; For the target-base station association indicator variable, if the first The monitoring data of the first aerial dynamic target needs to be transmitted to the second... One ground base station, then Otherwise, it is 0; The computing frequency of the ground base station edge server represents the computing power of the ground base station in processing offloading tasks.

[0030] Construct a total system energy consumption model:

[0031] ;

[0032] in, Indicates the first Within each time slot, monitor the total energy consumed by the drone in performing the integrated sensing and computing task; To monitor the effective capacitance coefficient of the UAV's onboard processor.

[0033] Preferably, in step S3, the optimization problem aimed at minimizing the weighted sum of the total system processing latency and total energy consumption specifically includes:

[0034] Construct the optimization objective function:

[0035] ;

[0036] in, To monitor the spatial trajectory variables of drones, , Indicates the first Each time slot monitors the three-dimensional spatial coordinate vector of the drone. , indicating the time slot index, For the total set of time slots, The total number of time slots divided for the total task time; For beamforming variables, , For communication beamforming vectors, To sense beamforming vectors; To calculate the unloading ratio variable, , To calculate the decision variables for the unloading ratio; This is the time delay weighting coefficient; An index for dynamic aerial targets; The total number of moving targets in the air; To detect the processing latency of the data; Energy consumption weighting coefficient; Indicates the first Within each time slot, monitor the total energy consumed by the drone in performing the integrated sensing and computing task;

[0037] The constraints include: UAV dynamics constraints, safety collision avoidance constraints, transmit power constraints, mission real-time constraints, and decision variable boundary constraints, among which:

[0038] UAV dynamics constraints: ;

[0039] Safety collision avoidance restraints: ;

[0040] Transmit power constraints: ;

[0041] Task real-time constraints: ;

[0042] Boundary constraints on decision variables: ;

[0043] in, To monitor the maximum flight speed of the drone; Indicates the first Each time slot represents the three-dimensional spatial coordinate vector of a dynamic aerial target; To monitor the minimum safe collision avoidance distance between drones and dynamic aerial targets; For the index of ground base stations; A collection of ground base stations; An index for dynamic aerial targets; A set of dynamic aerial targets; To monitor the maximum total transmit power of the drone; Duration for each time slot.

[0044] Preferably, in step S4, the step of solving the optimization problem based on a hybrid decision-making mechanism and deep reinforcement learning to obtain the optimal UAV trajectory design and resource allocation strategy specifically includes:

[0045] Step S41: Model the interaction process between the monitoring UAV and the dynamic aerial target as a Markov decision process, and define the state space, action space and reward function;

[0046] Step S42: Initialize the parameters of the policy network, value network and target network, and initialize the experience replay pool;

[0047] Step S43: The agent acquires the current state sequence, outputs Gaussian distribution parameters from the policy network, and samples to obtain a subset of reinforcement learning actions;

[0048] Step S44: Using the subset of reinforcement learning actions as known parameters, solve the computational unloading resource allocation subproblem using numerical analytical methods to obtain the analytically optimized subset of actions, and combine it with the subset of reinforcement learning actions to form a complete action vector;

[0049] Step S45: Execute the action vector to obtain the state and immediate reward for the next moment, and store the experience tuple into the experience replay pool;

[0050] Step S46: Randomly sample a batch of samples from the experience replay pool to update the value network, policy network, and temperature parameters;

[0051] Step S47: Repeat steps S43 to S46 until the algorithm converges to obtain the optimal trajectory design and resource allocation strategy for monitoring drones.

[0052] Preferably, the state space defined in step S41 Includes current time and history Observation sequence at each time point ,in This is an instantaneous observation vector, which includes the UAV's own three-dimensional position, the relative position estimates of all dynamic aerial targets, and the current channel state information;

[0053] The action space A hybrid decision-making mechanism is adopted to decouple actions into a subset of reinforcement learning actions. With parsing and optimizing action subsets Two parts, namely the total action vector ,in To monitor the spatial trajectory variables of drones, For beamforming variables, To calculate the unloading ratio variable;

[0054] The reward function Designed as follows:

[0055] ;

[0056] in, This is the time delay weighting coefficient; An index for dynamic aerial targets; The total number of moving targets in the air; To detect the processing latency of the data; Energy consumption weighting coefficient; Indicates the first Within each time slot, monitor the total energy consumed by the drone in performing the integrated sensing and computing task; This represents the number of constraints violated in the current time slot. To constrain the penalty constant for violations.

[0057] Preferably, in step S43, the policy network According to the state sequence Output the mean and standard deviation of a Gaussian distribution, and obtain a subset of reinforcement learning actions through reparameterization techniques. ;

[0058] In step S44, the reinforcement learning action subset is... Substituting known constants into the computational unloading resource allocation subproblem, a closed-form solution is derived using numerical theory, and a subset of analytically optimized actions is obtained. Finally, the two are combined to form a complete motion vector. .

[0059] Preferably, steps S43 to S45 further include:

[0060] A gated recurrent unit network layer is introduced at the front end of the policy network and the value network to extract spatiotemporal features from historical observation sequences;

[0061] In the time slot The gated recurrent unit layer receives the current short sequence observation vector. And the hidden state of the previous time slot Output a new hidden state vector ;

[0062] The multilayer perceptron part of the policy network receives the hidden state vector. Output action probability distribution parameters;

[0063] The multilayer perceptron part of the value network receives the spliced ​​vector. Output state action value estimation, where To reinforce the learning of action subsets.

[0064] Preferably, in step S46, the value network is updated by minimizing the mean square error; the policy network is updated by maximizing the weighted sum of the current Q value and the policy entropy; the temperature parameter... The policy entropy is automatically adjusted through gradient descent to approach the preset target entropy value; the target network is softly updated using a multi-segment moving average method.

[0065] ;

[0066] in, This is the soft update coefficient. For the current network parameters, These are the target network parameters.

[0067] This invention also provides a joint optimization system for trajectory design and resource allocation for UAVs with integrated sensor-computation capabilities. This system is used to implement the aforementioned joint optimization method for trajectory design and resource allocation for UAVs with integrated sensor-computation capabilities, specifically including:

[0068] The model building and initialization module is used to build a low-altitude surveillance system model that includes monitoring drones, ground base stations, and dynamic aerial targets, and to initialize system parameters;

[0069] The mathematical model building module is used to build system mathematical models, including communication models, sensing models, computing models, and energy consumption models.

[0070] The optimization problem construction and decomposition module is used to construct an optimization problem based on the system mathematical model, with the goal of minimizing the weighted sum of the total system processing delay and total energy consumption, and decompose the optimization problem into a joint optimization sub-problem of trajectory and beamforming and a sub-problem of computational offloading resource allocation.

[0071] The hybrid decision-making module is used to solve the optimization problem based on a hybrid decision-making mechanism and deep reinforcement learning, so as to obtain the optimal trajectory design and resource allocation strategy for monitoring drones.

[0072] As can be seen from the above technical solutions, this invention application has the following beneficial effects:

[0073] First, this invention constructs a deep reinforcement learning framework based on a hybrid decision-making mechanism, decoupling the original optimization problem into a joint optimization sub-problem of trajectory and beamforming and a sub-problem of calculating unloading resource allocation. By using reinforcement learning to output macroscopic actions and combining analytical optimization to solve the calculation unloading ratio, the dimensionality of the action space is significantly reduced. This solves the convergence difficulty and local optima problem caused by the excessively large action space in pure end-to-end reinforcement learning, and greatly improves the learning efficiency and convergence stability of the algorithm.

[0074] Second, this invention constructs a spatiotemporal feature extraction mechanism based on historical observation sequences by introducing a gated recurrent unit network layer at the front end of the policy network and value network. This enables the agent to effectively capture the motion trend of dynamic targets in the air (such as high-order features like direction, speed, and acceleration), giving the UAV the ability to predict target trajectories. It overcomes the beam alignment deviation caused by decision lag in traditional decision-making mechanisms, significantly improves perception accuracy and communication link stability, avoids the aggressive high-power transmission strategy used to compensate for link loss, and thus greatly reduces the system's ineffective energy consumption.

[0075] Third, this invention constructs a system mathematical model integrating communication, sensing, computing, and energy consumption, and designs a hybrid reward function that includes latency, energy consumption, and constraint violation penalties. This achieves the joint minimization of total system processing latency and total energy consumption in the optimization objective. Simulation results show that, compared with existing technologies, this invention achieves lower overall system energy consumption (e.g., 0.326 J) while maintaining processing latency, effectively balancing real-time performance and energy efficiency. It has significant engineering application value on lightweight UAV platforms where computing power and battery capacity are both limited. Attached Figure Description

[0076] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly described below. Referring to the drawings will make the features and advantages of the present invention clearer. The drawings are illustrative and should not be construed as limiting the present invention in any way. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0077] Figure 1 This is a low-altitude wireless network application scenario for urban surveillance.

[0078] Figure 2 The present invention provides a flowchart of a joint optimization method for trajectory design and resource allocation for unmanned aerial vehicles (UAVs) that integrates sensing and computation.

[0079] Figure 3 This is a schematic diagram of the simulation experimental environment in this invention;

[0080] Figure 4 This is a performance comparison chart between the present invention and various benchmark algorithms during the training process;

[0081] Figure 5 This is a comparison chart of the total latency and total energy consumption of different algorithms in this invention;

[0082] Figure 6 This is a block diagram of a joint optimization system for trajectory design and resource allocation for unmanned aerial vehicles (UAVs) that integrates sensing, computing, and communication. Detailed Implementation

[0083] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0084] Example 1: This example addresses the technical problems in existing UAV integrated dynamic surveillance systems, such as the difficulty in joint optimization due to strong coupling of multi-dimensional resources, and poor tracking performance and high energy consumption due to the lack of effective prediction of target movement trends. For instance... Figure 1 As shown, this invention considers the application scenario of urban surveillance using low-altitude wireless networks. The system consists of a monitoring drone with integrated sensing and computing capabilities, several ground base stations, and the aerial dynamic targets to be monitored. Figure 2 As shown, the method of the present invention includes the following steps:

[0085] Step S1: Construct a low-altitude surveillance system model that includes monitoring drones, ground base stations, and dynamic aerial targets, and initialize system parameters.

[0086] First, a three-dimensional Cartesian coordinate system is constructed to describe the spatial structure of the low-altitude surveillance area. Within this area, a set of ground base stations is defined. With aerial dynamic target set ,in and These represent the total number of ground base stations and aerial dynamic targets, respectively.

[0087] For monitoring drones performing missions, their antenna systems are configured as uniform planar arrays (UPAs), which consist of... The antenna consists of [number] antennas, among which This represents the number of rows in the antenna array. The number of columns in the antenna array is set to half a wavelength, which aims to support the concurrent transmission of communication signals and sensing signals.

[0088] The system is configured to operate within a discrete-time horizon, and the total task time is divided into... There are 1 time slot, and the duration of each time slot is 1. .

[0089] Initialize monitoring drone platform parameters: Set the maximum flight altitude of the monitoring drone to [value missing]. Maximum flight speed is The minimum safe collision avoidance distance that must be maintained between the monitoring drone and any dynamic aerial target is: The maximum total transmission power of the monitoring drone is The maximum computing frequency of the onboard processor of the monitoring drone is At the same time, the computing frequency of the ground base station edge server is set to... .

[0090] Step S2: Construct a mathematical model of the system, including a communication model, a sensing model, a computing model, and an energy consumption model.

[0091] (1) Communication Model: Construct an uplink communication rate model between the monitoring UAV and the ground base station. Considering that the signal transmitted by the monitoring UAV includes both communication and sensing beams, and that the channel is affected by Ricean fading and multi-user interference, establish the first... The monitoring drone directed to the first The uplink transmission rate model for each ground base station is as follows:

[0092] ;

[0093] in, For uplink communication rate, To monitor the index of drones, For the index of ground base stations, To represent the time slot index; For system bandwidth; This is the conjugate transpose of the channel gain vector. Represents the conjugate transpose operation; For communication beamforming vectors; Indicates service to other base stations Communication beamforming vector; To sense beamforming vectors, An index for dynamic aerial targets; The numerator represents the expected signal power received by the ground base station, while the denominator represents the communication beam interference serving other base stations, the sensing beam interference serving dynamic targets in the air, and the environmental background noise power, respectively.

[0094] (2) Perception Model: A perception rate model for monitoring UAVs of dynamic aerial targets is constructed. The UAV receives radar echo signals from dynamic aerial targets. Considering perception interference between multiple targets, the estimated information rate is used as the evaluation index, and the perception rate model is established as follows:

[0095] ;

[0096] in, For sensing rate; This is a two-way radar channel matrix that includes the radar cross-section characteristics of the target; This is the conjugate transpose of the two-way radar channel matrix; Indicates targeting other objectives The sensing beamforming vector. The numerator represents the effective echo signal power reflected by the target, and the denominator represents the sum of the reflection interference power generated at the current target by the sensing beam pointing to other dynamic air targets, and the background noise power.

[0097] (3) Computational Model: Construct a processing latency model for the sensing data. A partial offloading strategy is adopted, dividing the computational task into local processing and offloading processing, which are executed in parallel. The local computation latency is determined by the CPU frequency of the monitoring UAV; the offloading computation latency consists of the wireless transmission latency and the ground base station processing latency; the effective processing latency is the maximum value of the local and offloading processes. Establish the first... The task processing latency model for a single dynamic aerial target is as follows:

[0098] ;

[0099] in, To detect the processing latency of the data; To calculate the decision variables for the unloading ratio; To perceive the amount of task data; To calculate density, it represents the number of CPU clock cycles required to process each bit of data; To monitor the computing frequency of the drone's onboard processor, this indicates the monitoring of the drone's local CPU's processing speed; For the target-base station association indicator variable, if the first The monitoring data of the first aerial dynamic target needs to be transmitted to the second... One ground base station, then Otherwise, it is 0; The computing frequency of the ground base station edge server represents the computing power of the ground base station in handling offloading tasks.

[0100] (4) Energy Consumption Model: Construct a total system energy consumption model. The total energy consumption of the monitoring UAV performing the integrated sensing, communication, and computing task consists of three parts: sensing energy consumption, which depends on the sensing beam power and sensing duration; communication energy consumption, which depends on the communication beam power and data transmission duration; and computing energy consumption, which depends on the effective capacitance coefficient and operating frequency of the monitoring UAV's local CPU. The total system energy consumption model is established as follows:

[0101] ;

[0102] in, Indicates the first Within each time slot, monitor the total energy consumed by the drone in performing the integrated sensing and computing task; To monitor the effective capacitance coefficient of the UAV's onboard processor.

[0103] Step S3: Based on the mathematical model of the system, construct an optimization problem with the goal of minimizing the weighted sum of the total system processing delay and total energy consumption, and decompose the optimization problem into a joint optimization sub-problem of trajectory and beamforming and a sub-problem of computational offloading resource allocation.

[0104] (1) Optimization Objective: Construct a system weighted cost function to minimize the weighted sum of total system processing latency and total energy consumption over the entire time period, so as to achieve the optimal combination of efficiency and energy efficiency. The optimization objective function is defined as follows:

[0105] ;

[0106] in, To monitor the spatial trajectory variables of drones, , Indicates the first Each time slot monitors the three-dimensional spatial coordinate vector of the drone. For the total set of time slots, The total number of time slots divided for the total task time; For beamforming variables, ; To calculate the unloading ratio variable, ; This is the time delay weighting coefficient; This is the energy consumption weighting coefficient.

[0107] (2) Constraints: Establish constraints including UAV dynamics constraints, safety collision avoidance constraints, launch power constraints, mission real-time constraints, and decision variable boundary constraints:

[0108] UAV dynamics constraints: ;

[0109] Safety collision avoidance restraints: ;

[0110] Transmit power constraints: ;

[0111] Task real-time constraints: ;

[0112] Boundary constraints on decision variables: ;

[0113] in, Indicates the first Each time slot represents the three-dimensional spatial coordinate vector of a dynamic aerial target.

[0114] (3) Problem decomposition: This invention adopts a two-layer decoupling strategy based on a hybrid action space to decompose the original joint optimization problem into two hierarchical sub-problems: P1 is the joint trajectory design and beamforming problem (determining the...). and P2 is a computational resource allocation problem (deterministic). ).

[0115] Step S4: Solve the optimization problem based on a hybrid decision-making mechanism and deep reinforcement learning to obtain the optimal trajectory design and resource allocation strategy for the monitoring drone.

[0116] This step is based on the Hybrid Recurrent SoftActor-Critic (HR-SAC) algorithm proposed in this invention, and specifically includes the following sub-steps:

[0117] Step S41: Model the interaction process between the monitoring UAV and the dynamic aerial target as a Markov decision process, and define the state space, action space and reward function.

[0118] state space To capture the dynamic characteristics of aerial targets, the first... The state of the time slot This is a sequence containing both current observations and historical information. The length of the historical sliding window. This is an instantaneous observation vector, which includes the three-dimensional position of the monitoring UAV itself, the relative position estimates of all dynamic aerial targets, and the current channel state information.

[0119] Action space A hybrid decision-making mechanism is adopted to decouple actions into subsets of reinforcement learning actions. With parsing and optimizing action subsets Two parts, namely the total action vector .

[0120] reward function Design a hybrid reward function to guide the agent to minimize system cost while satisfying constraints.

[0121] ;

[0122] in, This represents the number of constraints violated in the current time slot. To constrain the penalty constant for violations.

[0123] Step S42: Initialize the parameters of the policy network, value network and target network, and initialize the experience replay pool.

[0124] Initialize the Actor policy network (Parameters are) Two independent Critic action value networks and (The parameters are respectively) , To train stability, the corresponding Critic target network is initialized. and initial parameters Initialize the experience replay pool. and learnable temperature parameters (Weights used to control entropy).

[0125] Step S43: The agent obtains the current state sequence, outputs Gaussian distribution parameters from the policy network, and samples to obtain a subset of reinforcement learning actions.

[0126] In the time slot The agent obtains the current state sequence. Policy Network according to The output Gaussian distribution has a mean and standard deviation. A subset of reinforcement learning actions is obtained by sampling using the reparameterization trick. .

[0127] Step S44: Using the subset of reinforcement learning actions as known parameters, solve the computational unloading resource allocation subproblem using numerical analytical methods to obtain the analytically optimized subset of actions, and combine it with the subset of reinforcement learning actions to form a complete action vector.

[0128] Reinforce learning action subset Substituting these known constants into the computational unloading resource allocation subproblem P2, a closed-form solution is derived using numerical theory, and the subset of analytically optimized actions is quickly obtained. Finally, the two are combined to form a complete motion vector. .

[0129] Step S45: Execute the action vector to obtain the state and immediate reward for the next moment, and store the experience tuple into the experience replay pool.

[0130] Monitoring drones performing joint actions in the environment The system state transitions to the next time slot state. Calculate the immediate reward based on the reward function in step S41. Transform the state transition tuple Store in the experience replay pool middle.

[0131] Step S46: Randomly sample a batch of samples from the experience replay pool and update the value network, policy network, and temperature parameters.

[0132] When the experience replay pool Once the amount of data reaches a threshold, a batch of samples is randomly selected for network updates at each training step.

[0133] Critic Network Update: Utilizing the Next State Calculate the target Q-value (which includes the reward and the entropy of the next action). Update the parameters by minimizing the mean squared error between the current Q-value and the target Q-value. and .

[0134] Actor Network Update: Freeze the Critic network and update the policy network parameters by maximizing the weighted sum of the current Q-value (usually the smaller of the two Critic values ​​to mitigate overestimation) and the policy entropy. .

[0135] Temperature parameters Update: Automatic adjustment via gradient descent This makes the entropy of the strategy approach the preset target entropy value.

[0136] Target network soft update: The target network parameters are updated smoothly using a multi-segment moving average method. ,in This is the soft update coefficient. For the current network parameters, These are the target network parameters.

[0137] Step S47: Repeat steps S43 to S46 until the algorithm converges to obtain the optimal trajectory design and resource allocation strategy for monitoring drones.

[0138] Repeat steps S43 to S46 until the maximum number of training episodes is reached or the network loss function converges, ultimately obtaining the optimal joint strategy for monitoring UAV trajectories and beamforming.

[0139] Furthermore, to enhance the SAC network's perception of target mobility, this invention introduces a Gate Recurrent Unit (GRU) network layer at the front end of the policy network and value network to extract spatiotemporal features from historical observation sequences.

[0140] Specifically, in time slots The GRU layer receives the current short sequence observation vector. And the hidden state of the previous time slot Output a new hidden state vector The calculation formula can be expressed as: The multilayer perceptron (MLP) part of the policy network receives the hidden state vector. Output action probability distribution parameters; the multilayer perceptron part of the value network receives the spliced ​​vector. Output state action value estimation.

[0141] To verify the effectiveness and robustness of the proposed UAV trajectory and resource joint optimization method based on the Hybrid Regression Soft Actor-Critic Algorithm (HR-SAC), a three-dimensional urban low-altitude surveillance simulation experimental platform was built based on a high-fidelity dynamic target trajectory dataset for verification.

[0142] like Figure 3 As shown, the experiment constructed a 400m×400m×100m three-dimensional urban low-altitude airspace as the monitoring scenario. Within this area, one monitoring drone, three ground base stations, and three highly dynamic aerial targets were deployed. To realistically reproduce the dynamic changes of the aerial targets, the flight trajectories of the monitored targets in the experiment were derived from a high-fidelity open-source dataset generated by the Gazebo simulation platform, which integrates the PX4 autopilot and Robot Operating System 2 (ROS 2). The main simulation parameters were set as follows: maximum drone flight altitude = 100m, maximum flight speed = 20m / s; system communication bandwidth = 1MHz, maximum drone transmit power = 4W; drone local onboard computing frequency = 1GHz, ground base station edge computing frequency = 10GHz; CPU cycles required to process one bit = 1000 cycles / bit; latency weight coefficient in the system optimization objective = 1, energy consumption weight coefficient = 11 / 9.

[0143] Figure 4 This paper presents a comparison of the convergence performance of the present invention (HR-SAC) with various benchmark algorithms (including the time-series-less feature policy H-SAC, the pure end-to-end reinforcement learning policy R-SAC, the dual-delay deterministic policy HR-TD3, and the fixed-trajectory policy HR-FIX) during iterative training. The results show that the present invention significantly outperforms all benchmark algorithms tested in terms of convergence speed and steady-state reward performance. Specifically, compared to the pure end-to-end reinforcement learning method (R-SAC) which directly outputs all decision variables, the present invention benefits from an innovative hybrid decoupled decision mechanism, which greatly reduces the dimensionality of the action space and the learning burden on the agent, thus improving learning efficiency. Furthermore, compared to H-SAC and the deterministic policy HR-TD3, which lack temporal prediction, the present invention effectively suppresses oscillations and stagnation during training by deeply integrating the GRU feature extraction module with the maximum entropy exploration mechanism of SAC, successfully preventing the agent from getting trapped in local optima. Meanwhile, the experiment also proved that the adaptive temperature coefficient dynamic adjustment mechanism adopted in this invention can achieve a better exploration-exploitation balance than fixed parameter configuration, and demonstrates excellent and extremely stable global optimal strategy optimization ability in complex dynamic environments.

[0144] Figure 5The results show a comprehensive comparison between the present invention and various benchmark algorithms in terms of total system delay and total energy consumption. Experimental data shows that the present invention achieves the lowest total system energy consumption of 0.326J. In contrast, the benchmark algorithm (H-SAC) that removes the timing prediction module achieves a similar delay, but its total energy consumption is as high as 0.766J. This proves the necessity of introducing GRU spatiotemporal feature extraction in the present invention: by accurately predicting the target trajectory, the present invention achieves perfect beam alignment, converting all limited electrical energy into effective geometric array gain, avoiding the aggressive high-energy-consuming compensation strategies that traditional methods are forced to adopt due to decision lag and beam defocusing.

[0145] Example 2: Figure 6 As shown, this invention provides a joint optimization system for trajectory design and resource allocation for UAVs with integrated sensory and computational capabilities. This system is used to implement the joint optimization method for trajectory design and resource allocation for UAVs with integrated sensory and computational capabilities described in Embodiment 1 above, specifically including:

[0146] The model building and initialization module 100 is used to build a low-altitude surveillance system model that includes monitoring drones, ground base stations and dynamic aerial targets, and to initialize system parameters;

[0147] The mathematical model building module 200 is used to build the system's mathematical model, including the communication model, the sensing model, the computing model, and the energy consumption model.

[0148] The optimization problem construction and decomposition module 300 is used to construct an optimization problem based on the system mathematical model, with the goal of minimizing the weighted sum of the total system processing delay and total energy consumption, and decompose the optimization problem into a joint optimization sub-problem of trajectory and beamforming and a computational offloading resource allocation sub-problem.

[0149] The hybrid decision-making module 400 is used to solve the optimization problem based on a hybrid decision-making mechanism and deep reinforcement learning, so as to obtain the optimal trajectory design and resource allocation strategy for the monitoring UAV.

[0150] This embodiment presents a joint optimization system for trajectory design and resource allocation for UAVs with integrated sensory and computational capabilities. This system is used to implement the aforementioned joint optimization method for trajectory design and resource allocation for UAVs with integrated sensory and computational capabilities. Therefore, the specific implementation of the joint optimization system for trajectory design and resource allocation for UAVs with integrated sensory and computational capabilities can be found in the previous section on the embodiments of the joint optimization method for trajectory design and resource allocation for UAVs with integrated sensory and computational capabilities. For example, the model construction and initialization module 100, the mathematical model construction module 200, the optimization problem construction and decomposition module 300, and the hybrid decision-making solution module 400 are respectively used to implement steps S1, S2, S3, and S4 in the aforementioned joint optimization method for trajectory design and resource allocation for UAVs with integrated sensory and computational capabilities. Therefore, its specific implementation can be referred to the descriptions of the corresponding embodiments. To avoid redundancy, further details are omitted here.

[0151] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0152] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0153] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0154] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A method of trajectory design and resource allocation joint optimization for unmanned aerial vehicle (UAV) situational awareness (SA) integration, characterized in that, Includes the following steps: Step S1: Construct a low-altitude surveillance system model that includes monitoring drones, ground base stations, and dynamic aerial targets, and initialize system parameters; Step S2: Construct a mathematical model of the system, including a communication model, a sensing model, a computing model, and an energy consumption model; Step S3: Based on the mathematical model of the system, construct an optimization problem with the goal of minimizing the weighted sum of the total system processing delay and total energy consumption, and decompose the optimization problem into a joint optimization sub-problem of trajectory and beamforming and a sub-problem of computational offloading resource allocation; Step S4: Solve the optimization problem based on a hybrid decision-making mechanism and deep reinforcement learning to obtain the optimal trajectory design and resource allocation strategy for the monitoring drone; The optimization problem described, which aims to minimize the weighted sum of total system processing latency and total energy consumption, specifically includes: Construct the optimization objective function: ； in, To monitor the spatial trajectory variables of drones, , Indicates the first Each time slot monitors the three-dimensional spatial coordinate vector of the drone. , indicating the time slot index, For the total set of time slots, The total number of time slots divided for the total task time; For beamforming variables, , For communication beamforming vectors, To sense beamforming vectors; To calculate the unloading ratio variable, , To calculate the decision variables for the unloading ratio; This is the time delay weighting coefficient; An index for dynamic aerial targets; The total number of moving targets in the air; To detect the processing latency of the data; Energy consumption weighting coefficient; Indicates the first Within each time slot, monitor the total energy consumed by the drone in performing the integrated sensing and computing task; The constraints include: UAV dynamics constraints, safety collision avoidance constraints, launch power constraints, mission real-time constraints, and decision variable boundary constraints, among which: UAV dynamics constraints: ; Safety collision avoidance restraints: ; Transmit power constraints: ; Task real-time constraints: ; Boundary constraints on decision variables: ; in, To monitor the maximum flight speed of the drone; Indicates the first Each time slot represents the three-dimensional spatial coordinate vector of a dynamic aerial target; To monitor the minimum safe collision avoidance distance between drones and dynamic aerial targets; For the index of ground base stations; A collection of ground base stations; An index for dynamic aerial targets; A set of dynamic aerial targets; To monitor the maximum total transmit power of the drone; Duration for each time slot.

2. The method for joint optimization of trajectory design and resource allocation for UAVs based on the integration of sensing and computation as described in claim 1, characterized in that, In step S1, the construction of a low-altitude surveillance system model that includes monitoring drones, ground base stations, and dynamic aerial targets, and the initialization of system parameters specifically include: Construct a three-dimensional Cartesian coordinate system to describe the spatial structure of the low-altitude surveillance area, and define a set of ground base stations. With aerial dynamic target set ,in and These represent the total number of ground base stations and aerial dynamic targets, respectively. The monitoring drone antenna system is configured as a uniform planar array, consisting of... The antenna array consists of three antennas, with the spacing between the antenna elements set to half a wavelength. This represents the number of rows in the antenna array. This represents the number of columns in the antenna array; Divide the total task time into There are 1 time slot, and the duration of each time slot is 1. ; Set the maximum flight altitude for monitoring drones Maximum flight speed Minimum safe collision avoidance distance with dynamic aerial targets Maximum total transmission power and the maximum computing frequency of the onboard processor ; Configure the computing frequency of the base station edge server .

3. The method for joint optimization of trajectory design and resource allocation for UAVs based on the integration of sensing and computation as described in claim 1, characterized in that, In step S2, the construction of the system mathematical model specifically includes: Construct a model for the uplink communication rate between the monitoring drone and the ground base station: ； in, For uplink communication rate, To monitor the index of drones, For the index of ground base stations, To represent the time slot index; For system bandwidth; This is the conjugate transpose of the channel gain vector. Represents the conjugate transpose operation; For communication beamforming vectors; Indicates service to other base stations Communication beamforming vector; To sense beamforming vectors, An index for dynamic aerial targets; This represents the ambient background noise power. Construct a sensing rate model for monitoring UAVs of dynamic aerial targets: ； in, For sensing rate; This is a two-way radar channel matrix that includes the radar cross-section characteristics of the target; This is the conjugate transpose of the two-way radar channel matrix; Indicates targeting other objectives Perceived beamforming vector; Construct a processing latency model for sensing data: ； in, To detect the processing latency of the data; To calculate the decision variables for the unloading ratio; To perceive the amount of task data; To calculate density, it represents the number of CPU clock cycles required to process each bit of data; To monitor the computing frequency of the drone's onboard processor, this indicates the monitoring of the drone's local CPU's processing speed; For the target-base station association indicator variable, if the first The monitoring data of the first aerial dynamic target needs to be transmitted to the second... One ground base station, then Otherwise, it is 0; The computing frequency of the ground base station edge server represents the computing power of the ground base station in processing offloading tasks. Construct a total system energy consumption model: ； in, Indicates the first Within each time slot, monitor the total energy consumed by the drone in performing the integrated sensing and computing task; To monitor the effective capacitance coefficient of the UAV's onboard processor.

4. The method for joint optimization of trajectory design and resource allocation for UAVs based on the integration of sensing and computation as described in claim 1, characterized in that, In step S4, the step of solving the optimization problem based on a hybrid decision-making mechanism and deep reinforcement learning to obtain the optimal UAV trajectory design and resource allocation strategy specifically includes: Step S41: Model the interaction process between the monitoring UAV and the dynamic aerial target as a Markov decision process, and define the state space, action space and reward function; Step S42: Initialize the parameters of the policy network, value network and target network, and initialize the experience replay pool; Step S43: The agent acquires the current state sequence, outputs Gaussian distribution parameters from the policy network, and samples to obtain a subset of reinforcement learning actions; Step S44: Using the subset of reinforcement learning actions as known parameters, solve the computational unloading resource allocation subproblem using numerical analytical methods to obtain the analytically optimized subset of actions, and combine it with the subset of reinforcement learning actions to form a complete action vector; Step S45: Execute the action vector to obtain the state and immediate reward for the next moment, and store the experience tuple into the experience replay pool; Step S46: Randomly sample a batch of samples from the experience replay pool to update the value network, policy network, and temperature parameters; Step S47: Repeat steps S43 to S46 until the algorithm converges to obtain the optimal trajectory design and resource allocation strategy for monitoring drones.

5. The method for joint optimization of trajectory design and resource allocation for UAVs based on the integration of sensing and computation as described in claim 4, characterized in that, The state space defined in step S41 Includes current time and history Observation sequence at each time point ,in This is an instantaneous observation vector, which includes the UAV's own three-dimensional position, the relative position estimates of all dynamic aerial targets, and the current channel state information; The action space A hybrid decision-making mechanism is adopted to decouple actions into a subset of reinforcement learning actions. With parsing and optimizing action subsets Two parts, namely the total action vector ,in To monitor the spatial trajectory variables of drones, For beamforming variables, To calculate the unloading ratio variable; The reward function Designed as follows: ； in, This is the time delay weighting coefficient; An index for dynamic aerial targets; The total number of moving targets in the air; To detect the processing latency of the data; Energy consumption weighting coefficient; Indicates the first Within each time slot, monitor the total energy consumed by the drone in performing the integrated sensing and computing task; This represents the number of constraints violated in the current time slot. To constrain the penalty constant for violations.

6. The method for joint optimization of trajectory design and resource allocation for UAVs based on the integration of sensing and computation as described in claim 4, characterized in that, In step S43, the policy network According to the state sequence Output the mean and standard deviation of a Gaussian distribution, and obtain a subset of reinforcement learning actions through reparameterization techniques. ; In step S44, the reinforcement learning action subset is... Substituting known constants into the computational unloading resource allocation subproblem, a closed-form solution is derived using numerical theory, and a subset of analytically optimized actions is obtained. Finally, the two are combined to form a complete motion vector. .

7. The method for joint optimization of trajectory design and resource allocation for UAVs based on the integration of sensing and computation as described in claim 4, characterized in that, Steps S43 to S45 further include: A gated recurrent unit network layer is introduced at the front end of the policy network and the value network to extract spatiotemporal features from historical observation sequences; In the time slot The gated recurrent unit layer receives the current short sequence observation vector. And the hidden state of the previous time slot Output a new hidden state vector ; The multilayer perceptron part of the policy network receives the hidden state vector. Output action probability distribution parameters; The multilayer perceptron part of the value network receives the spliced ​​vector. Output state action value estimation, where To reinforce the learning of action subsets.

8. The method for joint optimization of trajectory design and resource allocation for UAVs based on the integration of sensing and computation as described in claim 4, characterized in that, In step S46, the value network is updated by minimizing the mean square error; the policy network is updated by maximizing the weighted sum of the current Q value and the policy entropy; temperature parameter The policy entropy is automatically adjusted through gradient descent to approach the preset target entropy value. The target network uses a multi-segment moving average method for soft updates: ； in, This is the soft update coefficient. For the current network parameters, These are the target network parameters.

9. A joint optimization system for trajectory design and resource allocation for unmanned aerial vehicles (UAVs) integrating sensing and computation, characterized in that, The system is used to implement the trajectory design and resource allocation joint optimization method for UAV sensor-computer interface integration as described in any one of claims 1 to 8, specifically including: The model building and initialization module is used to build a low-altitude surveillance system model that includes monitoring drones, ground base stations, and dynamic aerial targets, and to initialize system parameters; The mathematical model building module is used to build system mathematical models, including communication models, sensing models, computing models, and energy consumption models. The optimization problem construction and decomposition module is used to construct an optimization problem based on the system mathematical model, with the goal of minimizing the weighted sum of the total system processing delay and total energy consumption, and decompose the optimization problem into a joint optimization sub-problem of trajectory and beamforming and a sub-problem of computational offloading resource allocation. The hybrid decision-making module is used to solve the optimization problem based on a hybrid decision-making mechanism and deep reinforcement learning, so as to obtain the optimal trajectory design and resource allocation strategy for monitoring drones.

Images