AoI optimization method and system of communication and perception system supported by multiple unmanned aerial vehicles
By introducing information age and deep reinforcement learning algorithm DDPG-D3QN in the UAV communication perception system, the UAV trajectory and communication perception beam formation are optimized, and the problem of insufficient information freshness of ground equipment is solved, and delay reduction and resource reduction are achieved Improved utilization.
Patent Information
- Application Number
- CN202411184537.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-27
- Publication Date
- 2024-11-12
AI Technical Summary
The prior art ignores the information freshness of ground equipment in the integrated drone communication and perception system, resulting in a prolonged time delay and cannot meet the demand of intelligent applications for delay sensitivity.
Introduce Age of Information (AoI) to measure information freshness, optimize drone trajectory, user association, target sensing selection and communication sensing beamforming through deep reinforcement learning algorithm DDPG-D3QN to reduce the average information age of ground equipment .
On the premise of ensuring communication quality and perception requirements, it effectively reduces latency and improves resource utilization, and minimizes the average information age of ground equipment. The simulation results show that the reward value is increased by 10% to 30%.
Smart Images

Figure CN118945693A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of mobile communications, and in particular to an AoI optimization method and system for a communication and perception system supported by multiple unmanned aerial vehicles. Background Art
[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.
[0003] With the latest development of the sixth generation network (6G), the demand for communication and sensing in emerging industries such as autonomous driving, telemedicine and smart industry is growing, which has aroused great interest in the development of integrated sensing and communication technology in the academic community. In this context, integrated sensing and communication (ISAC) is a new key technology for the next generation of wireless networks. By providing the same spectrum resources and wireless infrastructure, it combines perception and communication, which can meet the needs of low-latency and high-reliability wireless transmission, significantly reduce costs and improve spectrum efficiency.
[0004] With the rapid development of drone technology and the high flexibility and maneuverability of drones, drones will play a very important role in the development of integrated perception and communication (ISAC) in the future. In the existing technology "UAV communication perception integrated energy efficiency optimization algorithm for large-scale Internet of Things", the system energy efficiency is maximized by jointly designing the three-dimensional trajectory and transmission power of the drone under the constraints of the mobility and perception capabilities of the drone. The existing technology "UAV interawareness integrated system trajectory and resource joint optimization based on NOMA" discusses improving the system throughput with the support of NOMA technology. However, the above studies all ignore the information freshness of ground equipment, which is not conducive to real-time application.
[0005] In recent years, with the rapid development of intelligent applications, research on integrated perception and communication supported by UAVs has continued to deepen. As more and more intelligent applications are becoming more and more sensitive to latency, it is extremely important to improve the information freshness of ground equipment in ISAC systems. Summary of the invention
[0006] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides an AoI optimization method and system for a communication and perception system supported by multiple UAVs, which solves the problem of system delay sensitivity, and introduces the age of information to measure the freshness of information to achieve the purpose of reducing latency. Under the premise of ensuring communication quality and perception requirements, the average age of information (Average AoI) of ground equipment is minimized by jointly optimizing UAV trajectories, user associations, target sensing selection, and communication and perception beamforming.
[0007] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:
[0008] In a first aspect, the present invention provides an AoI optimization method for a communication and perception system supported by multiple UAVs, comprising the following steps:
[0009] Build a communication and perception system model supported by multiple drones, jointly optimize the communication and perception beamforming, the connection relationship between drones and ground equipment and potential targets, and the dynamic trajectory of drones to minimize the average information age of ground equipment, so as to establish an optimization problem model;
[0010] Using a deep reinforcement learning algorithm, the optimization problem model is solved to continuously update the average information age, and the optimization is terminated after it is determined that the average information age of the ground equipment has not changed.
[0011] Among them, the deep reinforcement learning algorithm is the DDPG-D3QN algorithm, which constructs the optimization problem as a Markov decision process in a multi-agent environment, divides the action space in the Markov decision process into discrete action space and continuous action space, and processes the communication and perception beamforming in the continuous action space by calling DDPG, and processes the drone movement direction selection in the discrete action space by calling D3QN.
[0012] A further technical solution is that in the communication and perception system model supported by multiple drones, the drones simultaneously communicate and sense ground equipment and potential targets.
[0013] In a further technical solution, the Markov decision process includes a state space, an action space and a reward function, wherein the reward function refers to the system gain of taking an action under the system state in the nth time slot.
[0014] According to a further technical solution, the system status includes the position status of the UAV, the remaining energy status of the UAV, the amount of remaining user data and the number of perceived potential ground targets.
[0015] A further technical solution is that the deep reinforcement learning algorithm adopts the standard paradigm of centralized training and decentralized execution, inputs the state and actions of all drones into the action-value function of the centralized value network, and outputs the Q values of all drones.
[0016] A further technical solution is that the drone acts as an intelligent agent, each agent generates actions through its own action network, and the system obtains corresponding rewards by executing actions. In the action network, the drone updates continuous actions through the action value function with the maximum Q value.
[0017] A further technical solution is to use DDPG to update the parameters of the value network of D3QN by minimizing the loss function.
[0018] In a second aspect, the present invention provides an AoI optimization system for a communication and perception system supported by multiple drones, comprising:
[0019] An optimization problem establishment module, which is configured to: construct a communication and perception system model supported by multiple UAVs, jointly optimize communication and perception beamforming, the connection relationship between UAVs and ground equipment and potential targets, and the dynamic trajectory of UAVs to minimize the average information age of ground equipment, thereby establishing an optimization problem model;
[0020] An optimization problem solving module is configured to: use a deep reinforcement learning algorithm to solve the optimization problem model so that the average information age is continuously updated, and then terminate the optimization after determining that the average information age of the ground equipment is updated without change;
[0021] Among them, the deep reinforcement learning algorithm is the DDPG-D3QN algorithm, which constructs the optimization problem as a Markov decision process in a multi-agent environment, divides the action space in the Markov decision process into discrete action space and continuous action space, and processes the communication and perception beamforming in the continuous action space by calling DDPG, and processes the drone movement direction selection in the discrete action space by calling D3QN.
[0022] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the AoI optimization method for a communication and perception system supported by multiple drones as described in the first aspect.
[0023] In a fourth aspect, the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the AoI optimization method for a communication and perception system supported by multiple drones as described in the first aspect are implemented.
[0024] One or more of the above technical solutions have the following beneficial effects:
[0025] The present invention introduces the age of information to measure the freshness of information in order to reduce latency. Under the premise of ensuring communication quality and perception requirements, the average age of information (Average AoI) of ground equipment is minimized by jointly optimizing UAV trajectory, user association, target sensor selection, and communication and perception beamforming.
[0026] The present invention studies the communication and perception system supported by multiple UAVs and proposes a new balanced communication and perception mode (BISAC). In this mode, while the UAVs communicate with ground equipment (GEs), the perception duration is reasonably set according to the number of potential targets (PTs) and perception requirements, thereby achieving the purpose of reducing the mutual influence between communication and perception and improving resource utilization.
[0027] In order to obtain long-term AoI performance and effectively solve non-convex problems (optimization problems) containing continuous and discrete variables, the present invention introduces a deep reinforcement learning (DRL) solution based on multi-UAV trajectory optimization, which processes continuous and discrete variables in the system by calling deep deterministic policy (DDPG) and Dueling doubledeep Q networks (D3QN). Specifically, the actor-critic structure of DDPG is improved by combining D3QN, which uses the actor part of DDPG to search for the optimal communication and perception beams. At the same time, the critic part of DDPG is combined with D3QN to select the optimal flight direction of the UAV. Simulation results show that the proposed DDPG-D3QN algorithm has better stability and faster convergence speed, while obtaining higher rewards than existing DRL-based methods. Numerical results show that the reward value of the algorithm is 10% to 30% higher than the baseline scheme. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.
[0029] Figure 1 is a flow chart of an optimization method according to an embodiment of the present invention;
[0030] Figure 2 is a flow chart of the DDPG-D3QN algorithm in an embodiment of the present invention;
[0031] Figure 3 Schematic diagram of a multi-UAV supporting ISAC system in an embodiment of the present invention. DETAILED DESCRIPTION
[0032] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.
[0033] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.
[0034] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.
[0035] Terminology explanation:
[0036] The ISAC system, or integrated communication and perception system, is one of the important development directions of future communication technology. It closely combines communication and perception functions, aiming to improve spectrum utilization, reduce equipment costs, and promote technological innovation in multiple fields.
[0037] Communication and perception beamforming refers to the use of specific signal processing techniques, such as beamforming, in communication systems to optimize signal transmission and reception in order to improve communication quality and achieve more accurate environmental perception.
[0038] In the context of background technology, the present invention considers an integrated communication and perception system supported by multiple UAVs, in which the UAVs can communicate and sense ground equipment and potential targets simultaneously. Against the background of improving the information freshness of ground equipment in the ISAC system, a multi-UAV integrated communication and perception problem based on information freshness is studied. The goal is to minimize the average AoI of ground equipment by optimizing communication and sensing beamforming, user association and perception target selection, and UAV dynamic trajectory planning while ensuring the quality of communication and sensing services.
[0039] The present invention takes into account the delay sensitivity problem currently faced by various intelligent devices under the premise of random distribution of certain ground devices, and minimizes the average information age of each device by optimizing communication and sensor beamforming, the connection relationship between the UAV and ground devices and potential targets, and the trajectory of the UAV.
[0040] Embodiment 1
[0041] like Figure 1 As shown, this embodiment discloses an AoI optimization method for a communication and perception system supported by multiple drones, and the method includes the following steps:
[0042] S1: Build a communication and perception system model supported by multiple UAVs, jointly optimize the communication and perception beamforming, the connection relationship between UAVs and ground equipment and potential targets, and the dynamic trajectory of UAVs to minimize the average information age of ground equipment, so as to establish an optimization problem model;
[0043] In this embodiment, for the drone-supported ISAC system, the present invention considers a multi-drone-supported integrated communication perception system, i.e., a multi-drone-supported communication and perception system model, in which the drone can simultaneously communicate and sense ground equipment and potential targets. The present invention proposes a new communication and perception balance mode (BISAC), also called a synaesthesia balance mode, that is, according to the actual perception and communication requirements of potential targets and ground equipment, the perception and communication duration are flexibly configured, the mutual influence between communication and sensing is reduced, and resource utilization and information freshness are improved.
[0044] Specifically: They represent the number of time slots, the number of ground equipment (GE), the number of UAVs, and the number of potential targets (PTs), respectively. In the multi-UAV supported ISAC system, each UAV is equipped with a uniform linear array (ULA) with M antennas to communicate and sense the ground equipment and potential targets in the UAV service area.
[0045] In the UAV-supported ISAC system, the present invention introduces the age of information (AoI) to evaluate the freshness of UAV-assisted communication and perception. The average AoI is the average information age of all ground equipment GEs. Therefore, it can be expressed as:
[0046]
[0047] in, Indicates GE j AoI in the sth unit, N i (s) indicates UAV i The maximum waiting time in the sth unit, Represents UAV i DL duration of the transmitted signal, Represents UAV i The time delay to return to the starting point. i represents the i-th UAV, j represents the j-th ground device, s represents the s-th service unit, τ represents the time slot length, N s represents the initial time slot of the sth region, m represents the mth unit after the sth unit, s i Represents UAV i The set of units passed during the task execution, o i Represents UAV i The set of cells from the initial position to the sth cell.
[0048] Based on the proposed BISAC synaesthesia balance mode, the problem of multi-UAV supporting communication and perception is proposed to minimize the average age of information AoI of ground equipment GEs in the ISAC system by jointly optimizing communication perception beamforming, user association and perception target selection, and UAV dynamic trajectory. The present invention aims to minimize the average AoI of GEs by jointly optimizing communication perception beamforming, user association and perception target selection, and UAV dynamic trajectory. In addition, in this synaesthesia balance mode, the service area is divided into multiple small cells, and each cell is selectively traversed by adjusting the flight direction of the UAV according to energy and delay constraints.
[0049] The goal of this invention is to optimize communication and sensor beamforming UAV connection with ground equipment and potential targets And drone tracks To minimize the average information age of each device, the optimization problem can be expressed as:
[0050]
[0051] st C1:S k ≥ψ th
[0052]
[0053] C3:α i,j,s [n]∈{0,1}
[0054] C4:ξ i,s,k [n]∈{0,1}
[0055]
[0056]
[0057]
[0058]
[0059]
[0060]
[0061] Where w represents the transmit beamforming vector of UAV communication, n represents the nth time slot, e represents the transmit beamforming vector of UAV sensing, u represents the unit norm receive beamforming vector, k represents the kth potential target, and w i,j[n] represents the transmit beamforming vector of the communication between the i-th UAV and the j-th ground device, e i ,k[n] represents the transmit beamforming vector perceived by the i-th UAV and the k-th potential target, u i ,j[n] represents the unit norm receive beamforming vector of the jth ground device receiving the ith UAV; α represents whether the UAV and the ground device are connected, ξ represents whether the UAV senses the potential target, α i,j,s [n] indicates whether the i-th UAV is connected to the j-th ground device in the s-th cell in the n-th time slot, ξ i,s,k [n] indicates whether the i-th UAV perceives the k-th potential target in the s-th unit in the n-th time slot; q indicates the trajectory of all UAVs, q i [n] represents the trajectory of the i-th UAV in the n-th time slot. k represents the perceptual performance prediction model, ψ th represents the cumulative signal-to-noise ratio of the potential target, represents the total energy consumption, represents the maximum energy consumption, γ represents the signal-to-noise ratio of the ground equipment, Expressed as the signal-to-noise ratio of the jth ground device in the uplink, It is represented as the signal-to-noise ratio of the jth ground device in the downlink, K represents the number of potential targets, and P i max It is represented as the maximum transmission power of the i-th UAV.
[0062] Among them, constraint C1 means that the perceived target needs to meet the minimum perception requirements, and constraint C2 means that the UAV i The total energy consumption during task execution cannot exceed its maximum energy consumption Constraints C3 and C4 ensure that the relevant parameter α i,j,s [n], ξ i,s,k The value of [n] is only 0 or 1. Constraints C5 and C6 describe the QoS requirements for DL and UL communications, respectively. Constraint C7 indicates that the UAV i The transmission power cannot exceed its maximum value Constraints C8 and C9 indicate that a ground device or sensing target can only communicate or sense with one UAV in one grid. 10 is the unit norm constraint for the receive beamforming vector.
[0063] S2: Using a deep reinforcement learning algorithm, the optimization problem model is solved to continuously update the average information age, and the optimization is terminated after it is determined that the average information age of the ground equipment has not changed.
[0064] In this embodiment, the Markov decision process in a multi-agent environment is specifically constructed as follows:
[0065] There are three key elements to describe when modeling the optimization problem as a Markov decision process (MDP): state, action, and reward function. n Choose an action and then return that action to the environment to get the next state S n+1 And the reward R n , to evaluate the impact of the action on the environment.
[0066] The components of the MDP model for the optimization problem are defined as follows:
[0067] State space S n : The system state at the nth time slot is expressed as The system status includes the position status of the drone, the remaining energy status of the drone, the amount of remaining user data, and the number of perceived potential ground targets. Indicates the position status of the drone. Indicates the remaining energy status of the drone. Indicates the amount of remaining user data. Indicates the number of perceived potential ground targets. The detailed system status is defined as follows:
[0068] (1) definition in It is UAV i At the position of the nth time slot, denoted as q i (n) = [q xi (n),q yi (n)].
[0069] (2) definition in Represents UAV i The remaining energy when entering the nth time slot.
[0070] (3) definition in Indicates the amount of ground equipment data remaining in the sth unit at the beginning of the nth time slot.
[0071] (4) definition in represents the number of potential targets remaining unperceived in the sth unit at the beginning of the nth time slot.
[0072] Action Space A n: The action of the drone in the nth time slot can be expressed as For any UAV i Its action can be defined as
[0073] The system state transitions are as follows:
[0074] 1. Position state transition: UAV i Take action at the nth time slot The position of the next time slot after q i [n+1] can be expressed as:
[0075]
[0076] Among them, q i (n) = [q xi (n),q yi (n),H] represents UAV i At the position of the nth time slot, Represents UAV i In the direction state of the nth time slot, D, W represents the side length of the bounded area served by the drone, L W ×L D Represents the boundaries of the discretized small regions.
[0077] 2. Residual energy state conversion: UAV i Take action at the nth time slot The remaining energy of the subsequent n+1th time slot can be expressed as:
[0078]
[0079] in, Represents UAV i The maximum initial energy, represents the communication and sensing energy of the ith UAV in the sth area in the nth time slot, represents the hovering power of the UAV during communication and sensing, τ represents the time slot length, It represents the flight power of the UAV during communication and sensing. Indicates the speed of the drone.
[0080] 3. GEs data volume state transition: When UAV i When the data of the ground device is collected in the nth time slot, the ground device and the UAV i Disconnect immediately, i.e., α i,j,s [n+1] = 0, otherwise, the ground device will continue to communicate with the UAV in the next time slot. icommunication, α i,j,s [n+1]=1.
[0081]
[0082] Among them, α i,j,s [n]∈{0,1} represents the GE of the sth unit in the nth time slot. j Whether with UAV i communication, Indicates GE j Uplink transmission rate in the mth time slot (bit / s / Hz), Represents UAV i DL duration of the transmitted signal, L j Indicates GE j The amount of tasks generated, N s Indicates the starting time slot of the sth region.
[0083] Therefore, at the n+1th time slot, the amount of remaining ground equipment data in unit s can be expressed as:
[0084]
[0085] Where G is the general name of ground equipment, x is the xth time slot, Indicates the DL duration of the transmitted signal in the xth time slot, L j Indicates GE j The amount of tasks generated.
[0086] 4. Perceiving target state transitions: UAV i While communicating within the sth unit, the potential ground target is also being sensed. The perception performance is evaluated through the perception performance evaluation function. Specifically, when the evaluation model is higher than the threshold, the perception of the potential ground target is successful, otherwise the perception continues.
[0087]
[0088] Among them, PT k Denote as the kth potential target, ξ i,s,k [n]∈{0,1} represents the PT of the sth unit in the nth time slot k Whether it is UAV i Perception, Indicates PT k The signal-to-noise ratio at the nth time slot, ψ th Indicates the threshold for perception.
[0089] Therefore, the remaining potential ground targets in the n+1th time slot can be expressed as
[0090] Reward function R n :The reward in MDP refers to the reward in the nth time slot when the system state S n Take action A n The overall goal of the designed system in the present invention is to minimize the average age of information (AoI) of each ground device under the constraints of energy and sensing performance. However, according to the calculation formula of AoI, it can be found that the size of the average AoI is mainly determined by the transmission rate of the ground device and the perception performance of the UAV to potential ground targets. Considering that the overall goal is to minimize the average AoI of each ground device, the maximum transmission rate of the ground device in each time slot is taken as its equivalent gain. The specific formula is as follows:
[0091]
[0092] in, Indicates GE j Downlink transmission rate in the nth time slot (bit / s / Hz), N i (s) indicates UAV i The maximum waiting time in the sth unit, Represents UAV i The maximum perceived delay in the sth unit, Represents UAV i The maximum communication delay within the sth unit.
[0093] In this embodiment, the deep reinforcement learning algorithm for multi-UAV trajectory optimization is the DDPG-D3QN algorithm, which constructs the optimization problem as a Markov decision process in a multi-agent environment, divides the action space in the Markov decision process into discrete action space and continuous action space, and calls the deep deterministic strategy (DDPG) and Dueling doubledeep Qnetworks (D3QN) to process the communication and perception beamforming in the continuous action space and the UAV movement direction selection in the discrete action space. Specifically, each UAV as an agent first generates corresponding continuous actions according to the environment and through the actor network of DDPG, and then all UAVs share a common D3QN-critic network to evaluate the actions performed by the UAV, so as to obtain the action with the highest Q value, thereby obtaining the optimal communication and perception beamforming. The UAV movement direction selection is to determine the discrete action based on the continuous action obtained by the agent, and finally evaluate it through D3QN.
[0094] The interaction process between multi-agents and the environment is as follows:
[0095] Step 1: Divide the action space into two subspaces: discrete and continuous, namely is the continuous part of the action space, Therefore, the original action space becomes two feasible subspaces, which are computationally easier to handle.
[0096] Step 2: In the hybrid DRL-based framework, the standard paradigm of centralized training and decentralized execution in multi-agent DRL is used. That is, each drone takes action as a participant, and all drones share a common critic. Therefore, continuous action decisions can be shared between drones, and discrete decisions can be coordinated. The present invention uses D3QN composed of Dueling DQN and Double DQN to replace the initial DQN algorithm, thereby eliminating the problem of overestimation of Q value and estimating Q value more accurately. Specifically, each drone takes action as an agent and uses its own actor network To generate continuous actions Note that considering the inconsistency between the number of ground devices and potential targets, this will lead to dimensionality differences. Therefore, it is assumed that the number of ground devices is greater than the number of potential targets to make up for the dimensionality problem, that is, J ≥ K. In addition, the state S n And the actions of all drones are input into the action value function of the centralized critic network To output the Q value of all drones. Therefore, UAV can be obtained i The action formula is as follows:
[0097]
[0098]
[0099] Among them, argmax represents the maximum value function, Represents UAV i The action output Q value, w i,j [n]∈C M×1 、e i,k [n]∈C M×1 are the transmit beamforming vectors for DL communication and sensing, respectively, and u i,j [n]∈C M×1 represents the unit norm receive beamforming vector, and ε represents the critic network parameter.
[0100] Step 3: Update the actor network. Each agent generates actions through its actor network, and then the system obtains corresponding rewards by executing the actions. For the actor network, each UAV can update this part independently. iThe continuous actions are updated by the action value function with the maximum Q value. The specific formula is as follows:
[0101]
[0102] And the parameters of the actor network are updated by the actor part of DDPG as:
[0103]
[0104] in, is represented as the gradient function and δ is represented as the actor network parameter.
[0105] Step 4: Update the D3QN-critic network. Dueling DQN structure explicitly transforms the state value function and the action advantage function Divided into two independent parts, and then combined together to output the action value The specific formula is as follows
[0106]
[0107] Note that subtracting the mean in the above formula does not change the relative rank of the Π values, but helps to identify and enhance the stability of the optimization when the mean rate changes in the same pattern.
[0108] Considering that the current network in Dueling DQN, namely the D3QN-critic network, is very similar to the target network, it is easy to cause the algorithm to overestimate. Therefore, Double DQN is introduced to obtain S through the critic network. n+1 The action corresponding to the optimal action value in the state is then calculated using the target network to obtain the target value. This overestimation problem can be effectively avoided through the interaction of the two networks. Specifically, by using the target network as one of the value estimators, a strategy ζ is obtained to obtain the greedy maximization of the current network. Therefore, the target update is
[0109]
[0110] in, It is represented as the actual reward obtained by the agent in the nth time slot, ρ n represents the penalty function, ζ represents the strategy coefficient, and ε′ represents the target network coefficient of the critic network.
[0111] When there is a deviation in the updated value estimates of the target and policy networks, the policy will produce worse updated values, resulting in a lower action value, which will strengthen the optimization of the action in the next policy update. Therefore, before updating the policy, in order to minimize the deviation, it is necessary to set a higher update frequency for the value network. In addition, in order to avoid multiple update errors, we can set the update frequency of the target network to be lower than the update frequency of the value network.
[0112] For the update of the critic network of D3QN, DDPG is used to update its parameters by minimizing the following loss function.
[0113]
[0114] Update the critic's gradient:
[0115]
[0116] Step 5: Update the target network parameters. The parameters of the target network for the actor and critic can be updated as follows:
[0117]
[0118] Among them, δ′ represents the target network parameter of the actor network, and μ represents the update rate.
[0119] Embodiment 2
[0120] This embodiment provides an AoI optimization system for a communication and perception system supported by multiple drones, including:
[0121] An optimization problem establishment module, which is configured to: construct a communication and perception system model supported by multiple UAVs, jointly optimize communication and perception beamforming, the connection relationship between UAVs and ground equipment and potential targets, and the dynamic trajectory of UAVs to minimize the average information age of ground equipment, thereby establishing an optimization problem model;
[0122] An optimization problem solving module is configured to: use a deep reinforcement learning algorithm to solve the optimization problem model so that the average information age is continuously updated, and then terminate the optimization after determining that the average information age of the ground equipment is updated without change;
[0123] Among them, the deep reinforcement learning algorithm is the DDPG-D3QN algorithm, which constructs the optimization problem as a Markov decision process in a multi-agent environment, divides the action space in the Markov decision process into discrete action space and continuous action space, and processes the communication and perception beamforming in the continuous action space by calling DDPG, and processes the drone movement direction selection in the discrete action space by calling D3QN.
[0124] Embodiment 3
[0125] The purpose of this embodiment is to provide a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method of embodiment 1 when executing the program.
[0126] Embodiment 4
[0127] The purpose of this embodiment is to provide a computer-readable storage medium, a computer-readable storage medium having a computer program stored thereon, and when the program is executed by a processor, the steps of the method of embodiment 1 are performed.
[0128] The steps involved in the apparatus of the above embodiments 3 and 4 correspond to the method embodiment 1, and the specific implementation method can refer to the relevant description part of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.
[0129] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.
[0130] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
[0131] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.
Claims
1. AoI optimization method for communication and perception systems supported by multiple UAVs, characterized in that: The following steps are involved: Build a communication and perception system model supported by multiple drones, jointly optimize the communication and perception beamforming, the connection relationship between drones and ground equipment and potential targets, and the dynamic trajectory of drones to minimize the average information age of ground equipment, so as to establish an optimization problem model; Using a deep reinforcement learning algorithm, the optimization problem model is solved to continuously update the average information age, and the optimization is terminated after it is determined that the average information age of the ground equipment has not changed. Among them, the deep reinforcement learning algorithm is the DDPG-D3QN algorithm, which constructs the optimization problem as a Markov decision process in a multi-agent environment, divides the action space in the Markov decision process into discrete action space and continuous action space, and processes the communication and perception beamforming in the continuous action space by calling DDPG, and processes the drone movement direction selection in the discrete action space by calling D3QN.
2. The AoI optimization method for a communication and perception system supported by multiple UAVs as claimed in claim 1, characterized in that: In the multi-UAV supported communication and perception system model, the UAVs simultaneously communicate and sense ground equipment and potential targets.
3. The AoI optimization method for a communication and perception system supported by multiple UAVs as claimed in claim 1, characterized in that: The Markov decision process includes a state space, an action space and a reward function, wherein the reward function refers to a system gain of taking an action under a system state in an nth time slot.
4. The AoI optimization method for a communication and perception system supported by multiple UAVs as claimed in claim 3, characterized in that: The system status includes the position status of the UAV, the remaining energy status of the UAV, the amount of remaining user data and the number of perceived potential ground targets.
5. The AoI optimization method for a multi-UAV supported communication and perception system as claimed in claim 1, characterized in that: The deep reinforcement learning algorithm adopts the standard paradigm of centralized training and decentralized execution, inputs the state and actions of all drones into the action-value function of the centralized value network, and outputs the Q-values of all drones.
6. The AoI optimization method for a communication and perception system supported by multiple UAVs as claimed in claim 5, characterized in that: The drone acts as an intelligent agent, and each agent generates actions through its own action network. The system obtains corresponding rewards by executing actions. In the action network, the drone updates continuous actions through the action value function with the maximum Q value.
7. The AoI optimization method for a communication and perception system supported by multiple UAVs as claimed in claim 5, characterized in that: The parameters of the value network of D3QN are updated by minimizing the loss function in DDPG.
8. An AoI optimization system for communication and perception systems supported by multiple drones, characterized in that: include: An optimization problem establishment module, which is configured to: construct a communication and perception system model supported by multiple UAVs, jointly optimize communication and perception beamforming, the connection relationship between UAVs and ground equipment and potential targets, and the dynamic trajectory of UAVs to minimize the average information age of ground equipment, thereby establishing an optimization problem model; An optimization problem solving module is configured to: use a deep reinforcement learning algorithm to solve the optimization problem model so that the average information age is continuously updated, and then terminate the optimization after determining that the average information age of the ground equipment is updated without change; Among them, the deep reinforcement learning algorithm is the DDPG-D3QN algorithm, which constructs the optimization problem as a Markov decision process in a multi-agent environment, divides the action space in the Markov decision process into discrete action space and continuous action space, and processes the communication and perception beamforming in the continuous action space by calling DDPG, and processes the drone movement direction selection in the discrete action space by calling D3QN.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in the AoI optimization method of the multi-UAV supported communication and perception system as described in any one of claims 1 to 7 are implemented.
10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the AoI optimization method of the multi-UAV supported communication and perception system according to any one of claims 1 to 7 are implemented.
Citation Information
Cited By
MATD3-based multi-unmanned aerial vehicle sensing integrated trajectory and beam forming optimization method
CN120186624A
An AARIS-based method for optimizing the average information age
CN120456060B
Joint beam and trajectory design method for auxiliary sensing energy supply of multiple unmanned aerial vehicles
CN121463077A
