Unmanned aerial vehicle auxiliary communication anti-interference method based on multi-agent reinforcement learning
By adopting a proximity strategy optimization method of heterogeneous multi-agent reinforcement learning in the drone assisted communication network, coordinating the communication parameters of the drone and ground users, the failure to effectively utilize heterogeneous proxy performance in the prior art is solved. The problem is to optimize the system throughput and effectively deal with anti-interference.
Patent Information
- Application Number
- CN202410908587.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-08
- Publication Date
- 2024-11-08
AI Technical Summary
Existing multi-agent deep reinforcement learning methods fail to effectively utilize potential performance enhancements between heterogeneous agents in UAV-assisted communication networks, especially when defending against malicious interference, resulting in insufficient algorithms for handling complex tasks in an adversarial environment. , and the system complexity increases.
Adopting a proximal strategy optimization method based on heterogeneous multiagent reinforcement learning, the system modeling and partially observable Markov decision-making process are used to coordinate the communication parameters of drones and ground users, optimize the system throughput, and design unmanned Distributed optimization algorithms for aircraft flight trajectory, ground node transmission power and channel selection, design reward functions for UAV agents and ground node agents respectively to promote collaborative decision-making.
It realizes communication parameter coordination with complete models and clear physical significance, optimizes system throughput, avoids interference, maximizes network performance, and significantly improves the multi-domain optimization capabilities of the UAV-assisted communication network in confrontation scenarios.
Smart Images

Figure CN118921099A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of wireless communication anti-interference, and in particular to a UAV-assisted communication anti-interference method based on multi-agent reinforcement learning. Background Art
[0002] In communication systems, drones are widely used to provide emergency communication services due to their flexibility, simplicity, and low deployment cost (NC Luong, DT Hoang, S. Gong, D. Niyato, P. Wang, Y. Liang et al., "Applications of Deep Reinforcement Learning in Communications and Networking: A Survey," IEEE Communications Surveys\&Tutorials, vol. 21, no. 4, pp. 3133-3174, Fourth quarter 2019.). When used as a base station, drones can establish an effective line-of-sight (LoS) link with ground users, thereby reducing energy consumption and achieving reliable connection. In addition, drones can be deployed dynamically to improve service quality, and using multiple drones as aerial base stations represents a general approach to extending network coverage and enhancing overall communication capabilities (E. Rodrítez, B. Otero, N. Gutiérrez and R. Canal, "ASurvey of Deep Learning Techniques for Cybersecurity in Mobile Networks," IEEE Communications Surveys\&Tutorials, vol. 23, no. 3, pp. 1920-1955, third quarter 2021.).
[0003] Although the collaborative performance of communication tasks by multi-UAV-assisted communication systems is promising, there are still challenges such as spectrum congestion, serious mutual interference, and complex resource optimization problems (Y. Gao, Y. Wu, Z. Cui, W. Yang, G. Huan and S. Xu, "Robust Trajectory and Communication Design for Angle-constrained Multi-UAV Communications in the Presence of Jammers," China Communications, vol. 19, no. 2, pp. 131-147, Feb 2022.). In addition, the frequent movement of drones leads to dynamic changes in the spectrum environment (H. Albinsaid, K. Singh, S. Biswas and C.-P. Li, "Multi-Agent Reinforcement Learning-Based Distributed Dynamic Spectrum Access," IEEE Transactions on Cognitive Communications and Networking, vol. 8, no. 2, pp. 1174-1185, Jun 2022.) and subsequent problems, which increases the complexity of the problem, and therefore requires a regular real-time updated control signal mechanism.
[0004] Multi-agent deep reinforcement learning (MADRL) is a promising approach to address the above immediate challenges. The authors (Y.Ju, Y.Chen, Z.Cao, L.Liu, Q.Pei, M.Xiao et al., "Joint Secure Offloading and Resource Allocation for Vehicular Edge Computing Network: A Multi-Agent DeepReinforcement Learning Approach," IEEE Transactions on IntelligentTransportation Systems, vol.24, no.5, pp.5555-5569, May 2023.) proposed a vehicular network resource allocation algorithm based on an extended dual deep Q-learning framework, which jointly optimizes transmission energy consumption, channel selection, and computing resource allocation. The authors proposed a hierarchical deep Q-learning approach to solve the communication resource allocation problem in cooperative UAV networks in (A.Kaur, J.Thakur, M.Thakur, K.Kumar, A.Prakash and R.Tripathi, “Deep Recurrent Reinforcement Learning-Based Distributed Dynamic Spectrum Access in Multichannel Wireless Networks With Imperfect Feedback,” IEEE Transactions on Cognitive Communications and Networking, vol.9, no.2, pp.281-292, Apr 2023.). In addition, the authors proposed a covert communication problem based on a multi-agent deep Q-network (MADQN) in (A.Pourranjbar, G.Kaddoum, A.Ferdowsi and W.Saad, “Reinforcement Learning for Deceiving Reactive Jammers in Wireless Networks,” IEEE Transactions on Communications, vol.69, no.6, pp.3682-3697, Jun 2021.) to deceive jammers.
[0005] However, these approaches often assume homogeneity among agents, inadvertently ignoring the potential performance enhancements gained through agent heterogeneity, especially when defending against malicious jammers. This issue causes the algorithms to be insufficient in effectively handling complex tasks in adversarial environments. For example, in drone-assisted networks, drones can design their flight trajectories and ground nodes can choose transmit power and channels to counter jammers. In addition, these works often utilize system reward design as a collective incentive for all agents, which may hinder individual agents from exploiting their unique features and attributes. The inherent interdependencies among heterogeneous agents create more complex dynamic environments, where the behavior of one class of agents can trigger cascading consequences on another class of agents, thereby increasing the complexity of the system. Summary of the invention
[0006] The purpose of the present invention is to provide a method with a complete model, clear physical meaning, and the ability to coordinate communication parameters between UAVs and ground users to resist interference and optimize system throughput.
[0007] The technical solution to achieve the purpose of the present invention is:
[0008] Step 1: System modeling of the multi-UAV assisted ground user communication network to facilitate subsequent problem analysis; the system modeling includes the coordinate modeling of the UAV, the communication parameter modeling between the user and the UAV, and the expected transmission rate modeling of the system;
[0009] Under the jammer attack, multiple drones and ground nodes collaborate to perform communication tasks; the drone set is represented as The user set is represented as The channel set is represented as The time slot is represented by Assume L>M;
[0010] The drones in the system optimize their trajectories. Users perceive the current channel quality through broadband spectrum and optimize the transmission power and channel. The multi-agent reinforcement learning algorithm based on heterogeneous proximal strategy optimization HAPPO is used to adjust the strategy of each agent.
[0011] Specifically, in each time slot t, each ground node and drone selects the appropriate transmission power, channel and drone trajectory according to the proximal strategy optimization PPO algorithm. The ground node can only stably access the network within the coverage of a specific drone. The service quality of the ground node at the corresponding moment is related to the location of the drone, the transmission channel and power of the ground node. The strategy network is updated according to the communication quality until the maximum number of iterations is reached, and the strategy of each agent is finally obtained. This achieves the effect of avoiding interference and maximizing system throughput.
[0012] To upload data, all ground nodes compete to access L channels. To ensure the communication quality of ground nodes, the drone dynamically optimizes its own trajectory. At the same time, due to the existence of external jammers, the frequency environment is more complex. The position of drone n at time t is expressed as H represents the flying height of the UAV, assuming that the flying height of the UAV is fixed;
[0013] Drone location The relationship with speed V is as follows:
[0014]
[0015] in, represents the position of UAV n at time t-1;
[0016] In addition, the trajectories of any UAV n and n' need to comply with the predefined collision avoidance constraints, which are expressed as:
[0017]
[0018] Among them, l min Indicates the distance threshold to avoid collision between drones.
[0019] For the communication model and channel model, the channel power gain is calculated using the free space path loss. Each ground node occupies a specific channel for data transmission. In addition, if multiple adjacent nodes choose the same channel, the ground nodes can share the channel. At time slot t, the channel gain between UAV n and ground node m is:
[0020]
[0021] Where ρ0 represents the channel gain at the reference distance d0, d n,m (t) represents the distance between UAV m and ground node n at time slot t;
[0022] The transmission power of ground node m is configured within a predefined range. Let p m (t) and p J (t) represents the transmission power of ground node m and the interference power of the jammer in time slot t, ξ represents the free fading coefficient of the channel; the signal-to-noise SINR received by drone m from ground node n is n,m (t) is expressed as:
[0023]
[0024] Among them, p m max represents the maximum transmission power of ground node m, δ(f m (t) = f J(t)) indicates that if the ground node m and the jammer select the same channel, δ(f m (t) = f J (t))=1, otherwise it is equal to 0, represents the channel power gain between malicious jammer J and drone n, N0 represents the power spectral density of additive white Gaussian noise, and B represents the channel bandwidth.
[0025] Expected transmission rate R n,m (t) is expressed in the form of:
[0026]
[0027] Where SINR th represents the SINR threshold of the received signal at the receiver, and B represents the channel bandwidth.
[0028] The main optimization goal is to maximize the network throughput. The optimization goal of the network is expressed as:
[0029]
[0030] Step 2: Model the process of UAV-assisted ground user communication as a partially observable Markov process, model the observations, actions, and rewards of the UAV agent and the ground user agent respectively, and design reward functions for the UAV agent and the ground user agent respectively to promote the collaborative completion of communication tasks.
[0031] In a dynamic unknown environment, the spectrum waterfall diagram of the drone receiving end is used as the user's input state, and the user's service quality is used as the drone's input state. The frequency domain and energy domain characteristics of the interference are calculated, and finally the strategy to maximize the network throughput is obtained, as follows:
[0032] In a dynamic and unknown UAV-assisted communication environment, the flight trajectory, transmission power, and channels of the UAV and the ground nodes are jointly optimized; the collaborative decision-making between the UAV and the ground nodes is an important factor affecting the throughput of the entire system. In addition, each action performed in the current time slot triggers the transition of the system from the current state to the subsequent state. Therefore, the complex UAV trajectory and network resource collaborative optimization problem is modeled as a partially observable Markov decision process (POMDP). More precisely, the UAV and the ground users are regarded as two heterogeneous agents, each with a unique observation space and action space, making decisions independently in each time slot with the goal of optimizing the system throughput. To simplify the representation, let represents the index of the agent. For each time slot t, let Indicates the current environment status information. Represented as the set of all states, the agent A different observation is received at time t All agents are in policy π i (a i |o i ) to take joint action This results in a transition from the current state s(t) to the state s(t+1) at time (t+1).
[0033] The constructed Markov description is as follows:
[0034] Observation: Due to partial observability and the limited perception distance of the agent, the observation of the drone agent n includes its current seat and adjacent drone coordinates and whether users in the coverage area are connected to the network. n,m (t);
[0035] The observation of drone agent n is expressed as:
[0036]
[0037] For each ground node agent m, the observations include its transmit power p m (t), channel selection f m (t) and whether to access the drone auxiliary network a m,n (t), then the observation value of the ground agent m is expressed as:
[0038]
[0039] Action: The drones and ground nodes are modeled as heterogeneous agents with different collaborative action spaces to optimize the overall throughput of the system; at time slot t, the action of the drone agent n is expressed as:
[0040]
[0041] In the formula represents the flight direction of the drone agent n in time slot t; for the ground node agent m, the tasks include avoiding spectrum conflicts between other ground nodes and defending against malicious interference attacks;
[0042] In addition, the actions of the ground node agent are described as:
[0043]
[0044] That and represents the channel selection and transmit power set by the ground node agent m; since the action space of the drone agent is continuous, while the action space of the ground node agent is discrete; considering that learning optimal actions in continuous action spaces is still a complex task in multi-agent reinforcement learning, it is assumed that all continuous spaces are discretized;
[0045] Reward: Considering that the goal of the UAV-assisted communication system based on multi-agent reinforcement learning is to optimize the system throughput during the entire service period, the value of the reward needs to be positively correlated with the objective function; in addition, the ground node access constraints and collision constraints must be considered, and the agents that violate the constraints must be punished. Therefore, the rewards of the UAV agent and the ground node agent are designed separately to optimize the total system throughput;
[0046] The reward is expressed as:
[0047]
[0048] Among them, a n,m (t) is an indicator function. At time slot t, if the ground node agent n accesses the network of the drone agent m, a n,m (t)=1, otherwise it is a n,m (t)=0,R col (t) represents the collision penalty and R access (t) means additional reward for user visit.
[0049] Step 3, determine the proximal policy optimization resource method for UAV-assisted communication based on heterogeneous multi-agents, including multiple heterogeneous agents interacting with a multi-UAV-assisted anti-interference environment; the proximal policy optimization resource is based on a centralized training and distributed execution framework, and each UAV agent and ground node agent executes the collectively adopted strategy to select actions, thereby accumulating experience data until a predefined time threshold is reached, and then calculates the corresponding action state mapping and advantage function respectively to complete the optimization of the policy network.
[0050] The proximal policy optimization resource method for heterogeneous multi-agent UAV-assisted communication networks includes the following steps:
[0051] The input is the actor network and the critic network, the experience pool, and the number of training steps;
[0052] The output is the trained UAV actor network and ground user actor network;
[0053] Step 31, initialize the policy network πθ and value network parameters V φ , initialize the experience pool D, set the number of cycles, step size and obtain the initial state of the environment s(1);
[0054] Step 32, each drone agent and ground node agent performs an action A(t) according to πθ and obtains a joint reward R(t). At the same time, the environment is transferred to the state s(t+1) at the next moment; A(t) represents the action set of the drone agent and the ground user agent; R(t) represents the reward set of the drone agent and the ground user agent;
[0055] Step 33, store the experience {s(t), A(t), R(t), s(t+1)} into the experience pool;
[0056] Step 34, calculate the Q function and calculate the advantage function;
[0057] Step 35, randomly extract the minimum batch of experience B for training and update π θ and V φ The hidden layer parameters in ;
[0058] Step 36, adaptively updating θ according to L(θ) and updating φ according to L(φ);
[0059] Steps 32 to 36 are repeated until the maximum number of iterations is reached and the algorithm ends.
[0060] The multi-agent reinforcement learning framework consists of multiple agents interacting with a multi-UAV assisted environment, where each UAV and ground node agent corresponds to an agent that executes the PPO algorithm. Each agent runs through two different phases: centralized training and decentralized execution. In the training phase, the critic plays the role of a central controller and predicts the state of the agent based on the comprehensive global state information Q(s,a1,a2,...,a N+M |φ) and observations to calculate the action value of the set, evaluate the behavior of the actor from a global perspective, and guide the agent to improve its decision. The critic network updates the parameters θ by minimizing the loss function c :
[0061]
[0062] Where φ is the parameter of the evaluation network, s' and φ' are the updated state of the target network and the updated parameter of the evaluation network, respectively. HAPPO is the expected reward value, loss(θ c ) is the loss function and E is the expected value. The actor network updates the parameters θ and takes actions based on the centralized Q function and its own observations.
[0063] In the execution step, only the trained actor drone network and actor ground node network make decisions based on local observations. Each agent can independently obtain near-global optimal behavior without directly understanding the behavior or state of other agents.
[0064] The training process is divided into two different phases: experience accumulation phase and strategy improvement phase. In the experience accumulation phase, each UAV agent and ground node agent executes the collectively adopted strategy to select actions, thereby accumulating experience data until a predefined time threshold is reached, denoted as T, and then the corresponding action-state mapping and advantage function are calculated respectively.
[0065] In the strategy improvement phase, the strategy network π θ The loss function L(θ) is used to perform iterative optimization for K epochs, and the state value of the same small batch of data extracted from the experience memory buffer is optimized K times by the loss function L(φ). In order to ensure the stability and consistency of the training process, the constituent data samples in the experience pool D will be randomly reindexed.
[0066] Compared with the prior art, the present invention has the following significant advantages: (1) considering the multi-domain optimization model of UAV-assisted communication network in confrontation scenarios, a distributed optimization algorithm based on heterogeneous multi-agent reinforcement learning is designed under this model; (2) in the UAV-assisted network, interference is countered by undesigning the flight trajectory of UAVs, the transmission power of ground nodes and channels; (3) reward functions are designed for UAV agents and ground node agents respectively. Compared with the unified reward in other works, different reward functions can better promote the strategy learning of heterogeneous agents. BRIEF DESCRIPTION OF THE DRAWINGS
[0067] Figure 1 It is a structural schematic diagram of a multi-UAV auxiliary communication system in a confrontation scenario in the present invention.
[0068] Figure 2 It is a diagram of the updating process of the distributed decision-making framework based on the multi-agent reinforcement learning algorithm in the present invention.
[0069] Figure 3 This is the convergence time curve when the number of users is 4, the number of jammers is 2, and the number of channels is 6 in the present invention.
[0070] Figure 4 It is a trajectory diagram of the UAV in the present invention. DETAILED DESCRIPTION
[0071] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.
[0072] This paper proposes an anti-interference method for UAV-assisted communication based on multi-agent reinforcement learning, which is transformed into a partially observable Markov decision process (POMDP), in which the flight trajectory of the UAV, the transmission power control of the ground node and the channel selection are jointly designed. At the same time, in order to solve the decision-making problem between different types of agents, a heterogeneous distributed decision-making algorithm based on the multi-agent proximal policy optimization (HAPPO) framework is proposed.
[0073] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.
[0074] Figure 1 This is a model diagram of a multi-UAV assisted communication system in a confrontation scenario. In this model, there are multiple UAVs and multiple ground users, and users need to upload data to the UAVs. Multiple agents can control the user's frequency selection and power adjustment, and multiple agents can control the UAV to plan the trajectory. There are multiple jammers in the system to interfere with user communications.
[0075] Figure 2 It is a distributed decision framework based on multi-agent reinforcement learning algorithm. The framework consists of multiple actor networks and critic networks. The actor network makes decisions based on its local observation information, and the critic network guides the actor network to optimize its network parameters based on global information, thereby obtaining the best network optimization decision in the current environment. The update of deep neural network parameters depends on real-time recording of the current input state, the selected decision description, and training through multi-agent reinforcement learning, rewards, and the next environmental state.
[0076] The framework is characterized as follows: multiple UAVs work together to ensure communications for ground users, and one or more jammers interfere with user communications; in a dynamic unknown environment, the UAV receiver position, signal-to-noise ratio and access status are used as the input state of the UAV neural network, and the service quality and frequency selection of ground users are used as the input state of the ground user neural network. The frequency domain and time domain characteristics of the interference are calculated, and finally the strategy for network resource optimization is obtained. The PPO algorithm is used to calculate the Q function and advantage function as the decision basis of the model.
[0077] The updating process adopts the idea of experience replay. When the agent's training experience is large enough, the target value is randomly extracted from the experience matrix; and the system parameters are updated by the gradient descent method.
[0078] Furthermore, the signal-to-noise ratio of the drone receiving end of the distributed decision-making framework based on the multi-agent reinforcement learning algorithm is expressed as:
[0079]
[0080] In formula (1), f m (t) and pm (t) represent the frequency and transmission power selected by user m under the guidance of the agent, t represents the current time, p m max represents the maximum transmission power of ground node m, δ(f m (t) = f J (t)) indicates that if the ground node m and the jammer select the same channel, δ(f m (t) = f J (t))=1, otherwise it is equal to 0, represents the channel power gain between malicious jammer J and drone n, N0 represents the power spectral density of additive white Gaussian noise, and B represents the spectrum bandwidth.
[0081] Furthermore, the expected transmission rate R of ground node m accessing UAV n is n,m (t) is expressed in the form of:
[0082]
[0083] Where SINR th It represents the SINR threshold of the received signal at the drone receiver.
[0084] Furthermore, based on the distributed decision-making framework of the multi-agent reinforcement learning algorithm, the communication environment is dynamic and unknown, and the network resource optimization problem in the model is modeled as a partially observable Markov decision process. The complex interference pattern in the environment is related to historical information, so the environment state is defined as S t ={s t ,s t-1 ,...,s t-T+1}, where T represents the number of historical states to be traced back, S t Represents a T×N two-dimensional matrix, composed of S t The matrix constructs the environment information of each agent.
[0085] Furthermore, the constructed POMDP can be described as follows:
[0086] Observation: Due to partial observability and the limited perception distance of the agent, the observation of the drone agent n includes its current seat and adjacent drone coordinates and whether users in its coverage area are connected to the network. n,m (t), therefore, the observation of drone agent n can be expressed as:
[0087]
[0088] For each ground node agent m, its observations include its transmit power p m (t), channel selection f m (t) and whether to access the drone auxiliary network a m,n (t), then the observation value of the ground agent m can be expressed as:
[0089]
[0090] Action: The drones and ground nodes are modeled as heterogeneous agents with different collaborative action spaces to optimize the overall throughput of the system. At time slot t, the action of the drone agent n is expressed as:
[0091]
[0092] In the formula represents the flight direction of drone agent n in time slot t. For ground node agent m, the tasks include avoiding spectrum conflicts between other ground nodes and defending against malicious interference attacks. In addition, the action of the ground node agent is described as:
[0093]
[0094] That and represents the channel selection and transmit power set by the ground node agent m. Since the action space of the drone agent is continuous, while the action space of the ground node agent is discrete. Considering that learning optimal actions in continuous action spaces is still a complex task in multi-agent reinforcement learning, it is assumed that all continuous spaces are discretized.
[0095] Reward: Considering that the goal of the drone-assisted communication system based on multi-agent reinforcement learning is to optimize the system throughput during the entire service period. Therefore, the value of the reward needs to be positively correlated with the objective function. In addition, it is also necessary to consider the ground node access constraints and collision constraints, and punish the agents that violate the constraints. Therefore, the rewards of the drone agent and the ground node agent are designed separately to optimize the total system throughput. Therefore, the reward can be expressed as:
[0096]
[0097] Among them, a n,m (t) is an indicator function. At time slot t, if the ground node agent n accesses the network of the drone agent m, a n,m (t)=1, otherwise it is a n,m (t)=0,R col (t) represents the collision penalty and R access (t) means additional reward for user visit.
[0098] The present invention is based on a heterogeneous multi-agent proximal strategy optimization algorithm and includes the following steps:
[0099] Step 1: Initialize the policy network π θ And the value of network parameters V φ , initialize the experience pool D, set the number of cycles, step size and obtain the initial state of the environment s(1);
[0100] Step 2: Each drone agent and ground node agent uses π θ Execute an action A(t) and obtain a reward R(t). At the same time, the environment moves to the next state s(t+1);
[0101] Step 3, store the experience {s(t), A(t), R(t), s(t+1)} into the experience pool;
[0102] Step 4, calculate the Q function and calculate the advantage function;
[0103] Step 5: Randomly extract the minimum batch of experience B for training and update π θ and V φ The hidden layer parameters in ;
[0104] Step 6, adaptively update θ according to L(θ) and update φ according to L(φ);
[0105] Repeat steps 2 to 6 until the maximum number of iterations is reached and the algorithm ends.
[0106] The multi-agent reinforcement learning framework consists of multiple agents interacting with a multi-UAV assisted environment, where each UAV and ground node agent corresponds to an agent executing the PPO algorithm. Each agent runs through two different phases: centralized training and decentralized execution. In the training phase, the critic plays the role of a central controller and predicts the state of the agent based on the comprehensive global state information Q(s,a1,a2,...,a N+M |φ) and the observation results to calculate the action value in the set. Note that this step is to evaluate the behavior of the actor from a global perspective and guide the agent to improve its decision. The critic network updates its parameters θ by minimizing the loss function c :
[0107]
[0108] Where φ is the parameter of the evaluation network, s' and φ' are the updated state of the target network and the updated parameter of the evaluation network, respectively. The actor network updates its parameters θ and takes actions based on the centralized Q function and its own observations.
[0109] Example 1
[0110] The first embodiment of the present invention is described in detail as follows. The system simulation adopts Python language and is based on the pytorch deep learning framework. The parameter setting does not affect the generality. The embodiment verifies the effectiveness of the proposed model and method. Figure 3 Verify the convergence of the algorithm against fixed interference patterns. The parameters are set as follows: A multi-UAV assisted network consisting of 4 users and 2 UAVs is considered. All agents are in a 1000x1000m 2 In the area, the coverage range of each drone is 125m. Assume that the initial positions of the drones are (150, 0, 300) and (225, 700, 300), the positions of the ground nodes are (150, 300, 0), (250, 300, 0), (150, 575, 0) and (250, 575, 0), and the positions of the jammers are (400, 600, 0), (200, 200, 0) and (600, 400, 0). In each time slot, each drone can only serve 3 ground nodes, and the speed of the drone is set to 5m / s. The allocated channel bandwidth is 1MHz, and the transmission power of each ground node is set to {23, 15, 10, 5dbm}. The path loss factor is 4 and the background noise is -80dbm. In Example 1, two fixed interference modes are considered: 1. swept frequency interference with a sweep speed of 1 GHz / s; 2. comb interference with three fixed interference frequencies, namely 2 MHz, 10 MHz, and 18 MHz.
[0111] As can be seen from the figure, the cumulative reward increases as the episode progresses, showing convergence in both configurations. Compared with C-MAPPO (optimizing only trajectories and channels) and FT-PPO (optimizing only the transmit power of ground nodes and fixed trajectories, fixed sequence communication frequency), the method provided by this application jointly optimizes the transmit power, channel, and trajectory. This shows that the method provided by this application converges slowly, but its performance is significantly improved due to its ability to coordinate the behaviors of different agents.
[0112] Example 2
[0113] The second embodiment of the present invention is described in detail as follows. The system simulation adopts Python language and is based on the pytorch deep learning framework. The parameter setting does not affect the generality. The embodiment verifies the effectiveness of the proposed model and method. Figure 4 Verify the flight trajectory of drones when adversarial fixed interference patterns. The parameters are set as follows: A multi-drone assisted network consisting of 4 users and 2 drones is considered. All agents are in a 1000x1000m 2In the area, the coverage range of each drone is 125m. Assume that the initial positions of the drones are (150, 0, 300) and (225, 700, 300), the positions of the ground nodes are (150, 300, 0), (250, 300, 0), (150, 575, 0) and (250, 575, 0), and the positions of the jammers are (400, 600, 0), (200, 200, 0) and (600, 400, 0). In each time slot, each drone can only serve 3 ground nodes, and the speed of the drone is set to 5m / s. The allocated channel bandwidth is 1MHz, and the transmission power of each ground node is set to {23, 15, 10, 5dbm}. The path loss factor is 4 and the background noise is -80dbm. Consider a dynamic interference mode: randomly switch between sweep interference and comb interference every 100ms.
[0114] As can be seen from the figure, through trajectory design, the drone agents learn to collaboratively provide stable services to different ground nodes. The blue lines represent the trajectories of the drones, and the red dots represent the ground nodes. Drone-1 and Drone-2 maneuver and maintain hovering positions near different ground users, with the goal of promoting high-quality communication services. The goal of each drone is to serve as many users as possible. However, their coverage area is inherently limited, and this limitation usually excludes stationary hovering, but instead forces drones to constantly adjust their positions and dynamically reposition themselves to positions closer to unserved users to expand their service range and alleviate coverage limitations.
[0115] The above-described embodiments of the present application do not constitute a limitation on the protection scope of the present application.
Claims
1. A UAV-assisted communication anti-interference method based on multi-agent reinforcement learning, characterized in that: The method comprises: Step 1: System modeling of the multi-UAV assisted ground user communication network; the system modeling includes coordinate modeling of the UAV, communication parameter modeling between the user and the UAV, and expected transmission rate modeling of the system; Step 2: Model the process of UAV-assisted ground user communication as a partially observable Markov process, model the observation, action, and reward of the UAV agent and the ground user agent respectively, and design reward functions for the UAV agent and the ground user agent respectively to promote the collaborative completion of the communication task; Step 3, determine the proximal policy optimization resource method for UAV-assisted communication based on heterogeneous multi-agents, including multiple heterogeneous agents interacting with a multi-UAV-assisted anti-interference environment; the proximal policy optimization resource method is based on a centralized training and distributed execution framework, and each UAV agent and ground node agent executes the collectively adopted strategy to select actions, thereby accumulating experience data until a predefined time threshold is reached, and then calculates the corresponding action state mapping and advantage function respectively to complete the optimization of the policy network.
2. The method according to claim 1, characterized in that: In step 1, the coordinates of the drone are modeled as follows: Under the jammer attack, multiple drones and ground nodes collaborate to perform communication tasks; the drone set is represented as The user set is represented as The channel set is represented as The time slot is represented by Assume L>M; All ground nodes compete to access L channels, and the drones dynamically optimize their own trajectories. The position of drone n at time t is expressed as H represents the flying height of the UAV, assuming that the flying height of the UAV is fixed; Drone location The relationship with speed V is as follows: in, represents the position of UAV n at time t-1; In addition, the trajectories of any UAV n and n' need to comply with the predefined collision avoidance constraints, which are expressed as: Among them, l min Indicates the distance threshold to avoid collision between drones.
3. The method according to claim 2, characterized in that The communication parameters between the user and the drone are modeled as follows: For the communication model and channel model, the channel power gain is calculated using the free space path loss. Each ground node occupies a specific channel for data transmission. In addition, if multiple adjacent nodes choose the same channel, the ground nodes can share the channel. At time slot t, the channel gain between UAV n and ground node m is: Where ρ0 represents the channel gain at the reference distance d0, d n,m (t) represents the distance between UAV m and ground node n at time slot t; The transmission power of ground node m is configured within a predefined range. Let p m (t) and p J (t) represents the transmission power of ground node m and the interference power of the jammer in time slot t, ξ represents the free fading coefficient of the channel; the signal-to-noise SINR received by drone m from ground node n is n,m (t) is expressed as: Among them, p m max represents the maximum transmission power of ground node m, δ(f m (t) = f J (t)) indicates that if the ground node m and the jammer select the same channel, δ(f m (t) = f J (t))=1, otherwise it is equal to 0, represents the channel power gain between malicious jammer J and drone n, N0 represents the power spectral density of additive white Gaussian noise, and B represents the channel bandwidth.
4. The method according to claim 3, characterized in that The expected transfer rate is modeled as follows: Expected transmission rate R n,m (t) is expressed in the form of: Where SINR th represents the threshold of SINR of the received signal at the receiver, and B represents the channel bandwidth; The optimization goal is to maximize the network throughput. The optimization goal of the network is expressed as:
5. The method according to claim 1, characterized in that Step 2: Model the process of UAV-assisted ground user communication as a partially observable Markov process, model the observation, action, and reward of the UAV agent and the ground user agent respectively, and design reward functions for the UAV agent and the ground user agent respectively to promote the collaborative completion of the communication task, including: In a dynamic unknown environment, the spectrum waterfall diagram of the drone receiver is used as the user's input state, and the user's service quality is used as the drone's input state. The frequency domain and energy domain characteristics of the interference are calculated, and finally the strategy to maximize the network throughput is obtained: The UAV and ground users are considered as two heterogeneous agents, each with a unique observation space and action space, making independent decisions in each time slot with the goal of optimizing system throughput. represents the index of the agent. For each time slot t, let Indicates the current environment status information. Represented as the set of all states, the agent A different observation is received at time t All agents are in policy π i (a i |o i ) to take joint action This results in a transition from the current state s(t) to the state s(t+1) at time (t+1). The constructed Markov description is as follows: Observation: The observation of drone agent n includes its current and adjacent drone coordinates and whether users in the coverage area are connected to the network. n,m (t); The observation of drone agent n is expressed as: For each ground node agent m, the observations include its transmit power p m (t), channel selection f m (t) and whether to access the drone auxiliary network a m,n (t), then the observation value of the ground agent m is expressed as: Action: The drones and ground nodes are modeled as heterogeneous agents with different collaborative action spaces to optimize the overall throughput of the system; at time slot t, the action of the drone agent n is expressed as: In the formula represents the flight direction of the drone agent n in time slot t; for the ground node agent m, the tasks include avoiding spectrum conflicts between other ground nodes and defending against malicious interference attacks; In addition, the actions of the ground node agent are described as: That and represents the channel selection and transmit power set by the ground node agent m; assuming that all continuous spaces are discretized; Reward: The goal of the UAV-assisted communication system based on multi-agent reinforcement learning is to optimize the system throughput during the entire service period. Therefore, the value of the reward is positively correlated with the objective function. In addition, the ground node access constraints and collision constraints are considered, and the agents that violate the constraints are punished. Therefore, the rewards of the UAV agent and the ground node agent are designed separately to optimize the total system throughput. The reward is expressed as: Among them, a n,m (t) is an indicator function. At time slot t, if the ground node agent n accesses the network of the drone agent m, a n,m (t)=1, otherwise it is a n,m (t)=0,R col (t) represents the collision penalty and R access (t) means additional reward for user visit.
6. The method according to claim 1, characterized in that In the proximal policy optimization resource method based on heterogeneous multi-agent for drone-assisted communication, the input is the actor network and critic network, the experience pool, and the number of training steps; The output is the trained UAV actor network and ground user actor network; The method comprises the following steps: Step 31, initialize the policy network π θ And the value of network parameters V φ , initialize the experience pool D, set the number of cycles, step size and obtain the initial state of the environment s(1); Step 32, each drone agent and ground node agent according to π θ Execute an action A(t) and obtain a joint reward R(t). At the same time, the environment is transferred to the state s(t+1) at the next moment. A(t) represents the action set of the drone agent and the ground user agent. R(t) represents the reward set of the drone agent and the ground user agent. Step 33, store the experience {s(t), A(t), R(t), s(t+1)} into the experience pool; Step 34, calculate the Q function and calculate the advantage function; Step 35, randomly extract the minimum batch of experience B for training and update π θ and V φ The hidden layer parameters in ; Step 36, adaptively updating θ according to L(θ) and updating φ according to L(φ); Steps 32 to 36 are repeated until the maximum number of iterations is reached and the algorithm ends.
7. The method according to claim 6, characterized in that The multi-agent reinforcement learning framework includes multiple agents interacting with a multi-UAV assisted environment, where each UAV and ground node agent corresponds to an agent that executes the PPO algorithm; Each agent runs through two distinct phases: centralized training and decentralized execution; In the training phase, the critic plays the role of a central controller and N+M ∣φ) and observations to calculate the action value in the set, evaluate the actor’s behavior from a global perspective, and guide the agent to improve its decision-making; the critic network updates the parameter θ by minimizing the loss function c : Where φ is the parameter of the evaluation network, s' and φ' are the updated state of the target network and the updated parameter of the evaluation network, respectively. HAPPO is the expected reward value, loss(θ c ) is the loss function, E is the expected value; the actor network updates the parameter θ and takes actions based on the centralized Q function and its own observations; In the execution step, only the trained actor drone network and the actor ground node network make decisions through local observations; The training process is divided into two different phases: experience accumulation phase and strategy improvement phase; In the experience accumulation phase, each UAV agent and ground node agent executes the collectively adopted strategy to select actions, thereby accumulating experience data until a predefined time threshold is reached, denoted as T, and then the corresponding action-state mapping and advantage function are calculated respectively; In the strategy improvement phase, the strategy network π θ The loss function L(θ) is used to perform iterative optimization for K epochs, and the state value of the same small batch data extracted from the experience memory buffer is optimized K times by the loss function L(φ); the constituent data samples in the experience pool D will be randomly reindexed.
Citation Information
Cited By
Multi-agent robot cooperative control method and system based on distributed reinforcement learning
CN119115968A
Method, system and device for optimizing cooperative communication of multiple unmanned aerial vehicles
CN119496553A
Optimization method, system and device for cooperative communication of multiple unmanned aerial vehicles
CN119496553B
Cognitive unmanned aerial vehicle network spectrum sharing method based on multi-agent reinforcement learning
CN119697646A
Unmanned aerial vehicle group intelligent resource scheduling and interference coordination optimization method and device
CN120018293A
