Unmanned aerial vehicle safety communication optimization method and system based on graph attention reinforcement learning

Abstract

The invention belongs to the technical field of unmanned aerial vehicle communication and artificial intelligence, and particularly relates to an unmanned aerial vehicle safety communication optimization method and system based on graph attention reinforcement learning, and the method comprises the steps: obtaining the parameter information of a system model and an intelligent agent, and constructing a dynamic heterogeneous graph state based on the parameter information; performing action decision on the dynamic heterogeneous graph state based on a graph attention network to obtain an execution action; after the execution action is implemented, resource allocation is carried out based on a security hierarchical bandwidth allocation algorithm or a self-adaptive hybrid power allocation algorithm; and calculating a composite reward value according to the system performance after the resource allocation, and guiding the adaptive learning of the intelligent agent based on the composite reward value until the strategy of the intelligent agent converges or reaches a preset training round number. Compared with a method based on a traditional convolutional neural network (CNN), the method has the advantage that the total reward, the system throughput and the average secrecy rate are remarkably improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of unmanned aerial vehicle communication and artificial intelligence, and particularly relates to an unmanned aerial vehicle safe communication optimization method and system based on graph attention reinforcement learning. BACKGROUND

[0002] Unmanned aerial vehicle (UAV) communication, as a key technology in the fifth generation (5G) and future sixth generation (6G) wireless networks, has shown great potential in expanding network coverage and emergency communication recovery due to its high mobility, flexible deployment, and ability to establish line-of-sight (LoS) links. In order to fully utilize the effectiveness of unmanned aerial vehicles, their flight trajectories must be accurately planned, as the trajectories directly determine the wireless channel environment, user fairness, and system total throughput.

[0003] Existing technologies mainly use analytical modeling, convex optimization, and heuristic algorithms to optimize unmanned aerial vehicle trajectories. However, these methods usually have high computational complexity and are difficult to adapt to large-scale and dynamically changing communication environments. In addition, the broadcast nature of unmanned aerial vehicle air-to-ground links makes their communication content vulnerable to illegal eavesdropping, posing a serious threat to data confidentiality. To address this challenge, physical layer security (PLS) technology has emerged, which utilizes the physical properties of wireless channels to ensure information security. Although existing research has explored trajectory and resource optimization strategies with PLS awareness, many works separate trajectory design and resource allocation or rely on overly simplified channel models, limiting their application effectiveness in complex real-world scenarios.

[0004] In recent years, deep reinforcement learning (DRL) has provided a new approach to solving unmanned aerial vehicle trajectory optimization problems in uncertain environments. However, most DRL schemes use flattened vectors or rasterized matrices to represent environmental states, which makes it difficult to effectively capture the complex topological relationships and dynamic interactions between unmanned aerial vehicles, users, and eavesdroppers. Graph neural networks (GNNs) offer a powerful tool for modeling such complex relationships due to their natural advantages in handling graph-structured data, but their application in unmanned aerial vehicle safe communication and intelligent trajectory planning remains insufficient. Therefore, there is an urgent need for a new method that can effectively model network topology, dynamically adapt to environmental changes, and achieve joint optimization of multiple objectives such as communication efficiency, fairness, and security. SUMMARY

[0005] The present application aims to provide an unmanned aerial vehicle safe communication optimization method and system based on graph attention reinforcement learning to address the technical problems of high computational complexity, poor environmental adaptability, difficulty in capturing complex relationships between network entities, and inability to jointly optimize communication efficiency, fairness, and security in existing technologies.

[0006] In order to achieve the above object, the present application adopts the following technical solutions: In the first aspect of the present application, a method for optimizing safe communication of unmanned aerial vehicles based on graph attention reinforcement learning is provided, comprising: Obtaining parameter information of a system model and an agent, and constructing a dynamic heterogeneous graph state based on the parameter information; The dynamic heterogeneous graph state is used for action decision based on a graph attention network, and an execution action is obtained; After implementing the execution action, resource allocation is performed based on a safe hierarchical bandwidth allocation algorithm or an adaptive hybrid power allocation algorithm; According to the system performance after resource allocation, a composite reward value is calculated, and the adaptive learning of the agent is guided based on the composite reward value until the agent policy converges or the preset training number of rounds is reached.

[0007] Preferably, the dynamic heterogeneous graph state is constructed based on the parameter information, that is, the instantaneous state of the system is mapped into a dynamic heterogeneous graph state at each decision time step t.

[0008] Preferably, the dynamic heterogeneous graph state is used for action decision based on a graph attention network, and an execution action is obtained, specifically comprising: The dynamic heterogeneous graph state is input into an online Q network of the agent, and key entity features are extracted through a multi-layer graph attention network; The key entity features are aggregated into a vector representing the state of the whole graph through global pooling, and the vector is output into a Q value vector through a fully connected network, the Q value vector corresponding to the value evaluation of a plurality of discrete actions; The agent selects an execution action from the set of discrete actions according to the Q value vector using -greedy strategy.

[0009] Preferably, the resource allocation is performed based on the safe hierarchical bandwidth allocation algorithm, specifically comprising: After the unmanned aerial vehicle implements the execution action, the parameter information of the system model and the agent changes, and the composite utility value of each user equipment in the system model is calculated; After allocating the minimum bandwidth to each user equipment, the remaining bandwidth is determined in proportion to the incremental bandwidth of each user equipment based on the composite utility value, and it is judged whether the sum of the minimum bandwidth and the incremental bandwidth of each user equipment exceeds a preset value, if not, the process is ended, if yes, the re-allocation is performed.

[0010] Preferably, the re-allocation specifically comprises: The bandwidth exceeding the preset value is recovered, and is reallocated to the user equipment not reaching the upper limit according to the weighting factor defined by the user priority and the bandwidth gap.

[0011] Preferably, the resource allocation is based on the adaptive hybrid power allocation algorithm, and specifically includes: After the UAV implements the execution action, the parameter information of the system model and the agent changes, and the safety-aware effective channel gain of each user equipment in the system model is calculated; After each user equipment is allocated a minimum guarantee power, the remaining total power is allocated as additional power based on the safety-aware effective channel gain.

[0012] Preferably, the adaptive learning of the agent is guided based on the composite reward value, and specifically includes: According to the target Q value calculation method and the mean square error loss function of the double DQN, the parameters of the online Q network of the agent are updated by gradient descent based on the composite reward value ; The parameters of the online Q network are copied to the target Q network of the agent in a soft update manner.

[0013] In the second aspect of the present application, a UAV safety communication system based on graph attention reinforcement learning is provided, comprising: A construction unit is configured to obtain parameter information of a system model and an agent, and construct a dynamic heterogeneous graph state based on the parameter information. An action decision unit is configured to make an action decision based on a graph attention network using the dynamic heterogeneous graph state to obtain an execution action. A resource allocation unit is configured to implement the execution action, and perform resource allocation based on a safety layered bandwidth allocation algorithm or an adaptive hybrid power allocation algorithm. An adaptive learning unit is configured to calculate a composite reward value according to the system performance after resource allocation, guide the adaptive learning of the agent based on the composite reward value, and stop until the agent policy converges or a preset training number of rounds is reached.

[0014] Preferably, in the resource allocation unit, the resource allocation is based on the safety layered bandwidth allocation algorithm, and specifically includes: After the UAV implements the execution action, the parameter information of the system model and the agent changes, and the composite utility value of each user equipment in the system model is calculated. After each user equipment is allocated a minimum bandwidth, the remaining bandwidth is proportionally determined as the incremental bandwidth of each user equipment based on the composite utility value, and it is determined whether the sum of the minimum bandwidth and the incremental bandwidth of each user equipment exceeds a preset value, if not, the process is ended, and if yes, the process is re-allocated.

[0015] Preferably, in the resource allocation unit, the resource allocation is based on the adaptive hybrid power allocation algorithm, and specifically includes: After the UAV implements the action, the system model and the parameter information of the agent change, and the security-aware effective channel gain of each user equipment in the system model is calculated; After each user equipment is allocated a minimum guarantee power, the remaining total power is allocated as additional power based on the security-aware effective channel gain.

[0016] In a third aspect, the present application provides an electronic device, comprising a processor and a memory, wherein the processor is configured to execute a computer program stored in the memory to implement the method for optimizing the security communication of the UAV based on the graph attention reinforcement learning according to any one of the preceding aspects.

[0017] In a fourth aspect, the present application provides a computer readable storage medium, wherein the computer readable storage medium stores at least one instruction, and the at least one instruction is executed by a processor to implement the method for optimizing the security communication of the UAV based on the graph attention reinforcement learning according to any one of the preceding aspects.

[0018] Compared with the prior art, the present application has the following beneficial effects: 1) The intelligence and globality of decision-making are improved: the present application innovatively models the complex communication scenario as a dynamic heterogeneous graph, and processes it by using a graph attention network (GAT), which can deeply capture the time-varying topological relationship and deep dependence among the UAV, the user and the eavesdropper. Compared with the traditional method, the agent can reason from a global perspective and formulate a more far-sighted long-term strategy, which significantly improves the overall performance.

[0019] 2) The collaborative optimization of communication efficiency and fairness is realized: the security hierarchical bandwidth allocation (SHBA) algorithm proposed in the present application can prioritize high-quality users while compensating for users with poor channel conditions through a composite utility function containing service priority, channel gain and inverse channel gain terms, effectively avoiding the "user starvation" problem and improving the overall fairness and sustainability of the system.

[0020] 3) The active defense capability of physical layer security is enhanced: the adaptive hybrid power allocation (AHPA) algorithm proposed in the present application endogenizes the influence of eavesdropping channels as a communication cost through the unique "security-aware effective channel gain" (SAE-CG) index. When allocating power, the algorithm actively reduces the power tilt on high eavesdropping risk links, forming a "power hole" in space and actively and intelligently avoiding eavesdropping threats from the source of resource allocation.

[0021] 4) Online and adaptive decision making: Based on the deep reinforcement learning framework, this method does not require an accurate environment model, and can learn and make decisions through real-time interaction with the environment. This makes it well adapted to dynamic changes in wireless channels and user demand, with strong robustness and scalability, significantly better than traditional optimization algorithms with high computational complexity and poor adaptability. BRIEF DESCRIPTION OF DRAWINGS

[0022] The accompanying drawings, which form a part of this specification, are included to provide a further understanding of the application, and are incorporated in and constitute a part of this specification. The embodiments of the application, together with their Figure 1 Method flowchart of the embodiment of the application; Figure 2 Unmanned aerial vehicle safety communication system model schematic diagram applied by the embodiment of the application; Figure 3 Overall architecture schematic diagram of the attention Q network of the embodiment of the application; Figure 4 Flowchart of the safety hierarchical bandwidth allocation (SHBA) algorithm of the embodiment of the application; Figure 5 Flowchart of the adaptive hybrid power allocation (AHPA) algorithm of the embodiment of the application; Figure 6 System structure block diagram of the embodiment of the application; Figure 7 Structure block diagram of an electronic device according to an embodiment of the application. DETAILED DESCRIPTION

[0023] The application will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.

[0024] The following detailed description is exemplary and is intended to provide further details of the application. Unless otherwise defined, all technical terms used in the present application have the same meanings as generally understood by those skilled in the art to which the present application belongs. The terms used in the present application are only for the purpose of describing the specific embodiments of the present application, and are not intended to limit the exemplary embodiments according to the present application.

[0025] Specifically, a method for jointly optimizing three-dimensional flight trajectory, transmission power and bandwidth allocation of unmanned aerial vehicles to enhance physical layer security. The purpose of the present application is to provide a method for optimizing the safety of unmanned aerial vehicle communication based on graph attention reinforcement learning, which aims to solve the technical problems of high computational complexity, poor environmental adaptability, difficulty in capturing complex relationships between network entities, and inability to optimize communication efficiency, fairness and security.

[0026] Referring to Figure 1 The application discloses a method for optimizing safe communication of a UAV based on a graph attention reinforcement learning, comprising: Obtaining parameter information of a system model and an agent, and constructing a dynamic heterogeneous graph state based on the parameter information; The dynamic heterogeneous graph state is used for action decision based on a graph attention network, and an execution action is obtained; After the execution action is implemented, resource allocation is performed based on a safe hierarchical bandwidth allocation algorithm or an adaptive hybrid power allocation algorithm; According to the system performance after the resource allocation, a composite reward value is calculated, and the composite reward value is used to guide adaptive learning of the agent until the agent policy converges or a preset training round number is reached.

[0027] In some embodiments, the dynamic heterogeneous graph state is constructed based on the parameter information, that is, at each decision time step t, the instantaneous state of the system is mapped into a dynamic heterogeneous graph state.

[0028] In some embodiments, the dynamic heterogeneous graph state is used for action decision based on a graph attention network, and an execution action is obtained, specifically comprising: The dynamic heterogeneous graph state is input into an online Q network of the agent, and key entity features are extracted through a multi-layer graph attention network; The key entity features are aggregated into a vector representing the state of the whole graph through global pooling, and the vector is output into a Q value vector through a fully connected network, the Q value vector corresponding to the value evaluation of a plurality of discrete actions; The agent selects an execution action from the discrete action set according to the Q value vector using a greedy strategy.

[0029] In some embodiments, the resource allocation is performed based on the safe hierarchical bandwidth allocation algorithm, specifically comprising: After the UAV implements the execution action, the parameter information of the system model and the agent changes, and a composite utility value of each user equipment in the system model is calculated; After the minimum bandwidth is allocated to each user equipment, the remaining bandwidth is used to determine the incremental bandwidth of each user equipment in proportion to the composite utility value, and it is judged whether the sum of the minimum bandwidth and the incremental bandwidth of each user equipment exceeds a preset value, if not, the process is ended, and if yes, redistribution is performed.

[0030] Further preferably, the redistribution specifically comprises: The bandwidth exceeding the preset value is recovered, and is reallocated to the user equipment not reaching the upper limit according to a weighting factor defined according to the user priority and the bandwidth gap.

[0031] In some embodiments, the resource allocation based on the adaptive hybrid power allocation algorithm specifically comprises: After the UAV performs the action, the system model and the parameter information of the agent change, and the safety-aware effective channel gain of each user equipment in the system model is calculated; After each user equipment is allocated a minimum guarantee power, the remaining total power is allocated as additional power based on the safety-aware effective channel gain.

[0032] In some embodiments, the adaptive learning of the agent based on the composite reward value specifically comprises: According to the target Q value calculation method of the double DQN and the mean square error loss function, the parameters of the online Q network of the agent are updated by gradient descent based on the composite reward value ; The parameters of the online Q network are copied to the target Q network of the agent in a soft update manner.

[0033] Embodiment 1: The present application provides a kind of based on graph attention reinforcement learning's unmanned plane safety communication optimization method.The method can be deployed in a system by computer (such as unmanned plane on-board processor and / or ground control station) is executed.It is shown in figure as Figure 2 The system includes a UAV as a flying base station, K ground user equipment (UEs), and a potential ground eavesdropper (EVE).The task of the UAV is to move in three-dimensional space while providing downlink communication services for user equipment, and to optimize system throughput, user fairness and communication secrecy rate in the process.

[0034] The complete steps of the present application are as follows: S1, system model and agent initialization First, the physical parameters of the communication system and the hyperparameters of the agent are initialized.

[0035] 1) Communication environment parameters: set a three-dimensional Cartesian coordinate system.In a pre-set geographical area (for example 100m × 100m, flight height 20m-100m), randomly deploy K UEs (for example K=20), and the two-dimensional coordinates of the UEs remain fixed.The initial position of the UAV is set in the center of the area.

[0036] 2) Channel model parameters: the channel gain between the UAV and the ground node (user k or eavesdropper E) is calculated using the free space path loss model, whose formula is where and Antenna gains of UAV and ground users (e.g., 30 dB and 3 dB, respectively), Signal wavelength (e.g., 0.05 m), Real-time distance.

[0037] 3) Wireless resource parameters: Set the total system bandwidth (e.g., 5 MHz) and divide it into C orthogonal subcarriers. Set the maximum total transmit power of UAV (e.g., 3 W). Set the noise power spectral density to -174 dBm / Hz.

[0038] 4) Agent initialization: Construct a dual deep Q network (DDQN) agent based on graph attention network (GAT). The agent contains an online Q network and a target Q network, and the internal structure is as shown in Figure 2 The network input is graph structure data, and the output is the Q value of discrete action. At the same time, initialize an experience replay memory M, and hyperparameters such as learning rate (e.g., 0.001), discount factor (e.g., 0.99), exploration rate (initially 1.0), etc.

[0039] S2, construct a dynamic heterogeneous graph state At each decision time step t, the instantaneous state of the system is mapped into a dynamic heterogeneous graph , which serves as the input of the agent.

[0040] 1) Node set : The node set contains 1 UAV node, K UE nodes, and 1 EVE node.

[0041] 2) Node features: Each node is associated with a feature vector.

[0042] UAV node features: Contain its three-dimensional coordinates , signal-to-noise ratio (SNR), etc. global information.

[0043] UE node k features: Contain its coordinates , service priority , channel gain , and allocated bandwidth .

[0044] EVE node features: Contain its coordinates , eavesdropping channel SNR, and distance from UAV .

[0045] 3) Edge set : The edge represents the physical relationship between nodes, and its attribute is the Euclidean distance between nodes.

[0046] S3, action decision based on graph neural network The graph state constructed in S2 is input to an online Q network of the agent (e.g. as shown in FIG. 3). Figure 3

[0047] 1) Feature extraction: The graph data is first processed by multiple GAT layers (GATConv1, GATConv2). The self-attention mechanism of GAT can dynamically assign different attention weights to different neighbor nodes for the agent, so as to focus on the most critical entities for the current decision.

[0048] 2) Global pooling and decision: After the GAT layer processing, the features of all nodes are aggregated into a vector representing the global graph state by a global pooling layer (Pooling). The vector is then processed by a fully connected network (Output) to finally output a Q value vector, which corresponds to the value evaluation of multiple discrete actions (up, down, left, right, front, back, hover).

[0049] 3) Action selection: The agent selects an action from the discrete action set according to the current exploration rate , using an -greedy strategy .

[0050] S4, hierarchical security resource allocation After the UAV performs an action , its position is updated, causing the channel environment to change. At this time, the system triggers the following two resource allocation algorithms: S401, security hierarchical bandwidth allocation (SHBA), the process of which is shown in FIG. 4: Figure 4 1) Calculate the composite utility: Calculate a composite utility value for each user k. Wherein is the service priority, is the legitimate channel gain, is the eavesdropping channel gain (as a security penalty term), is the inverse channel gain (as a fairness compensation term). The weight coefficients are adjustable (e.g. 0.6, 0.4, 0.7, 1.0).

[0051] 2) Baseline and proportional allocation: First, allocate a minimum guaranteed bandwidth (e.g. 0.12MHz) to each user. Then allocate the remaining bandwidth according to the composite utility value of each user to obtain the incremental bandwidth .​​

[0052] 3) Upper bound and redistribution: Check the allocation result to ensure that the bandwidth obtained by a single user does not exceed a certain proportion of the total bandwidth (e.g. 20%). The part exceeding the upper bound is recycled to the resource pool and redistributed to users who have not reached the upper bound according to the user priority and the weighting factor defined by the bandwidth gap .

[0053] S402, adaptive hybrid power allocation (AHPA), the process is shown in Figure 5 : 1) Calculate the security-aware effective channel gain: calculate the security-aware effective channel gain (SAE-CG) for each user k, . Where, is the noise power, is the security penalty factor (e.g. 0.1). This indicator internalizes the eavesdropping risk as part of the channel quality.

[0054] 2) Two-stage power allocation: first, allocate a minimum guarantee power (e.g. 0.15W) to all users to ensure basic communication fairness. Then, allocate the remaining total power using a security-aware iterative water-filling algorithm. This water-filling algorithm is not based on the traditional channel gain , but on the SAE-CG value , and the additional power allocated is , where is the optimal water level, first determine the set composed of the first j user equipment (UEs) with the maximum value , where is the value of the remaining power divided by j, and the sum of the first j , then increase the value of j by iteration until the condition that for all user equipment i in the set , there is , and is met, at this time the obtained is the optimal water level . This makes more power be intelligently allocated to users with good and safe channels.

[0055] S5, calculate the multi-objective composite reward According to the system performance after resource allocation in S4, a composite reward value is calculated to guide the learning of the agent. The reward function is designed as a weighted sum of multiple objectives: ​

[0056] wherein: are the weights of each performance indicator (e.g. 15, 5, 4).

[0057] is the total throughput of the system.

[0058] is the total secrecy rate of the system.

[0059] is the minimum throughput among users, to ensure fairness.

[0060] is the penalty term when the UAV flies out of the preset boundary.

[0061] S6, model training based on the DRL framework 1) Store experience: a complete experience tuple (wherein is the terminal state flag) is stored in the experience replay memory M.

[0062] 2) Sampling and training: when the number of experiences in the memory exceeds a batch size, training begins. From M, a small batch of experience data is obtained according to the priority.

[0063] 3) Network update: using the sampled batch data, according to the double DQN (DDQN) target Q value calculation method ( ) and the mean square error loss function, the parameters of the online Q network are updated by gradient descent. Finally, through soft update, the parameters of the online Q network are slowly copied to the target Q network to ensure the stability of the training.

[0064] 4) Repeat S2 to S6 until the agent policy converges or the preset number of training rounds (e.g. 400 rounds) is reached.

[0065] Example 2 The method of the application is verified in a simulation scenario including 20 users and 1 eavesdropper. At a certain decision-making moment, the UAV is located at (50, 50, 50). The S2 step constructs the information of the 22 entities into a 22-node graph. In the S3 step, the GAT network processes the graph, outputs a 7-dimensional Q value vector, and the agent selects the optimal action, for example, “fly forward”. After the UAV moves to a new position, the S4 step is triggered. According to the channel conditions and security risks of all users, the SHBA algorithm calculates the composite utility of each user, and differentiates the allocation of a total of 333 subcarriers. For example, users with good channel conditions and security may be allocated 25 subcarriers, while users with poor channel conditions or high risk may be allocated only 8 security subcarriers. Then, the AHPA algorithm calculates the SAE-CG values of all users and allocates a total power of 3W. For example, a user located in the direction of the eavesdropper and with a low SAE-CG value (such as User 20 in the paper Figure 5 ) is allocated only 0.09W power, while a user in a safe area and with a high SAE-CG value may be allocated 0.17W power. The S5 step calculates the total reward according to this resource allocation. The S6 step stores this complete interaction process in the experience pool for training. Experimental results show that the method of the application has achieved significant performance improvement in total reward, system throughput, and average secrecy rate compared to the method based on the traditional convolutional neural network (CNN).

[0066] Embodiment 3 As shown in Figure 6 , based on the same inventive concept as the above embodiments, the application further provides a UAV secure communication system based on graph attention reinforcement learning, comprising: A construction unit is configured to obtain parameter information of a system model and an agent, and construct a dynamic heterogeneous graph state based on the parameter information; An action decision unit is configured to perform action decision based on a graph attention network using the dynamic heterogeneous graph state, to obtain an execution action; A resource allocation unit is configured to perform resource allocation based on a security hierarchical bandwidth allocation algorithm or an adaptive hybrid power allocation algorithm after the execution of the action; An adaptive learning unit is configured to calculate a composite reward value according to the system performance after the resource allocation, and guide the adaptive learning of the agent based on the composite reward value, until the agent policy converges or a preset number of training rounds is reached.

[0067] In some embodiments, in the resource allocation unit, the resource allocation is performed based on the security hierarchical bandwidth allocation algorithm, specifically including: After the UAV executes the action, the parameter information of the system model and the agent changes, and the composite utility value of each user equipment in the system model is calculated; After allocating the minimum bandwidth to each user equipment, the remaining bandwidth is used to determine the incremental bandwidth of each user equipment in proportion to the composite utility value, and it is determined whether the sum of the minimum bandwidth and the incremental bandwidth of each user equipment exceeds a preset value, if not, the process is ended, if yes, the allocation is performed again.

[0068] In some embodiments, the resource allocation unit allocates resources based on an adaptive hybrid power allocation algorithm, specifically including: After the UAV performs the action, the parameter information of the system model and the agent changes, and the safety-aware effective channel gain of each user equipment in the system model is calculated. After allocating a minimum guaranteed power to each user equipment, the remaining total power is allocated as additional power based on the safety-aware effective channel gain.

[0069] Embodiment 4 As shown in Figure 7 The application also provides an electronic device 100 for implementing the method for optimizing the safe communication of the UAV based on the graph attention reinforcement learning. The electronic device 100 includes a memory 101, at least one processor 102, a computer program 103 stored in the memory 101 and executable on the at least one processor 102, and at least one communication bus 104.

[0070] The memory 101 can be used to store the computer program 103, and the processor 102 can realize the steps of the method for optimizing the safe communication of the UAV based on the graph attention reinforcement learning by running or executing the computer program stored in the memory 101 and calling the data stored in the memory 101.

[0071] The memory 101 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required by a function (such as a sound playing function, an image playing function, etc.), etc.; and the data storage area can store data (such as audio data) created according to the use of the electronic device 100, etc. In addition, the memory 101 can include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices.

[0072] The at least one processor 102 can be a central processing unit (CPU), and can also be other general-purpose processors, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, and the like. The processor 102 can be a microprocessor or the processor 102 can also be any conventional processor, and the like. The processor 102 is a control center of the electronic device 100, and is connected to various parts of the electronic device 100 through various interfaces and lines.

[0073] The memory 101 in the electronic device 100 stores a plurality of instructions to implement a method for optimizing safe communication of a UAV based on graph attention reinforcement learning. The processor 102 can execute the plurality of instructions to implement the following. Obtain parameter information of a system model and an agent, and construct a dynamic heterogeneous graph state based on the parameter information. The dynamic heterogeneous graph state makes an action decision based on a graph attention network, and obtains an execution action. After implementing the execution action, resource allocation is performed based on a safe hierarchical bandwidth allocation algorithm or an adaptive hybrid power allocation algorithm. According to the system performance after the resource allocation, a composite reward value is calculated, and the adaptive learning of the agent is guided based on the composite reward value until the agent policy converges or a preset number of training rounds is reached.

[0074] Embodiment 5 If the modules / units integrated in the electronic device 100 are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, the computer program can implement the steps of each method embodiment described above. The computer program includes computer program code, which can be in the form of source code, object code, an executable file, or some intermediate form. The computer-readable medium can include any entity or device capable of carrying computer program code, a recording medium, a U disk, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0075] Those skilled in the art will appreciate that embodiments of the present application can be devised for a method, a system, or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer readable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code thereon.

[0076] The present application is described in reference to the drawings, which are as follows. Figure 1 Figure 1

[0077] Figure 1 Figure 1

[0078] Figure 1 Figure 1

[0079] ​​​​​​​​​

[0080] It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the present application but not to limit it. Although the present application has been described in detail with reference to the above embodiments, it should be understood by those skilled in the art that the specific embodiments of the present application can be modified or equivalently replaced without departing from the spirit and scope of the present application, and any modification or equivalent replacement should be covered within the protection scope of the claims of the present application.

Claims

1. A method for optimizing secure communication of unmanned aerial vehicles (UAVs) based on graph attention reinforcement learning, characterized in that, include: Obtain parameter information of the system model and intelligent agents, and construct a dynamic heterogeneous graph state based on the parameter information; Dynamic heterogeneous graph states make action decisions based on graph attention networks to obtain the actions to be executed. After the execution action is carried out, resources are allocated based on the security layered bandwidth allocation algorithm or the adaptive hybrid power allocation algorithm; Based on the system performance after resource allocation, a composite reward value is calculated. The agent's adaptive learning is guided by the composite reward value until the agent's policy converges or the preset number of training rounds is reached.

2. The method for optimizing secure communication of unmanned aerial vehicles based on graph attention reinforcement learning according to claim 1, characterized in that, The construction of the dynamic heterogeneous graph state based on parameter information is as follows: at each decision time step t, the instantaneous state of the system is mapped to a dynamic heterogeneous graph state.

3. The method for optimizing secure communication of unmanned aerial vehicles based on graph attention reinforcement learning according to claim 1, characterized in that, The dynamic heterogeneous graph state is used to make action decisions based on a graph attention network to obtain the action to be executed, specifically including: The dynamic heterogeneous graph state is input into the agent's online Q-network, and key entity features are extracted through a multi-layer graph attention network. Key entity features are aggregated into a vector representing the state of the entire graph through global pooling. This vector is then passed through a fully connected network to output a Q-value vector, which corresponds to the value assessment of multiple discrete actions. The agent uses the Q-value vector to... The -greedy strategy selects one action from a set of discrete actions to perform.

4. The method for optimizing secure communication of unmanned aerial vehicles based on graph attention reinforcement learning according to claim 1, characterized in that, The resource allocation based on the security-layered bandwidth allocation algorithm specifically includes: After the drone performs its actions, the parameter information of the system model and the intelligent agent changes, and the composite utility value of each user device in the system model is calculated. After allocating minimum bandwidth to each user device, the remaining bandwidth is used to determine the incremental bandwidth for each user device proportionally based on the composite utility value. It is then determined whether the sum of the minimum bandwidth and incremental bandwidth for each user device exceeds a preset value. If it does not exceed the preset value, the process ends; otherwise, it is redistributed.

5. The method for optimizing secure communication of unmanned aerial vehicles based on graph attention reinforcement learning according to claim 4, characterized in that, The redistribution specifically refers to: Bandwidth exceeding the preset value is reclaimed and redistributed to user devices that have not reached the limit, based on user priority and a weighting factor defined by the bandwidth gap.

6. The method for optimizing secure communication of unmanned aerial vehicles based on graph attention reinforcement learning according to claim 1, characterized in that, The resource allocation based on the adaptive hybrid power allocation algorithm specifically includes: After the drone performs its actions, the parameter information of the system model and the agent changes, and the effective channel gain for security perception of each user device in the system model is calculated. After allocating a minimum guaranteed power to each user equipment, the remaining total power is allocated as additional power based on the security-aware effective channel gain.

7. The method for optimizing secure communication of unmanned aerial vehicles based on graph attention reinforcement learning according to claim 1, characterized in that, The adaptive learning of the agent guided by the composite reward value specifically includes: Based on the objective Q-value calculation method and mean squared error loss function of dual DQN, the parameters of the online Q-network of the agent are updated by gradient descent based on the composite reward value. ; The parameters of the online Q network are copied to the agent's target Q network via a soft update.

8. A secure communication system for unmanned aerial vehicles (UAVs) based on graph attention reinforcement learning, characterized in that, include: The building unit is used to acquire parameter information of the system model and the agent, and to construct a dynamic heterogeneous graph state based on the parameter information. The action decision unit is used to make action decisions based on a graph attention network using dynamic heterogeneous graph states to obtain the action to be executed. The resource allocation unit is used to allocate resources based on a security-layered bandwidth allocation algorithm or an adaptive hybrid power allocation algorithm after the execution action is performed. An adaptive learning unit is used to calculate a composite reward value based on the system performance after resource allocation, and guide the agent's adaptive learning based on the composite reward value until the agent's policy converges or reaches a preset number of training rounds.

9. A UAV secure communication system based on graph attention reinforcement learning according to claim 8, characterized in that, The resource allocation unit performs resource allocation based on a security-layered bandwidth allocation algorithm, specifically including: After the drone performs its actions, the parameter information of the system model and the intelligent agent changes, and the composite utility value of each user device in the system model is calculated. After allocating minimum bandwidth to each user device, the remaining bandwidth is used to determine the incremental bandwidth for each user device proportionally based on the composite utility value. It is then determined whether the sum of the minimum bandwidth and incremental bandwidth for each user device exceeds a preset value. If it does not exceed the preset value, the process ends; otherwise, it is redistributed.

10. A UAV secure communication system based on graph attention reinforcement learning according to claim 8, characterized in that, The resource allocation unit performs resource allocation based on an adaptive hybrid power allocation algorithm, specifically including: After the drone performs its actions, the parameter information of the system model and the agent changes, and the effective channel gain for security perception of each user device in the system model is calculated. After allocating a minimum guaranteed power to each user equipment, the remaining total power is allocated as additional power based on the security-aware effective channel gain.

Images