A joint optimization method for flight self-organizing network-assisted communication

Abstract

The present invention belongs to the field of wireless communication technology, and proposes a joint optimization method for flying self-organizing network-assisted communication. By real-time optimizing the flight trajectory and data routing path of a drone group to form a FANET, the relay communication of ground wireless sensors is assisted to meet the application requirements of WSN for low-latency communication in the field of real-time monitoring. The steps are as follows: (1) constructing a multi-drone and WSN data routing transmission system model; (2) establishing a joint optimization problem of FANET-assisted communication, with minimizing the maximum end-to-end delay of the transmission system as the optimization goal; (3) based on MDP, the joint optimization problem of FANET-assisted communication is expressed as a sequential decision problem; (4) based on a deep reinforcement learning graph neural network, a drone decision model for solving the sequential decision problem is established and trained; (5) applying the trained drone decision model to realize the joint optimization of flying self-organizing network-assisted communication.

Description

Technical Field

[0001] The present invention relates to the field of wireless communication technology, and in particular to a joint optimization method for flight self-organizing network-assisted communication. Background Art

[0002] With the rapid development of digitalization and intelligence, wireless sensor networks (WSNs) have become essential in fields such as intelligent transportation, environmental monitoring, and industrial automation. These networks need to process and transmit data in real time to ensure the efficiency and responsiveness of the system. One of the key performance indicators is end-to-end latency, which is the total transmission time of data from sender to receiver. In WSN, end-to-end latency is affected by multiple factors, including propagation delay, node processing delay, and network congestion caused by transmission rate and power limitations. Therefore, optimizing these factors is crucial to reducing the maximum end-to-end latency of terrestrial networks and improving overall performance.

[0003] Drones are considered an important supplement to ground wireless communication networks due to their high mobility and flexible deployment capabilities. In scenarios with high traffic demand and heavy network load, drones can be used as aerial communication platforms to provide relay communication services for ground equipment and improve the coverage and service quality of ground networks. The flying self-organizing communication network (FANET) composed of drone swarms can provide auxiliary communication services for ground networks with rapid deployment, dynamic adjustment and load balancing. This method effectively reduces the maximum end-to-end delay of the ground network. However, due to the high mobility of drones, the topology of FANET changes frequently, resulting in a complex and changeable network environment. The dynamically changing environment makes it difficult to select the data packet transmission path and plan the trajectory of drones. Summary of the invention

[0004] In order to cope with the challenge of FANET-assisted communication in dynamic environments to reduce the maximum end-to-end delay of WSN, this paper proposes a joint optimization method for flying self-organizing network-assisted communication. This method constructs an auxiliary communication model, adopts the method of real-time optimization of the flight trajectory and data routing path of the drone group, formulates the problem as an MDP sequence decision problem, and solves it using the deep reinforcement learning algorithm combined with GCN and MAPPO to reduce the maximum end-to-end delay of WSN.

[0005] The technical solution of the present invention is as follows:

[0006] A joint optimization method for flight self-organizing network assisted communication, comprising the following steps:

[0007] Step S1: construct a multi-UAV and WSN data routing transmission system model;

[0008] Step S2: Establish a joint optimization problem for FANET-assisted communication, with minimizing the maximum end-to-end delay of the transmission system as the optimization goal;

[0009] Step S3: Based on MDP, the joint optimization problem of FANET-assisted communication is formulated as a sequential decision problem;

[0010] Step S4: Establish a UAV decision model for solving sequential decision problems based on deep reinforcement learning graph neural network and perform training;

[0011] Step S5: Apply the trained UAV decision model to achieve joint optimization of flight self-organizing network-assisted communication.

[0012] Furthermore, the step S1 specifically includes:

[0013] Step S1-1: The UAV and WSN data routing transmission system model includes a FANET composed of N UAVs, a WSN composed of M ground nodes and a ground access point. Each ground node has multiple data packets, and each ground node needs to transmit all data packets to the ground access point; the ground routing method of the WSN is given by the Primm minimum spanning tree algorithm;

[0014] make Represents ground node At time step Data packet transmission path selection:

[0015] (1)

[0016] in, , I represents a drone set consisting of N drones;

[0017] (2)

[0018] Wherein, J represents a ground node set consisting of M ground nodes;

[0019] make Indicates drone At time step Transmitting data packets to the drone , and , then

[0020] (3)

[0021] Among them, drones For drones Neighborhood drones;

[0022] make Indicates drone At time step Transmitting data to a ground access point, we have

[0023] (4)

[0024] (5)

[0025] Command drone Flying at a fixed altitude H, flying or hovering at a constant speed ‖v(t)‖, the position of drone i is It is expressed as:

[0026] (6)

[0027] Among them, Δt represents the time step;

[0028] (7)

[0029] in, Indicates the minimum distance at which the drone does not collide; Indicates drone At time step A collection of neighbor drones, (8);

[0030] in, and Respectively represent drones and drones The effective communication area radius;

[0031] Step S1-2: Assume that all communication channels are orthogonal channels without inter-channel interference. Then the channel model when ground node j in WSN transmits a data packet to ground node j' at time step t is It is expressed as:

[0032] (9)

[0033] in, Indicates the reference distance The path loss when Represents ground node To ground node The straight-line distance, represents the path loss parameter of non-line-of-sight transmission, represents the small-scale fading component of the cyclically symmetric complex Gaussian distribution with unit variance at time step t;

[0034] The transmission rate of data packets transmitted from ground node j to ground node j' at time step t It is expressed as:

[0035] (10)

[0036] in, is the subchannel bandwidth, is the transmission power of ground node j, is the unilateral power spectral density of Gaussian white noise;

[0037] Channel model between ground node j and UAV i at time step t It is expressed as:

[0038] (11)

[0039] in, represents the ground node at time step t The straight-line distance to drone i, Represents the path loss parameter for line-of-sight transmission;

[0040] Channel model between UAV i and ground access point at time step t It is expressed as:

[0041] (12)

[0042] in, represents the straight-line distance from UAV i to the ground access point at time step t;

[0043] Step t ground node The transmission rate of data packets transmitted to UAV i It is expressed as:

[0044] (13)

[0045] The transmission rate of the data packet transmitted by the UAV to the ground access point at time step t It is expressed as:

[0046] (14)

[0047] in, Indicates the channel bandwidth used by the drone to transmit data packets. Constant power for transmitting packets to drones, shaping antenna gain for directional beamforming;

[0048] Time step t UAV and drones Channel Model for:

[0049] (15)

[0050] Time step t UAV To the drone The transmission rate of the transmitted data packets is:

[0051] (16)

[0052] Furthermore, the step S2 specifically includes:

[0053] make Represents ground node At time step The number of packets, Indicates drone At time step The number of data packets, then the joint optimization problem is expressed as:

[0054] (17)

[0055] (18)

[0056] in, is the time step when all packets arrive at the ground access point.

[0057] Furthermore, the step S3 specifically includes: establishing an MDP model of a multi-UAV and WSN data routing transmission system model, thereby achieving the joint optimization problem of FANET-assisted communication as a sequential decision problem;

[0058] The MDP model includes local observation states ,Behavior , state transition probability matrix P, reward function R and discount factor γ;

[0059] Among them, the local observation state Includes: The position of drone i at time step t , the number of data packets from ground nodes within the communication range of UAV i , the location of other drones within the communication range of drone i And the number of data packets of other drones within the communication range of drone i ;

[0060] ;

[0061] Reward Function :

[0062] (19)

[0063] in, Represents the reward when all packets reach the ground access point , where s=1,2,3,4, Indicates the node access reward, Indicates no data node access penalty, represents the auxiliary data transmission reward, Indicates mobile rewards, represents the time step reward.

[0064] Furthermore, the step S4 specifically includes:

[0065] Step S4-1: Use an undirected graph G to represent the routing network between drones. , V represents the vertex set in the undirected graph G, i.e., the drone set, represents the edge set, that is, the set of communication links between two drones, and the adjacency matrix ;

[0066] Step S4-2: Establish a drone decision model, and its deep reinforcement learning algorithm is as follows:

[0067] (1) Input drone features at time step t , represents the drone features of drone i, including the location of drone i, the number of data packets of drone i, and the number of data packets of other drones within the communication range of drone i. N represents the number of drones in the drone set N. is the dimension of the drone’s characteristics;

[0068] (2) Computational drones and drones The similarity coefficient of

[0069] (20)

[0070] is a learnable parameter, Indicates the splicing of the transformed drone features. Indicates mapping the concatenated high-dimensional drone features to real numbers;

[0071] (3) Calculate the attention coefficient, that is

[0072] (twenty one)

[0073] Among them, LeakyReLu() represents the activation function;

[0074] (4) According to the attention coefficient, the weighted sum of the adjacent drone features is performed to obtain the fused drone feature of drone i that integrates the adjacent drone features:

[0075] (twenty two)

[0076] in, represents the activation function;

[0077] (5) Introduce a multi-tap attention mechanism and establish multiple attention mechanisms for the same data sample:

[0078] (twenty three)

[0079] Where k represents the A tap, represents the fused UAV features of UAV i obtained by the k-th tap, Represents multi-head attention characteristics;

[0080] Step S4-3: Train the UAV decision model using the local observation state and adjacency matrix as input.

[0081] Furthermore, the step S5 specifically includes:

[0082] Step S5-1: Given the initial state S0 of the environment;

[0083] Step S5-2: Each UAV inputs the current local observation state into the UAV decision model to obtain the corresponding behavior;

[0084] Step S5-3: Apply the selected behavior to the environment, obtain rewards from environmental feedback, and update the environment to a new state;

[0085] Step S5-4: Determine whether the environment has reached the termination state, that is, whether all data packets have been transmitted to the ground access point. If it has not reached the termination state, return to step S5-2; if it has reached the termination state, end the application process.

[0086] Compared with the prior art, the present invention has the following beneficial effects:

[0087] 1. This paper constructs a multi-UAV and WSN data routing transmission system model, models the multi-UAV assisted WSN communication scenario as an undirected graph, better captures the characteristics of the self-organizing network topology, provides support for the dynamic changes of the network topology, and enables the system to maintain stable communication in various complex environments.

[0088] 2. This paper establishes a joint optimization problem for FANET-assisted communication, using the high mobility of drones and their ability to establish line-of-sight links to improve the transmission performance of self-organizing networks. By comprehensively considering multiple optimization objectives such as flight trajectory and data routing, the network transmission delay is effectively reduced, providing end-to-end delay guarantee for time-sensitive data.

[0089] 3. This invention implements the sequential decision problem formulation based on MDP, provides a powerful intelligent agent decision framework, and optimizes the decision process of UAVs in dynamic environments. By defining detailed communication environment states, flight actions, and reward functions, it promotes UAVs to make optimal global decisions based on the current local environment state, and clarifies the path and goal of solving the optimization problem.

[0090] 4. The present invention adopts an improved deep reinforcement learning algorithm to solve joint optimization problems and improve the performance of the algorithm in complex joint optimization problems. GCN captures the topological relationship between drones to extract dynamic features, and MAPPO achieves fast online decision-making in a multi-drone environment, which helps to improve the learning efficiency and decision-making quality of intelligent agents, realize the collaborative work of drones in a highly dynamic environment, and reduce end-to-end latency to improve the overall operation efficiency of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. The features and advantages of the present invention can be more clearly understood by referring to the drawings. The drawings are schematic and should not be understood as limiting the present invention in any way. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0092] Figure 1 It is a flow chart of the joint optimization method of flight self-organizing network assisted communication of the present invention.

[0093] Figure 2 It is a schematic diagram of a multi-UAV and WSN data routing transmission system model constructed by the present invention.

[0094] Figure 3 This is a feature extraction flow chart of the graph neural network used in the present invention.

[0095] Figure 4 This is a graph of the MAPPO algorithm based on graph neural networks. DETAILED DESCRIPTION

[0096] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0097] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0098] The present invention proposes a joint optimization method for flight self-organizing network assisted communication. Figure 1 As shown in the figure, this method constructs a multi-UAV and WSN data routing transmission system model, adopts the real-time optimization method of UAV swarm flight trajectory and data routing path, formulates the problem as an MDP sequence decision problem, and uses the deep reinforcement learning algorithm combining GCN and MAPPO to solve it, thereby reducing the maximum end-to-end delay of WSN. This is achieved through the following steps: first, a multi-UAV and WSN data routing transmission system model is constructed, and the UAV swarm is used as an aerial relay device to receive data packets from ground nodes. The UAV swarm and the ground access point jointly establish a line-of-sight communication link; second, a joint optimization problem of FANET-assisted communication is established, considering constraints such as neighbor node distance, number of ground node access, and UAV routing path, and expressing the objective function to be optimized; then, based on MDP, the sequential decision problem is expressed, the local observation state includes the UAV position and the number of data packets, the behavior includes the UAV path selection and data transmission decision, and the reward function includes the duration penalty and the data packet completion transmission reward; finally, a UAV decision model is established, and an improved deep reinforcement learning algorithm is used to solve the joint optimization problem. The local observation state of the UAV is aggregated into a graph message embedding through GCN, the dynamic relationship between the UAV and the ground node is captured, and the MAPPO algorithm is applied for policy optimization to select the optimal UAV flight path and routing strategy.

[0099] Example 1

[0100] A joint optimization method for flight self-organizing network assisted communication, comprising the following steps:

[0101] 1. Constructing a multi-UAV and WSN data routing transmission system model

[0102] 1. The present invention considers that in a rectangular area of ​​L×W, there is a FANET composed of N drones, a WSN composed of M ground nodes, and a ground access point. Each ground node has multiple data packets, and each ground node needs to transmit all data packets to the ground access point (such as Figure 2 As shown in FIG. 1 ). Assume that the ground routing method is predetermined by using the Primm minimum spanning tree algorithm, that is, a minimum spanning tree is constructed for randomly generated ground nodes and ground access points, and the ground routing method is obtained from the minimum spanning tree.

[0103] First, the time is discretized into time steps of Δt, and the data packets at the ground node are transmitted sequentially in each time step. The ground node can choose to transmit the data packet to the ground access point via other ground nodes through WSN routing, or it can choose to transmit it to the ground access point through FANET routing. If the ground node changes the transmission object of the data packet before the transmission of the data packet is completed, the data packet needs to be retransmitted, and each ground node can only access one communication channel for data transmission. For each data packet, the end-to-end delay is the sum of the delays of each intermediate ground node along the transmission path. Let Represents ground node At time step The data packet transmission path selection is:

[0104] (1)

[0105] in, , I represents a drone set consisting of N drones;

[0106] The ground node can only access one UAV at the same time step, i.e.

[0107] (2)

[0108] Wherein, J represents a ground node set consisting of M ground nodes;

[0109] make Indicates drone At time step Transmitting data packets to the drone , and , then

[0110] (3)

[0111] Among them, drones For drones Neighborhood drones;

[0112] make Indicates drone At time step Transmitting data to ground access points, we have

[0113] (4)

[0114] A drone can only access one drone or ground access point at the same time step, i.e.

[0115] (5)

[0116] Assume that the drone takes off from a random initial position in the rectangular area and flies at a fixed height H with a speed Flying at a constant speed or hovering, it has four motion states to adjust its flight trajectory. The position of drone i can be expressed as ,in represents the x-axis coordinate of drone i, Indicates the y-axis coordinate of drone i. The position of drone i satisfy

[0117] (6)

[0118] To avoid collisions between drones,

[0119] (7)

[0120] in It is the minimum distance that the drones can travel without causing collisions. At time step Neighbor Drone Collection It can be expressed as

[0121] (8)

[0122] in, and It's a drone and drones The effective communication area radius.

[0123] 2. Assume that all communication channels are orthogonal channels without inter-channel interference. For ground communication links, the present invention considers the Sub-6GHz frequency band and the free space transmission path loss model under Rayleigh fading. The channel model when ground node j transmits a data packet to ground node j' in time step tWSN is It can be expressed as:

[0124] (9)

[0125] in Indicates the reference distance The path loss when Represents ground node To ground node The straight-line distance, represents the path loss parameter of non-line-of-sight transmission, represents the small-scale fading component of the cyclically symmetric complex Gaussian distribution with unit variance at time step t.

[0126] Considering the presence of Gaussian white noise in the environment, the Shannon formula is used to give the ground node at time step t Given a constant transmission power Down, toward the ground node Channel capacity for transmitting data packets As its transmission rate ,Right now:

[0127] (10)

[0128] in is the sub-channel bandwidth in the Sub-6G band, is the unilateral power spectral density of Gaussian white noise.

[0129] Different from the ground communication channel, the UAV can establish line-of-sight link communication with the ground node and ground access point, which increases the effectiveness of communication. Under the line-of-sight link communication model, the channel gain between the UAV and the ground node and ground access point obeys the free space path loss model, and the path loss depends on the straight-line distance between the two. Then the channel model between the ground node j and the UAV i at time step t is It can be expressed as:

[0130] (11)

[0131] in represents the ground node at time step t The straight-line distance to drone i, Represents the path loss parameter for line-of-sight transmission.

[0132] Similarly, the channel model between UAV i and the ground access point at time step t is It can be expressed as:

[0133] (12)

[0134] in Represents the straight-line distance from UAV i to the ground access point at time step t.

[0135] Also consider the case where there is Gaussian white noise in the environment. At a constant transmission power Next, according to Shannon's formula, the ground node at time step t is given Channel capacity for transmitting data packets to UAV i , that is, as its transmission rate :

[0136] (13)

[0137] At time step t, the transmission rate of the data packet transmitted by the UAV to the ground access point is The channel capacity can also be calculated by Shannon's formula Given, that is:

[0138] (14)

[0139] in Indicates the channel bandwidth used by the drone to transmit data. Constant power for transmitting packets to drones, Antenna gain for directional beamforming.

[0140] The communication between UAVs is mainly dominated by line-of-sight links, so the time step is UAV and drones The channel model between is:

[0141] (15)

[0142] So the time step t drone To the drone The transmission rate of the transmitted data packets is

[0143] (16)

[0144] In fact, due to the channel bandwidth used by drones to transmit data Much greater than , and the UAV has a large antenna gain when transmitting data packets to the ground access point, so the speed at which the UAV transmits data packets to other UAVs or ground access points is very high.

[0145] 2. Joint Optimization Problem of Establishing FANET-Assisted Communications

[0146] The optimization goal of the present invention is to minimize the maximum end-to-end delay of the transmission system. Represents ground node At time step The number of packets, Indicates drone At time step In this model, the end-to-end delay of a data packet refers to the total time from time step t=0 to the time step when the data packet arrives at the ground receiving point, and also includes the waiting time of the data packet when each ground node waits for all the data packets in front of it in the ground node transmission data packet queue, and the maximum end-to-end delay of a data packet is the maximum value of the end-to-end delays of all data packets.

[0147] In the model used in the present invention, although there are multiple data packets arriving at a certain ground node at the same time, this article does not determine the order in which these data packets are transmitted to the next target at the ground node, so it is impossible to obtain the expression of the maximum end-to-end delay by describing the end-to-end delay of each data packet separately and then finding the maximum value. However, in this model, it is assumed that all data packets are transmitted from time step t=0. It can be found that the maximum end-to-end delay is the end-to-end delay of the last data packet that arrives at the ground access point, that is, the time taken for all data packets to arrive at the ground access point. Therefore, the present invention does not need to sort the data packets transmitted to a certain ground node at the same time to determine the order of their transmission. Therefore, the definition is the time step when all data packets arrive at the ground access point, satisfying

[0148] (17)

[0149] The end-to-end delay minimization problem can be formulated as:

[0150] (18)

[0151]

[0152] 3. Realizing Sequential Decision Problem Statement Based on MDP

[0153] The joint optimization problem to be solved by the present invention can be expressed as a sequential decision problem, which can be solved by a reinforcement learning algorithm. The reinforcement learning algorithm trains the intelligent agent by allowing the drone to interact with the environment continuously, and eventually enables the intelligent agent to learn the strategy that can obtain the maximum long-term return. The behavior of the intelligent agent is completely determined by the strategy. First, the MDP model of the transmission system is constructed, including the local observation state ,Behavior , state transition probability matrix P, reward function R and discount factor γ.

[0154] 1) Local observation state : In time step t, the local observation state of the transmission system consists of the following three parts: 1. The position of the UAV , including x i (t) and y i (t); 2. Number of data packets from ground nodes within the communication range of the drone ; 3. The location of other drones within the drone’s communication range and the number of packets .

[0155] 2) Behavior :The present invention considers the joint optimization of multi-UAV trajectory planning and communication route planning, so that the behavior of MDP includes three aspects of behavior: In terms of UAV trajectory planning, within time step t, the UAV adjusts its flight trajectory by changing its flight direction (front, back, left, right, etc.) or hovering, corresponding to the behavior ; In terms of ground node access planning, the drone can select a ground node to receive data packets, and the corresponding behavior ; In terms of FANET routing communication planning, drones can choose to transmit data packets to other drones or ground access points within the communication range, corresponding to the behavior .

[0156] 3) Reward Function :The setting of the reward function is closely related to the optimization goal. In order to achieve the goal of sending all data packets to the ground access point with minimum delay, a penalty is set for each time step. , and set a reward when all packets reach the ground access point To reduce the negative impact of sparse rewards, the rewards can be given some prior knowledge that the drone collecting packets from ground nodes where more packets are waiting in the queue should help reduce latency. Assume that the drone At time step Select the ground node to visit ,Right now . Set rewards . Represents ground node At time step If a ground node without a data packet is selected to access the drone, a penalty should be imposed .set up ,in, is a sign function. In addition, transmitting more data packets to the ground access point through multiple drones at each time step helps reduce latency, so if the drone At time step Transmit data packets to ground access points and set rewards , is the number of data packets transmitted by multiple drones i to the ground access point. Using prior knowledge, considering that drones will select nearby ground nodes to collect data packets, this helps to improve the transmission rate. Let the reward .in Indicates drone At time step Connect to the ground node Therefore, the reward function is defined as

[0157] (19)

[0158] in, represents the reward function, , , , , , , is a positive hyperparameter.

[0159] 4. Establish a UAV decision-making model for solving sequential decision-making problems, and use an improved deep reinforcement learning algorithm to solve the joint optimization problem.

[0160] The present invention adopts multi-agent deep reinforcement learning based on graph neural network to solve the joint optimization problem. The multi-agent deep reinforcement learning method adopted is not unique. The present invention takes the MAPPO algorithm as an example to solve the problem. The scenario of multi-UAV assisted WSN communication is modeled as an undirected graph. Each UAV is regarded as an agent. At each time step t, the adjacency matrix and local observation state O of the agent are input. The agent extracts the features of its adjacent nodes through the graph neural network. This solves the problem of multi-UAV assisted WSN data transmission, thereby minimizing end-to-end delay. After training, the neural network in MAPPO can fit the input state transformation to the state value and action strategy, which is used to evaluate the pros and cons of each behavior in each state, so as to obtain the strategy for the agent to make decisions.

[0161] Using an undirected graph Represents the routing network between drones, expressed as , the graph consists of a set of vertices and edge set In this system, the vertex set It is composed of various drones, and the edge is gathered It is composed of two communication links between drones. If the drone and drones There is a communication link between them, which can be expressed as or On this basis, adjacency is used to indicate that two drones are connected by an edge and are adjacent to each other. is a square matrix whose elements are either 0 or 1. Each element in the adjacency matrix Indicates drone and the relationship between drone p, if drone If there is a communication link between P and UAV p, then , otherwise, For an undirected graph , the adjacency matrix D is symmetric, that is, D .

[0162] The graph attention network is a type of graph neural network that uses an attention mechanism to assign different weights to different nodes. The training relies on pairs of adjacent nodes instead of the specific network structure. In the deep reinforcement learning algorithm based on the graph attention network, the drone features at time step t are first input. , including the location of drone i, the number of data packets of drone i, and the number of data packets of other drones within the communication range of drone i. Where N is the number of nodes, that is, the number of drones in the drone set N, is the dimension of the drone feature, and calculates the drone and drones The similarity coefficient of

[0163] (20)

[0164] is a learnable parameter, Indicates the splicing of the transformed drone features. It means mapping the concatenated high-dimensional drone features to real numbers, and then calculating the attention coefficient, that is,

[0165] (twenty one)

[0166] Among them, LeakyReLu() represents the activation function.

[0167] Finally, the adjacent drone features are weighted and summed according to the attention coefficient, and the fused drone features that fuse the adjacent drone features are output, that is,

[0168] (twenty two)

[0169] in, represents the activation function;

[0170] In order to improve the fitting ability of the drone decision model, a multi-tap attention mechanism is introduced to establish multiple attention mechanisms for the same data sample, namely

[0171] (twenty three)

[0172] Where k represents the A tap, represents the fused UAV features of UAV i obtained by the k-th tap, Represents multi-head attention features.

[0173] The present invention uses a two-layer GAT to extract the characteristics of the network topology. The feature extraction operation is as follows: Figure 3As shown. The weight coefficients between the current node and each adjacent node are calculated by attention, and the features of the current node in the next iteration are obtained by weighted summation of each neighboring node (including the current node). The fully connected layer is connected after the graph attention layer, and the feature representation processed by the graph attention layer is nonlinearly transformed and further processed, and finally the strategy or environment state value suitable for the reinforcement learning task is output. In addition, a normalization layer is added between the layers to normalize the input of each layer, solve the problem of internal covariate shift, improve training efficiency, accelerate convergence and enhance the generalization ability of the model.

[0174] The solution process can be divided into two steps: training process and application process. The specific solution process is as follows:

[0175] 1. Algorithm training process:

[0176] The training process is to model the environment and use the constructed environment to train the agent, that is, the agent constantly interacts with the environment and obtains rewards. The agent uses the rewards to update the parameters of the drone decision model, and finally enables the drone decision model to fit a more accurate Q value of each behavior according to the input state, thereby obtaining a convergent strategy. The flow chart of the training process is as follows Figure 4 As shown, the specific steps are as follows:

[0177] Step 1: Use the communication system simulation test platform to generate simulation scenarios of various communication applications; based on the generated simulation scenarios, set the initial state S0 of the environment, initialize the positions and number of data packets of the drone, ground nodes, and ground access points; initialize the Actor network parameters ϑ and Critic network parameters ω of the drone decision model.

[0178] Step 2: Initialize the experience replay pool .

[0179] Step 3: For each drone i, , get the current local observation state and the adjacency matrix , as the input of the UAV decision model, and output the behavior of the UAV , namely, UAV path planning and routing communication between UAV and ground nodes.

[0180] Step 4: Assign each drone selected behavior Act on the environment and get rewards from environmental feedback , and each UAV reaches the next local observation state and the adjacency matrix .

[0181] Step 5: Store the sample data from steps 3-4 into the experience replay pool middle.

[0182] Step 6: Determine whether the environment has been trained for the preset number of steps. If so, the experience replay pool Calculate the advantage function based on the sample data in and update the drone decision model and return to step 2, otherwise return to step 3.

[0183] Step 7: Determine whether the preset number of rounds of training has been completed. If yes, end the training; otherwise, return to step 2.

[0184] 2. Algorithm application process:

[0185] After training, the drone decision model is used as a software function and embedded in the air-ground communication system to achieve real-time joint optimization of air-ground communication of drone-assisted WSN. According to the observation status of the drone, the drone decision model is input, and the mobile and routing communication actions are output to realize the auxiliary routing function until all data packets are transmitted to the ground access point and reach the termination state, thereby obtaining the maximum end-to-end delay of data packets during drone trajectory planning and ground node access planning. The specific steps of the application process are as follows:

[0186] Step 1: Given the initial state S0 of the environment, each drone obtains the local observation state through positioning, navigation and other equipment.

[0187] Step 2: Each drone will observe the current local state Input the drone decision model to get the corresponding behavior .

[0188] Step 3: Select the behavior Act on the environment and get rewards from environmental feedback , and update the environment to a new state S t+1 .

[0189] Step 4: Determine whether the environment has reached the termination state, that is, whether all data packets have been transmitted to the ground access point. If it has not reached the termination state, return to step 2. If it has reached the termination state, end the application process.

[0190] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0191] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may include that the first and second features are in direct contact, or may include that the first and second features are not in direct contact but are in contact through another feature between them. Moreover, a first feature being "above", "above" and "above" a second feature includes that the first feature is directly above and obliquely above the second feature, or simply indicates that the first feature is higher in level than the second feature. A first feature being "below", "below" and "below" a second feature includes that the first feature is directly below and obliquely below the second feature, or simply indicates that the first feature is lower in level than the second feature.

[0192] In the present invention, the terms "first", "second", "third", "fourth" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance. The term "plurality" refers to two or more, unless otherwise clearly defined.

[0193] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A joint optimization method for flight self-organizing network assisted communication, characterized in that: The following steps are involved: Step S1: construct a multi-UAV and WSN data routing transmission system model; Step S2: Establish a joint optimization problem for FANET-assisted communication, with minimizing the maximum end-to-end delay of the transmission system as the optimization goal; Step S3: Based on MDP, the joint optimization problem of FANET-assisted communication is formulated as a sequential decision problem; Step S4: Establish a UAV decision model for solving sequential decision problems based on deep reinforcement learning graph neural network and perform training; Step S5: Apply the trained UAV decision model to achieve joint optimization of flight self-organizing network-assisted communication; The step S1 specifically includes: Step S1-1: The UAV and WSN data routing transmission system model includes a FANET composed of N UAVs, a WSN composed of M ground nodes and a ground access point. Each ground node has multiple data packets, and each ground node needs to transmit all data packets to the ground access point; the ground routing method of the WSN is given by the Primm minimum spanning tree algorithm; make Represents ground node At time step Data packet transmission path selection: (1) in, , I represents a drone set consisting of N drones; (2) Wherein, J represents a ground node set consisting of M ground nodes; make Indicates drone At time step Transmitting data packets to the drone , and , then (3) make Indicates drone At time step Transmitting data to a ground access point, we have (4) (5) Command drone Flying at a fixed altitude H, flying or hovering at a constant speed ‖v(t)‖, the position of drone i is It is expressed as: (6) Among them, Δt represents the time step; (7) in, Indicates the minimum distance at which the drone does not collide; Indicates drone At time step A collection of neighbor drones, (8); in, and Respectively represent drones and drones The effective communication area radius; Step S1-2: Assume that all communication channels are orthogonal channels without inter-channel interference. Then the channel model when ground node j in WSN transmits a data packet to ground node j' at time step t is It is expressed as: (9) in, Indicates the reference distance The path loss when Represents ground node To ground node The straight-line distance, represents the path loss parameter of non-line-of-sight transmission, represents the small-scale fading component of the cyclically symmetric complex Gaussian distribution with unit variance at time step t; The transmission rate of data packets transmitted from ground node j to ground node j' at time step t It is expressed as: (10) in, is the subchannel bandwidth, is the transmission power of ground node j, is the unilateral power spectral density of Gaussian white noise; Channel model between ground node j and UAV i at time step t It is expressed as: (11) in, represents the ground node at time step t The straight-line distance to drone i, Represents the path loss parameter of line-of-sight transmission; Channel model between UAV i and ground access point at time step t It is expressed as: (12) in, represents the straight-line distance from UAV i to the ground access point at time step t; Time step t ground node The transmission rate of data packets transmitted to UAV i It is expressed as: (13) The transmission rate of the data packet transmitted by the UAV to the ground access point at time step t It is expressed as: (14) in, Indicates the channel bandwidth used by the drone to transmit data packets. Constant power for transmitting packets to drones, shaping antenna gain for directional beamforming; Time step t UAV and drones Channel Model for: (15) Time step t UAV To the drone The transmission rate of the transmitted data packets is: (16)； The step S2 specifically includes: make Represents ground node At time step The number of packets, Indicates drone At time step The number of data packets, then the joint optimization problem is expressed as: (17) (18) in, is the time step when all packets arrive at the ground access point; The step S3 specifically includes: establishing an MDP model of a multi-UAV and WSN data routing transmission system model, thereby expressing the joint optimization problem of FANET-assisted communication as a sequential decision problem; The MDP model includes local observation states ,Behavior , state transition probability matrix P, reward function R and discount factor γ; Among them, the local observation state Includes: The position of drone i at time step t , the number of data packets from ground nodes within the communication range of UAV i , the location of other drones within the communication range of drone i And the number of data packets of other drones within the communication range of drone i ; ； Reward Function : (19) in, Represents the reward when all packets reach the ground access point , where s=1,2,3,4, Indicates the node access reward, Indicates no data node access penalty, represents the auxiliary data transmission reward, Indicates mobile rewards, represents the time step reward; The step S4 specifically includes: Step S4-1: Use an undirected graph G to represent the routing network between drones. , V represents the vertex set in the undirected graph G, i.e., the drone set, represents the edge set, that is, the set of communication links between two drones, and the adjacency matrix ; Step S4-2: Establish a drone decision model, and its deep reinforcement learning algorithm is as follows: (1) Input drone features at time step t , represents the drone features of drone i, including the location of drone i, the number of data packets of drone i, and the number of data packets of other drones within the communication range of drone i. N represents the number of drones in the drone set N. is the dimension of the drone’s characteristics; (2) Computational drones and drones The similarity coefficient of (20) is a learnable parameter, Indicates the splicing of the transformed drone features. Indicates mapping the concatenated high-dimensional drone features to real numbers; (3) Calculate the attention coefficient, that is (21) Among them, LeakyReLu() represents the activation function; (4) According to the attention coefficient, the weighted sum of the adjacent drone features is performed to obtain the fused drone feature of drone i that integrates the adjacent drone features: (22) in, represents the activation function; (5) Introduce a multi-tap attention mechanism and establish multiple attention mechanisms for the same data sample: (23) Where k represents the A tap, represents the fused UAV features of UAV i obtained by the k-th tap, Represents multi-head attention characteristics; Step S4-3: Train the UAV decision model using the local observation state and adjacency matrix as input.

2. The joint optimization method for flight self-organizing network assisted communication according to claim 1, characterized in that: The step S5 specifically includes: Step S5-1: Given the initial state S0 of the environment; Step S5-2: Each UAV inputs the current local observation state into the UAV decision model to obtain the corresponding behavior; Step S5-3: Apply the selected behavior to the environment, obtain rewards from environmental feedback, and update the environment to a new state; Step S5-4: Determine whether the environment has reached the termination state, that is, whether all data packets have been transmitted to the ground access point. If it has not reached the termination state, return to step S5-2; if it has reached the termination state, end the application process.

Images