Flight ad hoc network distributed routing method based on multi-agent reinforcement learning

Abstract

The invention belongs to the field of unmanned aerial vehicle communication, and discloses a flight ad hoc network distributed routing method based on multi-agent reinforcement learning, and the method comprises the steps: building four network performance models: energy consumption, packet delivery rate, throughput and end-to-end time delay; establishing a node interaction mechanism; establishing a Markov decision process, defining definitions of a state space, an action space and a space state transition probability, and designing a reward function; based on an improved multi-agent depth deterministic policy gradient algorithm DE-MADDPG, designing a flight ad hoc network distributed routing method based on multi-agent reinforcement learning; each unmanned aerial vehicle modifies a running protocol and a protocol parameter according to local network information of each node, optimizes a main network and a subnetwork of the system, and improves the overall performance of the network. The method mainly solves the problems that a traditional flight ad hoc network protocol is not high in adaptive capacity, and the network environment is not matched with the protocol. Compared with a traditional method, the method has the advantages of being low in control delay, high in control precision and good in performance.

Description

Technical Field

[0001] The present invention relates to the field of unmanned aerial vehicle communication, and in particular to a flying ad hoc network distributed routing method and system based on multi-agent reinforcement learning. Background Art

[0002] With the large-scale application of drone swarms, the performance requirements of drone communication networks for routing protocols are becoming increasingly higher. When selecting the best-performing protocol, drone network systems often need to test the performance of all protocols, which is time-consuming and laborious. When using deep learning algorithms to predict the best protocol through the network environment, as the network environment of the flight ad hoc network changes, a single protocol cannot meet the needs; and the random switching operation protocol strategy of adaptive multi-mode routing limits the performance. After the introduction of deep reinforcement learning, there is a delay in the control center collecting global network status information and each node receiving the control instructions issued by the control center. The control accuracy of single-agent reinforcement learning is insufficient, and multi-parameter optimization cannot be achieved for a single node. Therefore, a drone network routing method with low control delay, high control accuracy and good performance is needed.

[0003] The introduction of multi-agent deep reinforcement learning algorithm can adjust the routing protocol of the node operation, optimize the sending interval of the protocol packet, make the main network and each subnet work in the best state, improve the system throughput and packet delivery rate, and reduce the end-to-end delay. At the same time, compared with the single-agent reinforcement learning algorithm, it can effectively solve the problem of control signal delay and reduce energy consumption.

[0004] The transmission delay of the agent control signal will cause the instability of the algorithm performance. The single-agent reinforcement learning algorithm limits the expansion of the network scale. The present invention combines multi-agent deep reinforcement learning with adaptive multi-mode routing to enable it to have the characteristics of distributed execution, which solves this problem and makes the network performance more superior and stable. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a distributed routing method for a flying ad hoc network based on multi-agent reinforcement learning in response to the defects involved in the background technology. The method is based on an improved multi-agent reinforcement learning algorithm DE-MADDPG to adjust the routing protocol running on the node, optimize the sending interval of the protocol packet, so that the main network and each subnet work in the best state, improve the system throughput and packet delivery rate, reduce the end-to-end delay, effectively solve the problem of control signal delay and reduce energy consumption.

[0006] The present invention adopts the following technical solutions to solve the above technical problems:

[0007] A distributed routing method for a flying ad hoc network based on multi-agent reinforcement learning, characterized in that it comprises the following steps:

[0008] S1. Establish four network performance models: energy consumption, packet delivery rate, throughput and end-to-end delay;

[0009] S2. Establish node interaction mechanism;

[0010] S3. Establish a Markov decision process, clarify the definitions of state space, action space, and spatial state transition probability, and design a reward function;

[0011] S4. Based on the improved multi-agent deep deterministic policy gradient algorithm DE-MADDPG, a distributed routing method for flying ad hoc networks based on multi-agent reinforcement learning is designed;

[0012] S5. Each drone modifies the running protocol and protocol parameters according to the local network information of its own node, optimizes the system main network and subnet, and improves the overall network performance;

[0013] Specifically, in step S1, the four performance models of energy consumption, packet delivery rate, throughput and end-to-end delay are:

[0014] (1) Energy consumption: The energy consumption of flying ad hoc network communication mainly depends on the energy required for transmission and reception. The energy required to transmit the signal is related to the transmission distance. The energy required for the i-th node to transmit the signal to the adjacent node is expressed as:

[0015]

[0016] in, To receive the required energy, is the energy required for transmission, d ij represents the distance between two nodes, ε is a constant coefficient, η represents the drain efficiency, σ is the path loss exponent;

[0017] The network contains K h l The total energy consumed by the hop-forwarding transmission path is expressed as:

[0018]

[0019] (2) Packet delivery rate: The packet delivery rate (PDR) is the ratio of the data packets successfully received by the destination node to the data packets sent by the source node. In the multi-hop environment of the flying ad hoc network, the packet delivery rate of the transmission path l can be regarded as the product of the success rate of each hop. The packet delivery rate is expressed as:

[0020]

[0021] in, is the number of data packets received by the k+1th node, The number of packets sent by the kth node;

[0022] (3) Throughput: Throughput is the actual transmission rate between the source node and the destination node. The throughput of the transmission path is limited by the minimum transmission speed in the path. The throughput calculation formula is:

[0023]

[0024] Where TH represents the average throughput, P γi Refers to the iteration period T of node i S The data packets sent in |P γi |Represents the payload size of the data packet;

[0025] (4) End-to-end delay: End-to-end delay refers to the time it takes for a data packet to reach the destination node from the source node. The delay of the transmission path is expressed as:

[0026]

[0027] in, and They represent the processing delay, forwarding queue delay, transmission delay and wireless signal propagation delay of node i respectively.

[0028] Specifically, in step S2, the node interaction mechanism simplifies the joint actions of each intelligent agent into the joint actions between the node and its adjacent nodes to obtain the status information of the adjacent nodes, and utilizes the mobile adaptive multi-mode routing framework combined with the sequence destination node distance vector routing protocol and the on-demand plane distance vector routing protocol. All nodes retain the header part of the data packet for updating the network status information of the neighbor table, and only forward the node and the destination node and store the data part.

[0029] Specifically, in step S3, the Markov decision process is defined by a seven-tuple: <N,S i ,A i ,P,R i ,O i ,γ>. Where N is the number of agents. S i , A i , R i and O i Represent the state, action, reward and observation set of the ith agent respectively. P is the state transition probability matrix, and γ is the discount factor.

[0030] State space, action space, state transition probability, and reward function refer to:

[0031] (1) State space: The protocol currently running on the node is part of the state. In addition, the parameters of the protocol need to be adjusted to reduce routing overhead and improve routing performance. Since the parameters of different protocols vary greatly, the balance between performance and overhead can be achieved by adjusting the interval between sending Hello packets. The logical location information of the node is used as the state quantity to ensure the consistency of the strategies of each node. Then, the state of node i can be expressed as the formula

[0032] s i =[c i ,m i ,T i ];

[0033] Among them, c i Represents the logical position of the current node in the link.

[0034] (2) Action space: The action of the agent is to modify the node's routing protocol and the Hello packet sending interval. Changes to the node's routing protocol may also cause changes in the node's position in the link. The action space of the multi-agent reinforcement learning algorithm can be continuous, that is, the algorithm can directly output the time interval without discretization. Then, the action of node i can be expressed as:

[0035]

[0036] (3) State transition probability: The state transition probability is determined by the system. The node should fully execute the decision made by the intelligent agent. However, the intelligent agent runs on the framework of adaptive multi-mode routing. The change of node working mode and the adjustment of the time interval for sending Hello packets are subject to the basic rules of adaptive multi-mode routing. For example, after the working mode of the cluster head node is changed, the subnet will reselect the cluster head node; the Hello packet sending interval is restricted. These rules are marked as f. Then, the state transition probability can be expressed as:

[0037]

[0038] (4) Reward function: Energy consumption can be regarded as the sum of all node signal transmission and reception and signal path loss. The energy consumption of signal transmission and reception is positively correlated with the number of received and sent data packets. The distance between two nodes can be calculated by the delay of the exchanged data packets. After normalization, the part of the node reward function representing energy consumption is defined as the formula

[0039]

[0040] in, are the number of packets received and sent by the node, t a ,ts are the arrival time and sending time of the interactive information, and v is the propagation speed of electromagnetic waves in the air.

[0041] Packet delivery rate is a basic indicator of network performance. The network requires that each link segment be delivered as much as possible. For a node, it is necessary to ensure that the packet loss rate of the currently used link is as low as possible, so it is directly introduced into the reward function:

[0042]

[0043] To make the throughput of the entire network as large as possible, the throughput of each node needs to be as large as possible. Adjustment of the link will also cause changes in throughput. However, the source node and destination node of the link will not change, and the throughput of the destination node is crucial to the system. Then, the logical position of the node also needs to be taken into account. Therefore, the throughput formula of the reward function is:

[0044]

[0045] Among them, λ ci is the location coefficient of node i.

[0046] End-to-end delay is the time required for a data packet to reach the destination node from the source node. Processing delay and transmission delay are determined by hardware and are fixed values. However, the forwarding queue delay is positively correlated with the queue length of the current node i, and the wireless signal propagation delay is proportional to the distance between nodes. These parameters can be changed by changing the working mode of the node and then changing the link. After normalization, the delay part of the reward function is defined as follows:

[0047]

[0048] Combined with the above parameters, the reward function of node i at time t is (the following coefficients are all taken as 1 during the experiment):

[0049]

[0050] Specifically, the present invention adopts a centralized training and distributed execution algorithm structure, deploys a strategy algorithm (agent) in each drone node, and deploys an evaluation algorithm in the control center. When executing a task, each drone strategy algorithm makes real-time decisions on route optimization based on its own network status information and the status of collected neighboring nodes. When the task is completed, the control center makes a global optimization of the strategies of all nodes to improve the performance of each node in the next task.

[0051] Furthermore, the DE-MADDPG algorithm adds an additional global value network to maximize the global reward.

[0052] Furthermore, the DE-MADDPG algorithm environment contains n agents, with π i represents the policy network of the ith agent, and its policy parameters are denoted as θ i , use Q i represents the value network of the ith agent, and its network parameters are expressed as Q G It represents the global value network, and its network parameter is denoted as ψ. Obviously, every drone node in the flying ad hoc network environment is equal, and the strategy should be consistent, so π1=···=π n , the same value judgment criteria should also be the same, that is, Q1 = ··· = Q n Let θ = θ i , The cumulative expected return of the i-th agent can be expressed as

[0053]

[0054] where p π represents the state distribution of node i, and t represents the current time. For the deterministic policy gradient algorithm, the policy gradient of the cumulative expected return can be obtained as the formula

[0055]

[0056] Among them, D is the experience pool, which is used to store the node status, observation values, and corresponding actions and rewards. It is the value function established for each node. It is a value function established globally. and According to the formula Constantly updated.

[0057] Specifically, in step S5, during the execution phase, each node deploys a policy network on the onboard computer. The onboard computer obtains the logical location of the node, the routing protocol being run, the Hello packet sending interval, and the status information of neighboring nodes, and takes action based on the strategy, changes the protocol of the node and the sending interval of the Hello packet, and records these interactive information <S, A, O, S′, R> into the experience pool. Each execution ends and enters the training phase, and the value network deployed in the control center and Randomly select experience from the experience pool, minimize the loss function and update the network parameters and ψ. The updated value network is used to calculate the policy gradient, which is used to update the policy parameter θ. After the update is completed, wait for the execution of the next task. The specific process is as follows:

[0058] Step 1: After initializing all parameters, loop through steps 2 to 17 until the round ends;

[0059] Step 2: Input node network status information (logical location, routing protocol, and Hello packet sending interval) i =[c i ,m i ,T i ], the node's observation value (the state information of the neighboring node) o i ={s j |j is adjacent to i};

[0060] Step 3: The state of any drone node at time j according to the strategy network θ and observations Take Action Take action

[0061] Step 4: Get the new status Observations and the reward value r i j ;

[0062] Step 5: Record Enter experience pool D;

[0063] Step 6: Loop through steps 2 to 4 until the task is completed;

[0064] Step 7: Process the data in the experience pool D to obtain global records And store it in the experience pool D′;

[0065] Step 8: Randomly sample from the experience pool D′, with a sample size of S;

[0066] Step 9: Order

[0067] Step 10: Minimize the loss function:

[0068] Step 11: Update the global value network Q ψ′ :ψ′=τψ+(1-τ)ψ′;

[0069] Step 12: Order

[0070] Step 13: Minimize the loss function:

[0071] Step 14: Update the value network:

[0072] Step 15: Update the policy network via policy gradient:

[0073]

[0074] Step 16: Update the policy network: θ′=τθ+(1-τ)θ′;

[0075] Step 17: Output the changed routing protocol and the interval for sending Hello packets

[0076] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0077] 1. The present invention establishes a distributed routing method for a flying ad hoc network based on multi-agent reinforcement learning, which solves the problems of unstable performance, high control delay and low control accuracy of traditional methods.

[0078] 2. The present invention constructs four network performance models of energy consumption, packet delivery rate, throughput and end-to-end delay, so as to design a reward function and build a multi-agent algorithm model, which can obtain the global information of the network system in a timely manner.

[0079] 3. The present invention establishes a node interaction mechanism, taking into account the joint actions of various drone nodes, so that network nodes can obtain status information of adjacent nodes.

[0080] 4. The present invention uses a multi-agent reinforcement learning algorithm to establish a Markov decision process, thereby optimizing the network performance of the network.

[0081] 5. This invention adopts the optimized DE-MADDPG algorithm and adds a global Critic network to maximize the rewards of each agent and the global reward. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Figure 1 It is a schematic diagram of the UAV node interaction mechanism of the present invention;

[0083] Figure 2 It is a schematic diagram of the framework of the distributed routing optimization algorithm of the present invention;

[0084] Figure 3 It is a flow chart of the DE-MADDPG routing optimization algorithm of the present invention. DETAILED DESCRIPTION

[0085] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings:

[0086] 1) Figure 1This is a schematic diagram of the drone node interaction mechanism. All nodes retain the header part of the data packet for updating the network status information of the neighbor table. Only the forwarding node and the destination node will store the data part. After the introduction of multi-agents, if each agent updates independently, it will lead to environmental instability. The joint actions of each drone node must be considered. Therefore, each node must interact with each other and be able to observe the network status information of each other. For network nodes, it is unrealistic to obtain the status information of all other nodes in the world, but it is completely feasible to obtain the status information of adjacent nodes. The joint action of each agent can be simplified to the joint action between the node and its adjacent nodes. The formula is

[0087]

[0088] 2) If Figure 2 As shown, the present invention adopts a distributed routing optimization algorithm framework. The present invention adopts a centralized training distributed execution algorithm structure, deploys a strategy algorithm (agent) at each drone node, and deploys an evaluation algorithm at the control center. When executing a task, each drone strategy algorithm makes real-time decisions on routing optimization based on its own network status information and the status of collected neighboring nodes. At the end of the task, the control center makes a global optimization of the strategies of all nodes to improve the performance of each node in the next task. The DE-MADDPG algorithm additionally adds a global value network to maximize global rewards.

[0089] The DE-MADDPG algorithm environment contains n agents, represented by π i represents the policy network of the ith agent, and its policy parameters are denoted as θ i , use Q i represents the value network of the ith agent, and its network parameters are expressed as Q G It represents the global value network, and its network parameter is denoted as ψ. Obviously, every drone node in the flying ad hoc network environment is equal, and the strategy should be consistent, so π1=···=π n , the same value judgment criteria should also be the same, that is, Q1 = ··· = Q n Let θ = θ i , The cumulative expected return of the i-th agent can be expressed as

[0090]

[0091] where p π represents the state distribution of node i, and t represents the current time. For the deterministic policy gradient algorithm, the policy gradient of the cumulative expected return can be obtained as the formula

[0092]

[0093] Among them, D is the experience pool, which is used to store the node status, observation values, and corresponding actions and rewards. It is the value function established for each node. It is a value function established globally. and According to the formula Constantly updated.

[0094] 3) If Figure 3 As shown, the present invention adopts a routing optimization algorithm based on DE-MADDPG. In the execution phase, each node deploys a policy network on the onboard computer. The onboard computer obtains the logical location of the node, the running routing protocol and the Hello packet sending interval, and the status information of the neighboring nodes, and takes action based on the strategy, changes the protocol of the node and the sending interval of the Hello packet, and records these interactive information <S, A, O, S′, R> into the experience pool. After each execution, the training phase begins, and the value network deployed in the control center and Randomly select experience from the experience pool, minimize the loss function and update the network parameters and ψ. The updated value network is used to calculate the policy gradient, which is used to update the policy parameter θ. After the update is completed, wait for the execution of the next task. The specific process is as follows:

[0095] Step 1: After initializing all parameters, loop through steps 2 to 17 until the round ends;

[0096] Step 2: Input node network status information (logical location, routing protocol, and Hello packet sending interval) i =[c i ,m i ,T i ], the node's observation value (the state information of the neighboring node) o i ={s j |j is adjacent to i};

[0097] Step 3: The state of any drone node at time j according to the strategy network θ and observations Take Action Take action

[0098] Step 4: Get the new status Observations and reward value

[0099] Step 5: Record Enter experience pool D;

[0100] Step 6: Loop through steps 2 to 4 until the task is completed;

[0101] Step 7: Process the data in the experience pool D to obtain global records And store it in the experience pool D′;

[0102] Step 8: Randomly sample from the experience pool D′, with a sample size of S;

[0103] Step 9: Order

[0104] Step 10: Minimize the loss function:

[0105] Step 11: Update the global value network Q ψ′ :ψ′=τψ+(1-τ)ψ′;

[0106] Step 12: Order

[0107] Step 13: Minimize the loss function:

[0108] Step 14: Update the value network:

[0109] Step 15: Update the policy network via policy gradient:

[0110]

[0111] Step 16: Update the policy network: θ′=τθ+(1-τ)θ′;

[0112] Step 17: Output the changed routing protocol and the interval for sending Hello packets The main protection technical points of this invention are:

[0113] (1) Distributed routing method for flying ad hoc networks based on multi-agent reinforcement learning.

[0114] (2) Establish four network performance models: energy consumption, packet delivery rate, throughput, and end-to-end delay, and then design a reward function and build a multi-agent algorithm model to obtain the global information of the network system.

[0115] (3) Establish a node interaction mechanism so that network nodes can obtain status information of adjacent nodes.

[0116] (4) A multi-agent reinforcement learning algorithm is used to establish a Markov decision process, which optimizes the network performance of the network.

[0117] (5) The optimized DE-MADDPG algorithm is adopted, and a global critic network is added to maximize the rewards of each agent and the global reward.

[0118] The present invention is suitable for route optimization of flying ad hoc networks. The method can enhance the stability of network performance and solve the problems of high control delay and low control accuracy of traditional methods through a multi-agent reinforcement learning algorithm.

[0119] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.

[0120] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A distributed routing method for flying ad hoc networks based on multi-agent reinforcement learning, characterized in that: The following steps are involved: S1. Establish four network performance models: energy consumption, packet delivery rate, throughput and end-to-end delay; S2. Establish node interaction mechanism; S3. Establish a Markov decision process, clarify the definitions of state space, action space, and spatial state transition probability, and design a reward function; S4. Based on the improved multi-agent deep deterministic policy gradient algorithm DE-MADDPG, a distributed routing method for flying ad hoc networks based on multi-agent reinforcement learning is designed; S5. Each drone modifies the running protocol and protocol parameters according to the local network information of its own node, optimizes the system main network and subnet, and improves the overall network performance.

2. The distributed routing method for flight ad hoc networks based on multi-agent reinforcement learning according to claim 1, characterized in that: The four performance models of energy consumption, packet delivery rate, throughput and end-to-end delay described in step S1 are: (1) Energy consumption: The energy consumption of flying ad hoc network communication mainly depends on the energy required for transmission and reception. The energy required to transmit the signal is related to the transmission distance. The energy required for the i-th node to transmit the signal to the adjacent node is expressed as: in, To receive the required energy, The energy required for transmission, d ij represents the distance between two nodes, ε is a constant coefficient, η represents the drain efficiency, σ is the path loss exponent; The network contains K h l The total energy consumed by the hop-forwarding transmission path is expressed as: (2) Packet delivery rate: The packet delivery rate (PDR) is the ratio of the data packets successfully received by the destination node to the data packets sent by the source node. In the multi-hop environment of the flying ad hoc network, the packet delivery rate of the transmission path l can be regarded as the product of the success rate of each hop. The packet delivery rate is expressed as: in, is the number of data packets received by the k+1th node, The number of packets sent by the kth node; (3) Throughput: Throughput is the actual transmission rate between the source node and the destination node. The throughput of the transmission path is limited by the minimum transmission speed in the path. The throughput calculation formula is: Where TH represents the average throughput, P γi Refers to the iteration period T of node i S The data packets sent in |P γi |Represents the payload size of the data packet; (4) End-to-end delay: End-to-end delay refers to the time it takes for a data packet to reach the destination node from the source node. The delay of the transmission path is expressed as: Among them, D i proc , D i q , and D i prop They represent the processing delay, forwarding queue delay, transmission delay and wireless signal propagation delay of node i respectively.

3. The distributed routing method for flying ad hoc networks based on multi-agent reinforcement learning according to claim 1 is characterized in that: The node interaction mechanism described in step S2 is to simplify the joint actions of each intelligent body into the joint actions between the node and its adjacent nodes to obtain the status information of the adjacent nodes, and use the mobile adaptive multi-mode routing framework combined with the sequence destination node distance vector routing protocol and the on-demand plane distance vector routing protocol. All nodes retain the header part of the data packet for updating the network status information of the neighbor table, and only forward the node and the destination node and store the data part.

4. The distributed routing method for flight ad hoc networks based on multi-agent reinforcement learning according to claim 1, characterized in that: The Markov decision process described in step S3 is defined by a seven-tuple: <N,S i ,A i ,P,R i ,O i ,γ>; where N is the number of agents; S i , A i , R i and O i They represent the state, action, reward and observation set of the i-th agent respectively; P is the state transition probability matrix, and γ is the discount factor. The state space, action space, state transition probability, and reward function are: (1) State space: The protocol currently running on the node is part of the state. In addition, the protocol parameters need to be adjusted to reduce routing overhead and improve routing performance. Since the parameters of different protocols vary greatly, the balance between performance and overhead can be achieved by adjusting the interval between sending Hello packets. The logical location information of the node is used as the state quantity to ensure the consistency of the strategies of each node. Then, the state of node i can be expressed as the formula s i =[c i ,m i ,T i ]; Among them, c i Represents the logical position of the current node in the link; (2) Action space: The action of the agent is to modify the node's routing protocol and the Hello packet sending interval. Changes to the node's routing protocol may also cause changes in the node's position in the link. The action space of the multi-agent reinforcement learning algorithm can be continuous, that is, the algorithm can directly output the time interval without discretization. Then, the action of node i can be expressed as: (3) State transition probability: The state transition probability is determined by the system. The node should fully execute the decision made by the intelligent agent. However, the intelligent agent runs on the framework of adaptive multi-mode routing. The change of node working mode and the adjustment of the Hello packet sending time interval are constrained by the basic rules of adaptive multi-mode routing. For example, after the cluster head node working mode is changed, the subnet will reselect the cluster head node. The Hello packet sending interval is restricted. These rules are marked as f. Then, the state transition probability can be expressed as: (4) Reward function: Energy consumption can be regarded as the sum of all node receiving and sending signals and signal path losses. The energy consumption of receiving and sending signals is positively correlated with the number of received and sent data packets. The distance between two nodes can be calculated by the delay of the exchanged data packets. After normalization, the part of the node reward function representing energy consumption is defined as the formula in, are the number of packets received and sent by the node, t a ,t s are the arrival time and sending time of the interactive information, v is the propagation speed of electromagnetic waves in the air; Packet delivery rate is a basic indicator of network performance. The network requires that each link segment be delivered as much as possible. For nodes, it is necessary to ensure that the packet loss rate of the currently used link is as low as possible. Therefore, it is directly introduced into the reward function: To make the throughput of the entire network as large as possible, the throughput of each node needs to be as large as possible; the adjustment of the link will also cause the throughput to change; however, the source node and destination node of the link will not change, and the throughput of the destination node is crucial to the system; then, the logical position of the node also needs to be taken into account; therefore, the throughput formula of the reward function is: Among them, λ ci is the location coefficient of node i; End-to-end delay is the time required for a data packet to reach the destination node from the source node; processing delay and transmission delay are determined by hardware and are fixed values; however, the forwarding queue delay is positively correlated with the queue length of the current node i, and the wireless signal propagation delay is proportional to the distance between nodes; these parameters can be changed by changing the working mode of the node and then changing the link; after normalization, the delay part of the reward function is defined as follows: Combined with the above parameters, the reward function of node i at time t is (the following coefficients are all taken as 1 during the experiment):

5. The distributed routing method for flying ad hoc networks based on multi-agent reinforcement learning according to claim 1, characterized in that: The present invention adopts a centralized training and distributed execution algorithm structure, deploys a strategy algorithm (intelligent agent) at each drone node, and deploys an evaluation algorithm at the control center; when executing a task, each drone strategy algorithm makes a real-time decision on route optimization based on its own network status information and the status of collected neighboring nodes; when the task is completed, the control center makes a global optimization of the strategies of all nodes to improve the performance of each node in the next task.

6. The distributed routing method for flying ad hoc networks based on multi-agent reinforcement learning according to claim 5, characterized in that: The DE-MADDPG algorithm adds an additional global value network to maximize the global reward.

7. The distributed routing method for flying ad hoc networks based on multi-agent reinforcement learning according to claim 6 is characterized in that: The DE-MADDPG algorithm environment contains n agents, represented by π i represents the policy network of the ith agent, and its policy parameters are denoted as θ i , use Q i represents the value network of the ith agent, and its network parameters are expressed as Q G Represents the global value network, and its network parameter is denoted as ψ. Obviously, every drone node in the flying ad hoc network environment is equal, and the strategy should be consistent, so π1=···=π n , the same value judgment criteria should also be the same, that is, Q1 = ··· = Q n ; Let θ = θ i , The cumulative expected return of the i-th agent can be expressed as where p π represents the state distribution of node i, and t represents the current moment; for the deterministic policy gradient algorithm, the policy gradient of the cumulative expected return can be obtained as the formula Among them, D is the experience pool, which is used to store the node's status, observations, and corresponding actions and rewards; is the value function established for each node. is a value function established globally, and According to the formula Constantly updated.

8. The distributed routing method for flying ad hoc networks based on multi-agent reinforcement learning according to claim 1, characterized in that: Step S5: In the execution phase, each node deploys a strategy network on the onboard computer. The onboard computer obtains the logical location of the node, the routing protocol and Hello packet sending interval, and the status information of the neighboring nodes, and takes action based on the strategy, changes the protocol of the node and the sending interval of the Hello packet, and records these interaction information <S, A, O, S′, R> into the experience pool; each execution ends and enters the training phase, and the value network deployed in the control center and Randomly select experience from the experience pool, minimize the loss function and update the network parameters and ψ; use the updated value network to calculate the policy gradient, which is used to update the policy parameter θ. After the update is completed, wait for the execution of the next task; the specific process is as follows: Step 1: After initializing all parameters, loop through steps 2 to 17 until the round ends; Step 2: Input node network status information (logical location, routing protocol, and Hello packet sending interval) i =[c i ,m i ,T i ], the node's observation value (the state information of the neighboring node) o i ={s j |j is adjacent to i}; Step 3: The state of any drone node at time j according to the strategy network θ and observations Take Action Take action Step 4: Get the new status Observations and the reward value r i j ; Step 5: Record Enter experience pool D; Step 6: Loop through steps 2 to 4 until the task is completed; Step 7: Process the data in the experience pool D to obtain global records And store it in the experience pool D′; Step 8: Randomly sample from the experience pool D′, with a sample size of S; Step 9: Order Step 10: Minimize the loss function: Step 11: Update the global value network Q ψ′ :ψ′=τψ+(1-τ)ψ′; Step 12: Order Step 13: Minimize the loss function: Step 14: Update the value network: Step 15: Update the policy network via policy gradient: Step 16: Update the policy network: θ′=τθ+(1-τ)θ′; Step 17: Output the changed routing protocol and the interval for sending Hello packets

Images