Flight ad hoc network adaptive routing method based on deep reinforcement learning

Abstract

The invention belongs to the field of unmanned aerial vehicle communication, and discloses a flight ad hoc network adaptive routing method based on deep reinforcement learning, and the method comprises the steps: building a multi-mode routing framework; constructing a flight ad hoc network adaptive multi-mode routing model; and designing a flight ad hoc network adaptive route selection method based on deep reinforcement learning according to the deep Q network. The method mainly solves the problems that a traditional flight ad hoc network protocol is poor in adaptive capacity and poor in large-scale network application scene effect. According to the method, the routing working mode of each unmanned aerial vehicle is automatically adjusted through deep reinforcement learning, and a large network is decomposed into a main network and a plurality of small heterogeneous networks connected with the main network, so that the system complexity can be reduced, the local performance is optimal, and the overall network performance is improved.

Description

Technical Field

[0001] The present invention relates to the field of unmanned aerial vehicle communications, and in particular to a flying ad hoc network adaptive routing selection method based on deep reinforcement learning. Background Art

[0002] With the rapid development of modern drone technology, the demand for high performance and strong environmental adaptability in drone cluster communication is becoming increasingly obvious. In drone cluster communication, the large number of drones and frequent information exchange will cause high latency and high load problems; while the large coverage range, high bandwidth, and strong computing power requirements will cause high power consumption and high cost problems. Therefore, a technical method with high performance, low hardware requirements, and strong environmental adaptability is needed, while minimizing the cost of use.

[0003] For large-scale network application scenarios with a large number of nodes, the performance of traditional routing protocols has significantly decreased. The network environment is complex and diverse, and it is difficult to establish a reasonable and complete mathematical model between network node status information and network performance. Traditional mathematical modeling methods face the difficulties of establishing accurate system models, unifying models for different tasks, and quantifying environmental factors.

[0004] The introduction of deep reinforcement learning can solve the control optimization problem of the routing system, achieve an overall improvement in network performance, and reduce the load on the console, thereby ensuring high performance and strong environmental adaptability at a low cost. The intelligent agent of deep reinforcement learning can collect the network status of each node in real time, learn corresponding strategies for different network environments, and set each node to the optimal working mode.

[0005] The present invention adopts a flying ad hoc network adaptive routing selection method based on deep reinforcement learning, which can provide efficient routing services in various application scenarios. Even in large-scale network scenarios, its performance is still better than that of traditional protocols. Summary of the invention

[0006] The technical problem to be solved by the present invention is to address the defects involved in the background technology and provide a flying self-organizing network adaptive routing selection method based on deep reinforcement learning. The method automatically adjusts the routing working mode of each drone through deep reinforcement learning, decomposes the large network into a main network and several small heterogeneous networks connected to it, reduces the system complexity, optimizes the local performance, and improves the performance of the entire network.

[0007] The present invention adopts the following technical solutions to solve the above technical problems:

[0008] A method for adaptive routing selection in a flying ad hoc network based on deep reinforcement learning, characterized in that it comprises the following steps:

[0009] S1: Establish a multi-modal routing framework;

[0010] S2: Construct an adaptive multi-mode routing model for flight ad hoc networks;

[0011] S3: Design a deep reinforcement learning-based adaptive routing selection method for flying ad hoc networks according to the deep Q network. Specifically, in step S1, a single routing node in the multi-mode routing framework includes a monitoring component and a routing protocol component.

[0012] The routing protocol component includes a protocol parsing module and a routing protocol package. When a node receives a routing protocol package, the protocol parsing module will parse the type of the protocol package and make a decision to ignore or respond based on the current working mode of the node. All basic routing protocols will exist as processes in the system and be in a suspended state. The routing node will activate the corresponding routing protocol process according to the set mode, and the process will make corresponding operations based on the received and parsed protocol package.

[0013] Multi-mode routing integrates active routing protocols and reactive routing protocols. Multi-mode routing nodes have data structures of both, including routing tables, neighbor tables, connection information storage, and packet buffers. When working, nodes will record link information based on the information in the routing protocol packet and save it in the neighbor table for monitoring component statistics. At a certain moment, the specific functions performed by the node and the modules called are jointly determined by the set mode and the received protocol packet.

[0014] The monitoring component is the execution component of the adaptive algorithm; inside the node, the monitoring component is responsible for controlling the working mode of the node and collecting and counting the network monitoring information of the node. The monitoring component is connected to the external agent through an interface, uploads the statistical information of the node's iteration cycle to the agent, and resets the node's working mode according to the actions performed by the agent. The agent counts and processes the data uploaded by each node, summarizes the status and rewards of the entire system, performs corresponding operations according to the learned strategy, and distributes control instructions to each routing node.

[0015] Specifically, in step S2, the Markov decision process constructed by the flight ad hoc network adaptive multi-mode routing model includes an agent and an environment. The Markov decision process is defined as <S, A, P, R, γ>, where S refers to the state space, A refers to the action space, P refers to the state transition probability matrix, R refers to the reward, and γ is the discount factor.

[0016] State space: Within a certain time range, the drone network environment tends to be stable. At this stage, the current node state can be in working mode m i Uniquely represents and satisfies m i∈M, M is the set of modes of all protocols integrated in the implemented framework (including mode 1, mode 2, mode 3 and the suspended mode 0 that is not connected to the agent). Then, the set of all node states at time t can be expressed as the state of the flying ad hoc network at time t, that is,

[0017] S t =[m1,m2......m N ],

[0018] Where N is the maximum number of supported nodes in the network. Then, the state space S can be defined as

[0019] S={s|s=[m1,m2……,m N ],m i ∈M}.

[0020] Action space: According to the definition of state space, the action space is set to A, which must satisfy S×A→S; due to the characteristics of the system itself, the operation of the network changing from one state to another is unique and definite. Therefore, the action can be uniquely represented by the state pair, that is, action a=(s i ,s j ), a∈A,s i ≠s j ,s i ,s j ∈S. In addition, it is necessary to define the state-invariant operation a 0 ,Right now s∈S, there is a 0 =(s,s). Then, the action space A can be defined as:

[0021] A={a|a=(s i ,s j ),s i ≠s j ,s i ,s j ∈S}∪{a 0}.

[0022] State transition probability: Considering that nodes strictly execute control instructions, however, the new state is still subject to some basic rules, such as nodes in mode 0 cannot execute any commands of the agent. When there is no node in mode 3 in the divided small network, the node with the smallest number will be automatically selected to change its mode to 3, etc. These can be defined as rule f, then the state transition probability is determined as follows:

[0023]

[0024] The reward function refers to: introducing the QoS indicator into the value function, and then constructing the reward function. To construct the reward function, first introduce the relevant definitions:

[0025] Definition 1 (throughput): Throughput is the sum of the payload sizes of UDP packets received by the destination node per unit time:

[0026] TH=∑|Pγ i | / T,

[0027] Where TH represents the average throughput, Pγ i Refers to the data packets received by the destination node within the iteration cycle time T, |Pγ i | represents the payload size of the data packet. After normalization, the throughput part of the cost function is defined as:

[0028] r th =2 / (1+e -TH )-1.

[0029] Definition 2 (Packet Delivery Rate): The packet delivery rate PDR is calculated as the number of packets received by the destination node divided by the number of packets sent by the originating node. It is directly defined as the packet delivery rate part of the reward function:

[0030] r PDR =PDR=∑Pγ i / ∑P i ,

[0031] where Pγ i Refers to the data packets received by the destination node within the iteration cycle time T, P i It is the data packet sent by the source node.

[0032] Definition 3 (end-to-end delay): End-to-end delay is the time required for a data packet to pass through the network from source to destination. The average end-to-end delay is defined as:

[0033] EED=∑(T a (Pγ i )-T s (Pγ i )) / ∑Pγ i ,

[0034] Where T a (Pγ i ) represents the arrival time of the received data packet, T s (Pγ i ) represents the sending time of the received data packet. After normalization, the delay part of the cost function is as follows:

[0035] r d= -2arctan(EED) / π;

[0036] Combining the above parameters, the reward function at time t can be expressed as the formula:

[0037] R t =αr th +βr PDR +ηr d .

[0038] Specifically, in step S3, the deep Q network combines reinforcement learning and deep neural network, learns the strategy through high-dimensional sensor input, and uses the deep convolutional network to approximate the optimal Q-value function:

[0039]

[0040] Furthermore, the basic framework of the present invention is mainly divided into environment, replay memory pool, neural network and loss function. When the system is running, the replay memory pool is first initialized, the network node is connected, the network state s is obtained and input into the neural network, and the action value a is selected using the ε-greedy strategy; the network node adjusts the working mode according to the action a, and the network enters the new state s′, and the reward value is calculated according to the statistical information returned by the network; then the memory <s, a, R, s′> is stored in the memory pool, a random batch of memory is extracted from the replay memory pool, and then the loss function is calculated, and then the gradient descent method is used to minimize the loss function, and the parameters of the current neural network are updated, and the above steps are repeated until the task is completed. The specific algorithm flow is:

[0041] Step 1: Get the network node status m i The default state of the unconnected network node is the suspended state. 0 , generate the network state matrix s;

[0042] Step 2: State s selects action value a according to the ε-greedy strategy, and the agent distributes control commands;

[0043] Step 3: The network node changes its working mode according to the action value a executed in state s, and the entire network enters the new state s′. The agent obtains the statistical information uploaded by the network node and calculates the value according to formula R t =αr th +βr PDR +ηr d Calculation is rewarded;

[0044] Step 4: Store the memory <s,a,R,s′> in the replay memory pool. If there is a consistent memory <s,a,s′>, update the R value;

[0045] Step 5: Draw random batches of samples from the memory pool;

[0046] Step 6: According to the formula Calculate the loss function,

[0047] Step 7: According to the formula Update network parameters;

[0048] Step 8: Repeat the above steps until the task is completed.

[0049] Compared with the prior art, the present invention adopts the above technical solution and has the following beneficial effects:

[0050] 1. The present invention establishes a flying ad hoc network adaptive routing selection method based on deep reinforcement learning, which reduces the system complexity and improves the performance of the entire network.

[0051] 2. The present invention automatically adjusts the routing working mode of each drone through deep reinforcement learning, realizing a mode of decomposing a large network into a main network and several small heterogeneous networks connected to it, thereby ensuring that the local performance is optimal.

[0052] 3. The present invention constructs a mathematical model of Markov decision process MDP according to the application scenario of adaptive multi-mode routing in flight ad hoc networks, constructs a corresponding value function, and introduces QoS indicators into the value function, including system throughput, packet delivery rate and average end-to-end delay, which not only improves the throughput of data transmission, but also ensures end-to-end reliable transmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a schematic diagram of the framework of the multi-mode routing node of the present invention;

[0054] Figure 2 It is a schematic diagram of the working principle of the multi-mode routing of the present invention;

[0055] Figure 3 It is a schematic diagram of the adaptive multi-mode routing model of the flight ad hoc network of the present invention;

[0056] Figure 4 It is a schematic diagram of the adaptive routing selection method of the present invention. DETAILED DESCRIPTION

[0057] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings:

[0058] 1) If Figure 1As shown in the figure, a single routing node is mainly divided into two components, namely the monitoring component and the routing protocol component. The routing protocol component includes a protocol parsing module and a routing protocol package. When a node receives a routing protocol package, the protocol parsing module will parse the type of the protocol package and make a decision to ignore or respond based on the current working mode of the node. All basic routing protocols will exist as processes in the system and be in a suspended state. The routing node activates the corresponding routing protocol process according to the set mode, and the process makes corresponding operations based on the received and parsed protocol package.

[0059] Multimode routing integrates active routing protocols and reactive routing protocols. It is necessary to be able to execute active protocol programs to regularly maintain routing tables, and to be able to execute reactive routing protocol programs to respond to link establishment requests and detect link failures. Therefore, multimode routing nodes must have data structures for both, including routing tables, neighbor tables, connection information storage, and packet buffers. When the node is working, it will record link information based on the information in the routing protocol packet and save it in the neighbor table for monitoring component statistics. At a certain moment, the specific functions performed by the node and the modules called are jointly determined by the set mode and the received protocol packet.

[0060] The monitoring component is the execution component of the adaptive algorithm. Inside the node, the monitoring component is responsible for controlling the working mode of the node and collecting and counting the network monitoring information of the node. The monitoring component is connected to the external agent through an interface, uploads the statistical information of the node's iteration cycle to the agent, and resets the node's working mode according to the actions performed by the agent. The agent counts and processes the data uploaded by each node, summarizes the status and rewards of the entire system, performs corresponding operations according to the learned strategy, and distributes control instructions to each routing node.

[0061] 2) If Figure 2The figure shows the multi-mode working routing, where Figure a shows the initialization of the system, when there are not many nodes and the network nodes are basically in mode 1, that is, the default nodes are in the quasi-on-demand plane distance vector routing mode; Figure b shows that after a period of time, as the number of network nodes increases and the network scale increases, the performance is not high if only relying on mode 1. At this time, the nodes with relatively stable links, slow movement speeds, and adjacent nodes are divided into small networks, and the working state is set to mode 2, that is, the relatively stable local network is converted into a quasi-sequence destination node distance vector routing network, and a node in the small network is selected and set to mode 3 (if there is no control command, the one with the smallest number is selected) as the cluster head of the network. At this time, the main body of the network is the quasi-on-demand plane distance vector routing network composed of nodes in mode 1, and multiple small networks are connected to the quasi-on-demand plane distance vector routing network as a whole with their cluster head nodes as windows. The quasi-sequence destination node distance vector routing subnet in the system does not need to perform the routing discovery process, and the delay is greatly reduced. The number of nodes contained in the quasi-on-demand plane distance vector routing network as the main body of the network is significantly reduced, the routing discovery overhead is reduced, and the performance is also improved.

[0062] 3) If Figure 3 The figure shows the adaptive multi-mode routing model of the flight ad hoc network of the present invention. By introducing the reinforcement learning algorithm, a general intelligent solution is provided for the complex system of the flight ad hoc network. There is no need to establish an accurate system model. The environmental impact can be observed and perceived by the intelligent agent. The intelligent agent directly learns the strategy from the learning, and adjusts the working mode of the network node to achieve the function of adaptive routing optimization.

[0063] The mathematical model of Markov decision process MDP is constructed according to the application scenario of adaptive multi-mode routing in flying ad hoc networks. MDP contains a set of interacting objects, agents and environments. The Markov decision process is defined as <S, A, P, R, γ>, where S refers to the state space, A refers to the action space, P refers to the state transition probability matrix, R refers to the reward, and γ is the discount factor.

[0064] State space: Within a certain time range, the drone network environment tends to be stable. At this stage, the current node state can be in working mode m i Uniquely represents and satisfies m i ∈M, M is the set of modes of all protocols integrated in the implemented framework (including mode 1, mode 2, mode 3 and the suspended mode 0 that is not connected to the agent). Then, the set of all node states at time t can be expressed as the state of the flying ad hoc network at time t, that is,

[0065] S t =[m1,m2……m N ],

[0066] Where N is the maximum number of supported nodes in the network. Then, the state space S can be defined as

[0067] S={s|s=[m1,m2……,m N ],m i ∈M}.

[0068] Action space: According to the definition of state space, the action space is set to A, which must satisfy S×A→S; due to the characteristics of the system itself, the operation of the network changing from one state to another is unique and definite. Therefore, the action can be uniquely represented by the state pair, that is, action a=(s i ,s j ), a∈A,s i ≠s j ,s i ,s j ∈S. In addition, it is necessary to define the state-invariant operation a 0 ,Right now s∈S, there is a 0 =(s,s). Then, the action space A can be defined as:

[0069] A={a|a=(s i ,s j ),s i ≠s j ,s i ,s j ∈S}∪{a 0}.

[0070] State transition probability: Considering that nodes strictly execute control instructions, however, the new state is still subject to some basic rules, such as nodes in mode 0 cannot execute any commands of the agent. When there is no node in mode 3 in the divided small network, the node with the smallest number will be automatically selected to change its mode to 3, etc. These can be defined as rule f, then the state transition probability is determined as follows:

[0071]

[0072] The reward function refers to: introducing the QoS indicator into the value function, and then constructing the reward function. To construct the reward function, first introduce the relevant definitions:

[0073] Definition 1 (throughput): Throughput is the sum of the payload sizes of UDP packets received by the destination node per unit time:

[0074] TH=∑|Pγ i | / T,

[0075] Where TH represents the average throughput, Pγ iRefers to the data packets received by the destination node within the iteration cycle time T, |Pγ i | represents the payload size of the data packet. After normalization, the throughput part of the cost function is defined as:

[0076] r th =2 / (1+e -TH )-1.

[0077] Definition 2 (Packet Delivery Rate): The packet delivery rate PDR is calculated as the number of packets received by the destination node divided by the number of packets sent by the originating node. It is directly defined as the packet delivery rate part of the reward function:

[0078] r PDR =PDR=∑Pγ i / ∑P i ,

[0079] where Pγ i Refers to the data packets received by the destination node within the iteration cycle time T, P i It is the data packet sent by the source node.

[0080] Definition 3 (end-to-end delay): End-to-end delay is the time required for a data packet to pass through the network from source to destination. The average end-to-end delay is defined as:

[0081] EED=∑(T a (Pγ i )-T s (Pγ i )) / ∑Pγ i ,

[0082] Where T a (Pγ i ) represents the arrival time of the received data packet, T s (Pγ i ) represents the sending time of the received data packet. After normalization, the delay part of the cost function is as follows:

[0083] r d = -2arctan(EED) / π;

[0084] Combining the above parameters, the reward function at time t can be expressed as the formula:

[0085] R t =αr th +βr PDR +ηr d .

[0086] The Deep Q Network combines reinforcement learning and deep neural networks to learn strategies through high-dimensional sensor inputs and use deep convolutional networks to approximate the optimal Q-value function:

[0087]

[0088] 4) If Figure 4 The figure shows a schematic diagram of the adaptive routing selection method of the present invention. The basic framework is mainly divided into the environment, replay memory pool, neural network and loss function. When the system is running, the replay memory pool is first initialized, the network node is connected, the network state s is obtained and input into the neural network, and the action value a is selected using the ε-greedy strategy; the network node adjusts the working mode according to the action a, and the network enters the new state s′, and the reward value is calculated according to the statistical information returned by the network; then the memory <s, a, R, s′> is stored in the memory pool, a random batch of memory is extracted from the replay memory pool, and then the loss function is calculated, and then the gradient descent method is used to minimize the loss function, and the parameters of the current neural network are updated, and the above steps are repeated until the task is completed. The specific algorithm flow is:

[0089] Step 1: Get the network node status m i The default state of the unconnected network node is the suspended state. 0 , generate the network state matrix s;

[0090] Step 2: State s selects action value a according to the ε-greedy strategy, and the agent distributes control commands;

[0091] Step 3: The network node changes its working mode according to the action value a executed in state s, and the entire network enters the new state s′. The agent obtains the statistical information uploaded by the network node and calculates the value according to formula R t =αr th +βr PDR +ηr d Calculation is rewarded;

[0092] Step 4: Store the memory <s,a,R,s′> in the replay memory pool. If there is a consistent memory <s,a,s′>, update the R value;

[0093] Step 5: Draw random batches of samples from the memory pool;

[0094] Step 6: According to the formula Calculate the loss function,

[0095] Step 7: According to the formula Update network parameters;

[0096] Step 8: Repeat the above steps until the task is completed.

[0097] The main protection technical points of this invention are:

[0098] (1) An adaptive switching method for routing protocols in large-scale network application scenarios.

[0099] (2) By automatically adjusting the routing working mode of each UAV through the intelligent agent, a large network can be decomposed into a main network and several small heterogeneous networks connected to it, ensuring that the local performance is optimal.

[0100] (3) A Markov decision model was constructed, and the relevant state space, action space, and value function were defined. The QoS indicator was introduced into the value function, which improved the data transmission throughput and ensured end-to-end reliable transmission.

[0101] (4) The framework of the adaptive routing method is implemented through the deep Q network DQN. It does not require complicated data processing and can directly learn strategies through high-dimensional sensor input. It not only improves the data transmission throughput but also ensures end-to-end reliable transmission.

[0102] The present invention is suitable for large-scale network application scenarios. The scheme can be used to perform adaptive switching of routing protocols, and by building a Markov decision model combined with a DQN network, the data transmission throughput is increased while ensuring the real-time performance of the transmission.

[0103] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.

[0104] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for adaptive routing selection in a flying ad hoc network based on deep reinforcement learning, characterized in that: The following steps are involved: S1: Establish a multi-modal routing framework; S2: Construct an adaptive multi-mode routing model for flight ad hoc networks; S3: Design an adaptive routing selection method for flying ad hoc networks based on deep reinforcement learning according to the deep Q network.

2. The distributed routing method for flight ad hoc networks based on deep reinforcement learning according to claim 1, characterized in that: A single routing node in the multi-mode routing framework described in step S1 includes a monitoring component and a routing protocol component; The routing protocol component includes a protocol parsing module and a routing protocol package. When a node receives a routing protocol package, the protocol parsing module will parse the type of the protocol package and make a decision to ignore or respond based on the current working mode of the node. All basic routing protocols will exist as processes in the system and be in a suspended state. The routing node activates the corresponding routing protocol process according to the set mode, and the process performs corresponding operations based on the received and parsed protocol package. Multi-mode routing integrates active routing protocols and reactive routing protocols. Multi-mode routing nodes have data structures of both, including routing tables, neighbor tables, connection information storage, and packet buffers. When working, nodes will record link information based on the information in the routing protocol packet and save it in the neighbor table for monitoring component statistics. At a certain moment, the specific functions performed by the node and the modules called are determined by the set mode and the received protocol packet. The monitoring component is the execution component of the adaptive algorithm. Inside the node, the monitoring component is responsible for controlling the working mode of the node and collecting and statisticizing the network monitoring information of the node. The monitoring component is connected to the external intelligent agent through an interface, uploads the statistical information of the node's iteration cycle to the intelligent agent, and resets the node's working mode according to the actions performed by the intelligent agent; the intelligent agent counts and processes the data uploaded by each node, summarizes the status and rewards of the entire system, performs corresponding operations according to the learned strategy, and distributes control instructions to each routing node.

3. The distributed routing method for flight ad hoc networks based on deep reinforcement learning according to claim 1, characterized in that: The Markov decision process constructed in the flight ad hoc network adaptive multi-mode routing model described in step S2 includes an agent and an environment; the Markov decision process is defined as <S, A, P, R, γ>, where S refers to the state space, A refers to the action space, P refers to the state transition probability matrix, R refers to the reward, and γ is the discount factor; State space: Within a certain time range, the drone network environment tends to be stable; at this stage, the state of the current node can be in working mode m i Uniquely represents and satisfies m i ∈M, M is the set of modes of all protocols integrated in the implemented framework (including mode 1, mode 2, mode 3 and the suspended mode 0 that is not connected to the agent); then, the set of all node states at time t can be expressed as the state of the flying ad hoc network at time t, that is, S t =[m1,m2......m N ], Where N is the maximum number of supported nodes in the network; then, the state space S can be defined as S={s|s=[m1,m2……,m N ],m i ∈M}; Action space: According to the definition of state space, the action space is set to A, which must satisfy S×A→S. Due to the characteristics of the system itself, the operation of the network changing from one state to another is unique and definite. Therefore, the action can be uniquely represented by the state pair, that is, action a=(s i ,s j ), a∈A,s i ≠s j ,s i ,s j ∈S; In addition, it is necessary to define the state-invariant operation a 0 ,Right now There is a 0 =(s,s); then, the action space A can be defined as: A={a|a=(s i ,s j ),s i ≠s j ,s i ,s j ∈S}∪{a 0 }; State transition probability: Considering that the nodes strictly execute the control instructions, however, the new state is still subject to some basic rules, such as nodes in mode 0 cannot execute any commands of the agent; when there is no node in mode 3 in the divided small network, the node with the smallest number will be automatically selected to change its mode to 3, etc. These can be defined as rule f, then the state transition probability is determined as follows: The reward function refers to: introducing the QoS indicator into the value function, and then constructing the reward function; to construct the reward function, first introduce the relevant definitions: Definition 1 (throughput): Throughput is the sum of the payload sizes of UDP packets received by the destination node per unit time: TH=∑|Pγ i | / T, Where TH represents the average throughput, Pγ i Refers to the data packets received by the destination node within the iteration cycle time T, |Pγ i | represents the payload size of the data packet; after normalization, the throughput part of the cost function is defined as: r th ＝2 / (1+e -TH )-1； Definition 2 (Packet Delivery Rate): The packet delivery rate PDR is calculated as the number of packets received by the destination node divided by the number of packets sent by the origin node; it is directly defined as the packet delivery rate part of the reward function: r PDR =PDR=∑Pγ i / ∑P i , where Pγ i Refers to the data packets received by the destination node within the iteration cycle time T, P i It is the data packet sent by the source node; Definition 3 (end-to-end delay): End-to-end delay is the time required for a data packet to pass through the network from source to destination. The average end-to-end delay is defined as: EED=∑(T a (Pγ i )-T s (Pγ i )) / ∑Pγ i , Where T a (Pγ i ) represents the arrival time of the received data packet, T s (Pγ i ) represents the sending time of the received data packet; after normalization, the delay part in the cost function is as follows: r d =-2arctan(EED) / π; Combining the above parameters, the reward function at time t can be expressed as the formula: R t =αr th +βr PDR +ηr d 。 4. The distributed routing method for flight ad hoc networks based on deep reinforcement learning according to claim 1, characterized in that: The deep Q network described in step S3 combines reinforcement learning and deep neural networks, learns strategies through high-dimensional sensor input, and uses deep convolutional networks to approximate the optimal Q-value function:

5. The distributed routing method for flying ad hoc networks based on deep reinforcement learning according to claims 1-4, characterized in that: The basic framework is mainly divided into environment, replay memory pool, neural network and loss function. When the system is running, the replay memory pool is first initialized, the network node is connected, the network state s is obtained and input into the neural network, and the action value a is selected using the ε-greedy strategy. The network node adjusts the working mode according to the action a, and the network enters the new state s′. The reward value is calculated according to the statistical information returned by the network. Then the memory <s,a,R,s′> is stored in the memory pool, a random batch of memory is extracted from the replay memory pool, and then the loss function is calculated. The gradient descent method is used to minimize the loss function, and the parameters of the current neural network are updated. The above steps are repeated until the task is completed. The specific algorithm flow is as follows: Step 1: Get the network node status m i The default state of the unconnected network node is the suspended state. 0 , generate the network state matrix s; Step 2: State s selects action value a according to the ε-greedy strategy, and the agent distributes control commands; Step 3: The network node changes its working mode according to the action value a executed in state s, and the entire network enters the new state s′. The agent obtains the statistical information uploaded by the network node and calculates the value according to formula R t =αr th +βr PDR +ηr d Calculation is rewarded; Step 4: Store the memory <s,a,R,s′> in the replay memory pool. If there is a consistent memory <s,a,s′>, update the R value; Step 5: Draw random batches of samples from the memory pool; Step 6: According to the formula Calculate the loss function, Step 7: According to the formula Update network parameters; Step 8: Repeat the above steps until the task is completed.

Images