A method, system, device and medium for unmanned aerial vehicle network intelligent routing based on D3QN

Abstract

This invention provides a method, system, device, and medium for intelligent routing of unmanned aerial vehicle (UAV) networks based on D3QN. The method includes representing the UAV network topology as an undirected graph, which includes a set of UAV nodes and communication link geometry; selecting a routing path based on bandwidth load; calculating the maximum bandwidth utilization and transmission latency of the routing path; and constructing a UAV network routing decision problem model. This model is capable of handling latency-sensitive battlefield services, such as real-time control and target tracking data transmission, thus ensuring the model's reliability. Then, the fully distributed multi-agent D3QN algorithm is used to model the UAV network routing decision problem model as a partially observable Markov decision process, forming a deep reinforcement learning training framework. Training is performed based on this deep reinforcement learning training framework to obtain the final routing decision result for the real target under test.

Description

Technical Field

[0001] This invention belongs to the technical field of intelligent routing for unmanned aerial vehicle (UAV) networks, specifically relating to a method, system, device, and medium for intelligent routing of UAV networks based on D3QN. Background Technology

[0002] With the deep application of information and intelligent technologies in the military field, modern air warfare is accelerating its transformation towards intelligence and collaboration. The performance leaps of manned / unmanned aerial platforms and the empowerment of intelligent technologies are driving the evolution of air combat systems from single platforms to networked and autonomous systems. Under this trend, unmanned aerial vehicles (UAVs), with their high maneuverability, strong stealth, low cost, and high cost-effectiveness, have rapidly become a core force on the modern battlefield. However, factors such as equipment failure, electromagnetic interference, or enemy tactical strikes often lead to frequent fluctuations or even complete failures of communication links, severely weakening system responsiveness and potentially causing mission interruptions and irreparable battlefield losses. Faced with the core challenges of highly dynamic UAV network topologies, easily interrupted links, and limited node resources, recent years have seen research on routing strategies focusing on load balancing and QoS assurance, resulting in typical solutions. Among these, the ARdeep protocol proposed by Liu et al. is a significant advancement. This protocol models routing decisions as a Markov decision process, implements distributed routing selection through a Deep Q Network (DQN), and comprehensively considers multi-dimensional characteristics such as the distance between neighboring nodes and the destination, and the expected link connection time in the state design. It also introduces the concept of "reliable distance" as the core indicator of the reward function, thereby achieving adaptive routing in environments with frequent topology changes. Yu Jun et al. proposed a game theory-based QoS-aware UAV swarm wireless communication routing method. This method establishes a buffer queue queuing model for UAV ad hoc networks and periodically collects and statistically analyzes QoS characteristics such as the hot service types and congestion status of node relays. The statistical results are used as a comprehensive criterion for path selection, which effectively improves network throughput and scheduling efficiency.

[0003] However, existing technologies lack the ability to jointly optimize multi-dimensional routing performance indicators for drone swarms. Most focus on local parameters such as node location or channel status, lacking multi-dimensional integration of network performance indicators such as link latency and bandwidth utilization. This makes it difficult to simultaneously meet the low latency and congestion prevention requirements in multi-service concurrent scenarios. Specifically:

[0004] (1) Existing technologies have high information perception costs. Most of them require the collection of the entire network link status information of the UAV cluster before they can make routing decisions. The information transmission overhead is large and the routing decision delay is high, making it difficult to achieve lightweight deployment in UAV nodes with limited resources. Existing technologies lack the ability to jointly optimize multi-dimensional routing performance indicators of UAV clusters. Most of them focus on local parameters such as node location or channel status, and lack multi-dimensional integration of network performance indicators such as link latency and bandwidth utilization. It is difficult to take into account the low latency and congestion prevention requirements in multi-service concurrent scenarios.

[0005] (2) As the scale of UAV clusters continues to expand, the design requirements for multi-formation collaborative combat scenarios are increasing, and the communication network environment is becoming more and more complex. The traditional DQN algorithm model adopts a scheme of maximizing the Q value of a single network to guide decision-making, which is prone to introducing overestimation bias and is very easy to accumulate in the dynamic and ever-changing battlefield environment, affecting the routing decision performance. Summary of the Invention

[0006] To address the challenge of balancing low latency and congestion prevention in multi-service concurrent scenarios during multi-dimensional routing of drone swarms in existing technologies, this invention provides a method, system, device, and medium for intelligent routing of drone networks based on D3QN.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] In a first aspect, embodiments of this disclosure provide a method for intelligent routing of unmanned aerial vehicle (UAV) networks based on D3QN, comprising the following steps:

[0009] Step S1: Represent the UAV network topology as an undirected graph, which includes the set of UAV nodes and the geometry of communication links, and select a routing path based on bandwidth load;

[0010] Step S2: Calculate the maximum bandwidth utilization and transmission delay of the routing path, and construct a model for the UAV network routing decision problem;

[0011] Step S3: The UAV network routing decision problem model is modeled as a partially observable Markov decision process using the fully distributed multi-agent D3QN algorithm, forming a deep reinforcement learning training framework. The model is then trained based on the deep reinforcement learning training framework to obtain the final routing decision result of the real target to be tested.

[0012] Furthermore, the undirected graph include:

[0013] ;

[0014] In the formula, Represents a set of drone nodes. , It is the first A drone, It represents the total number of drones in the network; Represents a set of communication links. , Indicates drone and drones The link between them;

[0015] When selecting a routing path based on bandwidth load, the following steps are included:

[0016] Step A1: Calculate bandwidth utilization based on transmission delay and total bandwidth ,include:

[0017] ;

[0018] In the formula, It is a link The bandwidth currently in use is [not specified]. It is a link Total bandwidth, bandwidth utilization Used to reflect the status of link load and bandwidth utilization. The higher the value, the heavier the link load and the higher the bandwidth utilization. The smaller the value, the lighter the link load;

[0019] Step A2: Select the minimum bandwidth utilization The link is used as the routing path, and the routing path includes drone elements. and links drone elements , ,in, The drone representing the source node, This refers to a relay drone at an intermediate node. This represents the command center or attack drone of the target node; drone element. The two ends are used as source nodes. and destination node The drones, command centers, and attack drones all serve as intermediate nodes. relay drones, links , ,in, Indicates the first step that constitutes the routing path. One link.

[0020] Furthermore, the model for the UAV network routing decision problem includes the following steps:

[0021] Step B1: Calculate the transmission delay of the routing path. ,include:

[0022] ;

[0023] In the formula, Indicates the first Link Transmission delay;

[0024] Step B2: Calculate the maximum bandwidth utilization of the routing path. ,include:

[0025] ;

[0026] In the formula, Indicates the k-th link Bandwidth utilization;

[0027] Step B3: Transmission delay based on the routing path and maximum bandwidth utilization The model for the UAV network routing decision problem is obtained, including:

[0028] ;

[0029] ;

[0030] ;

[0031] In the formula, and All are weighting coefficients; for low-latency business data, increase the weighting coefficient. To ensure low latency, for large-volume business data, increase To ensure bandwidth utilization; Weight constraints; For delay constraints; This indicates the total end-to-end path delay limit for communication services.

[0032] Furthermore, the deep reinforcement learning training framework performs the following training steps:

[0033] Step C1: Randomly generate source nodes in the UAV network topology routing decision problem model. and destination node As a routing requirement for routing decisions, it originates from the source node. Start making routing decisions one by one, obtain the routing decision results for each training round, and pass the current routing requirement to the next training round;

[0034] Step C2: After each routing decision, based on the total reward function Obtain the reward value for the current decision and record the current action. and the corresponding state The state of the next jump and reward value As an experience sample, it is stored in the experience replay pool;

[0035] Step C3: In each training round, multiple empirical samples are randomly sampled from the empirical replay pool and input into the estimation network. The deviation value is calculated based on the TD deviation, and the Q value of the estimation network is updated based on the deviation value. The target network copies all network parameters of the estimation network every preset training steps. And assign the value to the corresponding network parameter. until the destination node is reached. One routing session ends;

[0036] Step C4: Repeatedly generate source nodes randomly in the UAV network topology routing decision problem model. and destination node Repeat steps C1-C3 for training. After reaching the preset number of training rounds, the optimal advantage function is obtained. and optimal state value function And based on the optimal advantage function and optimal state value function The final routing decision result of the actual target to be tested is obtained.

[0037] Furthermore, in step C1, when obtaining the routing decision result for each training round, it is based on... - Greedy strategy for routing decisions:

[0038] ;

[0039] in, For training steps; As a regulating factor; This is the probability threshold, with a value ranging from 0 to 1; The state for the next jump: The next jump action;

[0040] Among them, drone nodes Current state ,include:

[0041] ;

[0042] In the formula, Indicates whether the destination node has been reached. ; Indicates the current drone node to the destination node The shortest number of hops; Representing neighboring nodes to the destination node The shortest number of hops; and Representing the current drone nodes to neighboring nodes Link latency and bandwidth; Represents the current drone node to neighboring nodes Link bandwidth utilization;

[0043] drone nodes Current action ,include:

[0044] ;

[0045] In the formula, Indicates the neighbor node number, when When, it indicates the selection of the first. neighboring nodes As the action for the next jump.

[0046] Furthermore, the total reward function ,include:

[0047] ;

[0048] In the formula, , and All are positive weights and satisfy the following conditions: ; This represents the time-delay-related reward function. This represents the load-related reward function. This represents the routing-related reward function. For action space The corresponding reward function;

[0049] Wherein, the time-delay related reward function ,include:

[0050] ;

[0051] In the formula, It is a proportionality coefficient. It is a negative value, used to penalize non-compliance. The required decision, Quality of service (QoS) for routing metrics is determined based on different communication service transmission requirements, including latency, throughput, and packet loss rate. This indicates the minimum number of hops in the current end-to-end service transmission;

[0052] The load-related reward function ,include:

[0053] ;

[0054] In the formula, It is the proportionality coefficient;

[0055] The routing-related reward function ,include:

[0056] ;

[0057] In the formula, It is the proportionality coefficient.

[0058] Furthermore, the estimation network and the target network are dual. Network structure, estimating the network's current usage The network is used to select actions; the target network uses a copy of the current one. Updates received from the network The network is used to compute the updated estimate. Estimated value;

[0059] Based on target network estimated value The TD deviation calculation formula is used. The deviation values ​​are obtained, including:

[0060] ;

[0061] In the formula, This represents the sample size value for random sampling. This indicates the computation of states in the target network. Take the next jump action. The goal obtained later Estimate the value and choose the state in the next hop. The largest target network generated below estimated value The corresponding action becomes the next action. ;

[0062] Among them, the target network estimated value ,include:

[0063] ;

[0064] In the formula, Indicates the state Take the current action The reward value obtained; To estimate the current state of the network Network parameters; Updates for the target network The network parameters; γ∈[0,1] represents the discount factor;

[0065] Based on the parameters of the estimated network Parameters of the target network When performing an update, the following are included:

[0066] ;

[0067] In the formula, This represents the adjustment factor, used to control the network parameter update. The degree of influence of the value;

[0068] Optimal state value function ,include:

[0069] ;

[0070] In the formula, Indicates the adoption of a strategy Time state Corresponding value;

[0071] Optimal advantage function ,include:

[0072] ;

[0073] In the formula, the optimal advantage function Representing the optimal Q-function and the optimal state value function The difference; This represents the optimal Q-function.

[0074] In a second aspect, embodiments of this disclosure provide a system for intelligent routing of unmanned aerial vehicle (UAV) networks based on D3QN, comprising:

[0075] The initialization unit is configured to represent the UAV network topology as an undirected graph, which includes a set of UAV nodes and communication link geometry, and select routing paths based on bandwidth load.

[0076] The preprocessing unit is configured to calculate the maximum bandwidth utilization and transmission delay of the routing path, and to construct a model for the UAV network routing decision problem.

[0077] The output unit is configured to model the UAV network routing decision problem model as a partially observable Markov decision process using the fully distributed multi-agent D3QN algorithm, forming a deep reinforcement learning training framework, and to train based on the deep reinforcement learning training framework to obtain the final routing decision result of the real target to be tested.

[0078] In a third aspect, embodiments of this disclosure provide an electronic device, characterized in that the electronic device comprises:

[0079] At least one processor; and,

[0080] The memory is communicatively connected to the at least one processor; wherein,

[0081] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the D3QN-based intelligent routing method for unmanned aerial vehicle networks.

[0082] In a fourth aspect, embodiments of this disclosure provide a non-transitory computer-readable storage medium, characterized in that the non-transitory computer-readable storage medium stores computer instructions for causing the computer to execute the method for intelligent routing of UAV networks based on D3QN.

[0083] Compared with the prior art, the present invention has the following beneficial technical effects:

[0084] This invention provides a method, system, device, and medium for intelligent routing of unmanned aerial vehicle (UAV) networks based on D3QN. The method includes representing the UAV network topology as an undirected graph, which includes a set of UAV nodes and communication link geometry; selecting a routing path based on bandwidth load; calculating the maximum bandwidth utilization and transmission delay of the routing path; and constructing a UAV network routing decision problem model. This model is capable of handling latency-sensitive battlefield operations, such as real-time control and target tracking data transmission, thus ensuring the model's reliability. Then, the fully distributed multi-agent D3QN algorithm is used to model the UAV network routing decision problem model as a partially observable Markov decision process, forming a deep reinforcement learning training framework. Training is performed based on this deep reinforcement learning training framework to obtain the real target data for testing. The final routing decision result of the target is achieved by employing a fully distributed multi-agent D3QN algorithm, where each UAV node is treated as an independent agent. Each agent only needs to collect state information from neighboring nodes and links. Due to the distributed deployment architecture, the agents on each UAV node only need to decide on the next-hop node and forward the routing request information after receiving it. The next-hop node then repeats this process until the destination node is reached, ensuring the smooth execution of routing decisions and improving network resilience and adaptability. Furthermore, modeling the routing decision problem for each UAV as a partially observable Markov decision process allows each agent to train independently without sharing parameters and decision information, interacting only indirectly through link states. This avoids the risk of being intercepted by the enemy due to frequent information exchange, reduces communication overhead, improves resource utilization, and is more adaptable to the highly dynamic battlefield environment. This method enables each UAV node to learn from local perception and historical data under decentralized control, solving the problems of high signaling overhead and high latency in routing decisions for large-scale networks. Based on fully distributed multi-agent D3QN, this method jointly optimizes link latency and link utilization to achieve efficient and intelligent routing decisions, avoiding focusing on a single link factor. This allows for the uniform load distribution of a cluster of hundreds of UAVs while ensuring low latency, avoiding local congestion, and improving the stability of the UAV communication network. Attached Figure Description

[0085] Figure 1 A flowchart of a method for intelligent routing of unmanned aerial vehicle networks based on D3QN, according to an embodiment of this disclosure, is shown.

[0086] Figure 2 A block diagram of a method for intelligent routing of unmanned aerial vehicle networks based on D3QN, according to an embodiment of this disclosure, is shown.

[0087] Figure 3 A path latency performance comparison chart of embodiments of this disclosure is shown;

[0088] Figure 4A performance comparison chart of maximum link utilization of embodiments of this disclosure is shown;

[0089] Figure 5 A diagram of an apparatus for a method of intelligent routing of unmanned aerial vehicle (UAV) networks based on D3QN, according to an embodiment of this disclosure, is shown. Detailed Implementation

[0090] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0091] Figure 1 A flowchart 100 of a method for intelligent routing of unmanned aerial vehicle (UAV) networks based on D3QN, as provided in the first embodiment of this application, is shown. Figure 1 As shown, it includes the following steps:

[0092] In step S101, the UAV network topology is represented as an undirected graph, which includes a set of UAV nodes and communication link geometry, and a routing path is selected based on bandwidth load.

[0093] Specifically, the undirected graph include:

[0094] ;

[0095] In the formula, Represents a set of drone nodes. , It is the first A drone, It represents the total number of drones in the network; Represents a set of communication links. , Indicates drone and drones The link between them;

[0096] Specifically, when selecting a routing path based on bandwidth load, the following steps are included:

[0097] Step A1: Calculate bandwidth utilization based on transmission delay and total bandwidth ,include:

[0098] ;

[0099] In the formula, It is a link The bandwidth currently in use is [not specified]. It is a link Total bandwidth, bandwidth utilization Used to reflect the status of link load and bandwidth utilization. The higher the value, the heavier the link load and the higher the bandwidth utilization. The smaller the value, the lighter the link load;

[0100] Step A2: Select the minimum bandwidth utilization The link is used as the routing path, and the routing path includes drone elements. and links drone elements , ,in, The drone representing the source node, This refers to a relay drone at an intermediate node. This represents the command center or attack drone of the target node; drone element. The two ends are used as source nodes. and destination node The drones, command centers, and attack drones all serve as intermediate nodes. relay drones, links , ,in, Indicates the first step that constitutes the routing path. One link.

[0101] Next, proceed to step S102.

[0102] In step S102, the maximum bandwidth utilization and transmission delay of the routing path are calculated, and a model for the UAV network routing decision problem is constructed.

[0103] Specifically, in modern warfare, UAV network routing algorithms need to plan paths for arbitrary source-destination node pairs, such as reconnaissance UAVs to command centers or attack UAVs to target location nodes, under dynamic topology and interference environments. The core objective is to reduce link load and avoid communication interruptions caused by local congestion while ensuring that transmission latency meets the real-time requirements of the battlefield. Therefore, this embodiment includes the following steps in constructing the UAV network routing decision problem model:

[0104] Step B1: Calculate the transmission delay of the routing path. ,include:

[0105] ;

[0106] In the formula, Indicates the first Link Transmission delay;

[0107] Step B2: Calculate the maximum bandwidth utilization of the routing path. ,include:

[0108] ;

[0109] In the formula, Indicates the k-th link Bandwidth utilization;

[0110] Step B3: Transmission delay based on the routing path and maximum bandwidth utilization The model for the UAV network routing decision problem is obtained, including:

[0111] ;

[0112] ;

[0113] ;

[0114] In the formula, and All are weighting coefficients; for low-latency business data, increase the weighting coefficient. To ensure low latency, for large-volume business data, increase To ensure bandwidth utilization; Weight constraints; For delay constraints; This indicates the total end-to-end path delay limit for communication services.

[0115] Next, proceed to step S103.

[0116] In step S103, the fully distributed multi-agent D3QN algorithm is used to model the UAV network routing decision problem as a partially observable Markov decision process, forming a deep reinforcement learning training framework. The training is performed based on the deep reinforcement learning training framework, and the final routing decision result of the real target to be tested is obtained.

[0117] Specifically, such as Figure 2 As shown, the deep reinforcement learning training framework performs the following training steps:

[0118] Step C1: Randomly generate source nodes in the UAV network topology routing decision problem model. and destination node As a routing requirement for routing decisions, it originates from the source node. Start making routing decisions one by one, obtain the routing decision results for each training round, and pass the current routing requirement to the next training round;

[0119] Step C2: After each routing decision, based on the total reward function Obtain the reward value for the current decision and record the current action. and the corresponding state The state of the next jump and reward value As an experience sample, it is stored in the experience replay pool;

[0120] Step C3: In each training round, multiple empirical samples are randomly sampled from the empirical replay pool and input into the estimation network. The deviation value is calculated based on the TD deviation, and the Q value of the estimation network is updated based on the deviation value. The target network copies all network parameters of the estimation network every preset training steps. And assign the value to the corresponding network parameter. until the destination node is reached. One routing session ends;

[0121] Step C4: Repeatedly generate source nodes randomly in the UAV network topology routing decision problem model. and destination node Repeat steps C1-C3 for training. After reaching the preset number of training rounds, the optimal advantage function is obtained. and optimal state value function And based on the optimal advantage function and optimal state value function The final routing decision result of the actual target to be tested is obtained.

[0122] It should be noted that, in solving the aforementioned UAV network topology routing decision problem model, a fully distributed multi-agent architecture performs well to adapt to the dynamic changes in UAV network topology and the uncertainty of battlefield interference. Compared with centralized routing algorithms, this architecture does not require the collection of global network information. Each UAV node, as an independent agent, only needs to collect the state information of neighboring nodes and links, such as neighbor node load, link latency, and interference intensity, to make autonomous decisions. Due to the distributed deployment architecture, the agents on each UAV node only need to decide the next-hop node after receiving a routing request and forward the routing request information. The next-hop node then repeats this process until the destination node is reached, ensuring the smooth execution of routing decisions and improving network resilience and adaptability.

[0123] The routing decision problem for each UAV is modeled as a partially observable Markov decision process (POMDP). Each agent is trained independently and does not share parameters and decision information. They only interact indirectly through link states, which avoids the risk of being intercepted by the enemy due to frequent information exchanges. At the same time, it can reduce communication overhead, improve resource utilization, and is more adaptable to the highly dynamic battlefield environment.

[0124] Considering the state-space continuity of the unmanned aerial vehicle (UAV) network topology routing decision problem model, the two The Q-function of a network structure cannot be explicitly represented, therefore a neural network is used to fit the Q-function. In the DQN algorithm, during iterative training, the agent gradually tends to choose the action with the largest current Q-value, and the Q-value continues to participate in Q-value iteration as the neural network backpropagates. This leads to the continuous accumulation of estimation errors, eventually causing Q-value overestimation. By introducing a dual... The network structure is optimized by creating a copy of the Q-network with an identical structure as the target network. The current network is responsible for selecting actions, while the target network is responsible for calculating and updating the target's Q-value. This separates action selection from value estimation, reducing the impact of overestimating the Q-value on the original network. Every so often or during training epochs, the current network parameters are copied to the target network and kept unchanged in the next epoch. This delayed update measure also largely avoids the bootstrapping error and overestimation error of the traditional DQN algorithm, improving algorithm performance.

[0125] In this embodiment, at the start of training, the experience replay pool, state space, current network, and target network are first initialized. Source and destination node pairs are randomly generated in the UAV network topology routing decision problem model as routing decision requirements. Routing decisions are made sequentially starting from the source node agent, and the routing requirements are passed to the next hop node agent. In step C1, the routing decision results for each training epoch are obtained based on... - Greedy strategy for routing decisions:

[0126] ;

[0127] in, For training steps; As a regulating factor; The probability threshold is a value ranging from 0 to 1, and a random probability threshold is generated in each iteration. and with Whether to use a random strategy or an algorithmic strategy to determine the size of the comparison will increase with the number of training rounds; The state for the next jump: The next jump action;

[0128] Among them, drone nodes Current state ,include:

[0129] ;

[0130] In the formula, Indicates whether the destination node has been reached. ; Indicates the current drone node to the destination node The shortest number of hops; Representing neighboring nodes to the destination node The shortest number of hops; and Representing the current drone nodes to neighboring nodes Link latency and bandwidth; Represents the current drone node to neighboring nodes Link bandwidth utilization;

[0131] drone nodes Current action ,include:

[0132] ;

[0133] In the formula, Indicates the neighbor node number, when When, it indicates the selection of the first. neighboring nodes As the action for the next jump.

[0134] In this embodiment, the total reward function ,include:

[0135] ;

[0136] In the formula, , and All are positive weights and satisfy the following conditions: ; This represents the time-delay-related reward function. This represents the load-related reward function. This represents the routing-related reward function. For action space The corresponding reward function;

[0137] Wherein, the time-delay related reward function ,include:

[0138] ;

[0139] In the formula, It is a proportionality coefficient. It is a negative value, used to penalize non-compliance. The required decision, Quality of service (QoS) for routing metrics is determined based on different communication service transmission requirements, including latency, throughput, and packet loss rate. This indicates the end-to-end path total latency limit for communication services. This indicates the minimum number of hops in the current end-to-end service transmission;

[0140] The load-related reward function ,include:

[0141] ;

[0142] In the formula, It is the proportionality coefficient;

[0143] The routing-related reward function ,include:

[0144] ;

[0145] In the formula, It is the proportionality coefficient.

[0146] Specifically, in each iteration, several experience samples are randomly and repeatedly drawn from the experience replay pool and input into each agent. The current state and action are input into the estimation network, and the next state is input into the target network, thereby exploring the latency and bandwidth attributes of each link in the network environment. The estimation network decides the next hop node according to a greedy strategy, and based on the obtained reward, refers to the temporarily fixed optimization objective of the target network, calculates the deviation value according to the TD deviation formula, and updates the network parameters through gradient descent. The network parameters of the estimation network are transmitted to the target network for replication every 10 training iterations. At the same time, the next hop node decided by the estimation network will serve as the action input environment and receive state and reward feedback from the environment. The state corresponding to the current action, the next hop state, and the total reward value are recorded as experience samples and stored in the experience replay pool. Finally, the algorithm converges after completing the specified number of training iterations.

[0147] The estimation network and the target network are dual. Network structure, estimating the network's current usage The network is used to select actions; the target network uses a copy of the current one. Updates received from the network The network is used to compute the updated estimate. Estimated value;

[0148] Based on target network estimated value The TD deviation calculation formula is used. The deviation values ​​are obtained, including:

[0149] ;

[0150] In the formula, This represents the sample size value for random sampling. This indicates the computation of states in the target network. Take the next jump action. The goal obtained later Estimate the value and choose the state in the next hop. The largest target network generated below estimated value The corresponding action becomes the next action. ;

[0151] Among them, the target network estimated value ,include:

[0152] ;

[0153] In the formula, Indicates the state Take the current action The reward value obtained; To estimate the current state of the network Network parameters; Updates for the target network The network parameters; γ∈[0,1] represents the discount factor;

[0154] Based on the parameters of the estimated network Parameters of the target network When performing an update, the following are included:

[0155] ;

[0156] In the formula, This represents the adjustment factor, used to control the network parameter update. The degree of influence of the value;

[0157] Adversarial networks improve the DQN network structure by introducing an optimal advantage function, thus enabling dual-advantage networks to achieve their intended purpose. The output of the network structure is decomposed into two subnetworks, each representing the optimal dominance function. and optimal state value function .

[0158] Among them, the optimal state value function Defined as the state under the optimal policy The corresponding value includes:

[0159] ;

[0160] In the formula, Indicates the adoption of a strategy Time state Corresponding value;

[0161] Optimal advantage function Defined as the difference between the optimal Q-function and the optimal state-value function, including:

[0162] ;

[0163] In the formula, the optimal advantage function Representing the optimal Q-function and the optimal state value function The difference; This represents the optimal Q-function, which refers to the bipolar function that is achieved when a preset number of training epochs are reached. Network structure.

[0164] Finally, the output parameters of the neural network are decomposed into:

[0165] ;

[0166] in These are parameters specific to the optimal advantage network; These are parameters specific to the optimal state value network; These are network parameters for the common parts.

[0167] By splitting the double Network structure, extracting dual The portion of the network structure update and iteration process that improves the state value function helps the agent deeply learn how different actions contribute to increasing the reward value, which has strong practical application significance. Regarding the above formula... and Adding and subtracting the same number will yield the same result. This will lead to the optimal advantage function and optimal state value function The network parameters are not unique, causing instability in the training process. Therefore, the optimal dominance function can be forced. If the value at the selected action is 0, the above formula can be rewritten as:

[0168] ;

[0169] Taking into account habitual experience, The averaging operation is often more efficient than the maximization operation. Therefore, the following formula is usually used to replace the above formula, resulting in the Q-value update formula for the D3QN algorithm:

[0170] .

[0171] At the start of training, the agent tends to randomly select the next-hop node to explore the latency and bandwidth attributes of each link in the network environment. As training progresses, the agent's probability of selecting the action with the highest Q-value as the next hop increases. After each decision, the agent records the state corresponding to the current action, the next-hop state, and the reward as experience and stores them in the experience replay pool. In each training round, several experience samples are randomly sampled from the experience replay pool and input into the estimation network. The network weights are updated according to the deviation value obtained from the TD deviation calculation formula. The target network updates the network weights from the estimation network every fixed number of training steps. This process continues until the destination node is reached, at which point one routing cycle ends. The random generation of source and destination node pairs is repeated for training. After completing the specified number of training rounds, the algorithm converges, resulting in the KMDE-D3QN UAV network intelligent clustering routing algorithm. Specific simulation comparison results are given below.

[0172] In this embodiment, the neural network has four layers, including one input layer, two output layers, and one hidden layer. The ReLU function is used as the activation function, and the Adam optimizer is employed. The discount factor is set to 0.99, the empirical replay pool size is set to 10000, a pre-designed UAV network topology is imported, and link delay and load parameters are randomly generated. The network is trained using the multi-agent D3QN algorithm, with 10000 randomly generated data streams as training data. After the algorithm converges, a routing decision model is obtained. 100 source-destination node pairs are generated to cover different inter-formation communication scenarios. Routing tests are performed using Dijkstra's algorithm, Dijkstra-QU, and the method proposed in this embodiment. Simulation results are presented. Figure 3 and Figure 4 As shown, Dijkstra's shortest path algorithm calculates the shortest path latency, but its maximum bandwidth utilization is the highest, indicating that the algorithm has poor load balancing capabilities for UAV nodes. The Dijkstra-QU algorithm optimizes bandwidth utilization, resulting in a better load situation for the path, but the path latency is relatively high, which also fails to meet the low latency requirements of UAV networks. In contrast, the method proposed in this embodiment ensures low latency while also effectively responding to network load conditions, reducing the maximum bandwidth utilization of links, and achieving joint optimization of routing path latency and load performance, demonstrating excellent practical application value.

[0173] The second embodiment of the present invention also provides a system for intelligent routing of unmanned aerial vehicle (UAV) networks based on D3QN, comprising:

[0174] The initialization unit is configured to represent the UAV network topology as an undirected graph, which includes a set of UAV nodes and communication link geometry, and select routing paths based on bandwidth load.

[0175] The preprocessing unit is configured to calculate the maximum bandwidth utilization and transmission delay of the routing path, and to construct a model for the UAV network routing decision problem.

[0176] The output unit is configured to model the UAV network routing decision problem model as a partially observable Markov decision process using the fully distributed multi-agent D3QN algorithm, forming a deep reinforcement learning training framework, and to train based on the deep reinforcement learning training framework to obtain the final routing decision result of the real target to be tested.

[0177] The third embodiment of the present invention also provides an electronic device, the electronic device comprising:

[0178] At least one processor; and,

[0179] The memory is communicatively connected to the at least one processor; wherein,

[0180] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the D3QN-based intelligent routing method for unmanned aerial vehicle networks according to any of the foregoing embodiments.

[0181] The fourth embodiment of the present invention also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the method for intelligent routing of UAV networks based on D3QN as described in any of the foregoing embodiments.

[0182] The fifth embodiment of the present invention also provides a computer program product, which includes a computing program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions that, when executed by a computer, cause the computer to perform the method of intelligent routing of UAV networks based on D3QN according to any of the foregoing embodiments.

[0183] Figure 5 The illustration shows a method or device 1000 implementing an embodiment of the present invention. In some embodiments, more or fewer devices may be included than illustrated. In some embodiments, it may be implemented using a single or multiple devices. In some embodiments, it may be implemented using cloud-based or distributed devices.

[0184] like Figure 5As shown, device 1000 includes a processor 1001 for performing various appropriate operations and processes based on programs and / or data stored in read-only memory (ROM) 1002 or programs and / or data loaded from storage portion 1008 into random access memory (RAM) 1003. Processor 1001 may be a multi-core processor or may contain multiple processors. In some embodiments, processor 1001 may include a general-purpose main processor and one or more special coprocessors, such as a central processing unit (CPU), graphics processing unit (GPU), neural network processor (NPU), digital signal processor (DSP), etc. Various programs and data required for the operation of device 1000 are also stored in RAM 1003. Processor 1001, ROM 1002, and RAM 1003 are interconnected via bus 1004. Input / output (I / O) interface 1005 is also connected to bus 1004.

[0185] The processor and memory described above are used together to execute a program stored in the memory. When the program is executed by a computer, it can implement the methods, steps, or functions described in the above embodiments.

[0186] The following components are connected to I / O interface 1005: an input section 1006 including a keyboard, mouse, touchscreen, etc.; an output section 1007 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to I / O interface 1005 as needed. A removable medium 1011, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 1010 as needed so that computer programs read from it can be installed into storage section 1008 as needed. Figure 5 The diagram only shows a portion of the components and does not imply that the device 1000 only includes... Figure 5 The components shown.

[0187] The systems, devices, modules, or units described in the above embodiments can be implemented by a computer or its associated components. The computer may be, for example, a mobile terminal, smartphone, personal computer, laptop computer, in-vehicle human-machine interface device, personal digital assistant, media player, navigation device, game console, tablet computer, wearable device, smart TV, Internet of Things system, smart home, industrial computer, server, or a combination thereof.

[0188] Although not shown, in this embodiment of the invention, a computer-readable storage medium is provided having a computer program / instruction stored thereon, which, when executed by a processor, implements the method for intelligent routing of UAV networks based on D3QN as described in the embodiment.

[0189] Storage media in embodiments of the present invention include articles that are permanent and non-permanent, removable and non-removable, capable of storing information by any method or technology. Examples of storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0190] Although not shown, embodiments of the present invention also provide a computer program product, including: a computer program / instructions that, when executed by a processor, implement the method for intelligent routing of UAV networks based on D3QN as described in the embodiments.

[0191] The methods, programs, systems, apparatuses, etc., in embodiments of the present invention can be executed or implemented in one or more networked computers, or practiced in a distributed computing environment. In the embodiments of this specification, in these distributed computing environments, tasks can be performed by remote processing devices connected via a communication network.

Claims

1. A method for intelligent routing in unmanned aerial vehicle (UAV) networks based on D3QN, characterized in that, Includes the following steps: Step S1: Represent the UAV network topology as an undirected graph, which includes the set of UAV nodes and the geometry of communication links, and select a routing path based on bandwidth load; Step S2: Calculate the maximum bandwidth utilization and transmission delay of the routing path, and construct a model for the UAV network routing decision problem; Step S3: The UAV network routing decision problem model is modeled as a partially observable Markov decision process using the fully distributed multi-agent D3QN algorithm, forming a deep reinforcement learning training framework. The model is then trained based on the deep reinforcement learning training framework to obtain the final routing decision result of the real target to be tested.

2. The method for intelligent routing of unmanned aerial vehicle (UAV) networks based on D3QN according to claim 1, characterized in that, The undirected graph include: ； In the formula, Represents a set of drone nodes. , It is the first A drone, It represents the total number of drones in the network; Represents a set of communication links. , Indicates drone and drones The link between them; When selecting a routing path based on bandwidth load, the following steps are included: Step A1: Calculate bandwidth utilization based on transmission delay and total bandwidth ,include: ； In the formula, It is a link The bandwidth currently in use is [not specified]. It is a link Total bandwidth, bandwidth utilization Used to reflect the status of link load and bandwidth utilization. The higher the value, the heavier the link load and the higher the bandwidth utilization. The smaller the value, the lighter the link load; Step A2: Select the minimum bandwidth utilization The link is used as the routing path, and the routing path includes drone elements. and links drone elements , ,in, The drone representing the source node, This refers to a relay drone at an intermediate node. This represents the command center or attack drone of the target node; drone element. The two ends are used as source nodes. and destination node The drones, command centers, and attack drones all serve as intermediate nodes. relay drones, links , ,in, Indicates the first step that constitutes the routing path. One link.

3. The method for intelligent routing of unmanned aerial vehicle networks based on D3QN according to claim 2, characterized in that, The steps involved in constructing the model for the UAV network routing decision problem are as follows: Step B1: Calculate the transmission delay of the routing path. ,include: ； In the formula, Indicates the first Link Transmission delay; Step B2: Calculate the maximum bandwidth utilization of the routing path. ,include: ； In the formula, Indicates the k-th link Bandwidth utilization; Step B3: Transmission delay based on the routing path and maximum bandwidth utilization The model for the UAV network routing decision problem is obtained, including: ； ； ； In the formula, and All are weighting coefficients; for low-latency business data, increase the weighting coefficient. To ensure low latency, for large-volume business data, increase To ensure bandwidth utilization; Weight constraints; For delay constraints; This indicates the total end-to-end path delay limit for communication services.

4. The method for intelligent routing of unmanned aerial vehicle networks based on D3QN according to claim 3, characterized in that, The deep reinforcement learning training framework performs the following training steps: Step C1: Randomly generate source nodes in the UAV network topology routing decision problem model. and destination node As a routing requirement for routing decisions, it originates from the source node. Start making routing decisions one by one, obtain the routing decision results for each training round, and pass the current routing requirement to the next training round; Step C2: After each routing decision, based on the total reward function Obtain the reward value for the current decision and record the current action. and the corresponding state The state of the next jump and reward value As an experience sample, it is stored in the experience replay pool; Step C3: In each training round, multiple experience samples are randomly sampled from the experience replay pool and input into the estimation network. The deviation value is calculated based on the TD deviation, and the Q value of the estimation network is updated based on the deviation value. The target network copies all network parameters of the estimation network every preset number of training steps. And assign the value to the corresponding network parameter. until the destination node is reached. One routing session ends; Step C4: Repeatedly generate source nodes randomly in the UAV network topology routing decision problem model. and destination node Repeat steps C1-C3 for training. After reaching the preset number of training rounds, the optimal advantage function is obtained. and optimal state value function And based on the optimal advantage function and optimal state value function The final routing decision result of the actual target to be tested is obtained.

5. The method for intelligent routing of unmanned aerial vehicle networks based on D3QN according to claim 4, characterized in that, In step C1, the routing decision result for each training round is obtained based on... - Greedy strategy for routing decisions: ； in, For training steps; As a regulating factor; This is the probability threshold, with a value ranging from 0 to 1; The state for the next jump: The next jump action; Among them, drone nodes Current state ,include: ； In the formula, Indicates whether the destination node has been reached. ; Indicates the current drone node to the destination node The shortest number of hops; Representing neighboring nodes to the destination node The shortest number of hops; and Representing the current drone nodes to neighboring nodes Link latency and bandwidth; Represents the current drone node to neighboring nodes Link bandwidth utilization; drone nodes Current action ,include: ； In the formula, Indicates the neighbor node number, when When, it indicates the selection of the first. neighboring nodes As the action for the next jump.

6. The method for intelligent routing of unmanned aerial vehicle networks based on D3QN according to claim 4, characterized in that, The total reward function ,include: ； In the formula, , and All are positive weights and satisfy the following conditions: ; This represents the time-delay-related reward function. This represents the load-related reward function. This represents the routing-related reward function. For action space The corresponding reward function; Wherein, the time-delay related reward function ,include: ； In the formula, It is a proportionality coefficient. It is a negative value, used to penalize non-compliance. The required decision, Quality of service (QoS) for routing metrics is determined based on different communication service transmission requirements, including latency, throughput, and packet loss rate. This indicates the minimum number of hops in the current end-to-end service transmission; The load-related reward function ,include: ； In the formula, It is the proportionality coefficient; The routing-related reward function ,include: ； In the formula, It is the proportionality coefficient.

7. The method for intelligent routing of unmanned aerial vehicle networks based on D3QN according to claim 3, characterized in that, The estimation network and the target network are dual. Network structure, estimating the network's current usage The network is used to select actions; the target network uses a copy of the current one. Updates received from the network The network is used to compute the updated estimate. Estimated value; Based on target network estimated value The TD deviation calculation formula is used. The deviation values ​​are obtained, including: ； In the formula, This represents the sample size value for random sampling. This indicates the computation of states in the target network. Take the next jump action. The goal obtained later Estimate the value and choose the state in the next hop. The largest target network generated below estimated value The corresponding action becomes the next action. ; Among them, the target network estimated value ,include: ； In the formula, Indicates the state Take the current action The reward value obtained; To estimate the current state of the network Network parameters; Updates for the target network The network parameters; γ∈[0,1] represents the discount factor; Based on the parameters of the estimated network Parameters of the target network When performing an update, the following are included: ； In the formula, This represents the adjustment factor, used to control the network parameter update. The degree of influence of the value; Optimal state value function ,include: ； In the formula, Indicates the adoption of a strategy Time state Corresponding value; Optimal advantage function ,include: ； In the formula, the optimal advantage function Representing the optimal Q-function and the optimal state value function The difference; This represents the optimal Q-function.

8. A system for intelligent routing of unmanned aerial vehicle (UAV) networks based on D3QN, characterized in that, The method for intelligent routing of unmanned aerial vehicle (UAV) networks based on D3QN according to any one of claims 1-7 includes: The initialization unit is configured to represent the UAV network topology as an undirected graph, which includes a set of UAV nodes and communication link geometry, and select routing paths based on bandwidth load. The preprocessing unit is configured to calculate the maximum bandwidth utilization and transmission delay of the routing path, and to construct a model for the UAV network routing decision problem. The output unit is configured to model the UAV network routing decision problem model as a partially observable Markov decision process using the fully distributed multi-agent D3QN algorithm, forming a deep reinforcement learning training framework, and to train based on the deep reinforcement learning training framework to obtain the final routing decision result of the real target to be tested.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and, The memory is communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method for intelligent routing of unmanned aerial vehicle networks based on any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions for causing the computer to perform the method of intelligent routing of unmanned aerial vehicle networks based on D3QN as described in any one of claims 1 to 7.

Images