A blockchain-enabled intelligent trusted routing method for drone networks

Abstract

This paper discloses a blockchain-enabled intelligent trusted routing method for drone networks. This method aims to minimize the total end-to-end routing latency. By building a trusted network environment through a blockchain-based trust management mechanism, the trusted routing problem is reformulated as a distributed partially observable Markov decision process. This method utilizes a multi-agent deep reinforcement learning algorithm to solve the problem, taking into account the multidimensional observable information of nodes. This method does not rely on difficult-to-obtain global information, is adaptable to dynamic environments, and offers low computational cost. It can obtain trusted routing paths in a relatively short time, effectively addressing the trusted routing problem in drone networks where malicious nodes exist due to deliberate attacks and other situations.

Description

Technical Field

[0001] The present invention relates to the field of drone routing technology, and in particular to a blockchain-enabled intelligent and trusted routing method for drone networks. Background Art

[0002] As a key component of the integrated air-space-ground network of 6G communication technology, drones have experienced rapid development and are widely used in a variety of tasks, including disaster relief and real-time monitoring. In these applications, drones can serve as aerial base stations for data collection and transmission, providing low-cost, flexible, and versatile services. However, due to the complex application environments, high dynamics, and distributed topology of drone networks, they are vulnerable to malicious attacks. When malicious drones are present, hostile activity can lead to degraded routing performance. To ensure drone safety, it is necessary to strengthen research on drone security technologies and applications to improve their safety and reliability.

[0003] Evaluating and managing malicious drone behavior is an effective approach. However, some current work focuses on establishing a node trust table through a centralized manager based on a single node. However, this manager lacks fault tolerance and is easily tampered with. Therefore, to improve routing security, blockchain technology has been introduced to establish a decentralized trust mechanism. Specifically, blockchain is a distributed ledger composed of blocks of blocks that stores transaction details and records, making it difficult to tamper with.

[0004] Drone routing algorithms are a key research area for ensuring drone security. In scenarios where multiple drones must collaborate to complete a mission, routing algorithms can coordinate data transmission and reception between them and plan the shortest, most secure data transmission paths to improve the efficiency and security of multi-drone collaboration. The distributed topology of drone networks makes them vulnerable to attacks that can disrupt routing, making the design of efficient, adaptive, and dynamic routing algorithms crucial. Summary of the Invention

[0005] The purpose of this invention is to propose a blockchain-enabled intelligent trusted routing method for drone networks. This method aims to minimize the total end-to-end routing latency. By building a trusted network environment through a blockchain-based trust management mechanism, the trusted routing problem is reformulated as a distributed partially observable Markov decision process. This method utilizes a multi-agent deep reinforcement learning algorithm to solve the problem, taking into account the multidimensional observable information of nodes. This method does not rely on difficult-to-obtain global information, is adaptable to dynamic environments, and offers low computational cost. It can obtain trusted routing paths in a relatively short time, effectively addressing the trusted routing problem in drone networks where malicious nodes exist due to deliberate attacks and other situations.

[0006] In order to achieve the above technical objectives, the technical solution adopted by the present invention is:

[0007] A blockchain-enabled intelligent and trusted routing method for drone networks, comprising the following steps:

[0008] S1. Design a network architecture for a time-varying drone swarm. Use the network architecture to describe a trusted routing scenario with malicious drones having low trust values. This includes: dividing the time period into T time slots. In each time slot, determine the movement of drones, normal drone nodes and malicious drone nodes, the routing data to be transmitted and its source-destination drone pair, the communication range and network topology link status of each drone node, and the original and recovered trusted routing paths. The source and destination drone nodes are both normal drone nodes.

[0009] S2, based on the communication range of each drone node, select normal drone nodes within its communication range from its neighboring drone nodes and use them as candidate forwarding drone nodes; based on the candidate forwarding drone node set of each normal drone node, analyze all routing paths from the source drone node to the destination drone node in the network architecture and use them as candidate routing paths;

[0010] S3, a deliberate attack model is designed. Based on the node importance ranking results, drone nodes that are more important to the network are attacked first. This simulates the scenario where an attacker causes network performance to degrade faster by prioritizing key nodes, thereby constructing a drone network environment with malicious nodes. The importance of nodes is evaluated by calculating the node degree and link weight value.

[0011] S4, in response to the presence of malicious nodes in the drone network, a decentralized node trust evaluation mechanism based on blockchain is designed to manage drone trust values. Some drones with high security values ​​are selected as consensus nodes to maintain and update node trust values. Drones with trust values ​​below the threshold are removed to update normal drone nodes and malicious drone nodes in the network, thus building a trusted network environment.

[0012] S5, based on the trusted network environment built on the blockchain-based trust management mechanism, designs a free space path loss model between UAV nodes. Under the premise that the total energy consumption of UAV movement and communication cannot exceed the UAV energy constraint, the maximum allowable single-hop transmission rate between each UAV node and the next-hop UAV node in each candidate routing path is obtained;

[0013] S6, based on the communication distance between each UAV node and the next-hop UAV node and the number of data packets waiting to be transmitted in the next-hop UAV node, calculates the single-hop data transmission delay and queuing delay respectively, and then calculates the end-to-end delay model from the source UAV node to the destination UAV node based on the routing path; transforms the routing problem of the UAV network into a problem of minimizing the end-to-end delay, and constructs the delay minimization objective function;

[0014] S7 uses a distributed partially observable Markov decision process to reconstruct the delay minimization objective function problem, and uses a multi-agent deep reinforcement learning algorithm to solve the minimization objective function problem modeled as a distributed partially observable Markov decision process, obtaining the optimal strategy with the smallest end-to-end delay value. The routing path selected based on the optimal strategy restores the data transmission between the source drone node and the destination drone node.

[0015] Step S1 further comprises:

[0016] Suppose N drone nodes are randomly and evenly deployed in the drone swarm network, and the network is modeled as a weighted undirected graph in: U r and denote the source, relay and destination UAV nodes respectively, i∈U represents UAV node i; ε={e ij ,i=1,2,…,N,j=1,2,…,N},e ij ∈{0, 1} is a binary variable that represents the link status between drone node i and drone node j. If e ij =1, indicating that there is a communication link between the two UAV nodes. If e ii =0, indicating that there is no communication link between the two UAV nodes;

[0017] Using tuples Represents a request r∈R, where and L r Represent the source drone, destination drone and data size of the request r respectively; Indicates that the drone is from the source To the corresponding destination drone The complete routing path for the transmission request r.

[0018] Step S2 further comprises:

[0019] The time period is divided into T time slots, and the length of each time slot is τ. At the beginning of time slot t∈T, the UAV adjusts its direction by accelerating or decelerating instantaneously, and then flies at a constant speed in a fixed flight direction.

[0020] The position of drone node i at time t is expressed as coordinates Indicates that when the Euclidean distance between drone nodes i and j is Greater than d max When the value is , the two drone nodes cannot communicate; and when Less than d min When the value is , a collision will occur; then the candidate forwarding drone node set of each drone node i

[0021] Furthermore, in step S3, the node importance is calculated using the following formula:

[0022]

[0023] in, Represents the UAV node i to link e ii The important contribution represents the node degree of drone node i.

[0024] Furthermore, in step S4, the trust value evaluation mechanism of each drone is:

[0025]

[0026] in, and They represent the initial trust value of UAV i, the trust value based on the delivery rate, and the trust value based on the correct transmission line, respectively. and Respectively and The weight value of and Expressed as:

[0027]

[0028]

[0029] in, and They represent the number of requests received and sent by UAV i respectively; and They represent the number of non-designated transmission paths and the total number of transmission paths of UAV i in time slot t, respectively.

[0030] Furthermore, in step S4, the process of designing a blockchain-based decentralized node trust evaluation model to manage the drone trust value includes the following steps:

[0031] Select the 3n+1 drones with the largest trust value as the consensus node c∈U c, and select the drone with the highest trust value as the leader. The consensus drone set and candidate consensus drone set are denoted as U c and

[0032] In the consensus drone set U c After confirmation, the transaction consensus process begins. The non-consensus drone uploads the transaction record information to the nearest consensus drone, which forwards it to the leader for verification. The leader initiates a round of negotiation among all consensus drones. In order to reduce the communication load between consensus nodes, a U is executed every M rounds of consensus. c Update process; if there is a drone c∈U c The trust value is lower than the safety threshold Then it is from U c Delete it from the candidate set Invite a new consensus drone to complete the update of the formula node.

[0033] Furthermore, in step S5, based on the trusted network environment constructed based on the trust management mechanism of blockchain, a free space path loss model between drone nodes is designed to calculate the communication channel transmission rate between two drones i and j.

[0034]

[0035] in, and are the channel bandwidth and signal-to-noise ratio between UAVs i and j, respectively; in, and are transmission power and noise power respectively; is the channel path loss value, which is in, represents the path loss exponent, is the distance between UAVs i and j, f c represents the carrier frequency, c is the speed of light; in addition, the transmission capacity constraint is expressed as: in, Expressed as the data length of the requirement r∈R, is a binary variable indicating whether the demand r passes through the link e(i, j) at time slot t:

[0036]

[0037] Total energy consumption of drones Satisfy the constraints: in represents the energy capacity of UAV i, and β is the empirical power efficiency.

[0038] Furthermore, in step S6, the single-hop data transmission delay and queuing delay are calculated based on the communication distance between each drone node and the next-hop drone node and the amount of data packets waiting to be transmitted in the next-hop drone node; when a request r∈R arrives at drone i, the queuing delay for transmission to the next-hop drone j is Expressed as:

[0039]

[0040] in, is the set of queued requests of drone i at time slot t, Expressed as the data length of the requirement r, is a binary variable indicating whether the demand r passes through the link e(i, k) in time slot t. is the transmission rate of the communication channel between two UAVs i and k, and the single-hop transmission delay of request r from UAV i to UAV j in time slot t is for:

[0041]

[0042] The total single-hop delay for transmitting request r from UAV i to UAV j in time slot t is:

[0043]

[0044] in, The value cannot exceed the maximum permissible delay of a single hop Right now:

[0045]

[0046] When the drone is sourced Drone to destination Routing path Determined, end-to-end latency The calculation formula is:

[0047]

[0048] Each source drone Can only communicate with one other drone:

[0049]

[0050] in, is a binary variable indicating whether the demand r passes the link at time slot t

[0051] For the intermediate relay UAV j at time t, t∈U r , the flow conservation satisfies:

[0052]

[0053] A request can only choose one routing path, namely:

[0054]

[0055] Destination Drone It can only receive a request r from another drone i, that is:

[0056]

[0057] in, is a binary variable indicating whether the demand r passes the link at time slot t

[0058] The objective function is to minimize the total delay of all requests r∈R, which can be expressed as:

[0059]

[0060]

[0061]

[0062]

[0063]

[0064]

[0065] in, It means that the drone nodes avoid forwarding data through malicious nodes.

[0066] Furthermore, in step S7, the distributed partially observable Markov decision process is composed of five tuples Composition, of which:

[0067] Agent Collection represents all agents, i.e. all drones;

[0068] The process of obtaining the state space S is:

[0069] At the beginning of each time slot t, each agent i∈U collects and obtains its own observation data from the environment Each drone can only observe neighbor nodes k∈Θ i ;

[0070] The observation data of agent i is:

[0071]

[0072] Where, is the position coordinate of agent k in time period t, It is energy consumption, Is the length of the data packet waiting to be transmitted. Defined as C k and l k They represent the number and size of packets queued for transmission by agent k respectively;

[0073] After obtaining the observation data of all agents, the joint state s at time t is summarized t , recorded as The state space S is represented as S = {s t , t∈T};

[0074] The process of obtaining the action space A is:

[0075] definition in, Represents a single-hop action of drone node i with k as the next-hop target. The size of the action space corresponds to the set Θ i The number of drone nodes in the ; the joint action of N agents at time t is The action space is A={a t , t∈T};

[0076] Reward value function Represents the instantaneous reward set of all agents, where the goal is to minimize the total delay of the routing path of the request r∈R from the source drone node to the destination drone node. Let the current state s t Next, agent i takes action a at time slot t t The instantaneous reward is:

[0077]

[0078] in, is the single-hop delay of transmitting request r from UAV i to UAV j in time slot t;

[0079] Discount factor γ = {γ i , i∈U} is the discount factor set of all agents. The larger the value of γ, the more emphasis is placed on long-term returns. i ∈(0, 1) is the discount factor when calculating the cumulative reward.

[0080] Furthermore, in step S7, the multi-agent deep reinforcement learning algorithm is used to solve the optimal strategy π that minimizes the end-to-end delay value. *, which corresponds to finding the optimal behavior value function value Q of the corresponding UAV node in each state of the distributed partially observable Markov decision process * , and go to the next state;

[0081] Each state s t The ε-greedy method is used to select action a t , randomly select an action from the action space with a preset probability value ε, or select the action that produces the maximum behavior value function value Q with a probability value 1-ε max action, where ε∈(0,1) is the probability of exploring the action;

[0082] The Q-value function is approximated by the online network and target network of agent i, with the observed state as input and the action as output. The parameters of the online network and the target network are θ i and By continuously updating the weights θ of the online network i , the loss function of agent i is minimized, calculating the Q function With Q target The mean square error between:

[0083]

[0084] The gradient of the loss function is recorded as Update θ by gradient descent i :

[0085]

[0086] Every W steps, the parameter Updated regularly to match the online network i Parameters:

[0087]

[0088] Compared with the prior art, the present invention has the following beneficial effects:

[0089] First, the blockchain-enabled drone network intelligent trusted routing method of the present invention proposes a blockchain-based trust evaluation mechanism, a node trust evaluation mechanism and a consensus drone update mechanism, which effectively improves the security of using the Byzantine algorithm and thus improves the security of the routing process.

[0090] Second, the blockchain-enabled intelligent trusted routing method for drone networks of the present invention depicts the routing process in a time-varying drone network with malicious nodes. To address the challenge of obtaining global information in a decentralized drone network, the routing problem is reconstructed into a distributed partially observable Markov decision process, and an algorithm based on multi-agent deep reinforcement learning is proposed to solve the problem.

[0091] Third, the blockchain-enabled drone network intelligent trusted routing method of the present invention was simulated in a drone network with attacked malicious nodes. The results showed that the designed algorithm based on multi-agent deep reinforcement learning is superior to other algorithms in many aspects such as convergence performance, delay, throughput and queue length, effectively solving the trusted routing problem in networks with malicious node drones. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] Figure 1 Schematic diagram of routing recovery in a time-varying UAV network according to the present invention;

[0093] Figure 2 Schematic diagram of the trusted practical Byzantine fault-tolerant algorithm involved in the present invention; wherein (a) shows the node update process when there is a malicious consensus unmanned aerial vehicle (f=1); (b) shows a schematic diagram of the practical Byzantine fault-tolerant consensus process;

[0094] Figure 3 This is an experimental simulation diagram of the relationship between the number of requests and the cumulative rewards involved in the present invention;

[0095] Figure 4 This is an experimental simulation diagram of the relationship between routing delay and number of drones for the three algorithms involved in the present invention;

[0096] Figure 5 This is an experimental simulation diagram of the relationship between network throughput and the number of drones and data transmission requests for the three algorithms involved in the present invention;

[0097] Figure 6 This is an experimental simulation diagram of the relationship between routing delay, number of malicious drones, and number of drones involved in different algorithms of the present invention;

[0098] Figure 7 This is an experimental simulation diagram of the relationship between the average end-to-end delay and the number of malicious drones under different network scales involved in the present invention;

[0099] Figure 8 This is an experimental simulation diagram of the relationship between the average queue length and request quantity of each drone in drone networks of different sizes involved in the present invention. DETAILED DESCRIPTION

[0100] The embodiments of the present invention are described in further detail below with reference to the accompanying drawings.

[0101] Figure 1 A schematic diagram of routing recovery in a time-varying network with malicious drones is shown in a specific embodiment. The present invention discloses a blockchain-enabled intelligent and trusted routing method for drone networks, comprising the following steps:

[0102] Step 1) Design a network architecture for a time-varying drone swarm. The network architecture is used to describe a trusted routing scenario involving malicious drones with low trust values. This involves dividing the time period into T time slots. In each time slot, the movement of drones, normal and malicious drone nodes, the routing data to be transmitted and its source-destination drone pair, the communication range and network topology link status of each drone node, and the original and recovered trusted routing paths are determined.

[0103] In the method of this embodiment, both the source drone node and the destination drone node are normal drone nodes.

[0104] According to the communication range of each drone node, the drone nodes within its communication range are screened out from its adjacent drone nodes and used as candidate forwarding drone nodes.

[0105] Based on the candidate forwarding drone node set of each normal drone node, all routing paths from the source drone node to the destination drone node in the network architecture are analyzed and used as candidate routing paths.

[0106] Step 2), design a deliberate attack model, calculate the node importance by evaluating the node degree and link weight, and attack the drone nodes with high importance first according to the ranking.

[0107] Step 3) In response to the presence of malicious nodes in the drone network, a blockchain-based decentralized node trust evaluation model is designed to manage the drone trust value. Specifically, some drones with high security values ​​are selected as consensus nodes to maintain and update the node trust value, and drones with trust values ​​below the threshold are removed.

[0108] Step 4) Based on the free space path loss model between UAV nodes, the maximum allowable single-hop transmission rate between each UAV node and the next-hop UAV node in each candidate routing path is obtained; at the same time, the total energy consumption of UAV movement and communication cannot exceed the UAV energy constraint.

[0109] Step 5) Based on the communication distance between each UAV node and the next-hop UAV node and the number of packets queued for transmission at the next-hop UAV node, the single-hop data transmission delay and queuing delay are calculated. The end-to-end delay model from the source UAV node to the destination UAV node is then calculated based on the routing path. The routing problem of the UAV network is transformed into a problem of minimizing the above end-to-end delay, and a delay minimization objective function is constructed.

[0110] Step 6) Use a distributed partially observable Markov decision process to reconstruct the minimization objective function problem, and use a multi-agent deep reinforcement learning algorithm to solve the minimization objective function problem modeled as a distributed partially observable Markov decision process, and obtain a strategy with the minimum end-to-end delay value. The routing path selected based on the optimal strategy is used to restore the data transmission between the source drone node and the destination drone node.

[0111] Figure 1 A schematic diagram of routing recovery in a time-varying network with malicious drones in accordance with a specific embodiment is shown.

[0112] In the above process:

[0113] In step 1), N drone nodes are randomly and evenly deployed in the network of the drone swarm, and the network is modeled as a weighted undirected graph Among them: U=U s ∩U r ∩U d , U s 、U r and U d denote the source, relay and destination UAV nodes respectively, i∈U represents UAV node i; ε={e ij ,i=1,2,…,N,j=1,2,…,N},e ij ∈{0, 1} is a binary variable that represents the link status between drone node i and drone node j. If e ij =1, indicating that there is a communication link between the two UAV nodes. If e ij = 0, indicating that there is no communication link between the two UAV nodes. The request r∈R is represented as a tuple in and L r denote the source drone, destination drone and data size of request r respectively. In addition, Indicates that the drone is from the source To the corresponding destination drone The complete routing path for the transmission request r.

[0114] The time period is divided into T time slots, and the length of each time slot is τ. Assume that the drone can adjust its direction by accelerating or decelerating at the beginning of time slot t∈T, and then fly at a constant speed in a fixed flight direction. The position of the drone node i at time t is represented by the coordinates Indicates that when the Euclidean distance between drone nodes i and j is Greater than d max When the value is , the two drone nodes cannot communicate; and when Less than d min When the value is , a collision will occur; therefore, the candidate forwarding drone node set of each drone node i is

[0115] In step 2), a UAV network environment with malicious nodes is constructed through the intentional attack model. The importance of nodes is calculated by evaluating the node degree and link weight, and the UAV nodes with high importance are attacked first. Note f and U f Represent the number and set of malicious drones respectively. The node importance is defined as:

[0116]

[0117] in, Represents the UAV node i to link e ij The important contribution represents the node degree of drone node i.

[0118] In step 3), the malicious behavior of drones was analyzed from two aspects. When malicious drones receive requests, they may discard the requests or transmit the requests not according to the specific routing path. Therefore, considering the delivery rate and the correctness of the transmission path, the trust value evaluation mechanism for each drone is designed as follows:

[0119]

[0120] in, and They represent the initial trust value of drone i, the trust value based on delivery rate, and the trust value based on transmission correctness, respectively. and Respectively and The weight value of .

[0121] In order to improve the security of the Byzantine fault-tolerant consensus algorithm, a trusted Byzantine fault-tolerant consensus method based on the drone update consensus mechanism is proposed:

[0122] First, select the 3n+1 drones with the largest trust value as the consensus node c∈U c, and select the drone with the highest trust value as the leader. The consensus drone set and candidate consensus drone set are denoted as U c and

[0123] in U c After confirmation, the transaction consensus process begins, which mainly includes three stages: pre-preparation, preparation, and submission. Specifically, the non-consensus drone uploads the transaction recording information to the nearest consensus drone, which forwards it to the leader consensus drone for verification. The leader then initiates the three-stage process among all consensus drones. If M rounds of consensus are reached, the U c Update process. If there is a UAV c∈U c The trust value is lower than the safety threshold Then it is from U c Then, from the candidate set Invite a new consensus drone to complete the update of the formula node. Figure 2 A schematic diagram of a trusted practical Byzantine fault-tolerant algorithm is shown, including: (a) a diagram showing the node update process when there is a (f=1) malicious consensus UAV and (b) a diagram showing the practical Byzantine fault-tolerant consensus process.

[0124] In step 4), the communication channel transmission rate between two drones i and j is calculated based on the free space path loss model between drone nodes. The transmission capacity constraint is expressed as:

[0125]

[0126] Considering the UAV's mobility energy and communication energy, the total energy consumption of the UAV is Satisfy the constraints:

[0127]

[0128] In step 5), the single-hop data transmission delay and queuing delay are calculated based on the communication distance between each drone node and the next-hop drone node and the number of packets waiting to be transmitted in the next-hop drone node. When a request r∈R arrives at drone i, the queuing delay for transmission to the next-hop drone j is Expressed as:

[0129]

[0130] in, is the set of queued requests of UAV i at time slot t, and the single-hop transmission delay of request r from UAV i to UAV j at time slot t for:

[0131]

[0132] Therefore, the total single-hop delay for transmitting request r from UAV i to UAV j in time slot t is:

[0133]

[0134] in, The value cannot exceed the maximum permissible delay of a single hop Right now

[0135]

[0136] When the drone is sourced Drone to destination Routing path Determined, end-to-end latency The calculation formula is:

[0137]

[0138] Each source drone Can only communicate with one other drone:

[0139]

[0140] For the intermediate relay drone t ∈U r , flow conservation should satisfy:

[0141]

[0142] In addition, a request can only choose one routing path, i.e.

[0143]

[0144] Similar to the source drone, the destination drone It can only receive the demand r from another UAV i, i.e.

[0145]

[0146] The objective function is to minimize the total delay of all requests r∈R, which can be expressed as:

[0147]

[0148]

[0149]

[0150]

[0151]

[0152]

[0153] in, It means that the drone nodes avoid forwarding data through malicious nodes;

[0154] In step 6), the distributed partially observable Markov decision process consists of five tuples Composition, of which:

[0155] (a) Agent Set represents all agents, i.e. all drones;

[0156] (b) State space S: At the beginning of each time slot t, each agent i∈U collects and obtains its own observation data from the environment It should be noted that each drone can only observe neighbor nodes k∈Θ i ;

[0157] Therefore, the observation results of agent i are:

[0158]

[0159] is the position coordinate of agent k in time period t, It is energy consumption, Is the length of the data packet waiting to be transmitted. Defined as C k and l k They represent the number and size of packets queued for transmission by agent k respectively;

[0160] After obtaining the observation data of all agents, the joint state s at time t is summarized t , recorded as The state space S is represented as S = {s t , t∈T};

[0161] (c) Action space A: Definition in, Represents a single-hop action of drone node i with k as the next-hop target. The size of the action space corresponds to the set Θ i The number of drone nodes in the ; the joint action of N agents at time t is The action space is A={a t , t∈T};

[0162] (d) Reward value function Indicates that agent i performs an action at time slot t The instantaneous reward is to minimize the total delay of the routing path of the request r∈R from the source drone node to the destination drone node. Let the current state s t Next take action a t The instantaneous reward is:

[0163]

[0164] (e) Discount factor γ: γ i ∈(0,1) is the discount factor when calculating the cumulative reward, γ={γ i , i∈U} is the set of discount factors for all agents. The larger the value of γ, the more emphasis is placed on long-term returns;

[0165] Step 6) Use the multi-agent deep reinforcement learning algorithm to solve the optimal strategy π that minimizes the end-to-end delay value * , which corresponds to finding the optimal behavior value function value Q of the corresponding UAV node in each state of the distributed partially observable Markov decision process * , and go to the next state;

[0166] Each state s t The ε-greedy method is used to select action a t , randomly select an action from the action space with a preset probability value ε, or select the action that produces the maximum behavior value function value Q with a probability value 1-ε max action, where ε∈(0,1) is the probability of exploring the action.

[0167] The DDQN algorithm proposes to approximate the Q-value function through a DNN. With the observation state as input and the action as output, there are two DNNs in the DDQN algorithm, namely the online network of agent i and the target network, with parameters θ and θ respectively. i and By continuously updating the weights θ of the online network i , the loss function of agent i is minimized, usually calculated as the Q function With Q target The mean square error between:

[0168]

[0169] In addition, the gradient of the loss function is recorded as It can be used to update θ through gradient descent i ,Right now

[0170]

[0171] Every W steps, the parameter Will be updated regularly to match the online network i The parameters of , thus stabilizing the training and improving the convergence, that is,

[0172]

[0173] exist Figure 3-8 In this paper, the present invention uses the blockchain-based trust evaluation mechanism BTMM and the reinforcement learning algorithm MADDQN to solve the routing path recovery problem in a network with malicious drones. The results are compared and analyzed with two other reinforcement learning algorithms, MADQN and MAPPO, as well as methods that do not use the blockchain trust evaluation mechanism. The details are as follows:

[0174] Figure 3 The reward of the designed BTMM-MADDQN algorithm for different numbers of requests in a network of 20 drones is shown. It can be seen that when the number of requests is 40, the convergence speed is the slowest and the reward is the smallest. Meanwhile, as the number of requests decreases, the convergence speed is faster and the reward is larger. This is because the more requests, the more complex the environment perceived by the drone becomes, requiring more iterations to learn, resulting in slower convergence. Furthermore, because the designed reward is related to the negative value of the sum of the transmission delays of all requests, the reward decreases with more requests.

[0175] Figure 4 We evaluated the end-to-end latency of transmission requests. It can be seen that as the network scale increases and the number of requests increases, the data transmission latency of all three algorithms increases. Compared to the BTMM-MAPPO algorithm at {20, 25} and the BTMM-MADQN algorithm at {15, 20}, the proposed BTMM-MADDQN algorithm reduces latency by 13.39% and 12.74%, respectively.

[0176] Figure 5 The network throughput performance is shown for different numbers of drones and requests. As the number of drones increases, the network's communication channel resources become more abundant, allowing more requests to be transmitted per second, thereby improving throughput. When the number of drones reaches 10, the network throughput is lowest, at 67.94 MB / s. Furthermore, the BTMM-MADDQN algorithm achieves a throughput increase of 15.44% and 14.6% compared to BTMM-MADQN at {20, 25} and BTMM-MAPPO at {15, 20}, respectively.

[0177] Under different drone network scales, Figure 6The proposed BTMM-MADDQN algorithm demonstrates its superior latency in the presence of malicious drones. Specifically, in drone networks without malicious nodes, the MADDQN-based routing method exhibits excellent low-latency performance because drones do not engage in malicious behavior, such as transmitting data along incorrect paths and incurring additional delays. However, in drone networks with malicious nodes, the MADDQN-based method achieves the worst latency performance because malicious drones may transmit requests along non-optimal paths, resulting in increased latency costs. Furthermore, when malicious drones are present, the BTMM-MADDQN-based algorithm achieves a 16.6% reduction in latency compared to the MADDQN-based method.

[0178] Figure 7 The latency performance of drone networks of varying sizes and numbers of malicious drones is demonstrated. It can be seen that latency is minimized in networks without malicious drones, while latency increases significantly in networks with malicious drones. This indicates that the presence of malicious drones in routing paths significantly degrades routing latency performance. Furthermore, routing latency increases dramatically as the number of malicious drones increases. This is because existing routing paths are disrupted and replaced by suboptimal ones. Furthermore, as the number of drones increases, overall network resources become more abundant in terms of energy, channel, and queue capacity, thereby reducing routing latency costs.

[0179] Figure 8 This figure depicts the average queue lengths of drones for different requests in the network. It can be seen that as the number of requests increases, the queue length rapidly increases due to increased congestion in the drone network. Furthermore, when the same number of requests is transmitted in the network, the average queue length decreases with a greater number of drones, due to more abundant service resources. Furthermore, when the number of requests exceeds the number of drones, the average queue length per drone increases at a faster rate.

[0180] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code. The scheme in the embodiment of the present application can be implemented in various computer languages, for example, object-oriented programming language Java and literal translation scripting language JavaScript, etc.

[0181] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0182] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0183] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions for executing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0184] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0185] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A blockchain-enabled intelligent trusted routing method for drone networks, characterized by: The UAV network intelligent trusted routing method comprises the following steps: S1. Design a network architecture for a time-varying drone swarm. Use the network architecture to describe a trusted routing scenario with malicious drones having low trust values. This includes: dividing the time period into T time slots. In each time slot, determine the movement of drones, normal drone nodes and malicious drone nodes, the routing data to be transmitted and its source-destination drone pair, the communication range and network topology link status of each drone node, and the original and recovered trusted routing paths. The source and destination drone nodes are both normal drone nodes. S2, based on the communication range of each drone node, select normal drone nodes within its communication range from its neighboring drone nodes and use them as candidate forwarding drone nodes; based on the candidate forwarding drone node set of each normal drone node, analyze all routing paths from the source drone node to the destination drone node in the network architecture and use them as candidate routing paths; S3, a deliberate attack model is designed. Based on the node importance ranking results, drone nodes that are more important to the network are attacked first. This simulates the scenario where an attacker causes network performance to degrade faster by prioritizing key nodes, thereby constructing a drone network environment with malicious nodes. The importance of nodes is evaluated by calculating the node degree and link weight value. S4, in response to the presence of malicious nodes in the drone network, a decentralized node trust evaluation mechanism based on blockchain is designed to manage drone trust values. Some drones with high security values ​​are selected as consensus nodes to maintain and update node trust values. Drones with trust values ​​below the threshold are removed to update normal drone nodes and malicious drone nodes in the network, thus building a trusted network environment. S5, based on the trusted network environment built on the blockchain-based trust management mechanism, designs a free space path loss model between UAV nodes. Under the premise that the total energy consumption of UAV movement and communication cannot exceed the UAV energy constraint, the maximum allowable single-hop transmission rate between each UAV node and the next-hop UAV node in each candidate routing path is obtained; S6, based on the communication distance between each UAV node and the next-hop UAV node and the number of data packets waiting to be transmitted in the next-hop UAV node, calculates the single-hop data transmission delay and queuing delay respectively, and then calculates the end-to-end delay model from the source UAV node to the destination UAV node based on the routing path; transforms the routing problem of the UAV network into a problem of minimizing the end-to-end delay, and constructs the delay minimization objective function; S7 uses a distributed partially observable Markov decision process to reconstruct the delay minimization objective function problem, and uses a multi-agent deep reinforcement learning algorithm to solve the minimization objective function problem modeled as a distributed partially observable Markov decision process, obtaining the optimal strategy with the smallest end-to-end delay value. The routing path selected based on the optimal strategy restores the data transmission between the source drone node and the destination drone node.

2. The blockchain-enabled UAV network intelligent trusted routing method according to claim 1 is characterized in that: Step S1 further comprises: Suppose N drone nodes are randomly and evenly deployed in the drone swarm network, and the network is modeled as a weighted undirected graph in: U r and denote the source, relay and destination UAV nodes respectively, i∈U represents UAV node i; ε={e ij ,i=1,2,…,N,j=1,2,…,N},e ij ∈{0,1} is a binary variable that represents the link status between drone node i and drone node j. If e ij =1, indicating that there is a communication link between the two UAV nodes. If e ij =0, indicating that there is no communication link between the two UAV nodes; Using tuples represents a request r∈R, where and L r Represent the source drone, destination drone and data size of the request r respectively; Indicates that the drone is from the source To the corresponding destination drone The complete routing path for the transmission request r.

3. The blockchain-enabled UAV network intelligent trusted routing method according to claim 2 is characterized in that: Step S2 further comprises: The time period is divided into T time slots, and the length of each time slot is τ. At the beginning of time slot t∈T, the UAV adjusts its direction by accelerating or decelerating instantaneously, and then flies at a constant speed in a fixed flight direction. The position of drone node i at time t is expressed as coordinates Indicates that when the Euclidean distance between drone nodes i and j is Greater than d max When the value is , the two drone nodes cannot communicate; and when Less than d min When the value is , a collision will occur; then the candidate forwarding drone node set of each drone node i 4. The blockchain-enabled UAV network intelligent trusted routing method according to claim 2 is characterized in that: In step S3, the node importance is calculated using the following formula: in, Represents the UAV node i to link e ij The important contribution represents the node degree of drone node i.

5. The blockchain-enabled UAV network intelligent trusted routing method according to claim 1 is characterized in that: In step S4, the trust value evaluation mechanism for each drone is: in, and They represent the initial trust value of UAV i, the trust value based on the delivery rate, and the trust value based on the correct transmission line, respectively. and Respectively and The weight value of and Expressed as: in, and They represent the number of requests received and sent by UAV i respectively; and They represent the number of non-designated transmission paths and the total number of transmission paths of UAV i in time slot t, respectively.

6. The blockchain-enabled intelligent trusted routing method for drone networks according to claim 1 is characterized in that: In step S4, the process of designing a blockchain-based decentralized node trust evaluation model to manage the drone trust value includes the following steps: Select the 3n+1 drones with the largest trust value as the consensus node c∈U c , and select the drone with the highest trust value as the leader. The consensus drone set and candidate consensus drone set are denoted as U c and In the consensus drone set U c After confirmation, the transaction consensus process begins. The non-consensus drone uploads the transaction record information to the nearest consensus drone, which forwards it to the leader for verification. The leader initiates a round of negotiation among all consensus drones. In order to reduce the communication load between consensus nodes, a U is executed every M rounds of consensus. c Update process; if there is a drone c∈U c The trust value is lower than the safety threshold Then it is from U c Delete it from the candidate set Invite a new consensus drone to complete the update of the formula node.

7. The blockchain-enabled UAV network intelligent trusted routing method according to claim 2 is characterized in that: In step S5, based on the trusted network environment built by the trust management mechanism based on blockchain, a free space path loss model between drone nodes is designed to calculate the communication channel transmission rate between two drones i and j. in, and are the channel bandwidth and signal-to-noise ratio between UAVs i and j, respectively; in, and are transmission power and noise power respectively; is the channel path loss value, which is Where θ represents the path loss exponent, is the distance between UAVs i and j, f c represents the carrier frequency, c is the speed of light; in addition, the transmission capacity constraint is expressed as: in, Expressed as the data length of the requirement r∈R, is a binary variable indicating whether the demand r passes through the link e(i,j) at time slot t: Total energy consumption of drones Satisfy the constraints: in represents the energy capacity of UAV i, and β is the empirical power efficiency.

8. The blockchain-enabled UAV network intelligent trusted routing method according to claim 2 is characterized in that: In step S6, the single-hop data transmission delay and queuing delay are calculated based on the communication distance between each drone node and the next-hop drone node and the amount of data packets waiting to be transmitted in the next-hop drone node. When a request r∈R arrives at drone i, the queuing delay for transmission to the next-hop drone j is Expressed as: in, is the set of queued requests of drone i at time slot t, Expressed as the data length of the requirement r, is a binary variable indicating whether the demand r passes through the link e(i,k) in time slot t. is the transmission rate of the communication channel between two UAVs i and k, and the single-hop transmission delay of request r from UAV i to UAV j in time slot t is for: The total single-hop delay for transmitting request r from UAV i to UAV j in time slot t is: in, The value cannot exceed the maximum permissible delay of a single hop Right now: When the drone is sourced Drone to destination Routing path Determined, end-to-end latency The calculation formula is: Each source drone Can only communicate with one other drone: in, is a binary variable indicating whether the demand r passes the link at time slot t For the intermediate relay UAV j at time t, t ∈U r , the flow conservation satisfies: A request can only choose one routing path, namely: Destination Drone It can only receive a request r from another drone i, that is: in, is a binary variable indicating whether the demand r passes the link at time slot t The objective function is to minimize the total delay of all requests r∈R, which can be expressed as: in, It means that the drone nodes avoid forwarding data through malicious nodes.

9. The blockchain-enabled UAV network intelligent trusted routing method according to claim 1 is characterized in that: In step S7, the distributed partially observable Markov decision process is composed of five tuples Composition, of which: Agent Collection represents all agents, i.e. all drones; The process of obtaining the state space S is: At the beginning of each time slot t, each agent i∈U collects and obtains its own observation data from the environment Each drone can only observe neighbor nodes k∈Θ i ; The observation data of agent i is: Where, is the position coordinate of agent k in time period t, It is energy consumption, Is the length of the data packet waiting to be transmitted. Defined as C k and l k They represent the number and size of packets queued for transmission by agent k respectively; After obtaining the observation data of all agents, summarize the joint state s at time t t , recorded as The state space S is represented as S = {s t ,t∈T}; The process of obtaining the action space A is: definition in, Represents a single-hop action of drone node i with k as the next-hop target. The size of the action space corresponds to the set Θ i The number of drone nodes in the ; the joint action of N agents at time t is The action space is A={a t ,t∈T}; Reward value function Represents the instantaneous reward set of all agents, where the goal is to minimize the total delay of the routing path of the request r∈R from the source drone node to the destination drone node. Let the current state s t Next, agent i takes action a at time slot t t The instantaneous reward is: in, is the single-hop delay of transmitting request r from UAV i to UAV j in time slot t; Discount factor γ = {γ i ,i∈U} is the discount factor set of all agents. The larger the value of γ, the more emphasis is placed on long-term returns. i ∈(0,1) is the discount factor when calculating the cumulative reward.

10. The blockchain-enabled UAV network intelligent trusted routing method according to claim 1 is characterized in that: In step S7, the multi-agent deep reinforcement learning algorithm is used to solve the optimal strategy π that minimizes the end-to-end delay value. * , which corresponds to finding the optimal behavior value function value Q of the corresponding UAV node in each state of the distributed partially observable Markov decision process * , and go to the next state; Each state s t The ε-greedy method is used to select action a t , randomly select an action from the action space with a preset probability value ε, or select the action that produces the maximum behavior value function value Q with a probability value 1-ε max action, where ε∈(0,1) is the probability of exploring the action; The Q-value function is approximated by the online network and target network of agent i, with the observed state as input and the action as output. The parameters of the online network and the target network are θ i and By continuously updating the weights θ of the online network i , the loss function of agent i is minimized, calculating the Q function With Q target The mean square error between: The gradient of the loss function is recorded as Update θ by gradient descent i : Every W steps, the parameter Updated regularly to match the online network i Parameters:

Images