Unmanned aerial vehicle network intelligent trusted routing method based on blockchain enabling

Abstract

The invention discloses an unmanned aerial vehicle network intelligent trusted routing method based on blockchain enabling, which aims at minimizing routing total end-to-end time delay, constructs a trusted network environment through a trust management mechanism based on a blockchain, and emphasizes a trusted routing problem as a distributed partially observable Markov decision process. And a multi-agent deep reinforcement learning algorithm is used for solving, and multi-dimensional observable information of the nodes is comprehensively considered. The method does not depend on global information which is difficult to obtain, can adapt to a dynamic environment, is low in calculation cost, can obtain a trusted routing path in a short time, and effectively solves the trusted routing problem that malicious nodes exist in an unmanned aerial vehicle network due to deliberate attacks and the like.

Description

Technical Field

[0001] The present invention relates to the field of drone routing technology, and in particular to a blockchain-enabled drone network intelligent trusted routing method. Background Art

[0002] As a key component of the air-space-ground integrated network in 6G communication technology, drones have developed rapidly and are widely used in a variety of tasks such as disaster relief and real-time monitoring. In these applications, drones can be used as aerial base stations for data collection and transmission, providing low-cost, flexible, and multi-purpose services. However, due to the characteristics of drone networks such as complex application environments, strong dynamics, and distributed topology, they are vulnerable to malicious attacks. When malicious drones are present, hostile activities will cause routing performance to degrade. In order to ensure the safety of drones, it is necessary to strengthen research on drone security technologies and applications and improve the safety performance and reliability of drones.

[0003] Evaluating and managing the behavior of malicious drones is an effective way. However, some current work focuses on establishing a node trust table through a centralized manager based on a single node, but this manager is not fault-tolerant and can be easily tampered with. Therefore, in order to improve routing security, blockchain technology is introduced to establish a decentralized trust mechanism. Specifically, blockchain is a distributed ledger consisting of blocks, which stores transaction details and records and is difficult to tamper with.

[0004] UAV routing algorithms are one of the important research areas to ensure their safety. In some scenarios where multiple UAVs need to work together to complete a task, routing algorithms can coordinate the data transmission and reception between multiple UAVs and plan the shortest and safest data transmission path to improve the efficiency and security of multi-UAV collaboration. The distributed topology of the UAV network makes it vulnerable to attacks, which can interrupt routing. Therefore, it is of great significance to design efficient adaptive and dynamic routing algorithms. Summary of the invention

[0005] The purpose of the present invention is to propose an intelligent trusted routing method for drone networks based on blockchain empowerment, with the goal of minimizing the total end-to-end delay of routing. A trusted network environment is constructed through a trust management mechanism based on blockchain, and the trusted routing problem is restated as a distributed partially observable Markov decision process. The problem is solved using a multi-agent deep reinforcement learning algorithm, which comprehensively considers the multi-dimensional observable information of the nodes. The method of the present invention does not rely on global information that is difficult to obtain, can adapt to dynamic environments, has low computational cost, can obtain a trusted routing path in a relatively short time, and effectively solves the trusted routing problem of malicious nodes in drone networks caused by deliberate attacks and the like.

[0006] In order to achieve the above technical objectives, the technical solution adopted by the present invention is:

[0007] A blockchain-enabled UAV network intelligent trusted routing method, the UAV network intelligent trusted routing method comprising the following steps:

[0008] S1, design the network architecture of the time-varying drone swarm, and describe the trusted routing scenario of malicious drones with low trust values ​​through the network architecture, including: dividing the time period into T time slots, in each time slot, determining the movement of drones, normal drone nodes and malicious drone nodes, routing data to be transmitted and its source-destination drone pairs, the communication range and network topology link status of each drone node, and the original and restored trusted routing paths; the source drone node and the destination drone node are both normal drone nodes;

[0009] S2, according to the communication range of each drone node, from its adjacent drone nodes, filter out normal drone nodes within its communication range, and use them as candidate forwarding drone nodes; based on the candidate forwarding drone node set of each normal drone node, analyze all routing paths from the source drone node to the destination drone node in the network architecture, and use them as candidate routing paths;

[0010] S3, design a deliberate attack model, prioritize attacking drone nodes that are more important to the network according to the node importance ranking results, so as to simulate the scenario where the attacker makes the network performance degrade faster by prioritizing attacking key nodes, and then construct a drone network environment with malicious nodes; the importance evaluation of the node is obtained by calculating the node degree and link weight value;

[0011] S4, in view of the existence of malicious nodes in the drone network, a decentralized node trust evaluation mechanism based on blockchain is designed to manage the drone trust value, select some drones with high security values ​​as consensus nodes to maintain and update the node trust value, and remove drones with trust values ​​below the threshold to update normal drone nodes and malicious drone nodes in the network and build a trusted network environment;

[0012] S5, based on the trusted network environment built based on the trust management mechanism of blockchain, the free space path loss model between drone nodes is designed. Under the premise that the total energy consumption of drone movement and communication cannot exceed the energy constraint of drone, the maximum allowable transmission rate of a single hop between each drone node and the next hop drone node in each candidate routing path is obtained;

[0013] S6, according to the communication distance between each drone node and the next-hop drone node and the amount of data packets waiting to be transmitted in the next-hop drone node, the single-hop data transmission delay and queuing delay are calculated respectively, and then the end-to-end delay model from the source drone node to the destination drone node is calculated according to the routing path; the routing problem of the drone network is converted into the problem of minimizing the end-to-end delay, and the delay minimization objective function is constructed;

[0014] S7, uses a distributed partially observable Markov decision process to reconstruct the delay minimization objective function problem, and uses a multi-agent deep reinforcement learning algorithm to solve the minimization objective function problem modeled as a distributed partially observable Markov decision process, and obtains the optimal strategy with the smallest end-to-end delay value. The routing path selected based on the optimal strategy restores the data transmission between the source UAV node and the destination UAV node.

[0015] Step S1 further comprises:

[0016] Suppose N drone nodes are randomly and evenly deployed in the drone swarm network, and the network is modeled as a weighted undirected graph in: U r and denote the source, relay and destination UAV nodes respectively, i∈U represents UAV node i; ε={e ij ,i=1,2,…,N,j=1,2,…,N},e ij ∈{0, 1} is a binary variable representing the link status between drone node i and drone node j. If e ij =1, indicating that there is a communication link between the two drone nodes. If e ii =0, indicating that there is no communication link between the two UAV nodes;

[0017] Using Tuples represents a request r∈R, where and L r denote the source drone, destination drone and data size of request r respectively; Indicates that the drone is from the source To the corresponding destination drone The complete routing path for the transport request r.

[0018] Step S2 further comprises:

[0019] The time period is divided into T time slots, and the length of each time slot is τ. At the beginning of time slot t∈T, the drone adjusts its direction by accelerating or decelerating instantly, and then flies at a constant speed in a fixed flight direction;

[0020] The position of drone node i at time t is expressed as coordinates Indicates that when the Euclidean distance between drone nodes i and j is Greater than d max When the value is , the two drone nodes cannot communicate; Less than d min When the value is , a collision will occur; then the candidate forwarding drone node set of each drone node i

[0021] Furthermore, in step S3, the node importance is calculated using the following formula:

[0022]

[0023] in, Represents the UAV node i to link e ii The important contribution Represents the node degree of drone node i.

[0024] Furthermore, in step S4, the trust value evaluation mechanism of each drone is:

[0025]

[0026] in, and They represent the initial trust value of drone i, the trust value based on the delivery rate, and the trust value based on the correct transmission. and Respectively and The weight value of and It is expressed as:

[0027]

[0028]

[0029] in, and They represent the number of requests received and sent by UAV i respectively; and They represent the number of non-designated transmission paths and the total number of transmission paths of UAV i in time slot t, respectively.

[0030] Furthermore, in step S4, the process of designing a blockchain-based decentralized node trust evaluation model to manage the drone trust value includes the following steps:

[0031] Select 3n+1 drones with the largest trust value as consensus nodes c∈U c, and select the drone with the highest trust value as the leader. The consensus drone set and candidate consensus drone set are denoted as U c and

[0032] In the consensus drone set U c After confirmation, the transaction consensus process begins. The non-consensus drone uploads the transaction recording information to the nearest consensus drone, which forwards it to the leader for verification. The leader starts a round of negotiation process among all consensus drones. In order to reduce the communication load between consensus nodes, a U is executed every M rounds of consensus. c Update process; if there is a drone c∈U c The trust value is lower than the safety threshold Then it is from U c Delete it from the candidate set Invite a new consensus drone to complete the update of the formula node.

[0033] Furthermore, in step S5, according to the trusted network environment constructed based on the trust management mechanism of blockchain, a free space path loss model between drone nodes is designed to calculate the communication channel transmission rate between two drones i and j.

[0034]

[0035] in, and are the channel bandwidth and signal-to-noise ratio between UAVs i and j, respectively; in, and are transmission power and noise power respectively; is the channel path loss value, which is in, represents the path loss exponent, is the distance between drones i and j, f c represents the carrier frequency, c is the speed of light; in addition, the transmission capacity constraint is expressed as: in, It is expressed as the data length of the requirement r∈R, is a binary variable indicating whether demand r passes through link e(i, j) at time slot t:

[0036]

[0037] Total energy consumption of drone Satisfy the constraints: in represents the energy capacity of UAV i, and β is the empirical power efficiency.

[0038] Furthermore, in step S6, the single-hop data transmission delay and queuing delay are calculated based on the communication distance between each drone node and the next-hop drone node and the amount of data packets waiting to be transmitted in the next-hop drone node; when a request r∈R arrives at drone i, the queuing delay for transmission to the next-hop drone j is It is expressed as:

[0039]

[0040] in, is the set of queued requests of UAV i in time slot t, It is expressed as the data length of the requirement r, is a binary variable indicating whether the demand r passes through the link e(i, k) in time slot t. is the transmission rate of the communication channel between two UAVs i and k, and the single-hop transmission delay of request r from UAV i to UAV j in time slot t for:

[0041]

[0042] The sum of the single-hop delays for transmitting request r from UAV i to UAV j in time slot t is:

[0043]

[0044] in, The value cannot exceed the maximum permissible delay of a single hop Right now:

[0045]

[0046] When the drone is sourced Drone to destination The routing path Determined, end-to-end latency The calculation formula is:

[0047]

[0048] Each source drone Can only communicate with one other drone:

[0049]

[0050] in, is a binary variable indicating whether demand r passes through link at time slot t

[0051] For the intermediate relay UAV j at time t, t∈U r , the flow conservation satisfies:

[0052]

[0053] A request can only choose one routing path, namely:

[0054]

[0055] Destination Drone It can only receive a request r from another drone i, that is:

[0056]

[0057] in, is a binary variable indicating whether demand r passes through link at time slot t

[0058] The objective function is to minimize the total delay of all requests r∈R, which can be expressed as:

[0059]

[0060]

[0061]

[0062]

[0063]

[0064]

[0065] in, It means that the drone nodes avoid forwarding data through malicious nodes.

[0066] Further, in step S7, the distributed partially observable Markov decision process is composed of five tuples Composition, of which:

[0067] Agent Collection represents all agents, i.e. all drones;

[0068] The process of obtaining the state space S is:

[0069] At the beginning of each time slot t, each agent i∈U collects and obtains its own observation data from the environment Each drone can only observe neighbor nodes k∈Θ i ;

[0070] The observation data of agent i is:

[0071]

[0072] In the formula, is the position coordinate of agent k in time period t, It is energy consumption. is the length of the data packets waiting to be transmitted. Defined as C k and l k They represent the number and size of the data packets queued for transmission by agent k respectively;

[0073] After obtaining the observation data of all agents, summarize the joint state s at time t t , denoted as The state space S is represented by S = {s t , t∈T};

[0074] The process of obtaining the action space A is:

[0075] definition in, represents the single-hop action of drone node i with k as the next-hop target, and the size of the action space corresponds to the set Θ i The number of drone nodes in the game; the joint action of N agents at time t is The action space is A = {a t , t∈T};

[0076] Reward value function represents the instantaneous reward set of all agents, where the goal is to minimize the total delay of the routing path of the request r∈R from the source drone node to the destination drone node. Let the current state s t Next, agent i takes action a at time slot t t The instantaneous reward is:

[0077]

[0078] in, is the single-hop delay of transmitting request r from UAV i to UAV j in time slot t;

[0079] Discount factor γ = {γ i , i∈U} is the set of discount factors for all agents. The larger the value of γ, the more emphasis is placed on long-term returns. i ∈(0, 1) is the discount factor when calculating the cumulative reward.

[0080] Furthermore, in step S7, the multi-agent deep reinforcement learning algorithm is used to solve the optimal strategy π that minimizes the end-to-end delay value. *, which corresponds to finding the optimal behavior value function value Q of the corresponding drone node in each state of the distributed partially observable Markov decision process * , and go to the next state;

[0081] Each state s t The ε-greedy method is used to select action a t , randomly select an action from the action space with a preset probability value ε, or select the action that produces the maximum behavior value function value Q with a probability value 1-ε max action, where ε∈(0,1) is the probability of the exploration action;

[0082] The Q-value function is approximated by the online network and target network of agent i, with the observed state as input and the action as output. The parameters of the online network and the target network are θ i and By continuously updating the weights θ of the online network i , the loss function of agent i is minimized, calculating the Q function With Q target The mean square error between:

[0083]

[0084] The gradient of the loss function is recorded as Update θ by gradient descent i :

[0085]

[0086] Every W steps, parameter Updated regularly to match the online network i Parameters:

[0087]

[0088] Compared with the prior art, the present invention has the following beneficial effects:

[0089] First, the blockchain-enabled drone network intelligent trusted routing method of the present invention proposes a blockchain-based trust evaluation mechanism, a node trust evaluation mechanism and a consensus drone update mechanism, which effectively improves the security of using the Byzantine algorithm and thus improves the security of the routing process.

[0090] Second, the blockchain-enabled drone network intelligent trusted routing method of the present invention depicts the routing process in a time-varying drone network with malicious nodes. In response to the challenge of obtaining global information in a decentralized drone network, the routing problem is reconstructed into a distributed partially observable Markov decision process, and an algorithm based on multi-agent deep reinforcement learning is proposed to solve the problem.

[0091] Third, the blockchain-enabled drone network intelligent trusted routing method of the present invention is simulated in a drone network with attacked malicious nodes. The results show that the designed algorithm based on multi-agent deep reinforcement learning is superior to other algorithms in many aspects such as convergence performance, latency, throughput and queue length, and effectively solves the trusted routing problem in networks with malicious node drones. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] Figure 1 A schematic diagram of routing recovery in a time-varying UAV network according to the present invention;

[0093] Figure 2 A schematic diagram of a trusted practical Byzantine fault-tolerant algorithm according to the present invention; wherein (a) shows a node update process when there is a (f=1) malicious consensus unmanned aerial vehicle; (b) shows a schematic diagram of a practical Byzantine fault-tolerant consensus process;

[0094] Figure 3 This is an experimental simulation diagram of the relationship between the number of requests and the cumulative rewards involved in the present invention;

[0095] Figure 4 This is an experimental simulation diagram of the relationship between routing delay and number of drones for the three algorithms involved in the present invention;

[0096] Figure 5 It is an experimental simulation diagram of the relationship between network throughput and the number of drones and data transmission requests for the three algorithms involved in the present invention;

[0097] Figure 6 It is an experimental simulation diagram of the relationship between the routing delay of different algorithms involved in the present invention and the number of malicious drones and the number of drones;

[0098] Figure 7 This is an experimental simulation diagram of the relationship between the average end-to-end delay and the number of malicious drones under different network scales involved in the present invention;

[0099] Figure 8 This is an experimental simulation diagram of the relationship between the average queue length and request quantity of each drone in drone networks of different sizes involved in the present invention. DETAILED DESCRIPTION

[0100] The embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings.

[0101] Figure 1 A schematic diagram of routing recovery in a time-varying network with intentional drones in a specific embodiment is shown. The present invention discloses a blockchain-enabled drone network intelligent trusted routing method, comprising the following steps:

[0102] Step 1) Design the network architecture of the time-varying drone swarm, and describe the trusted routing scenario of malicious drones with low trust values ​​through the network architecture, including: dividing the time period into T time slots, and in each time slot, determining the movement of drones, normal drone nodes and malicious drone nodes, as well as the routing data to be transmitted and its source-destination drone pairs, the communicable range and network topology link status of each drone node, and the original and recovered trusted routing paths.

[0103] In the method of this embodiment, both the source drone node and the destination drone node are normal drone nodes.

[0104] According to the communication range of each UAV node, the UAV nodes within its communication range are screened out from its adjacent UAV nodes and used as candidate forwarding UAV nodes.

[0105] Based on the candidate forwarding drone node set of each normal drone node, all routing paths from the source drone node to the destination drone node in the network architecture are analyzed and used as candidate routing paths.

[0106] Step 2), design a deliberate attack model, calculate the node importance by evaluating the node degree and link weight, and attack the drone nodes with high importance first according to the ranking.

[0107] Step 3) In response to the presence of malicious nodes in the drone network, a decentralized node trust evaluation model based on blockchain is designed to manage the drone trust value. Specifically, some drones with high security values ​​are selected as consensus nodes to maintain and update the node trust value, and drones with trust values ​​below the threshold are removed.

[0108] Step 4) According to the free space path loss model between drone nodes, the maximum allowable transmission rate of a single hop between each drone node and the next hop drone node in each candidate routing path is obtained; at the same time, the total energy consumption of drone movement and communication cannot exceed the drone energy constraint.

[0109] Step 5) According to the communication distance between each drone node and the next-hop drone node and the amount of data packets waiting to be transmitted in the next-hop drone node, the single-hop data transmission delay and queuing delay are calculated respectively, and then the end-to-end delay model from the source drone node to the destination drone node is calculated according to the routing path. The routing problem of the drone network is transformed into the problem of minimizing the above end-to-end delay, and the delay minimization objective function is constructed.

[0110] Step 6) Use a distributed partially observable Markov decision process to reconstruct the minimization objective function problem, and use a multi-agent deep reinforcement learning algorithm to solve the minimization objective function problem modeled as a distributed partially observable Markov decision process, and obtain a strategy with the minimum end-to-end delay value. The routing path selected based on the optimal strategy restores the data transmission between the source drone node and the destination drone node.

[0111] Figure 1 A schematic diagram of routing recovery in a time-varying network with intentional drones in a specific embodiment is shown.

[0112] In the above process:

[0113] In step 1), N drone nodes are randomly and evenly deployed in the network of the drone swarm, and the network is modeled as a weighted undirected graph Among them: U=U s ∩U r ∩U d , U s , U r and U d denote the source, relay and destination UAV nodes respectively, i∈U represents UAV node i; ε={e ij ,i=1,2,…,N,j=1,2,…,N},e ij ∈{0, 1} is a binary variable representing the link status between drone node i and drone node j. If e ij =1, indicating that there is a communication link between the two drone nodes. If e ij = 0, indicating that there is no communication link between the two drone nodes. The request r∈R is represented as a tuple in and L r denote the source drone, destination drone and data size of request r respectively. In addition, Indicates that the drone is from the source To the corresponding destination drone The complete routing path for the transport request r.

[0114] The time period is divided into T time slots, and the length of each time slot is τ. Assume that the drone can adjust its direction by accelerating or decelerating at the beginning of time slot t∈T, and then fly at a constant speed in a fixed flight direction. The position of drone node i at time t is represented by coordinates Indicates that when the Euclidean distance between drone nodes i and j is Greater than d max When the value is , the two drone nodes cannot communicate; Less than d min When the value is , a collision will occur; therefore, the candidate forwarding drone node set of each drone node i is

[0115] In step 2), a UAV network environment with malicious nodes is constructed through the intentional attack model. The importance of nodes is calculated by evaluating the node degree and link weight, and the UAV nodes with high importance are attacked first. Note f and U f Represent the number and set of malicious drones respectively. The node importance is defined as:

[0116]

[0117] in, Represents the UAV node i to link e ij The important contribution Represents the node degree of drone node i.

[0118] In step 3), the malicious behavior of drones is analyzed from two aspects. When malicious drones receive requests, they may discard the requests or transmit the requests not according to a specific routing path; therefore, considering the delivery rate and the correctness of the transmission path, the trust value evaluation mechanism of each drone is designed as follows:

[0119]

[0120] in, and They represent the initial trust value of drone i, the trust value based on the delivery rate, and the trust value based on the correct transmission. and Respectively and The weight value of .

[0121] In order to improve the security of using the Byzantine fault-tolerant consensus algorithm, a trusted Byzantine fault-tolerant consensus method based on the drone update consensus mechanism is proposed:

[0122] First, select 3n+1 drones with the largest trust value as consensus nodes c∈U c, and select the drone with the highest trust value as the leader. The consensus drone set and candidate consensus drone set are denoted as U c and

[0123] in U c After confirmation, the transaction consensus process begins, which mainly includes three stages: pre-preparation, preparation, and submission. Specifically, the non-consensus drone uploads the transaction recording information to the nearest consensus drone, which forwards it to the leader consensus drone for verification, and then the leader starts the three-stage process in all consensus drones. If M rounds of consensus are reached, U c Update process. If there is a drone c∈U c The trust value is lower than the safety threshold Then it is from U c Then, from the candidate set Invite a new consensus drone to complete the update of the formula node. Figure 2 A schematic diagram of a trusted practical Byzantine fault-tolerant algorithm is shown, including: (a) showing the node update process when there is a (f=1) malicious consensus unmanned aerial vehicle and (b) showing a schematic diagram of the practical Byzantine fault-tolerant consensus process.

[0124] In step 4), the communication channel transmission rate between two drones i and j is calculated based on the free space path loss model between drone nodes. The transmission capacity constraint is expressed as:

[0125]

[0126] Considering the movement energy and communication energy of the drone, the total energy consumption of the drone is Satisfy the constraints:

[0127]

[0128] In step 5), the single-hop data transmission delay and queuing delay are calculated based on the communication distance between each drone node and the next-hop drone node and the number of packets waiting to be transmitted in the next-hop drone node. When a request r∈R arrives at drone i, the queuing delay for transmission to the next-hop drone j is It is expressed as:

[0129]

[0130] in, is the set of queued requests of UAV i at time slot t, and the single-hop transmission delay of request r from UAV i to UAV j at time slot t for:

[0131]

[0132] Therefore, the sum of the single-hop delays for transmitting request r from UAV i to UAV j in time slot t is:

[0133]

[0134] in, The value cannot exceed the maximum permissible delay of a single hop Right now

[0135]

[0136] When the drone is sourced Drone to destination The routing path Determined, end-to-end latency The calculation formula is:

[0137]

[0138] Each source drone Can only communicate with one other drone:

[0139]

[0140] For the intermediate relay drone t ∈U r , flow conservation should satisfy:

[0141]

[0142] In addition, a request can only choose one routing path, i.e.

[0143]

[0144] Similar to the source drone, the destination drone can only receive the demand r from another UAV i, i.e.

[0145]

[0146] The objective function is to minimize the total delay of all requests r∈R, which can be expressed as:

[0147]

[0148]

[0149]

[0150]

[0151]

[0152]

[0153] in, It means that the drone nodes avoid forwarding data through malicious nodes;

[0154] In step 6), the distributed partially observable Markov decision process consists of five tuples Composition, of which:

[0155] (a) Agent Collection represents all agents, i.e. all drones;

[0156] (b) State space S: At the beginning of each time slot t, each agent i∈U collects and obtains its own observation data from the environment It should be noted that each drone can only observe neighbor nodes k∈Θ i ;

[0157] Therefore, the observation result of agent i is:

[0158]

[0159] is the position coordinate of agent k in time period t, It is energy consumption. is the length of the data packets waiting to be transmitted. Defined as C k and l k They represent the number and size of the data packets queued for transmission by agent k respectively;

[0160] After obtaining the observation data of all agents, summarize the joint state s at time t t , denoted as The state space S is represented by S = {s t , t∈T};

[0161] (c) Action space A: Definition in, represents the single-hop action of drone node i with k as the next-hop target, and the size of the action space corresponds to the set Θ i The number of drone nodes in the game; the joint action of N agents at time t is The action space is A = {a t , t∈T};

[0162] (d) Reward value function Indicates that agent i performs an action at time slot t The instantaneous reward is to minimize the total delay of the routing path of the request r∈R from the source drone node to the destination drone node. Let the current state s t Take action a t The instantaneous reward is:

[0163]

[0164] (e) Discount factor γ: γ i ∈(0,1) is the discount factor when calculating the cumulative reward, γ={γ i , i∈U} is the set of discount factors for all agents. The larger the value of γ, the more emphasis is placed on long-term returns;

[0165] Step 6) Use the multi-agent deep reinforcement learning algorithm to solve the optimal strategy π that minimizes the end-to-end delay value * , which corresponds to finding the optimal behavior value function value Q of the corresponding drone node in each state of the distributed partially observable Markov decision process * , and go to the next state;

[0166] Each state s t The ε-greedy method is used to select action a t , randomly select an action from the action space with a preset probability value ε, or select the action that produces the maximum behavior value function value Q with a probability value 1-ε max action, where ε∈(0,1) is the probability of the exploration action.

[0167] The DDQN algorithm proposes to approximate the Q-value function through DNN. With the observed state as input and the action as output, there are two DNNs in the DDQN algorithm, namely the online network of agent i and the target network, with parameters θ and θ respectively. i and By continuously updating the weights θ of the online network i , the loss function of agent i is minimized, usually calculated as the Q function With Q target The mean square error between:

[0168]

[0169] In addition, the gradient of the loss function is recorded as It can be used to update θ through gradient descent i ,Right now

[0170]

[0171] Every W steps, parameter Will be updated regularly to match the online network i The parameters of , thus stabilizing the training and improving the convergence, that is,

[0172]

[0173] exist Figure 3-8 In this paper, the present invention uses the blockchain-based trust evaluation mechanism BTMM and the reinforcement learning algorithm MADDQN to solve the routing path recovery problem in a network with malicious drones, and compares and analyzes it with two other reinforcement learning algorithms MADQN and MAPPO, as well as a method that does not use the blockchain trust evaluation mechanism, as follows:

[0174] Figure 3 The reward of the designed BTMM-MADDQN algorithm under different numbers of requests in a network with 20 drones is shown. It can be seen that when the number of requests is 40, the convergence speed is the slowest and the reward is the smallest. At the same time, as the number of requests decreases, the convergence speed is faster and the reward is larger. This is because when there are more requests, the environment perceived by the drone becomes more complex, and more iterations are required to learn, resulting in slower convergence. In addition, since the designed reward is related to the negative value of the sum of the transmission delays of all requests, the more requests, the smaller the reward.

[0175] Figure 4 The end-to-end delay of the transmission request is evaluated. It can be seen that as the network scale increases and the number of requests increases, the data transmission delay of the three algorithms will increase. Compared with the BTMM-MAPPO algorithm at {20, 25} and the BTMM-MADQN algorithm at {15, 20}, the delay of the proposed BTMM-MADDQN algorithm is reduced by 13.39% and 12.74%, respectively.

[0176] Figure 5 The performance of network throughput under different numbers of drones and requests is shown. As the number of drones increases, the network's communication channel resources become more abundant, and more requests can be transmitted per second, thereby improving throughput. When the number of drones is 10, the network throughput is the lowest, at 67.94MB / s. In addition, the throughput of the BTMM-MADDQN algorithm increases by 15.44% and 14.6% compared to BTMM-MADQN at {20, 25} and BTMM-MAPPO at {15, 20}, respectively.

[0177] Under different drone network scales, Figure 6The superiority of the proposed BTMM-MADDQN algorithm in terms of latency is verified when malicious drones are present. Specifically, in drone networks without malicious nodes, the MADDQN-based routing method exhibits excellent low-latency performance because drones do not generate malicious behaviors, such as transmitting data through the wrong path and incurring additional delays. However, in drone networks with malicious nodes, the MADDQN-based method obtains the worst latency performance because malicious drones may transmit requests under non-optimal paths, resulting in increased latency costs. At the same time, when malicious drones are present, the latency based on the BTMM-MADDQN algorithm is reduced by 16.6% compared to the MADDQN-based method.

[0178] Figure 7 The latency performance of drone networks of different sizes with different numbers of malicious drones is shown. It can be seen that the latency is minimal in networks without malicious drones, while the latency increases significantly in networks with malicious drones. This indicates that malicious drones in the routing path significantly reduce the routing latency performance. In addition, as the number of malicious drones increases, the routing latency also increases dramatically. This is because the original routing path is destroyed and replaced by a suboptimal routing path. At the same time, as the number of drones increases, the total network resources are richer in terms of energy, channel, and queue capacity, thereby reducing the latency cost of routing.

[0179] Figure 8 The average queue length of drones under different requests in the network is described. It can be seen that as the number of requests increases, the queue length rises rapidly due to the increased congestion of the drone network. At the same time, when the same number of requests is transmitted in the network, the more drones there are, the smaller the average queue length is, which is due to more abundant service resources. In addition, when the number of requests is greater than the number of drones, the average queue length on each drone increases at a faster rate.

[0180] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code. The scheme in the embodiments of the present application can be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.

[0181] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0182] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0183] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0184] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0185] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A blockchain-enabled UAV network intelligent trusted routing method, characterized in that: The UAV network intelligent trusted routing method comprises the following steps: S1, design the network architecture of the time-varying drone swarm, and describe the trusted routing scenario of malicious drones with low trust values ​​through the network architecture, including: dividing the time period into T time slots, in each time slot, determining the movement of drones, normal drone nodes and malicious drone nodes, routing data to be transmitted and its source-destination drone pairs, the communication range and network topology link status of each drone node, and the original and restored trusted routing paths; the source drone node and the destination drone node are both normal drone nodes; S2, according to the communication range of each drone node, from its adjacent drone nodes, filter out normal drone nodes within its communication range, and use them as candidate forwarding drone nodes; based on the candidate forwarding drone node set of each normal drone node, analyze all routing paths from the source drone node to the destination drone node in the network architecture, and use them as candidate routing paths; S3, design a deliberate attack model, prioritize attacking drone nodes that are more important to the network according to the node importance ranking results, so as to simulate the scenario where the attacker makes the network performance degrade faster by prioritizing attacking key nodes, and then construct a drone network environment with malicious nodes; the importance evaluation of the node is obtained by calculating the node degree and link weight value; S4, in view of the existence of malicious nodes in the drone network, a decentralized node trust evaluation mechanism based on blockchain is designed to manage the drone trust value, select some drones with high security values ​​as consensus nodes to maintain and update the node trust value, and remove drones with trust values ​​below the threshold to update normal drone nodes and malicious drone nodes in the network and build a trusted network environment; S5, based on the trusted network environment built based on the trust management mechanism of blockchain, the free space path loss model between drone nodes is designed. Under the premise that the total energy consumption of drone movement and communication cannot exceed the energy constraint of drone, the maximum allowable transmission rate of a single hop between each drone node and the next hop drone node in each candidate routing path is obtained; S6, according to the communication distance between each drone node and the next-hop drone node and the amount of data packets waiting to be transmitted in the next-hop drone node, the single-hop data transmission delay and queuing delay are calculated respectively, and then the end-to-end delay model from the source drone node to the destination drone node is calculated according to the routing path; the routing problem of the drone network is converted into the problem of minimizing the end-to-end delay, and the delay minimization objective function is constructed; S7, uses a distributed partially observable Markov decision process to reconstruct the delay minimization objective function problem, and uses a multi-agent deep reinforcement learning algorithm to solve the minimization objective function problem modeled as a distributed partially observable Markov decision process, and obtains the optimal strategy with the smallest end-to-end delay value. The routing path selected based on the optimal strategy restores the data transmission between the source UAV node and the destination UAV node.

2. According to the blockchain-enabled UAV network intelligent trusted routing method according to claim 1, it is characterized in that: Step S1 further comprises: Suppose N drone nodes are randomly and evenly deployed in the drone swarm network, and the network is modeled as a weighted undirected graph in: U r and denote the source, relay and destination UAV nodes respectively, i∈U represents UAV node i; ε={e ij ,i=1,2,…,N,j=1,2,…,N},e ij ∈{0,1} is a binary variable representing the link status between drone node i and drone node j. If e ij =1, indicating that there is a communication link between the two drone nodes. If e ij =0, indicating that there is no communication link between the two UAV nodes; Using Tuples represents a request r∈R, where and L r denote the source drone, destination drone and data size of request r respectively; Indicates that the drone is from the source To the corresponding destination drone The complete routing path for the transport request r.

3. The blockchain-enabled UAV network intelligent trusted routing method according to claim 2 is characterized in that: Step S2 further comprises: The time period is divided into T time slots, and the length of each time slot is τ. At the beginning of time slot t∈T, the drone adjusts its direction by accelerating or decelerating instantly, and then flies at a constant speed in a fixed flight direction; The position of drone node i at time t is expressed as coordinates Indicates that when the Euclidean distance between drone nodes i and j is Greater than d max When the value is , the two drone nodes cannot communicate; Less than d min When the value is , a collision will occur; then the candidate forwarding drone node set of each drone node i 4. The blockchain-enabled UAV network intelligent trusted routing method according to claim 2 is characterized in that: In step S3, the node importance is calculated using the following formula: in, Represents the UAV node i to link e ij The important contribution Represents the node degree of drone node i.

5. The blockchain-enabled UAV network intelligent trusted routing method according to claim 1 is characterized in that: In step S4, the trust value evaluation mechanism of each drone is: in, and They represent the initial trust value of drone i, the trust value based on the delivery rate, and the trust value based on the correct transmission. and Respectively and The weight value of and It is expressed as: in, and They represent the number of requests received and sent by UAV i respectively; and They represent the number of non-designated transmission paths and the total number of transmission paths of UAV i in time slot t, respectively.

6. The blockchain-enabled UAV network intelligent trusted routing method according to claim 1 is characterized in that: In step S4, the process of designing a blockchain-based decentralized node trust evaluation model to manage the drone trust value includes the following steps: Select 3n+1 drones with the largest trust value as consensus nodes c∈U c , and select the drone with the highest trust value as the leader. The consensus drone set and candidate consensus drone set are denoted as U c and In the consensus drone set U c After confirmation, the transaction consensus process begins. The non-consensus drone uploads the transaction recording information to the nearest consensus drone, which forwards it to the leader for verification. The leader starts a round of negotiation process among all consensus drones. In order to reduce the communication load between consensus nodes, a U is executed every M rounds of consensus. c Update process; if there is a drone c∈U c The trust value is lower than the safety threshold Then it is from U c Delete it from the candidate set Invite a new consensus drone to complete the update of the formula node.

7. The blockchain-enabled UAV network intelligent trusted routing method according to claim 2 is characterized in that: In step S5, according to the trusted network environment constructed based on the trust management mechanism of blockchain, the free space path loss model between drone nodes is designed to calculate the communication channel transmission rate between two drones i and j. in, and are the channel bandwidth and signal-to-noise ratio between UAVs i and j, respectively; in, and are transmission power and noise power respectively; is the channel path loss value, which is Where θ represents the path loss exponent, is the distance between UAVs i and j, f c represents the carrier frequency, c is the speed of light; in addition, the transmission capacity constraint is expressed as: in, It is expressed as the data length of the requirement r∈R, is a binary variable indicating whether demand r passes through link e(i,j) at time slot t: Total energy consumption of drone Satisfy the constraints: in represents the energy capacity of UAV i, and β is the empirical power efficiency.

8. The blockchain-enabled UAV network intelligent trusted routing method according to claim 2 is characterized in that: In step S6, the single-hop data transmission delay and queuing delay are calculated based on the communication distance between each drone node and the next-hop drone node and the number of packets waiting to be transmitted in the next-hop drone node. When a request r∈R arrives at drone i, the queuing delay for transmission to the next-hop drone j is It is expressed as: in, is the set of queued requests of UAV i in time slot t, It is expressed as the data length of the requirement r, is a binary variable indicating whether the demand r passes through the link e(i,k) in time slot t. is the transmission rate of the communication channel between two UAVs i and k, and the single-hop transmission delay of request r from UAV i to UAV j in time slot t for: The sum of the single-hop delays for transmitting request r from UAV i to UAV j in time slot t is: in, The value cannot exceed the maximum permissible delay of a single hop Right now: When the drone is sourced Drone to destination The routing path Determined, end-to-end latency The calculation formula is: Each source drone Can only communicate with one other drone: in, is a binary variable indicating whether demand r passes through link at time slot t For the intermediate relay UAV j at time t, t ∈U r , flow conservation satisfies: A request can only choose one routing path, namely: Destination Drone It can only receive a request r from another drone i, that is: in, is a binary variable indicating whether demand r passes through link at time slot t The objective function is to minimize the total delay of all requests r∈R, which can be expressed as: in, It means that the drone nodes avoid forwarding data through malicious nodes.

9. The blockchain-enabled UAV network intelligent trusted routing method according to claim 1 is characterized in that: In step S7, the distributed partially observable Markov decision process consists of five tuples Composition, of which: Agent Collection represents all agents, i.e. all drones; The process of obtaining the state space S is: At the beginning of each time slot t, each agent i∈U collects and obtains its own observation data from the environment Each drone can only observe neighbor nodes k∈Θ i ; The observation data of agent i is: In the formula, is the position coordinate of agent k in time period t, It is energy consumption. is the length of the data packet waiting to be transmitted. Defined as C k and l k They represent the number and size of the data packets queued for transmission by agent k respectively; After obtaining the observation data of all agents, summarize the joint state s at time t t , denoted as The state space S is represented by S = {s t ,t∈T}; The process of obtaining the action space A is: definition in, represents the single-hop action of drone node i with k as the next-hop target, and the size of the action space corresponds to the set Θ i The number of drone nodes in the game; the joint action of N agents at time t is The action space is A = {a t ,t∈T}; Reward value function represents the instantaneous reward set of all agents, where the goal is to minimize the total delay of the routing path of the request r∈R from the source drone node to the destination drone node. Let the current state s t Next, agent i takes action a at time slot t t The instantaneous reward is: in, is the single-hop delay of transmitting request r from UAV i to UAV j in time slot t; Discount factor γ = {γ i , i∈U} is the set of discount factors for all agents. The larger the value of γ, the more emphasis is placed on long-term returns. i ∈(0,1) is the discount factor when calculating the cumulative reward.

10. The blockchain-enabled UAV network intelligent trusted routing method according to claim 1 is characterized in that: In step S7, a multi-agent deep reinforcement learning algorithm is used to solve the optimal strategy π that minimizes the end-to-end delay value. * , which corresponds to finding the optimal behavior value function value Q of the corresponding drone node in each state of the distributed partially observable Markov decision process * , and go to the next state; Each state s t The ε-greedy method is used to select action a t , randomly select an action from the action space with a preset probability value ε, or select the action that produces the maximum behavior value function value Q with a probability value 1-ε max action, where ε∈(0,1) is the probability of exploring the action; The Q-value function is approximated by the online network and target network of agent i, with the observed state as input and the action as output. The parameters of the online network and the target network are θ i and By continuously updating the weights θ of the online network i , the loss function of agent i is minimized, calculating the Q function With Q target The mean square error between: The gradient of the loss function is recorded as Update θ by gradient descent i : Every W steps, parameter Updated regularly to match the online network i Parameters:

Images