A distributed collaborative evolution method, UAV and intelligent routing method and device thereof

Abstract

The present invention discloses a distributed co-evolution method, UAV and its intelligent routing method and device, each UAV node is configured with a routing DQN model, each UAV node is configured with a routing evaluation model; each UAV node is configured with a block; a data packet is forwarded from a source node in the network to a destination node in sequence, forming a route from the source node to the destination node corresponding to the data packet; within an evolution cycle, the following steps are performed: S1: update routing DQN model parameters: S2: elect block nodes: S3: broadcast routing DQN model parameters; S4: parameter co-evolution. The distributed co-evolution routing technology based on blockchain and deep reinforcement learning of the present invention has lower end-to-end delay and higher packet delivery rate than traditional GPSR, and has a faster convergence speed than ordinary intelligent routing algorithms.

Description

Technical Field

[0001] The present invention relates to the field of unmanned aerial vehicle (UAV) self-organizing networks, and in particular to a distributed collaborative evolution method, a UAV and an intelligent routing method and device thereof. Background Art

[0002] In recent years, Unmanned Aerial Vehicle (UAV) networks have been widely used in various fields to complete many tasks that are difficult to implement in conventional ways, have complex environments, and are highly flexible and changeable. In this environment, UAVs need to collaborate efficiently and flexibly to ensure the effective completion of tasks. Therefore, the low-latency and high-reliability information transmission capability of UAV networks in complex communication environments is the basic prerequisite for the completion of application tasks. However, due to the rapid movement of UAVs, the communication environment and topology of UAV networks are in dynamic change at any time. How to use efficient and adaptive routing technology in this dynamic environment to ensure the low-latency and high-reliability information transmission capability of UAV networks and the effective completion of tasks has become a challenge for UAV networks.

[0003] Greedy Perimeter Stateless Routing (GPSR) only considers the distance between the next hop node and the destination node when routing. However, in UAV networks, due to the complex and dynamic network environment and topology, routing based only on geographic location may not be the best route. In addition, routing holes caused by the frequent movement of UAVs will increase end-to-end delay and waste routing resources.

[0004] In order to coordinate the models of different nodes and enable the entire network to achieve good routing results quickly, centralized parameter aggregation and optimization algorithms such as Federated Learning are usually used. However, this approach cannot avoid the fundamental disadvantages of centralized methods, such as the increase in routing overhead and latency caused by data collection and distribution, and the inability to adapt to rapidly changing dynamic environments. In addition, the centralized aggregation parameters do not take into account the diversity of the network environments in which different UAVs are located, and the learning model cannot make the best routing decisions based on the unique local environment.

[0005] Therefore, in response to the above problems, an intelligent, distributed co-evolutionary routing technology is designed for UAV networks to accelerate the convergence speed of deep reinforcement learning and realize routing co-evolution in UAV networks. This is of great significance for improving the performance of UAV network routing and its ability to adapt to dynamic environments. Summary of the invention

[0006] The purpose of the present invention is to propose a UAV network distributed co-evolutionary routing technology based on blockchain and deep reinforcement learning. The technology is first based on the deep reinforcement learning DQN (Deep Q-network) algorithm, which enables each UAV to make routing decisions that evolve with the environment and achieve autonomous evolution; on this basis, through blockchain technology, the best routing decision model of each round is integrated with the local model of each node, which accelerates the convergence speed of deep reinforcement learning, realizes routing co-evolution in the UAV network, and improves the overall routing performance.

[0007] A distributed co-evolution method is applied to a network consisting of multiple UAV nodes, the network including a verification committee consisting of at least one UAV node;

[0008] Each UAV node is configured with a routing DQN model;

[0009] Each UAV node is configured with a routing evaluation model;

[0010] Each UAV node is configured as a block;

[0011] The data packet is forwarded from the source node in the network to the destination node in sequence, forming a route from the source node to the destination node corresponding to the data packet;

[0012] In one evolution cycle, the following steps are performed:

[0013] S1: Update routing DQN model parameters:

[0014] The UAV node trains its own routing DQN model based on the samples generated by the routing and updates its own routing DQN model parameters;

[0015] S2: Elect block nodes:

[0016] Input the route into the route evaluation model to obtain the score of each node in the route, so that the verification committee selects the node with the highest score as the block generating node according to the score of each node in the route;

[0017] S3: Broadcast routing DQN model parameters: The block-producing node broadcasts its own routing DQN model parameters to the network;

[0018] S4: Parameter co-evolution: The non-block producing node receives the routing DQN model parameters of the block producing node and fuses them with its own routing DQN model parameters to obtain the evolved routing DQN model parameters.

[0019] Furthermore, the updating of routing DQN model parameters specifically includes:

[0020] When forwarding a data packet, the UAV node forwards the data packet to the next node with the maximum Q value based on the routing DQN model and generates training samples. The samples are used to train and update its own routing DQN model when idle, and generate updated routing DQN model parameters after iteration.

[0021] Furthermore, the block-producing node selection specifically includes:

[0022] When the destination node receives the data packet, it obtains the total score of the route based on the routing evaluation model, and generates a score verification package based on the total score. The score verification package in the network is transmitted back to the source node in sequence according to the corresponding routing path. The score verification package includes the score assigned to each node. When receiving the score verification package, the verification committee updates the score corresponding to each node, and elects the node with the highest score as the block-generating node in the election phase of the evolution cycle.

[0023] Furthermore, the route is input into the route evaluation model to obtain the score of each node in the route, specifically the following steps:

[0024] For route R h , the source node is The destination node is When receiving a data packet, the destination node generates a corresponding score verification packet according to the routing evaluation model, and the routing evaluation model specifically includes the following steps:

[0025] Calculate the distance d between the source node and the destination node when the source node sends a data packet to the destination node h ;

[0026] Calculate the total end-to-end delay T of the data packet from the source node to the destination node h ;

[0027] According to the distance d h , end-to-end total delay T h , calculate the route R h Total score

[0028]

[0029] Assign the total score to route R h The first n nodes on the UAV node In the routing R h The score obtained

[0030]

[0031] in, For UAV nodes In route R hThe score obtained above, τ is the coefficient for adjusting the score distribution ratio;

[0032] The destination node sends the data packet number, sending and receiving time, and the routing R h The nodes and the scores assigned to each node are signed and packaged to generate a score verification package.

[0033] Furthermore, upon receiving the score verification package, the verification committee verifies all signatures therein, and adds the score assigned to each UAV node to the current score of the corresponding UAV node and updates the score of the corresponding UAV.

[0034] Furthermore, the process of parameter fusion is as follows:

[0035] Get UAV node C k Current routing DQN model parameters

[0036] Get block node C sel The current routing DQN model parameters θ sel ;

[0037] According to the θ sel Calculate UAV node C k Routing DQN model parameters for the next evolution cycle

[0038]

[0039] in, and They are UAV nodes C k The number of routing DQN model parameters in the current evolution cycle and the next evolution cycle.

[0040] Furthermore, the strategy for generating a route includes utilizing a routing strategy or exploring a routing strategy, which specifically includes the following processes:

[0041] Before the UAV node forwards the data packet, it compares the random number built into the UAV node with the preset threshold:

[0042] If the random number built into the UAV node is greater than or equal to the preset threshold, the routing strategy is used, and the routing strategy is: the UAV node selects the neighbor node with the largest Q value as the next hop node based on the routing DQN model;

[0043] If the random number built into the UAV node is less than a preset threshold, an exploration routing strategy is performed, wherein the exploration routing strategy is: the UAV node randomly selects a neighbor node as the next hop.

[0044] An intelligent routing method based on a routing DQN model, when a UAV node receives a data packet, includes:

[0045] Get the current state space information of the UAV node;

[0046] Input the current state space information into a pre-trained routing DQN model to determine the next hop neighbor node;

[0047] The routing DQN model is trained based on the UAV node and the training samples corresponding to the UAV node; the training samples include: the current state space information, action, reward value and the maximum Q value in the next hop node during UAV node routing.

[0048] Furthermore, the training process of the routing DQN model includes:

[0049] Build the initial routing DQN model;

[0050] Obtain training samples corresponding to UAV nodes;

[0051] Inputting the training sample into the routing DQN model to obtain the Q value of the selected node;

[0052] Calculate the value of the preset loss function L(θ) according to the training sample and the Q value calculated by the routing DQN model;

[0053] The routing DQN model parameters are updated based on the gradient descent algorithm to minimize the loss function L(θ) and obtain a trained routing DQN model.

[0054] Furthermore, in the training samples:

[0055] The current state space information s={NT, VL}, where VL is an access list for recording nodes that have forwarded the data packet, and NT is a neighbor table consisting of neighbor node location information, the BER transmitted between each neighbor node and the current node, and the queue length of the data packets to be sent in the cache of each neighbor node;

[0056] The action represents selecting a neighboring node that has not been forwarded as the next hop;

[0057] The reward value is a weighted sum of a reference R value, a congestion penalty value, a forwarding angle reward value, a link quality value, and a distance ratio value.

[0058] A UAV:

[0059] The UAV node is configured with a routing DQN model

[0060] The UAV node is configured with a routing evaluation model;

[0061] The UAV node is configured with a block,

[0062] The UAV node specifically performs the following steps in one evolution cycle:

[0063] Update parameters:

[0064] Forward data packets based on the routing DQN model and generate training samples, which are used to train and update the routing DQN model configured by itself, and obtain the routing DQN model parameters in the current consensus cycle;

[0065] Routing score:

[0066] If the current UAV node is the destination node, the total score of the route is obtained based on the route evaluation model, and a score verification package is generated according to the total score and transmitted back to the previous node. The score verification package includes the scores assigned to each node;

[0067] If the current UAV node is not the destination node, the data packet is forwarded to the next node based on the routing DQN model;

[0068] Feedback rating:

[0069] If the current UAV node is a source node, the score verification package is sent to the verification committee;

[0070] If the current UAV node is not the source node, the score verification packet is transmitted back to the previous node, so that the score verification packet is sequentially transmitted back to the source node according to the corresponding routing path, and the source node is used to forward the received score verification packet to the verification committee;

[0071] Election Node:

[0072] If the current UAV node is a member of the verification committee, when receiving the score verification package, verify the signature of the score verification package and update the scores of each UAV node, and participate in the election of the node with the highest score as the block-producing node in the election phase of the evolution cycle;

[0073] Co-evolution:

[0074] If the current UAV node is a block-producing node, it broadcasts its own routing DQN model parameters in the current consensus cycle to the network;

[0075] If the current UAV node is a non-block-producing node, it receives the routing DQN model parameters of the block-producing node and fuses them with its own routing DQN model parameters.

[0076] A control device, comprising:

[0077] one or more processors;

[0078] A storage unit is used to store one or more programs, which, when executed by the one or more processors, enable the one or more processors to implement the distributed collaborative evolution method.

[0079] The present invention has the beneficial effects: the overall technical solution consists of two parts, namely, an intelligent routing algorithm based on the DQN model and a distributed collaborative evolution strategy based on blockchain. In the intelligent routing algorithm based on the DQN model, each UAV in the network acts as an intelligent agent, trains the DQN model individually, and makes routing decisions based on its own learning model. The DQN model enables the UAV to extract network information other than location information as features, and make better routing decisions than traditional GPSR based on a comprehensive consideration of these features. In the distributed collaborative evolution strategy based on blockchain, the core idea is to regularly select the most evolved learning model, and broadcast it to the network by the UAV that owns it, and other UAVs fuse their local parameters with it to achieve the purpose of co-evolution. Among them, the UAV self-organizing network serves as a blockchain, and each UAV corresponds to a single block. The UAV is selected by the consensus algorithm to publish the block, that is, to share the optimal parameters. The specific details are as follows:

[0080] In the intelligent routing algorithm based on the DQN model, a DQN model suitable for UAV distributed training and decision-making is designed in combination with the characteristics and requirements of UAV self-organizing network routing. Specifically, it includes: (1) The perception state space of UAV as an intelligent agent in the network environment and the action space for making routing decisions are designed. (2) According to the characteristics of UAV self-organizing network, a reward function is designed, which enables UAV to comprehensively consider the network environment and make better routing decisions. (3) According to the size of UAV capabilities, a suitable neural network model is established. UAV makes action decisions based on the current state and obtains corresponding rewards. The local neural network model will be updated according to the gradient descent algorithm.

[0081] In the distributed co-evolution strategy based on blockchain, an evaluation model is designed to evaluate the routing performance of each UAV in the current consensus cycle. The evaluation score of each node will be verified and counted by the verification committee selected by the Verifiable Random Function (VRF). The verification committee reaches a consensus on the UAV with the highest score in the current consensus cycle and broadcasts the result to the entire network. The selected UAV broadcasts its DQN parameters to the network, and other UAVs merge their local parameters with it to achieve co-evolution.

[0082] In the present invention, the intelligent routing algorithm based on DQN enables the UAV network routing to achieve lower end-to-end delay and higher data packet transmission rate compared to the traditional GPSR; the distributed co-evolution strategy based on blockchain accelerates the convergence speed of the DQN model and improves the overall routing performance during the UAV mission. Since the method of the present invention has a certain generalization ability and is applicable to routing scenarios in a variety of mobile self-organizing networks, it can also be extended to self-organizing networks such as Internet of Vehicles and wireless sensor networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 This is a schematic diagram of the distributed collaborative evolution strategy based on blockchain of the present invention;

[0084] Figure 2 It is a schematic diagram of the collection stage of the present invention;

[0085] Figure 3 It is a schematic diagram of the election stage of the present invention;

[0086] Figure 4 This is a schematic diagram of the block consensus stage of the present invention;

[0087] Figure 5 This is a schematic diagram of distributed co-evolutionary routing based on blockchain and deep reinforcement learning of the present invention;

[0088] Figure 6 A schematic diagram showing a performance comparison between the routing algorithm of the present invention and a traditional routing algorithm in terms of end-to-end delay;

[0089] Figure 7 A schematic diagram showing a performance comparison between the routing algorithm of the present invention and a traditional routing algorithm in terms of data packet delivery rate;

[0090] Figure 8 A schematic diagram showing a comparison of reward values ​​between the routing algorithm of the present invention and a common intelligent routing algorithm in training iterations; DETAILED DESCRIPTION

[0091] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present invention and its application or use. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0092] The relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless specifically stated otherwise.

[0093] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0094] Additionally, descriptions of well-known structures, functions, and configurations may be omitted for clarity and conciseness.One of ordinary skill in the art will recognize that various changes and modifications may be made to the examples described herein without departing from the spirit and scope of the present disclosure.

[0095] Technologies, methods, and apparatus known to ordinary technicians in the relevant field may not be discussed in detail, but where appropriate, such technologies, methods, and apparatus should be considered part of the authorization specification.

[0096] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0097] 1. Intelligent routing algorithm based on DQN model

[0098] The core of the DQN-based intelligent routing algorithm is that each UAV uses its own DQN model to comprehensively consider network environment information, making routing decisions more efficient and evolving with the environment, improving dynamic adaptability. This section first introduces the overall idea of ​​the algorithm, then details the setting of the reward function in the DQN model, and finally gives the pseudo code of the overall algorithm flow.

[0099] (1) Overall thinking

[0100] In the DQN-based intelligent routing algorithm, the UAV calculates the Q value of the neighbor node through the DQN model and selects the neighbor node with the largest Q value as the next hop. The DQN model extracts the access list, the bit error rate (BER) of the transmission, the location information of the neighbor node, and the queue length as features, and applies the reward function to directly act on the calculation of the Q value. In this way, the above environmental information is comprehensively considered in the calculation of the Q value of the neighbor node, and the routing decision becomes more reasonable and efficient. As the DQN model continues to converge, the Q value can better reflect the true state of the neighbor node. The extraction of features and the setting of the reward function will be specifically introduced in Section 2 of this part.

[0101] In this routing algorithm, the node only considers the next hop selection when routing, so we record the current node as C and its neighbor nodes as {N1, N2, N3, ..., N j,...}. The routing process can be regarded as a Markov Decision Process (MDP). MDP is represented by a four-tuple (S, A, P, R), where S is the state space, A is the action space, P is the state transition probability, and R is the reward value. The current node acts as an intelligent agent and makes the best routing decision based on the current state to obtain the maximum reward value. The specific definition of MDP is as follows:

[0102] State space, action space and reward value

[0103] State space: The state space of the current node C is defined as an access list and a neighbor list. The data packet will contain an access list (VL), which records the nodes that have forwarded this data packet. Whenever a node forwards a data packet, it will add itself to the access list of this packet. Let the neighbor node N j The location information is N j The BER of transmission between C is N j The queue length of the packets to be sent in the cache is The three form a neighbor vector The neighbor table consists of neighbor vectors, denoted as NT = {NI1, NI2, ..., NI j , ...}. The current node can learn the current position, queue length and transmission BER of the neighbor node through the regularly sent HELLO information and received ACK, and update its neighbor table. Therefore, the state space of the current node can be expressed as s = {NT, VL}.

[0104] Action space: The action space contains all actions that the agent can perform in the current state. Here, the action space is the routing decision space of the current node, that is, all the routing options for the next hop. The current node is defined to only select neighbor nodes that have not forwarded this packet to prevent wasting routing resources. Therefore, the action space only contains neighbor nodes that are not in the access list, denoted by A = {N j}-VL. Among them, action a j (a j ∈A) represents the selection of neighbor node N j For the next hop.

[0105] Reward value: The agent takes an action in the current state and receives a corresponding reward. The reward value reflects the quality of this action. Therefore, the design of the reward function is particularly important, as it directly determines the evolutionary direction of the agent. In this routing algorithm, the calculation of the reward function comprehensively considers the access list, transmission BER, queue length and location information of neighboring nodes, achieving traffic balance, avoiding the use of peripheral modes and improving communication quality. For the specific calculation method of the reward value, see "(2) Reward function design".

[0106] Training process

[0107] The current node C selects the neighbor node N j As the next hop, and obtain the reward value. j After successfully receiving the data packet, it replies ACK to the current node C, which contains its current location information, BER, queue length, and the maximum Q value among neighboring nodes. The current node C updates the neighbor table based on this information and communicates the state, decision, and reward value of the routing with the node N. j The maximum Q value of the next hop is combined into a four-tuple (s, a j , R,max a , Q(s′, a′)) is used as a training sample and stored in the sample pool.

[0108] During training, the node uses the memory playback mechanism to train its own DQN model, that is, randomly extracting training samples from the sample pool for training. The training updates the DQN model parameters through the gradient descent algorithm, and the goal is to minimize the loss function L(θ), that is, the mean square error between the evaluated Q value and the target Q value, which is calculated as follows:

[0109] L(θ i )=E[Q target -Q(s, a; θ i )] 2 #(1)

[0110] Among them, θ i represents the model parameters of the i-th iteration, Q target is the target Q value, which is calculated as follows:

[0111]

[0112] Among them, R total is the total reward value obtained, max a′ Q(s′, a′) is the maximum Q value of the next-hop neighbor node. The evaluation network parameters in the DQN model are updated with each iteration, while the target network parameters are fixed. After a certain number of iterations, the evaluation network synchronizes the parameters to the target network to ensure the stability and convergence of the DQN model.

[0113] After many iterations, the DQN model converges and the Q value can reflect the real state of neighbor nodes, so the nodes can make routing decisions that are better than traditional GPSR.

[0114] (2) Reward Function Design

[0115] Traditional GPSR only considers location information when routing, which has the following problems: (1) Traffic is concentrated on the best-located node, causing network congestion. (2) The use of peripheral patterns increases the number of hops and latency. (3) The quality of the communication link is ignored during routing, reducing routing efficiency. To solve the above problems, the access list, queue length and BER information are applied to the calculation of the reward value, so that the environmental information is comprehensively considered during node routing, achieving the effect of traffic balance, avoiding peripheral patterns and improving link quality. The specific design of the reward function is as follows:

[0116] Traffic balance

[0117] The traditional GPSR protocol only considers the node location when routing, so when routing, the traffic will be concentrated on the neighboring node closest to the destination node, causing network congestion. This will lead to high queuing delay and even packet loss.

[0118] To solve this problem, a traffic balancing strategy is applied in this routing algorithm. Each node can obtain local congestion information from its neighbors and apply it to routing decisions. Here, the queue length in the neighbor node cache is used to determine its congestion status. A larger queue length means a higher queue delay and a greater probability of packet loss. The current node can learn its current queue length from the HELLO message and the ACK reply sent regularly by the neighbor node, and update the neighbor information table in time. A penalty mechanism is applied to force the current node C to consider congestion factors. The congestion penalty acts on the Q value and is calculated as follows:

[0119]

[0120] in, is the neighbor node N j The queue length, is node N j The cache size, Q(C, N j ) is node N j The original Q value in node C, β1 and β2 are congestion penalty coefficients.

[0121] Avoid Peripheral Mode

[0122] In UAV networks, routing holes are likely to occur due to the high-speed movement of UAVs. In traditional GPSR, when the current node is in the local minimum position, it must switch to the peripheral mode to bypass the hole area, but the peripheral mode will bring more hops and longer delays. Therefore, this routing algorithm uses a direct transmission strategy to avoid the use of the peripheral mode.

[0123] The current node C will extract the access list in the data packet and know which neighbor nodes have been visited, and make better routing decisions based on this. Here we define ∠V f CNj (≤180°) is the forwarding angle, where V f is the neighbor node included in VL. f CN j The angle is calculated as follows:

[0124]

[0125] Among them, P C , and Represent nodes C and V respectively f and N j geographical location.

[0126] Based on the forwarding angle, the current node routes the data packet as straight as possible to the destination node, so that it can avoid the area that has been routed and reduce the use of surrounding modes. The forwarding angle is extracted as a feature and acts on the Q value with a reward function, which is calculated as follows:

[0127]

[0128] Where num represents V f The number of

[0129] Link Quality

[0130] In UAV ad hoc networks, the network environment is complex and changeable, for example, there may be severe electromagnetic interference. Since UAVs are distributed in a vast space, for a node, the link quality between it and different neighboring nodes will vary greatly. In this routing algorithm, BER is used to represent link quality and is used in the calculation of Q value, forcing nodes to consider link quality when routing. The reward is calculated as follows:

[0131]

[0132] in, Represents the current node C and its neighbor node N j The bit error rate of transmission between the two packets can be obtained from the HELLO message and the reply ACK.

[0133] In addition to the above three strategies, the distance between the next hop and the destination node is also considered, which is the core idea of ​​traditional GPSR, aiming to forward data packets to the vicinity of the destination node. The distance reward function is calculated as follows:

[0134]

[0135] in, and d C Represents node N j and the distance from C to the destination node

[0136] Therefore, the total reward value is calculated as:

[0137] R total =R+α1R con +α2R dir +α3R qua +α4R dis #(8)

[0138] Among them, α1, α2, α3 and α4 are weight coefficients, and R is calculated as follows:

[0139]

[0140] According to the context, the baseline R value is R and the congestion penalty value is R con , the forwarding angle reward value is R dir , link quality value R qua , the distance ratio is R dis .

[0141] (3) Algorithm Process

[0142] In the DQN-based intelligent routing algorithm, each UAV makes routing decisions through its own DQN model, and generates training samples for each routing process to train its own DQN model. With continuous iterations, the DQN model converges and the UAV can perform efficient routing.

[0143] When there is a routing task, UAV adopts the ∈ greedy strategy, that is, there are two options for routing decisions: exploitation or exploration. UAV selects the neighbor node with the largest Q value as the next hop, which is the exploitation strategy with a probability of 1-∈; or, UAV randomly selects a neighbor node as the next hop, which is the exploration strategy with a probability of ∈. Among them, ∈ decreases as the number of iterations increases. After sending to the neighbor node, UAV extracts the maximum Q value of the next hop from the replied ACK and generates training samples (s, a, R, max a′ Q(s′, a′)) and stored in the sample pool.

[0144] When idle, the UAV enters the training phase, randomly extracts a number of training samples from the sample pool for training, and updates the DQN model parameters by minimizing the loss function L(θ). After a certain number of iterations, the target network obtains the neural network parameters from the evaluation network and updates them.

[0145] The specific process is as follows:

[0146]

[0147]

[0148] 2. Distributed collaborative evolution strategy based on blockchain

[0149] The core of the distributed co-evolution strategy based on blockchain is to use the designed evaluation model to evaluate the routing performance of each UAV in the current consensus cycle, and the verification committee selects the UAV with the highest score. The selected UAV broadcasts its DQN model parameters to the network, and other UAVs merge their local parameters with it to achieve co-evolution. This section first introduces the evaluation model and then the overall consensus process.

[0150] (1) Evaluation Model

[0151] To facilitate the evaluation of routing performance, the following concepts are first defined:

[0152] Unit distance delay: In a UAV ad hoc network, the ratio of the total end-to-end delay of a data packet from the source node to the destination node during its routing process to the distance between the source and destination nodes of the data packet is called the unit distance delay of the data packet.

[0153] Average unit distance delay: The average unit distance delay of all data packets in the UAV ad hoc network.

[0154] Basic unit distance delay: The average unit distance delay in traditional GPSR is defined as the basic unit distance delay and is represented by t0.

[0155] On this basis, the present invention uses the basic unit distance delay t0 as a benchmark to measure the quality of the route experienced by a given data packet in the DQN-based UAV network. h , which starts from the source node To the destination node Passing through the intermediate nodes in sequence Let this route be R h Then the route R h The formula for calculating the routing performance score is:

[0156]

[0157] Among them, d h The source node sends a data packet P to the destination node. h The distance from the source node to the destination node, T h It is the data packet P h The total end-to-end delay from the source node to the destination node, μ is the coefficient.

[0158] For routing P h The total score of the route R hConsidering that in the routing process from the source node to the destination node, the contribution of each node in the middle to the overall routing performance is different, and the node with the earlier forwarding order has a greater contribution. Therefore, S h In route R h The allocation method on each node is as follows:

[0159]

[0160] in, For routing nodes In this route R h The score obtained above, τ is the coefficient for adjusting the score distribution ratio. No points are assigned.

[0161] When the source node forwards a data packet, it will indicate the forwarding time and the distance from the destination node at this time and sign it. Other nodes will sign in turn when forwarding. Therefore, when the destination node receives the data packet, it can calculate the total score based on the arrival time and assign scores to each node according to the signature order in the data packet. Then, the destination node generates a score verification package, which contains the number of the data packet just received, the sending and receiving time, the routing nodes passed, and the scores assigned to each node, and forwards the package to the previous node after signing. After checking that the score calculation and allocation are correct, each node signs and returns the package in turn to the source node. After checking that it is correct, the source node signs and forwards the package to the verification committee. During the return process, the network load increases significantly. Therefore, the co-evolution strategy can be carried out periodically according to the specific network situation, so as to balance the load increase and the co-evolution effect.

[0162] The verification committee is selected from all nodes by cryptographic lottery implemented by a verifiable random function (VRF). After receiving the score verification package, the verification committee first verifies the signature inside. After successful verification, the score assigned to each node is added to its current score. When the collection phase is over, the verification committee reaches a consensus on the node with the highest score and broadcasts it to the network. The verification committee for each round is selected by the previous committee, and the probability of each node being selected is proportional to its score at the end of the previous round. The number of validators on the verification committee can be adjusted according to the size of the UAV network and the number of potential attackers. At the beginning of a new consensus cycle, the score of each node is reset to zero.

[0163] (2) Consensus Process

[0164] In the distributed co-evolution strategy based on blockchain, the consensus algorithm process can be divided into the collection phase, the election phase and the block consensus phase. col ,t ele ,t conThey are the timing variables for the collection phase, election phase, and block consensus phase. When the timing is set to zero, it indicates the start of the corresponding phase. When the timing reaches a given time, it indicates that the corresponding phase ends and enters the next phase or the next consensus cycle begins. Set act_flag as the collection and election flag. When it is set to 0, it enters the collection phase, and when it is set to 1, it enters the election phase; con_flag is the block consensus flag. When it is set to 0, it enters the block consensus phase, and when it is set to 1, it is ready to enter the next consensus cycle. The UAV self-organizing network is regarded as the entire blockchain, and each drone corresponds to a block. Define the number of UAVs in the network as m(m∈N + ), each UAV is represented by {C1, C2, C3, ..., C m The UAV selected by the verification committee in this round is denoted as C sel The consensus algorithm process is as follows:

[0165] (a) Collection phase

[0166] During the collection phase, UAV C k Execute routing tasks and generate samples, and use the collected samples to train its DQN model. When receiving the score verification package, C k After verifying that the score assigned to itself is correct, it signs (not the verifier) ​​and sends the packet back to the previous node or forwards it to the nearest verifier. k It is a verifier, who is responsible for updating and maintaining the scores of all nodes. When receiving the score verification package, it needs to verify all signatures in it and add the assigned score to the corresponding node. The algorithm flow is as follows:

[0167]

[0168]

[0169] (b) Election stage

[0170] In the election phase, the verification committee first selects the verification committee for the next round through VRF and publishes the identity of the new verifier to the network. Then, the verification committee reaches a consensus on the node with the highest score in this round and publishes this node to the network. Among them, the message of announcing the new verification committee and the highest-scoring node is signed by all members of the verification committee. The algorithm flow is as follows:

[0171]

[0172]

[0173] (c) Block Consensus Phase

[0174] In the block consensus phase, C selected by the verification committeesel The signed DQN model parameters are broadcast to the entire network. Other nodes verify the signature after receiving it. If the verification is successful, the parameters are integrated. If the parameters are not received for a long time, other nodes can send a message to C sel Send the request.

[0175]

[0176]

[0177] When fusing parameters, a weighted average method is used. The weight parameter ρ is used to adjust the proportion of the optimal parameter in the fusion to ensure the stability of the local model. The parameter fusion calculation is as follows:

[0178]

[0179] in, and They are UAV C k The parameters in the rth and r+1th consensus cycles, θ sel The consensus algorithm selects C sel Parameters.

[0180] 3. Simulation Example

[0181] To illustrate the effect of the present invention, the performance of the present invention in a given scenario is further described below in conjunction with a specific simulation example.

[0182] This simulation example is based on the Pycharm platform and uses the Pytorch package to establish a convolutional neural network. The UAV self-organizing network in the simulation consists of fifty mobile nodes, which transmit data packets to each other according to preset parameters. In order to prove the superiority of the present invention, the routing algorithm proposed in the present invention will be compared with the traditional GPSR and the ordinary intelligent routing algorithm, in which the ordinary intelligent routing algorithm has the same DQN model as this routing algorithm, but lacks the distributed co-evolution strategy based on blockchain. In the simulation, there are three performance evaluation indicators, namely end-to-end delay, data packet delivery rate and reward value. By comparing the changes in the reward value of this routing algorithm and the ordinary intelligent routing algorithm, the effect of co-evolution can be observed.

[0183] The following are the simulation results:

[0184] Figure 6 The performance of the three routing algorithms in terms of end-to-end delay. It can be seen that after convergence, the end-to-end delay of the routing algorithm proposed by the present invention and the common intelligent routing algorithm is stable at about 200 milliseconds, while the end-to-end delay of the traditional GPSR is about 260 milliseconds. In addition, compared with the common intelligent routing algorithm, the delay of this routing algorithm decreases faster and converges earlier.

[0185] Figure 7 The performance of the three routing algorithms in terms of packet delivery rate. It can be seen that after convergence, the packet delivery rate of the routing algorithm proposed by the present invention and the ordinary intelligent routing algorithm is stable at about 72%, while the packet delivery rate of the traditional GPSR is about 67%. Similar to the end-to-end delay, the routing algorithm converges faster than the ordinary intelligent routing algorithm.

[0186] Figure 8 The reward value comparison between the routing algorithm proposed by the present invention and the common intelligent routing algorithm is shown in FIG. 1 . As shown, the reward value of the routing algorithm proposed by the present invention increases faster and is always higher than that of the common intelligent routing algorithm before convergence.

[0187] In summary, the distributed co-evolutionary routing technology based on blockchain and deep reinforcement learning of the present invention has lower end-to-end delay and higher packet delivery rate than traditional GPSR, and has faster convergence speed than ordinary intelligent routing algorithms.

[0188] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. According to the technical essence of the present invention, within the spirit and principles of the present invention, any simple modification, equivalent replacement and improvement made to the above embodiment still falls within the protection scope of the technical solution of the present invention.

Claims

1. A distributed collaborative evolution method, characterized in that: Applicable to a network composed of a plurality of UAV nodes, the network comprising a verification committee composed of at least one UAV node; Each UAV node is configured with a routing DQN model; Each UAV node is configured with a routing evaluation model; Each UAV node is configured as a block; The data packet is forwarded from the source node in the network to the destination node in sequence, forming a route from the source node to the destination node corresponding to the data packet; In one evolution cycle, the following steps are performed: S1: Update routing DQN model parameters: The UAV node trains its own routing DQN model based on the samples generated by the routing and updates its own routing DQN model parameters; S2: Elect block nodes: Input the route into the route evaluation model to obtain the score of each node in the route, so that the verification committee selects the node with the highest score as the block generating node according to the score of each node in the route; Input the route into the route evaluation model to obtain the score of each node in the route, specifically the following steps: For route R h , the source node is The destination node is When receiving a data packet, the destination node generates a corresponding score verification packet according to the routing evaluation model, and the routing evaluation model specifically includes the following steps: Calculate the distance d between the source node and the destination node when the source node sends a data packet to the destination node h ; Calculate the total end-to-end delay T of the data packet from the source node to the destination node h ; According to the distance d h , end-to-end total delay T h , calculate the total score of route Rh Among them, t0 represents the basic unit distance delay, and μ represents the coefficient; Assign the total score to route R h The first n nodes on the UAV node In the routing R h The score obtained in, For UAV nodes In route R h The score obtained above, τ is the coefficient for adjusting the score distribution ratio; The destination node sends the data packet number, sending and receiving time, and the routing R h The nodes and the scores assigned to each node are signed and packaged to generate a score verification package; The verification committee verifies all signatures in the score verification package upon receiving it, and adds the score assigned to each UAV node to the current score of the corresponding UAV node and updates the score of the corresponding UAV; S3: Broadcast routing DQN model parameters: The block-producing node broadcasts its own routing DQN model parameters to the network; S4: Parameter co-evolution: The non-block producing node receives the routing DQN model parameters of the block producing node and fuses them with its own routing DQN model parameters to obtain the evolved routing DQN model parameters.

2. A distributed collaborative evolution method according to claim 1, characterized in that: The updating routing DQN model parameters specifically include: When forwarding a data packet, the UAV node forwards the data packet to the next node with the maximum Q value based on the routing DQN model and generates training samples. The samples are used to train and update its own routing DQN model when idle, and generate updated routing DQN model parameters after iteration.

3. A distributed collaborative evolution method according to claim 1, characterized in that: The block node selection specifically includes: When the destination node receives the data packet, it obtains the total score of the route based on the routing evaluation model, and generates a score verification package based on the total score. The score verification package in the network is transmitted back to the source node in sequence according to the corresponding routing path. The score verification package includes the score assigned to each node. When receiving the score verification package, the verification committee updates the score corresponding to each node, and elects the node with the highest score as the block-generating node in the election phase of the evolution cycle.

4. A distributed collaborative evolution method according to claim 1, characterized in that: The process of parameter fusion is as follows: Get UAV node C k Current routing DQN model parameters Get block node C sel The current routing DQN model parameters θ sel ; According to the θ sel Calculate UAV node C k Routing DQN model parameters for the next evolution cycle in, and They are UAV nodes C k The routing DQN model parameters in the current evolution cycle and the next evolution cycle, ρ represents the weight parameter.

5. A distributed collaborative evolution method according to claim 1, characterized in that: The strategies for generating routes include using routing strategies or exploring routing strategies, which include the following processes: Before the UAV node forwards the data packet, it compares the random number built into the UAV node with the preset threshold: If the random number built into the UAV node is greater than or equal to the preset threshold, the routing strategy is used, and the routing strategy is: the UAV node selects the neighbor node with the largest Q value as the next hop based on the routing DQN model; If the random number built into the UAV node is less than a preset threshold, an exploration routing strategy is performed, wherein the exploration routing strategy is: the UAV node randomly selects a neighbor node as the next hop.

6. An intelligent routing method based on a routing DQN model, characterized in that: Applied to a UAV node in a distributed collaborative evolution method as claimed in claim 1, when the UAV node receives a data packet, the following steps are performed: Get the current state space information of the UAV node; Input the current state space information into a pre-trained routing DQN model to determine the next hop neighbor node; The routing DQN model is trained based on the UAV node and the training samples corresponding to the UAV node; the training samples include: the current state space information, action, reward value and the maximum Q value in the next hop node during UAV node routing.

7. The intelligent routing method based on the routing DQN model according to claim 6, characterized in that: The training process of the routing DQN model includes: Build the initial routing DQN model; Obtain training samples corresponding to UAV nodes; Inputting the training sample into the routing DQN model to obtain the Q value of the selected node; Calculate the value of the preset loss function L(θ) according to the training sample and the Q value calculated by the routing DQN model; The routing DQN model parameters are updated based on the gradient descent algorithm to minimize the loss function L(θ) and obtain a trained routing DQN model.

8. The intelligent routing method based on the routing DQN model according to claim 7, characterized in that: In the training samples: The current state space information s={NT, VL}, where VL is an access list for recording nodes that have forwarded the data packet, and NT is a neighbor table consisting of neighbor node location information, the BER transmitted between each neighbor node and the current node, and the queue length of the data packets to be sent in the cache of each neighbor node; The action represents selecting a neighboring node that has not been forwarded as the next hop node; The reward value is a weighted sum of a reference R value, a congestion penalty value, a forwarding angle reward value, a link quality value, and a distance ratio value.

9. A UAV, characterized in that: The UAV node is configured with a routing DQN model The UAV node is configured with a routing evaluation model; The UAV nodes are configured as blocks, The UAV node specifically performs the following steps in one evolution cycle: Update parameters: Forward data packets based on the routing DQN model and generate training samples, which are used to train and update the routing DQN model configured by itself, and obtain the routing DQN model parameters in the current consensus cycle; Routing score: If the current UAV node is the destination node, the total score of the route is obtained based on the route evaluation model, and a score verification package is generated according to the total score and transmitted back to the previous node. The score verification package includes the scores assigned to each node; If the current UAV node is not the destination node, the data packet is forwarded to the next node based on the routing DQN model; For route R h , the source node is The destination node is When receiving a data packet, the destination node generates a corresponding score verification packet according to the routing evaluation model, and the routing evaluation model specifically includes the following steps: Calculate the distance d between the source node and the destination node when the source node sends a data packet to the destination node h ; Calculate the total end-to-end delay T of the data packet from the source node to the destination node h ; According to the distance d h , end-to-end total delay T h , calculate the route R h Total score Among them, t0 represents the basic unit distance delay, and μ represents the coefficient; Assign the total score to route R h The first n nodes on the UAV node In the routing R h The score obtained in, For UAV nodes In route R h The score obtained above, τ is the coefficient for adjusting the score distribution ratio; The destination node sends the data packet number, sending and receiving time, and the routing R h The nodes and the scores assigned to each node are signed and packaged to generate a score verification package; Feedback rating: If the current UAV node is a source node, the score verification package is sent to the verification committee; If the current UAV node is not the source node, the score verification packet is transmitted back to the previous node, so that the score verification packet is sequentially transmitted back to the source node according to the corresponding routing path, and the source node is used to forward the received score verification packet to the verification committee; Election Node: If the current UAV node is a member of the verification committee, when receiving the score verification package, verify the signature of the score verification package and update the scores of each UAV node, and elect the node with the highest score as the block node in the election phase of the evolution cycle; Co-evolution: If the current UAV node is a block-producing node, it broadcasts its own routing DQN model parameters in the current consensus cycle to the network; If the current UAV node is a non-block-producing node, it receives the routing DQN model parameters of the block-producing node and fuses them with its own routing DQN model parameters.

10. A control device, characterized in that: include: one or more processors; A storage unit, used to store one or more programs, which, when executed by the one or more processors, enable the one or more processors to implement a distributed collaborative evolution method according to any one of claims 1 to 5.

Images