An adaptive message propagation method under a UAV ad hoc network

Abstract

The application discloses an adaptive message propagation method under a UAV ad hoc network, and comprises the following steps: in each round of communication of the UAV ad hoc network, a UAV containing a pre-trained model communicates with neighbors at t time slots to obtain preset strings and message redundancy of the t time slots of the neighbors and the UAV; the preset string of each UAV is used for recording the communication conditions between the UAV and each neighbor, and between each neighbor and the neighbors of the UAV from the start of the round of communication to the t time slots; the UAV obtains the t time slot observation space of each of the UAV and the neighbors and a plurality of forwarding strategies of the t time slots according to the preset string and the message redundancy of the t time slots; the UAV inputs the forwarding strategies and the processed t time slot observation space into the model, obtains target forwarding probability and target forwarding neighbors according to the principle of maximum Q value, and forwards the message to the target forwarding neighbors at the t time slots.

Description

Technical Field

[0001] This invention belongs to the field of unmanned aerial vehicle (UAV) communication technology, specifically relating to an adaptive message propagation method under an UAV ad hoc network. Background Technology

[0002] Unmanned aerial vehicles (UAVs), with their advantages of high maneuverability, low cost, and ease of deployment, are widely used in activities such as field research, disaster relief, industrial production, and agricultural plant protection. According to the "UAV Market Report 2020-2025" released by a German UAV research company, the global UAV market is expected to double to $42.8 billion by 2025. With the development and application of UAVs, UAV self-organizing networks have emerged to meet the increasingly complex needs of task collaboration. Without relying on any infrastructure, UAV self-organizing networks automatically connect and build a wireless multi-hop self-organizing network, with nodes able to send messages to each other. During task collaboration, UAVs need to broadcast critical control information or data via multiple hops; for example, they can use broadcasts to negotiate group keys to achieve secure UAV group communication.

[0003] Existing multi-hop broadcast protocols include flooding and gossip. Flooding offers optimal propagation latency but suffers from the problem of redundant message implosion. In contrast, the gossip protocol avoids this implosion problem, reduces drone power consumption, and is more suitable for resource-constrained drones. Initially, each drone, acting as a source node, sends its message packet to one or more neighboring nodes. In each subsequent round, the relay node receiving the message packet decides with a certain probability whether to forward it; if so, it randomly selects one or more neighboring nodes to forward it. Theoretically, the gossip protocol will continue to run for multiple rounds until all nodes have received the message packet. However, as... Figure 1 As shown, due to the random forwarding characteristic of gossip, messages are forwarded to only a limited number of nodes in each round, resulting in excessively long propagation delays. To reduce propagation delays, existing works have improved the gossip protocol from two aspects: forwarding probability decision and forwarding neighbor decision [4-9].

[0004] To reduce the communication overhead of gossip algorithms in information propagation, various methods exist in related technologies. For example, the Binary Exponential Backoff Gossip Algorithm (BEBG), which combines the binary exponential backoff algorithm with the classic gossip algorithm, employs a propagation strategy where the more times a node receives the same information, the lower the probability of continuing to propagate it. This is further reduced by incorporating a PULL mechanism to decrease redundant messages and reduce propagation latency. Another example is an improved gossip protocol called Brisa, which maintains a view called HyParView to discover special structures in the current topology, such as trees, directed acyclic graphs, and forests, and achieves more efficient propagation based on these structures. Yet another example is a hierarchical improved gossip protocol called ProFI, which constructs a dominance set to partition a hierarchical structure containing peer nodes and uses an energy consumption model to allow some nodes to switch to an idle state, thereby eliminating unnecessary gossip forwarding operations. Finally, there are methods that utilize a biomimetic attractor selection model, using vehicle position and speed as auxiliary information for neighbor forwarding decisions to improve gossip reachability and efficiency. For example, improvements can be achieved by introducing determinism into gossip forwarding neighbor selection. Specifically, a policy learning mechanism is introduced, based on the proposed utility function, allowing each node to effectively determine the optimal forwarding neighbor node based on the actions of other nodes and the loss pattern within the system. Another example is two improved gossip protocols based on distance measurements: the first, NNGossip, uses nearest neighbor distance measurement, while the second, CBGossip, uses city block distance measurement. These protocols calculate the scores of candidate neighbor nodes using different methods and select the forwarding neighbor based on the highest score.

[0005] However, the above schemes only consider forwarding probability constraints or forwarding neighbor constraints. Furthermore, when only forwarding probability constraints are considered, forwarding to neighbors remains completely random, failing to forward packets to nodes that haven't yet received them, resulting in low propagation efficiency. When only forwarding neighbor constraints are considered, duplicate packets are always forwarded, leading to redundant messages. Additionally, when applied to dynamic topology environments like drones, these schemes still forward packets to nodes that have already received them, resulting in low propagation efficiency. Summary of the Invention

[0006] To address the aforementioned problems in related technologies, this invention provides an adaptive message propagation method for unmanned aerial vehicle (UAV) ad hoc networks. The technical problem to be solved by this invention is achieved through the following technical solution:

[0007] This invention provides an adaptive message propagation method for unmanned aerial vehicle (UAV) ad hoc networks, comprising:

[0008] In each round of communication in the UAV ad hoc network, any UAV containing a pre-trained multi-branch agent reinforcement learning model communicates with neighboring UAVs in time slot t to obtain the preset strings for time slot t for both the neighboring UAVs and the message redundancy of time slot t for each UAV. The preset string for each UAV is used to record the message sending and receiving status between the UAV and each neighboring UAV, and between each neighboring UAV and its own neighboring UAV, from the start of this round of communication to time slot t. t is an integer greater than 0.

[0009] The drone obtains the observation space of the drone and each of the neighboring drones in time slot t based on the preset string and message redundancy of time slot t. Based on different preset forwarding probabilities, it determines multiple forwarding strategies for time slot t, and each forwarding strategy includes a preset forwarding and at least one forwarding neighbor.

[0010] After precoding the observation space of any UAV in time slot t, the precoding result is input into the model along with multiple forwarding strategies for time slot t, and the target forwarding strategy is obtained according to the principle of maximum Q value; the model is obtained by training the initial multi-branch agent reinforcement learning model using the observation space and forwarding strategies of multiple historical time slots of each UAV in the ad hoc network.

[0011] The drone forwards the message to the target neighbor in time slot t; the target neighbor is the forwarding neighbor corresponding to the target forwarding strategy, and is the drone with the highest probability of not receiving the message among all the neighboring drones of the drone.

[0012] The present invention has the following beneficial technical effects:

[0013] By maintaining a preset string within each drone for real-time updates of message sending and receiving between any drone and its neighboring drones, and between each neighboring drone and its own neighboring drones, and by obtaining the message redundancy of neighboring drones in real time, the historical reception status of message packets of all drones can be monitored in real time during communication among drones. Furthermore, by using the observation space obtained from the historical reception status of message packets of all drones monitored in real time during each round of communication, the model trained with the forwarding strategy determined in real time, and the historical reception status of message packets of all drones currently monitored by the drone, the neighboring drone to forward the message to can be determined. Thus, by considering the dependencies between each drone and its neighboring drones and the historical reception status of message packets of each drone, the neighboring drone that needs to receive the message can be accurately determined, thereby improving the message propagation efficiency between drones.

[0014] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0015] Figure 1 This is an exemplary diagram illustrating a broadcast scenario using a drone as the source node and employing flooding and gossip, provided as an embodiment of the present invention.

[0016] Figure 2 A flowchart of an adaptive message propagation method for unmanned aerial vehicle (UAV) ad hoc networks provided in an embodiment of the present invention;

[0017] Figure 3 A schematic diagram of a bitgraph of an exemplary drone A provided for an embodiment of the present invention;

[0018] Figure 4 An exemplary diagram of message broadcasting under Bitgraph provided for embodiments of the present invention;

[0019] Figure 5 The exemplary architecture diagram of an unmanned aerial vehicle (UAV) using a pre-trained multi-branch agent reinforcement learning model to communicate with neighboring nodes in an ad hoc network, and the network structure diagram of the pre-trained multi-branch agent reinforcement learning model, are provided for embodiments of the present invention.

[0020] Figure 6 A comparative diagram illustrating the number of times an exemplary message broadcast is required, provided for embodiments of the present invention;

[0021] Figure 7 A comparative diagram illustrating the time delay required for exemplary message broadcasting provided in embodiments of the present invention;

[0022] Figure 8 A comparative diagram illustrating the number of redundant message broadcast packets provided in an embodiment of the present invention;

[0023] Figure 9 An exemplary system architecture diagram of the present invention is provided for embodiments of the present invention. Detailed Implementation

[0024] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0025] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0026] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0027] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings, disclosure, and appended claims in carrying out the claimed invention. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0028] Figure 2 This is a flowchart of an adaptive message propagation method for unmanned aerial vehicle (UAV) ad hoc networks provided in an embodiment of the present invention, as shown below. Figure 2 As shown, the method includes the following steps:

[0029] S101. In each round of communication in the UAV ad hoc network, any UAV containing a pre-trained multi-branch agent reinforcement learning model communicates with neighboring UAVs in time slot t to obtain the preset strings of each neighboring UAV and the message redundancy of time slot t for each UAV. The preset string of each UAV is used to record the message sending and receiving status between the UAV and each neighboring UAV, and between each neighboring UAV and its own neighboring UAV, from the start of this round of communication to time slot t. t is an integer greater than 0.

[0030] Here, a drone ad hoc network can be a network composed of multiple drones, where each drone has multiple neighboring drones.

[0031] Here, each round of communication in a drone ad hoc network refers to the multiple communications between drones within the network from the start of executing a task until the task is completed. For example, when a drone ad hoc network needs to execute two different tasks, there will be two rounds of communication. `t` represents each time slot in a round of communication. Each time slot can be a preset time period or a preset moment.

[0032] Here, the default string length is N squared; N is the total number of drones in the ad hoc network; each drone corresponds to N characters, one of the N characters is used to record whether the drone itself has sent a message, and the remaining N-1 characters are used to record whether the drone has received messages from the remaining N-1 drones, where N is an integer greater than 2.

[0033] Here, the message sending and receiving status between any drone and its neighboring drones includes: whether the drone has sent its own message and whether the drone has received messages from its neighboring drones.

[0034] Here, before each round of communication in the ad hoc network, each drone initializes the value of its preset string to the first value; in each round of communication in the ad hoc network, when the preset string of any drone in the Rth time slot is the second value, it indicates that the drone has sent a message and received messages from each neighboring drone in this round of communication. At this time, the drone enters a silent state; R is an integer greater than t.

[0035] Specifically, the default string is "Bitgraph," a 01 string that provides a virtual global view of packet reception for the drone during message propagation. In the ad hoc network, all drones are logically ordered before deployment. The length of the Bitgraph string is the square of the number of members in the ad hoc network, and the string is partitioned and segmented according to the logical order of the members. For example, Figure 3 As shown, when the ad hoc network includes three drones, A, B, and C, the Bitgraph string is 9 characters long. Each drone has a corresponding area in logical order, and each area contains corresponding bits in logical order. For example, Figure 3 This is a schematic diagram of a bitgraph for drone A, where the underlined bits represent whether the corresponding drone has sent a message packet, such as... Figure 3As shown, when the underlined bit in drone A's partition is 1, it means drone A sent its own message packet; when it's 0, it means drone A did not send its own message packet. The second 1 indicates drone A received a message packet from drone B, and the third 0 indicates drone A did not receive a message packet from drone C. In drone B's partition, two 0s indicate drone B did not receive message packets from drones A and C respectively, and a 1 indicates drone B sent its own message packet. In drone C's partition, the first two 1s indicate drone C received message packets from drones A and B respectively, and the last 1 indicates drone C sent its own message packet. Before each round of communication in the ad hoc network, each drone initializes its own Bitgraph value to 0.

[0036] In some embodiments, the neighboring drones of any drone include: a first neighboring drone and a second neighboring drone; based on this, in S101 above, any drone containing a pre-trained multi-branch agent reinforcement learning model obtains the preset string and message redundancy of time slot t for both the neighboring drones and any drone by communicating with the neighboring drones in time slot t, including:

[0037] S1011. In time slot t, any UAV receives a message sent by at least one of its first neighboring UAVs, a preset string for time slot t, and a message redundancy of time slot t, thereby obtaining the preset string for time slot t and the message redundancy of time slot t for each first neighboring UAV.

[0038] For example, when a self-organizing network includes three drones A, B, and C, and they are neighbors, drone A can receive messages sent by drone C in time slot t, the preset string of drone C in time slot t, and the message redundancy of drone C in time slot t, thus obtaining the preset string of drone C in time slot t and the message redundancy of drone C in time slot t.

[0039] S1012. The UAV determines its own message redundancy in time slot t based on the message received in time slot t.

[0040] Continuing with the example above, UAV A determines its message redundancy in time slot t based on the messages received in time slot t.

[0041] Specifically, for any UAV i, in each time slot, each time UAV i receives a message, it determines the number of duplicate message packets it has received up to the present. In this way, the message redundancy of UAV i in each time slot can be obtained.

[0042] S1013. The drone sends the message received in time slot t and its own message to at least one of its second neighbor drones.

[0043] Continuing with the example above, drone A sends the message sent by drone C in time slot t, along with the message it wants to send out, to drone B.

[0044] S1014. The drone changes the values ​​of the first character of its own N characters in the preset string of its own t-1 time slot and the value of the at least one second character corresponding to at least one neighboring drone to the second value. Based on the values ​​of the characters in the preset string of each first neighboring drone in its t time slot other than the N characters corresponding to the drone itself, the drone updates the values ​​of the characters in the preset string of its own t-1 time slot other than the N characters corresponding to the drone itself, thus obtaining the preset string of the drone's t time slot.

[0045] Continuing with the example above, drone A sets the values ​​of the first character of its own three characters and the second character of drone C in its own time slot t-1 Bitgraph to 1. Based on the values ​​of the characters in drone C's time slot t Bitgraph other than the three characters corresponding to drone A, drone A updates the values ​​of the characters in its own time slot t-1 Bitgraph other than the three characters corresponding to drone A, thus obtaining drone A's time slot t Bitgraph.

[0046] For ease of explanation, and without loss of generality, we will use three drones as an example to illustrate the stopping rules of the Bitgraph-gossip protocol. Each of the three drones is a neighbor of the others, and it is stipulated that each drone selects only one neighbor node for message forwarding when propagating gossip. Figure 4 As shown, after the previous round of message broadcasting ends, the drone enters the initial state. First, the drone's Bitgraph is reset to zero. Then, each drone generates its own message packet, updates the corresponding bits of its Bitgraph from 0 to 1, and propagates its message packet and its own Bitgraph via the gossip protocol. For example, in this example, the gossip propagation process can be divided into six steps, as detailed below. Figure 4 As shown, the 01 string with light shading represents the latest Bitgraph of the drone, the string of all 1s with dark shading represents the drone's Bitgraph becoming all 1s in this step, and the circle with dark shading represents the drone actually receiving all message packets in this step.

[0047] The first step is that when drone B receives the message from drone A and drone A's current Bitgraph "100 000 000", drone B updates drone A's partition in the Bitgraph to 100, indicating that drone A has generated and sent its own message packet, and updates drone B's partition to 110, indicating that drone B has generated and sent its own message packet and received drone A's message packet. The result is that drone B's current Bitgraph is "100 110 000".

[0048] The second step is that when drone C receives the message from drone B and drone B's current Bitgraph "100 110 000", drone C updates drone A's partition in the Bitgraph to 100, indicating that drone A has generated and sent its own message packet, updates drone B's partition to 110, indicating that drone B has generated and sent its own message packet and received drone A's message packet, and updates drone C's partition to 111, indicating that drone C has collected all the message packets, thus obtaining drone C's current Bitgraph as "100 110 111".

[0049] Third, following the same logic, when drone A receives the message from drone C and drone C's current Bitgraph "100110 111", drone A updates its Bitgraph partition to 111, indicating that it has received all the message packets. It updates drone B's partition to 110, indicating that drone B has not yet received drone C's message packets. It updates drone C's partition to 111, indicating that drone C has received all the message packets. The result is that drone A's current Bitgraph is "111 110 111".

[0050] Fourth, drone A sends a message packet and its current Bitgraph "111 110111" to drone B. When drone B receives the message and its current Bitgraph from drone A, it updates its Bitgraph to "111 111 111". At this point, drone B has collected all the message packets and assumes that the other drones have also collected them. Drone B will forward the message packet one last time, and after sending the message, it will stop updating its gossip and Bitgraph.

[0051] Fifth, drone B sends a message packet and its current Bitgraph "111 111111" to drone C. When drone C receives the message and drone B's current Bitgraph, it updates its Bitgraph to "111 111 111". At this point, drone C has collected all the message packets and assumes that the other drones have also collected them. Drone C will forward the message packet one last time. After sending the message, drone C will stop gossip and Bitgraph updates. Similarly, drone C will forward the message packet one last time.

[0052] Step six: Drone C sends a message packet and its current Bitgraph "111 111111" to Drone A. Upon receiving the message and Drone C's current Bitgraph, Drone A updates its Bitgraph to "111 111 111". At this point, Drone A has received the message packet and assumes that the other drones have also received it. Similarly, Drone A will forward the message packet one last time before entering a silent state.

[0053] Due to link instability, gossip messages may experience packet loss. If drone A does not receive a message packet from drone C in step six, drone A's bitgraph will still be "111 110 111". In this case, drone A assumes that drone B has not yet received all the message packets, so drones A and B will synchronize their messages. Specifically, drone A will request drone B's bitgraph. Since the received bitgraph is "111 111 111", it means that drone B has received all the message packets, so drone A will update its own bitgraph to "111 111 111". At this point, the completeness check is complete, and all three drones have received all the message packets.

[0054] S102. Based on the preset string and message redundancy of time slot t, any UAV obtains the observation space of time slot t for itself and each of its neighboring UAVs, and determines multiple forwarding strategies for time slot t based on different preset forwarding probabilities; each forwarding strategy includes a preset forwarding and at least one forwarding neighbor.

[0055] Here, the observation space of each UAV i in time slot t is: Among them, R i B represents the message redundancy of time slot t for drone i. i The percentage of 1s in the preset string for time slot t of drone i, m i Let t represent whether drone i receives a message from any neighboring drone in time slot t (which can be represented by 0 / 1), and (·|·) represent a splicing operation.

[0056] Here, different preset forwarding probabilities can be 10%, 90%, 80%, 70%, ..., 10%. Drone i can use these 10 forwarding probabilities, along with neighboring drones within its current one-hop range, to perform permutations and combinations of forwarding probabilities, thus obtaining multiple combinations, i.e., multiple forwarding strategies. Each forwarding strategy corresponds to a forwarding probability and one or more neighboring drones. For example, the multiple forwarding strategies can be: forwarding strategy A, forwarding strategy B, forwarding strategy C, and forwarding strategy D.

[0057] S103. After precoding the observation space of any UAV in time slot t, the precoding result is input into the model along with multiple forwarding strategies in time slot t, and the target forwarding strategy is obtained according to the principle of maximum Q value. The model is trained on the initial multi-branch agent reinforcement learning model by using the observation space and forwarding strategies of multiple historical time slots of each UAV in the ad hoc network.

[0058] Here, the model generates the Q-value for each strategy based on the precoding results and multiple forwarding strategies for time slot t, and selects the forwarding strategy corresponding to the largest Q-value from the multiple Q-values ​​corresponding to the multiple forwarding strategies as the target forwarding strategy.

[0059] Here, the initial multi-branch agent reinforcement learning model includes: a graph attention network layer, a state value function operation layer, a first output layer, a second output layer, and a forwarding object prediction layer.

[0060] In some embodiments, S103 includes:

[0061] S1031. The observation space of any UAV in time slot t is input into a multilayer perceptron with a preset weight matrix and a preset bias vector to obtain the embedding space of time slot t. The embedding space of time slot t includes: the embedding space of time slot t of any UAV and the embedding space of time slot t of neighboring UAVs of any UAV.

[0062] Specifically, for each drone i, it can be controlled by a preset weight matrix W. e and preset bias vector b e The multilayer perceptron will Embedded into a latent vector, the embedding space of UAV i in time slot t is obtained, and the specific formula is as follows: σ represents the ReLU activation function.

[0063] For each drone i, the embedding space of time slot t of any of its neighboring nodes can be represented as e j , where A(i,j)=1, and A is the adjacency matrix of the UAV ad hoc network.

[0064] S1032. Input the embedding space of time slot t and various forwarding strategies into the pre-trained graph attention network layer, and output the feature vector of time slot t for each forwarding strategy.

[0065] Here, the pre-trained graph attention network layer includes: a first graph attention layer and a second graph attention layer with identical parameters, and a concatenation layer. Specifically, for example, for a forwarding strategy A, the first graph attention layer determines the attention weight of each forwarding neighbor for any UAV in forwarding strategy A based on the embedding space of any UAV in time slot t, the embedding space of each forwarding neighbor in time slot t in forwarding strategy A, and the pre-trained learning matrix; the first graph attention layer obtains a first aggregated feature based on the embedding space of each forwarding neighbor in time slot t, the attention weight of each forwarding neighbor for any UAV, and the pre-trained attention parameters; the second graph attention layer obtains a second aggregated feature based on the first aggregated feature; the concatenation layer concatenates the first and second aggregated features to obtain the feature vector of forwarding strategy A in time slot t.

[0066] For example, when any drone is drone i, the attention weight of each forwarding neighbor for any drone i is represented as follows: Where, α ij Let G represent the attention weight of the j-th neighbor node of drone i for drone i, T denote the transpose, exp(.) denotes the exponential function with base e, and G... i W represents the set of neighboring drones within one hop range of drone i. K and W Q This is the pre-trained learning matrix.

[0067] For example, for drone i, the first aggregated feature can be represented as: in, This is the first aggregation feature. For the neighbor list of drone i, W V W θ and b θ These are the attention parameters for pre-training. W Q W K W V These are embedding e i A learning matrix mapped to query, key, and value vectors. W θ b θ These are the model parameters learned by the graph attention network layer.

[0068] For example, for drone i, the second aggregated feature is represented as This illustrates the processing of the graph attention network layer.

[0069] For example, the feature vector for UAV i, time slot t is represented as:

[0070] S1033. Input the feature vector of time slot t for each forwarding strategy into the pre-trained state value function operation layer to obtain the state value of time slot t for each forwarding strategy.

[0071] Specifically, the pre-trained state value function operation layer can obtain an intermediate vector based on the pre-trained state parameters and the feature vector of time slot t for each forwarding strategy; after performing nonlinear activation processing on the intermediate vector, the state value of time slot t for each forwarding strategy is obtained.

[0072] For example, for drone i, the state value of slot t for each forwarding strategy can be represented as: Among them, W α and b α For the pre-trained state parameters, This is the intermediate vector.

[0073] S1034. Input the state value of time slot t and the feature vector of time slot t into the first output layer and the second output layer of the pre-trained system to obtain the first action value result value and the second action value result value of each forwarding strategy.

[0074] Specifically, for example, for forwarding strategy A, the pre-trained first output layer obtains the first advantage value of forwarding strategy A based on the feature vector of forwarding strategy A in time slot t and the pre-trained object parameters; based on the first advantage value of forwarding strategy A and the state value of forwarding strategy A in time slot t, it determines the first action value result value of forwarding strategy A; the pre-trained second output layer obtains the second advantage value of forwarding strategy A based on the feature vector of forwarding strategy A in time slot t and the pre-trained probability parameters; based on the second advantage value of forwarding strategy A and the state value of forwarding strategy A in time slot t, it determines the second action value result value of forwarding strategy A.

[0075] For example, for drone i, the first advantage value of forwarding strategy A is expressed as: in, and Represents the probability parameters of pre-training. Let represent the forwarding probability corresponding to forwarding strategy A for UAV i in time slot t. Here, "A" in forwarding strategy A is related to... The meaning of A in these cases is different.

[0076] For example, for drone i, the first action value of forwarding strategy A is: a1' is the value that maximizes A1

[0077] For example, for drone i, the second advantage value of forwarding strategy A is expressed as: in, and Indicates the parameters of the pre-trained object. Let A represent the forwarding neighbor corresponding to the forwarding strategy A of UAV i in time slot t.

[0078] For example, for drone i, the value of the second action of forwarding strategy A is: a'2 is the value that maximizes A2.

[0079] S1035. After summing the first action value result value and the second action value result value of the same forwarding strategy, input them into the forwarding object prediction layer, and output the forwarding strategy with the largest sum value as the target forwarding strategy.

[0080] Specifically, continuing with the above-mentioned forwarding strategies: forwarding strategy A, forwarding strategy B, forwarding strategy C, and forwarding strategy D, each of these strategies corresponds to one of four summation values ​​(i.e., four Q values). The forwarding strategy corresponding to the largest summation value can be used as the target forwarding strategy.

[0081] S104. The UAV forwards the message to the target neighbor in time slot t. The target neighbor is the forwarding neighbor corresponding to the target forwarding strategy, and is the UAV with the highest probability of not receiving the message among all the neighboring UAVs of the UAV.

[0082] Here, any UAV will send the message it received, as well as its own message, to all forwarding neighbors corresponding to the target forwarding policy by the time slot t.

[0083] For example, Figure 5 This diagram illustrates the architecture for a drone to communicate with neighboring nodes in an ad hoc network using a pre-trained multi-branch agent reinforcement learning model, and also shows the network structure of the pre-trained multi-branch agent reinforcement learning model. Figure 5 As shown, in the ad hoc network composed of UAV1 to UAV5, each UAV s, after obtaining the observation space and multiple forwarding strategies for the current time slot, performs pre-encoding on the observation space of the current time slot to obtain the observation embedding for the current time slot. Then, on the one hand, the observation embedding and multiple forwarding strategies for the current time slot are input into a pre-trained graph attention network layer (by...). Figure 5The system consists of a first graph attention layer, a second graph attention layer, and a concatenation layer to obtain the feature vector of the current time slot for each forwarding strategy. On the other hand, the observation embedding of the current time slot for each forwarding strategy is input into the advantage layer (Advantages 1) of the pre-trained first output layer, outputting the first result value A1 of the forwarding strategy, and into the advantage layer (Advantages 2) of the pre-trained second output layer, outputting the second result value A2 of the forwarding strategy. The pre-trained state value function layer processes the feature vector of the current time slot for each forwarding strategy to obtain the state value of the current time slot. This state value is then input into the first aggregation layer (Q-values ​​1) of the pre-trained first output layer to obtain the first action value result value of the current time slot for the forwarding strategy, and into the second aggregation layer (Q-values ​​2) of the pre-trained second output layer to obtain the second action value result value of the current time slot for the forwarding strategy. Afterwards, the forwarding object prediction layer (Action layer) sums the first action value result value and the second action value result value of each forwarding strategy. Then, from the multiple summation values ​​that correspond one-to-one with multiple forwarding strategies, it selects the forwarding strategy corresponding to the largest summation value as the target forwarding strategy. All forwarding neighbors corresponding to the target forwarding strategy are designated as the target forwarding neighbors of the UAV s in the current time slot. The UAV s then sends all its current message packets to the target forwarding neighbors in the current time slot.

[0084] In some embodiments, prior to S103 above, the method further includes:

[0085] S301. During the training phase k, obtain the observation space of slot z, the heuristic reward value of slot z, the forwarding probability of the target forwarding neighbor and the target forwarding neighbor corresponding to slot z, and the observation space of slot z+1 for each UAV in one round of communication in the ad hoc network; slot z is each slot in this round of communication; during the training phase k, each UAV contains a multi-branch agent reinforcement learning model in phase k-1; k is an integer greater than or equal to 1; the forwarding probability of the target forwarding neighbor and the target forwarding neighbor corresponding to slot z is a strategy randomly selected from a variety of forwarding strategies for the determined slot z.

[0086] Here, when k is 1, each drone contains an initial multi-branch agent reinforcement learning model during the first stage of training.

[0087] Here, at each stage, a preset number of training sessions can be performed, such as 5 times, 10 times, etc., which can be specifically set, and the embodiments of the present invention do not limit this.

[0088] Here, after the training of each stage is completed, the multi-branch agent reinforcement learning model obtained from this stage is reloaded into each UAV in the ad hoc network. Thus, these UAVs can generate training samples for the next stage of training based on the reloaded multi-branch agent reinforcement learning model. This process is repeated iteratively until the last training stage ends.

[0089] In some embodiments, obtaining the observation space of slot z, the heuristic reward value of slot z, the forwarding probability of the target forwarding neighbor and the target forwarding neighbor corresponding to the target forwarding neighbor in slot z, and the observation space of slot z+1 for each UAV in a round of communication in the ad hoc network in S301 above can be achieved through the following steps:

[0090] S3011. In a round of communication in a drone ad hoc network, a drone containing a multi-branch agent reinforcement learning model in the (k-1)th stage turns on its own timer when communication begins, obtains the corresponding timing time in time slot i, and obtains the preset string of time slot i and the message redundancy of time slot i for both the neighboring drone and the drone itself by communicating with neighboring drones in time slot i; i is an integer greater than 1.

[0091] Here, the Bitgraph of each drone in the ad hoc network is cleared before each round of message broadcasting in the ad hoc network. When clearing the Bitgraph, each drone i starts a local timer and keeps the timer running as the message broadcasting continues.

[0092] Here, the principle of obtaining the preset string of the neighboring drone and the message redundancy of the i-time slot by communicating with the neighboring drone in S3011 is the same as the principle of S101 above.

[0093] S3012. The UAV obtains the observation space of the UAV and each of the neighboring UAVs in the i-th time slot according to the preset string and message redundancy of the i-th time slot. Based on different preset forwarding probabilities, it determines multiple forwarding strategies for the i-th time slot. It then selects one forwarding strategy from the multiple forwarding strategies in the i-th time slot as the target forwarding neighbor and the forwarding probability corresponding to the target forwarding neighbor.

[0094] Here, the principle of S3012 is the same as that of S102 described above.

[0095] S3013. Determine the message broadcast delay reward for time slot i based on the preset string of time slot i and the timing of time slot i.

[0096] Here, in time slot i, after each drone obtains its own bitgraph for time slot i, if all bits in the bitgraph for time slot i are 1, it means that the drone believes that all drones in the ad hoc network have received all message packets. The drone then stops its local timer, and the timer is now t. i Then, based on the timing time t of time slot i i (Representing the time taken for the message broadcast observed locally by the drone), calculate the message broadcast delay reward for time slot i. If there is a 0 in the Bitgraph of time slot i, then r i bitclock =0.

[0097] S3014. Determine the message broadcast redundancy reward for time slot i based on the message redundancy of the UAV in time slot i and the message redundancy of the neighboring UAVs in time slot i.

[0098] Here, the message broadcast redundancy reward for each drone in slot i can be expressed as: Where, n onehop This refers to the number of drones in the group consisting of the drone and its neighboring drones within its one-hop communication range. This represents the sum of message redundancy in slot i for all drones in this group.

[0099] Here, message broadcast redundancy rewards can encourage neighboring drones to collaborate.

[0100] S3015. Determine the filling degree reward for time slot i based on the proportion of preset characters in the preset string of time slot i of the drone and the proportion of preset characters in the preset string of time slot i-1 of the drone.

[0101] Here, the fill rate reward for each drone's i-slot can be expressed as: Among them, b i This represents the percentage of 1s in the bitgraph of time slot i of the drone (the number of 1s divided by the length of the bitgraph string), b i-1 This shows the percentage of 1s in the Bitgraph of the i-1 time slot of the drone.

[0102] S3016. Based on the preset discount factor, the message broadcast delay reward of slot i, the message broadcast redundancy reward of slot i, and the filling degree reward of slot i, determine the heuristic reward value of slot i. Continue until the last slot of this round of communication ends, and obtain the observation space of slot i+n, the heuristic reward value of slot i+n, the forwarding probability of the target forwarding neighbor and the target forwarding neighbor corresponding to slot i+n, and the observation space of slot i+n+1; n is an integer greater than 1; z is an integer from 1 to i+n.

[0103] Specifically, when time slot i is not the last time slot of this round of communication, the heuristic reward value for time slot i of each drone s can be expressed as: Where γ represents the preset discount factor, γ i-1 This represents γ raised to the power of i-1, where i-1 represents the (i-1)th time slot, and the initial time slot is the first time slot.

[0104] Specifically, when the last time slot of this round of communication is the T-th time slot, the message broadcast latency reward for each drone in the T-th time slot is:

[0105] Here, the most important evaluation metric in message broadcasting is the global message broadcast latency, therefore r bitclock The required proportion is large, so it is more appropriate to choose a discount factor γ∈[0,1] close to 1.

[0106] Here, through S3011 to S3016 above, the observation space of each UAV s in time slot i during a round of communication in the ad hoc network can be obtained. Heuristic reward value of i-slot The target forwarding neighbor and its corresponding forwarding probability in time slot i Observation space of slot i+1

[0107] Here, by obtaining training samples containing heuristic reward values, the model's message propagation strategy can be trained more effectively.

[0108] S302. Take the observation space of each UAV in time slot z, the heuristic reward value of time slot z, the forwarding probability of the target forwarding neighbor and the target forwarding neighbor in time slot z, and the observation space of time slot z+1 as a set of training samples to obtain multiple sets of training samples in the k-th stage; the training samples are different in different stages.

[0109] Here, a set of training samples is...

[0110] S303. Using multiple sets of training samples in the k-th stage, train the multi-branch agent reinforcement learning model in the (k-1)-th stage to obtain the multi-branch agent reinforcement learning model in the k-th stage, until the multi-branch agent reinforcement learning model in the k+p-th stage is obtained under preset conditions, which is used as the pre-trained multi-branch agent reinforcement learning model; p is an integer greater than or equal to 1.

[0111] Specifically, after obtaining the multi-branch agent reinforcement learning model for the k-th stage, during training in the (k+1)-th stage, the observation space of each UAV in the next round of communication in the ad hoc network, the heuristic reward value of the z'-th slot, the forwarding probability of the target forwarding neighbor and its corresponding forwarding neighbor in the z'-th slot, and the observation space of the z'+1-th slot are obtained; the z'-th slot is each time slot in this round of communication; during training in the (k+1)-th stage, each UAV contains the multi-branch agent reinforcement learning model from the k-th stage; the observation space of each UAV in the z'-th slot, the heuristic reward value of the z'-th slot, and the forwarding probability of the target forwarding neighbor in the z'-th slot are obtained. The reward value, the forwarding probability of the target forwarding neighbor in slot z', and the observation space in slot z'+1 are used as a set of training samples to obtain multiple sets of training samples for stage k+1. The multiple sets of training samples for stage k+1 are used to train the multi-branch agent reinforcement learning model for stage k to obtain the multi-branch agent reinforcement learning model for stage k+1. This process continues until k+p equals the preset value or the model converges, resulting in the multi-branch agent reinforcement learning model for stage k+p, which is then used as the final pre-trained multi-branch agent reinforcement learning model.

[0112] In some embodiments, S301 to S303 described above can be executed by any one of the UAVs in the UAV ad hoc network, or by a ground control center communicating with the UAVs in the UAV ad hoc network; there is no limitation on this. When S301 to S303 are executed by the ground control center, each UAV can transmit multiple sets of training samples obtained for each stage back to the ground control center in real time. After the training for each stage is completed, the ground control center transmits the multi-branch agent reinforcement learning model obtained from this stage back to each UAV for updating the model in the UAVs.

[0113] In some embodiments, when the training of the multi-branch agent reinforcement learning model in each stage includes w times, during the first training in the k-th stage, the network parameters of the multi-branch agent reinforcement learning model in the (k-1)-th stage are copied to obtain the target model. During the first training, y is calculated using the target model. d The target model is used as the temporal difference objective. Then, the loss L is calculated using the (k-1)th stage multi-branch agent reinforcement learning model. Based on the loss L, backpropagation and gradient descent are performed to update the network parameters of the (k-1)th stage multi-branch agent reinforcement learning model, resulting in the first training model. Subsequently, during the second training, the target model is still used to calculate y. dThe first training model is used as the temporal difference objective. Then, the loss L is calculated using the first training model. Backpropagation and gradient descent are performed based on the loss L to update the network parameters of the first training model, resulting in the second training model. This process is iterated until the xth training iteration ends (x can be any set hyperparameter, such as 2, 5, 10, etc.). When the xth training model is obtained, the target model is updated to the xth training model. Then, the above principle is used to continue training for the (x+1)th iteration until the 2xth training iteration ends, resulting in the 2xth training model. At this point, the target model is updated to the 2xth training model, and the above principle is used to continue training for the (2x+1)th iteration. This process continues until the wth training iteration ends, resulting in the k-th stage multi-branch agent reinforcement learning model.

[0114] Specifically, during the above training, the loss value for each training iteration can be calculated using the following loss function: Where N represents the number of drones in the ad hoc network, and D represents the number of training samples in each stage. The selected training samples are used for training. The objective is time-series differencing, and γ is a preset discount factor. For the target model of branch d, ,

[0115] Here, after the model is trained, it will be mounted on each drone. After the drone takes off, it will interact with its neighbors to obtain the observation space before forwarding each message packet. After inputting the observation space into the model, it will obtain the target neighbors and perform message packet forwarding operations based on the target neighbors.

[0116] For example, Figure 9 This is a system structure diagram of the present invention, as shown below. Figure 9 As shown, the technical solution of this invention involves two modules: a deep reinforcement learning algorithm module and a topology-adaptive data fast propagation module. The deep reinforcement learning algorithm module designs a branched multi-agent deep reinforcement learning algorithm, with two branches corresponding to the forwarding probability and forwarding neighbors, respectively. A data structure called Bitgraph is proposed, and a reward function is designed based on this data structure, enabling the reinforcement learning algorithm to learn within the environment. The topology-adaptive data fast propagation module deploys the learned model on a drone, inputs the obtained observation space into the model, and uses the final forwarding neighbors output by the model to make forwarding decisions.

[0117] This invention uses different actor networks to make different actions (i.e., forwarding neighbors and their corresponding forwarding probabilities), reducing the dimension of the combined action space from O(n²) to O(n), and introduces a shared module to improve model convergence. Simultaneously, to handle different neighbor scenarios, a deep graph neural network is used to learn the dependencies between the drone and its neighbors under dynamic topology changes. The trained model thus exhibits high decision accuracy. Furthermore, this invention proposes a data structure called Bitgraph to record the drone's packet reception status, and designs a reward function based on it. Specifically, Bitgraph is a customized 0 / 1 string, updated via gossip about gossip. Through reward reshaping, the changes in a single Bitgraph are designed as the drone's intrinsic reward, while the final propagation delay and the number of redundant messages are designed as the drone's extrinsic rewards, guiding each gossip propagation decision.

[0118] This invention maintains a preset string within each UAV for real-time updates of message sending and receiving status between any UAV and its neighboring UAVs, and between each neighboring UAV and its own neighboring UAVs. It also obtains the message redundancy of neighboring UAVs in real time. This allows for real-time monitoring of the historical message packet reception status of all UAVs during communication. Furthermore, by employing an observation space obtained from the historical message packet reception status of all UAVs monitored by each UAV in previous rounds of communication, a model trained using a real-time determined forwarding strategy, and the current historical message packet reception status of all UAVs monitored by each UAV, the invention determines which neighboring UAV should forward the message to. Therefore, by considering the dependencies between each UAV and its neighboring UAVs and the historical message packet reception status of each UAV, the invention accurately determines which neighboring UAV needs to receive the message, thus improving the efficiency of message propagation between UAVs.

[0119] The following experimental data further illustrates the effectiveness of the method provided by this invention.

[0120] 1. Comparison of Schemes

[0121] We compare our proposed message broadcasting method with other message broadcasting methods based on flooding, random gossip, forwarding probability decision, and forwarding neighbor decision.

[0122] 1) Blind flooding based broadcasting approach (BFBA): Multiple source nodes send message packets to all their neighbors. When a relay node receives a new message packet, it saves the packet and forwards it to all its neighbors. The total time spent by a group of nodes forwarding a message packet and receiving some messages is called a round. The algorithm converges when all nodes in the network have received message packets from multiple source nodes.

[0123] 2) Random gossip-based broadcasting approach (RGBA). Multiple source nodes do not randomly forward data to all their neighbors; instead, they forward message packets with a 60% probability to three randomly selected neighbors. Relay nodes randomly select three neighbor nodes with a 60% probability to forward message packets. The convergence requirement is the same as above: the algorithm converges when all nodes have received all message packets.

[0124] 3) Changeable forwarding probability based broadcasting approach (CFPBA). In this approach, when a relay node receives a message, the decision-making principle regarding the forwarding probability is that the more times a node receives the same information, the lower the probability of it continuing to propagate the information.

[0125] 4) Suitable forwarding neighbor based broadcasting approach (SFNBA). This approach scores neighboring nodes based on the distance relationship between nodes and selects the most suitable forwarding neighbor node.

[0126] 2. Comparison Indicators

[0127] Message broadcast turn count and latency: The proposed solution aims for high efficiency in data transmission, one aspect of which is to transmit as quickly as possible while ensuring broadcast quality. Therefore, we tested the turn count and latency required for transmission of the proposed solution and a comparative solution.

[0128] Redundant Packet Count in Message Broadcast: The proposed solution aims for high efficiency in data propagation, one aspect of which is minimizing the number of redundant messages generated while ensuring broadcast quality. Therefore, we tested the number of redundant packets in the proposed solution and comparative solutions.

[0129] 1) Time delay required for message broadcasting

[0130] Figure 6 This represents the number of communications required to propagate a message throughout the entire network using the five broadcast methods (note: for BFBA and RGBA, we only count the cases where message broadcasting is successful). According to theoretical analysis, compared to other methods, BFBA, a flood-based broadcast method, has the optimal convergence speed because each node forwards the message packet to all neighbors instead of just one, requiring only 5 propagation rounds. Our experimental results also verify this conclusion. RGBA and CFPBA, because they still forward message packets randomly, have significantly higher propagation latency than BFBA. CFPBA, by introducing a forwarding strategy with decreasing probability of redundant messages, avoids sending some redundant messages and slightly reduces the number of propagation rounds. SFNBA introduces some fixed strategies when selecting neighbors, further accelerating the convergence speed, requiring only 11 propagation rounds. Compared to other solutions, we found that the broadcasting method trained by the model does not involve completely random probability of forwarding message packets to neighbors. The forwarding strategy learned by the model enables the drone to forward message packets with a higher probability to neighbor nodes that have never received the message packets before. Therefore, compared to randomly selecting a neighbor, our solution is more efficient in message packet forwarding. Experiments show that when the number of nodes is 125, our solution requires 20 rounds, which is better than CFPBA's 26 rounds but worse than SFNBA's 25 rounds.

[0131] In addition, we also calculated the specific latency required for the five broadcast methods in the corresponding rounds. Figure 7 The results show the required latency at different node counts. On average, our solution reduces latency by 36.1% compared to RGBA, 29.2% compared to CFPBA, and 22.6% compared to SFNBA.

[0132] 2) Number of redundant message broadcast packets

[0133] Figure 8 This demonstrates message broadcast redundancy packets for different numbers of drones (Note: For BFBA and RGBA, we only count cases where message broadcast is successful). From Figure 8As can be seen, BFBA suffers from message implosion due to its flooding method for propagating message packets. Since we fixed the spatial size of the drone distribution, the network density increases with the number of nodes, leading to an increase in the number of neighbors for each node, thus exacerbating message implosion. Because RGBA randomly selects 3 neighbors for message packet forwarding, the number of redundant packets does not increase dramatically with the increase in the number of nodes. Experiments show that when the number of nodes is 125, CFPBA significantly reduces the forwarding probability of duplicate redundant message packets, thus reducing the number of redundant message packets by 32.9% compared to RGBA. SFNBA, because it only considers neighbor selection, still forwards duplicate redundant message packets with a probability of 1, so the number of redundant message packets is only reduced by 17.8% compared to RGBA. Our trained model first sends packets to the necessary nodes, improving propagation efficiency. When multiple duplicate message packets are received, it also reduces the forwarding probability or even stops sending packets. Therefore, compared to RGBA, our message redundancy is reduced by 38.2%, compared to CFPBA by 6.9%, and compared to SFNBA by 20.7%.

[0134] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. An adaptive message propagation method under a UAV ad hoc network, characterized in that, The application relates to a method for realizing communication in a UAV ad hoc network. In each round of communication in the UAV ad hoc network, any UAV containing a pre-trained multi-branch intelligent agent reinforcement learning model obtains a preset string of each UAV and message redundancy of t time slots through communication with neighbor UAVs at t time slots; the preset string of each UAV is used for recording message sending and receiving between the UAV and each neighbor UAV, between each neighbor UAV and the neighbor UAV of the UAV, and between each neighbor UAV and the neighbor UAV of each neighbor UAV from the start of the round of communication to t time slots; t is an integer greater than 0; wherein the length of the preset string is N square; N is the total number of UAVs in the ad hoc network; each UAV corresponds to N characters, one of which is used for recording whether the UAV has sent a message, and the remaining N-1 characters are used for recording whether the UAV has received messages from the remaining N-1 UAVs; N is an integer greater than 2; The any UAV obtains an observation space of the any UAV and each UAV in the neighbor UAVs at t time slots according to the preset string at t time slots and the message redundancy at t time slots, and determines a plurality of forwarding strategies at t time slots according to different preset forwarding probabilities; each forwarding strategy contains a preset forwarding and at least one forwarding neighbor; The any UAV inputs the pre-coding result of the observation space at t time slots and the plurality of forwarding strategies at t time slots into the model, and obtains a target forwarding strategy according to the principle of maximum Q value; the model is trained based on the initial multi-branch intelligent agent reinforcement learning model by using the observation space and the forwarding strategy of each UAV in the ad hoc network at a plurality of historical time slots; The any UAV forwards a message to a target neighbor at t time slots; the target neighbor is a forwarding neighbor corresponding to the target forwarding strategy, and is a UAV with the highest probability of not receiving a message among all neighbor UAVs of the any UAV.

2. The adaptive message propagation method for UAV ad hoc networks according to claim 1, wherein, The message sending and receiving between the any UAV and each neighbor UAV includes whether the any UAV has sent its own message and whether the any UAV has received messages from each neighbor UAV.

3. The adaptive message propagation method for UAV ad hoc networks according to claim 1, wherein, When the ad hoc network includes N UAVs, each UAV corresponds to N characters in the preset string, and the N characters include a first character for recording whether the UAV has sent a message and N-1 second characters for recording whether the UAV has received messages from the corresponding N-1 UAVs; the neighbor UAVs include first neighbor UAVs and second neighbor UAVs; the any UAV containing the pre-trained multi-branch intelligent agent reinforcement learning model obtains the preset string of each UAV and the message redundancy of t time slots through communication with the neighbor UAVs at t time slots, including: At t time slots, the any UAV receives messages sent by at least one first neighbor UAV of the any UAV, the preset string at t time slots, and the message redundancy at t time slots, and obtains the preset string at t time slots and the message redundancy at t time slots of each first neighbor UAV. The any one UAV determines message redundancy of the self t time slot according to the message received in the t time slot; The any one UAV sends the message received in the t time slot and the self message to at least one second neighbor UAV of the self; The any one UAV changes the first character in the N characters corresponding to the self in the preset character string of the t-1 time slot of the self and the value of at least one second character corresponding to the at least one second neighbor UAV into a second value, and updates the values of the characters in the preset character string of the t-1 time slot of the self except the N characters corresponding to the self according to the values of the characters in the preset character string of the t time slot of each first neighbor UAV except the N characters corresponding to the self, to obtain the preset character string of the t time slot of the self.

4. The adaptive message propagation method for UAV ad hoc networks according to claim 1, wherein, The pre-trained multi-branch agent reinforcement learning model comprises a pre-trained graph attention network layer, a pre-trained state value function operation layer, a pre-trained first output layer, a pre-trained second output layer and a pre-trained forwarding object prediction layer; after the observation space of the t time slot is pre-encoded by the any one UAV, the pre-encoded result and multiple forwarding strategies of the t time slot are input into the model to obtain a target forwarding strategy according to a maximum Q value principle, which comprises: The any one UAV inputs the observation space of the t time slot into a multi-layer perception machine with a preset weight matrix and a preset bias vector to obtain an embedding space of the t time slot; the embedding space of the t time slot comprises an embedding space of the t time slot of the any one UAV and an embedding space of the t time slot of a neighbor UAV of the any one UAV; The embedding space of the t time slot and the multiple forwarding strategies are input into the pre-trained graph attention network layer to output a feature vector of the t time slot of each forwarding strategy; The feature vector of the t time slot of each forwarding strategy is input into the pre-trained state value function operation layer to obtain a state value of the t time slot of each forwarding strategy; The state value of the t time slot and the feature vector of the t time slot are input into the pre-trained first output layer and the pre-trained second output layer to correspondingly obtain a first action value result value of each forwarding strategy and a second action value result value of each forwarding strategy; The first action value result value and the second action value result value of the same forwarding strategy are summed and input into the pre-trained forwarding object prediction layer to output a forwarding strategy with a maximum sum value as the target forwarding strategy.

5. The adaptive message propagation method for UAV ad hoc networks according to claim 4, wherein, The pre-trained graph attention network layer comprises a first graph attention layer and a second graph attention layer with the same parameters and a splicing layer; the inputting of the embedding space of the t time slot and the multiple forwarding strategies into the pre-trained graph attention network layer to output the feature vector of the t time slot of each forwarding strategy comprises: The first graph attention layer determines an attention weight of each forwarding neighbor for the any one UAV according to the embedding space of the t time slot of the any one UAV, the embedding space of the t time slot of each forwarding neighbor in each forwarding strategy and a pre-trained learning matrix; The first graph attention layer obtains first aggregated features according to an embedding space of a t time slot of each forwarding neighbor, an attention weight of each forwarding neighbor for the any unmanned aerial vehicle, and pre-trained attention parameters; The second graph attention layer obtains second aggregated features based on the first aggregated features; The splicing layer splices the first aggregated features and the second aggregated features to obtain a feature vector of the t time slot of each forwarding strategy.

6. The adaptive message propagation method for UAV ad hoc networks according to claim 4, wherein, The inputting of the state value of the t time slot and the feature vector of the t time slot into the pre-trained first output layer and the pre-trained second output layer corresponds to obtaining a first action value result value of each forwarding strategy and a second action value result value of each forwarding strategy, including: The pre-trained first output layer obtains a first advantage value according to the feature vector of the t time slot and pre-trained object parameters; The first action value result value of each forwarding strategy is determined based on the first advantage value and the state value of the t time slot; The pre-trained second output layer obtains a second advantage value according to the feature vector of the t time slot and pre-trained probability parameters; The second action value result value of each forwarding strategy is determined based on the second advantage value and the state value of the t time slot.

7. The adaptive message propagation method for UAV ad hoc networks according to claim 1, wherein, Before the pre-encoding result and the multiple forwarding strategies of the t time slot are input into the model to obtain the target forwarding strategy according to the maximum Q value principle, including: In the kth stage training, the observation space of the z time slot of each unmanned aerial vehicle in a round of communication of the ad hoc network, the heuristic reward value of the z time slot, the target forwarding neighbor of the z time slot and the forwarding probability corresponding to the target forwarding neighbor, and the observation space of the z+1 time slot are obtained; the z time slot is each time slot in this round of communication; each unmanned aerial vehicle in the kth stage training includes the multi-branch intelligent agent reinforcement learning model of the k-1th stage; k is an integer greater than or equal to 1; the target forwarding neighbor of the z time slot and the forwarding probability corresponding to the target forwarding neighbor are a strategy randomly selected from the determined multiple forwarding strategies of the z time slot; The observation space of the z time slot of each unmanned aerial vehicle, the heuristic reward value of the z time slot, the target forwarding neighbor of the z time slot and the forwarding probability corresponding to the target forwarding neighbor, and the observation space of the z+1 time slot are taken as a set of training samples to obtain multiple sets of training samples of the kth stage; the training samples of different stages are different; The multi-branch intelligent agent reinforcement learning model of the kth stage is obtained by training the multi-branch intelligent agent reinforcement learning model of the k-1th stage using the multiple sets of training samples of the kth stage, until the multi-branch intelligent agent reinforcement learning model of the k+pth stage is obtained as the pre-trained multi-branch intelligent agent reinforcement learning model under a preset condition; p is an integer greater than or equal to 1.

8. The adaptive message propagation method for UAV ad hoc networks according to claim 7, wherein, The observation space of the z time slot of each unmanned aerial vehicle in a round of communication of the ad hoc network, the heuristic reward value of the z time slot, the target forwarding neighbor of the z time slot and the forwarding probability corresponding to the target forwarding neighbor, and the observation space of the z+1 time slot are acquired, including: In a round of communication of the unmanned aerial vehicle ad hoc network, the unmanned aerial vehicle containing the multi-branch intelligent agent reinforcement learning model of the k-1 stage opens its own timer when starting the communication, obtains the corresponding timing time at the i time slot, and obtains the preset string of each of the unmanned aerial vehicle and the neighbor unmanned aerial vehicle at the i time slot and the message redundancy at the i time slot through the communication with the neighbor unmanned aerial vehicle; i is an integer greater than 1; The unmanned aerial vehicle obtains the observation space of each of the unmanned aerial vehicle and the neighbor unmanned aerial vehicle at the i time slot according to the preset string at the i time slot and the message redundancy at the i time slot, and determines a plurality of forwarding strategies at the i time slot according to different preset forwarding probabilities, and selects one of the plurality of forwarding strategies at the i time slot as the target forwarding neighbor and the corresponding forwarding probability of the target forwarding neighbor at the i time slot; The message broadcast time delay reward at the i time slot of the unmanned aerial vehicle is determined according to the preset string at the i time slot and the timing time at the i time slot of the unmanned aerial vehicle; The message broadcast redundancy reward at the i time slot is determined according to the message redundancy at the i time slot of the unmanned aerial vehicle and the message redundancy at the i time slot of the neighbor unmanned aerial vehicle; The filling degree reward at the i time slot is determined according to the proportion of the preset character in the preset string at the i time slot of the unmanned aerial vehicle and the proportion of the preset character in the preset string at the i-1 time slot of the unmanned aerial vehicle; The heuristic reward value at the i time slot is determined according to the preset discount factor and the message broadcast time delay reward from the initial time slot to the i time slot, the message broadcast redundancy reward from the initial time slot to the i time slot, and the filling degree reward from the initial time slot to the i time slot, until the last time slot of the round of communication ends, and the observation space of i+n time slots, the heuristic reward value of i+n time slots, the target forwarding neighbor and the corresponding forwarding probability of the target forwarding neighbor of i+n time slots, and the observation space of i+n+1 time slots are obtained; n is an integer greater than 1; z is an integer from 1 to i+n.

9. The adaptive message propagation method for UAV ad hoc networks of claim 2, wherein, The method further comprises: Before each round of communication of the ad hoc network, the preset string value of any unmanned aerial vehicle is initialized to a first value; In each round of communication of the ad hoc network, when the preset string of the Rth time slot of any unmanned aerial vehicle is a second value, it indicates that the any unmanned aerial vehicle has sent a message and has received messages from each neighbor unmanned aerial vehicle in this round of communication, and the any unmanned aerial vehicle enters a silent state after communicating with the neighbor unmanned aerial vehicle; R is an integer greater than t.

Images