Dynamic route reconfiguration method, apparatus, device, and storage medium

Abstract

The application discloses a dynamic route reconstruction method and device, equipment and storage medium, relates to the networking routing technical field, and includes: based on the unmanned aerial vehicle cluster network, the route reconstruction problem is modeled, the network topology model and the route reconstruction model are generated; based on the unmanned aerial vehicle cluster network, node importance evaluation is carried out, and the corresponding deliberate attack model of the unmanned aerial vehicle cluster network is established; based on the network topology model, the route reconstruction model and the deliberate attack model, Markov decision process conversion is carried out, and the corresponding Markov decision process is generated; the Markov decision process is solved iteratively through the reinforcement learning algorithm, and the optimal route strategy of the unmanned aerial vehicle cluster network is output. The application obtains the optimal route strategy through the construction of network model and attack model, the Markov decision process and the reinforcement learning algorithm, realizes the fast dynamic route reconstruction and stable communication of network under the disturbance.

Description

Technical Field

[0001] This application relates to the field of network routing technology, and in particular to dynamic route reconstruction methods, apparatus, devices and storage media. Background Technology

[0002] Low-altitude cooperative communication networks require rapid and adaptive route reconfiguration capabilities in adversarial environments. Existing technologies for route reconfiguration in UAV ad hoc networks primarily employ static alternative path schemes or message-switching-based dynamic reconfiguration schemes. Static alternative path schemes pre-calculate multiple backup routes, switching to alternative paths when the primary path fails. Message-switching-based dynamic reconfiguration schemes dynamically update the routing table by periodically or event-triggered exchange of routing status information between nodes. However, these message-switching-based dynamic reconfiguration schemes have a significant drawback: as the network size increases and the topology changes dynamically, the number of routing status messages that need to be exchanged between nodes expands dramatically, leading to excessive network control overhead, severely consuming limited wireless channel resources, and thus reducing the effective transmission efficiency of data packets.

[0003] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main objective of this application is to provide a dynamic route reconstruction method, apparatus, device, and storage medium, aiming to solve the technical problem of low efficiency in dynamic route reconstruction.

[0005] To achieve the above objectives, this application proposes a dynamic route reconstruction method, the method comprising: A systematic model of the routing reconstruction problem is performed based on the UAV swarm network, generating a network topology model and a routing reconstruction model; Based on the aforementioned drone swarm network, node importance is assessed, and a deliberate attack model corresponding to the drone swarm network is established. Based on the network topology model, the route reconstruction model, and the intentional attack model, a Markov decision process transformation is performed to generate the corresponding Markov decision process. The Markov decision process is iteratively solved using a reinforcement learning algorithm to output the optimal routing strategy for the UAV swarm network.

[0006] In one embodiment, the step of systematically modeling the routing reconstruction problem based on the UAV swarm network and generating a network topology model and a routing reconstruction model includes: Obtain the node location information of each drone node in the drone swarm network, and construct a network topology model based on the node location information and node communication distance constraints; The routing reconstruction problem is modeled based on the network topology model, and a routing reconstruction model is generated.

[0007] In one embodiment, the step of assessing node importance based on the drone swarm network and establishing a deliberate attack model corresponding to the drone swarm network includes: Obtain the node degree and link importance from the node location information of each drone node; The node degree and the link importance are weighted and fused to obtain the comprehensive importance evaluation value of the UAV node; Attack priorities are determined based on the comprehensive importance assessment value, and a deliberate attack model for the drone swarm network is generated.

[0008] In one embodiment, the step of performing Markov decision process transformation based on the network topology model, the routing reconstruction model, and the deliberate attack model to generate the corresponding Markov decision process includes: Based on the network topology model, a state space is constructed; Based on the aforementioned intentional attack model, an action space is constructed; Based on the aforementioned route reconstruction model, a reward function is constructed; Based on the state space, the action space, and the reward function, a Markov decision process corresponding to the UAV swarm network is constructed.

[0009] In one embodiment, the step of iteratively solving the Markov decision process using a reinforcement learning algorithm to output the optimal routing strategy for the UAV swarm network includes: Initialize the action value function and qualification trace value; Update the node connection state in the network topology model according to the intentional attack model, and determine the current state of the current training round in the state space; In the current state, the current action corresponding to the current state is determined based on the target greedy strategy, and the immediate reward is obtained according to the reward function; Based on the current state, the current action, and the immediate reward, calculate the temporal difference error in the current state; The action value function and the qualification trace value are updated based on the time-series differential error. The current state is updated to the next state and iteratively executed until the action value function converges, thereby obtaining the optimal routing strategy of the UAV swarm network.

[0010] In one embodiment, the step of updating the action value function and the qualification trace value based on the temporal difference error includes: A decay coefficient is determined based on the discount factor and the qualification decay parameter, and the qualification trace values ​​corresponding to the current state and the current action are updated by decaying the decay coefficient. The action value function is calculated and updated based on the updated eligibility trace value and the temporal difference error.

[0011] In one embodiment, the step of determining the current state in the state space of the current training round by updating the node connection states in the network topology model according to the intentional attack model includes: The failure nodes that are attacked and rendered ineffective in the current training round are determined based on the intentional attack model. The failed node and the links connected to the failed node are overflowed from the network topology model to obtain an updated network topology model; Based on the updated network topology model, the current initial node of the current training round is determined, and the state information corresponding to the current initial node in the state space is determined as the current state.

[0012] Furthermore, to achieve the above objectives, this application also proposes a dynamic route reconstruction apparatus, which includes: The system modeling module is used to perform system modeling of the routing reconstruction problem based on the UAV swarm network, generating network topology models and routing reconstruction models; An attack modeling module is used to assess the importance of nodes based on the drone swarm network and establish a deliberate attack model corresponding to the drone swarm network. The decision process module is used to transform the Markov decision process based on the network topology model, the routing reconstruction model and the intentional attack model, and generate the corresponding Markov decision process. The strategy generation module is used to iteratively solve the Markov decision process using a reinforcement learning algorithm and output the optimal routing strategy for the UAV swarm network.

[0013] In addition, to achieve the above objectives, this application also proposes a dynamic route reconfiguration device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the dynamic route reconfiguration method as described above.

[0014] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a processor, implements the steps of the dynamic route reconfiguration method described above.

[0015] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the dynamic route reconfiguration method described above.

[0016] One or more technical solutions proposed in this application have at least the following technical effects: This application proposes a dynamic route reconstruction method, apparatus, device, and storage medium. It systematically models the route reconstruction problem based on a UAV swarm network, generating a network topology model and a route reconstruction model. Based on the UAV swarm network, it assesses node importance and establishes a corresponding deliberate attack model. Based on the network topology model, the route reconstruction model, and the deliberate attack model, it performs a Markov decision process transformation to generate a corresponding Markov decision process. Finally, it iteratively solves the Markov decision process using a reinforcement learning algorithm to output the optimal routing strategy for the UAV swarm network. This application achieves rapid dynamic route reconstruction and stable communication under disturbance conditions by constructing a network model and an attack model, and obtaining the optimal routing strategy through a Markov decision process combined with a reinforcement learning algorithm. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating an embodiment of the dynamic route reconstruction method of this application. Figure 2 This is a schematic diagram of the module structure of the dynamic route reconstruction device according to an embodiment of this application; Figure 3 This is a schematic diagram of the device structure of the hardware operating environment involved in the dynamic route reconstruction method in this application embodiment.

[0020] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0022] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0023] The main solution of this application embodiment is as follows: A systematic model of the routing reconstruction problem is performed based on the UAV swarm network, generating a network topology model and a routing reconstruction model; node importance is assessed based on the UAV swarm network, establishing a deliberate attack model corresponding to the UAV swarm network; a Markov decision process is transformed based on the network topology model, the routing reconstruction model, and the deliberate attack model to generate a corresponding Markov decision process; the Markov decision process is iteratively solved using a reinforcement learning algorithm to output the optimal routing strategy for the UAV swarm network.

[0024] In this embodiment, for ease of description, the dynamic route reconstruction device will be used as the execution subject in the following description.

[0025] Low-altitude cooperative communication networks require rapid and adaptive route reconfiguration capabilities in adversarial environments. Existing technologies for route reconfiguration in UAV ad hoc networks primarily employ static alternative path schemes or message-switching-based dynamic reconfiguration schemes. Static alternative path schemes pre-calculate multiple backup routes, switching to alternative paths when the primary path fails. Message-switching-based dynamic reconfiguration schemes dynamically update the routing table by periodically or event-triggered exchange of routing status information between nodes. However, these message-switching-based dynamic reconfiguration schemes have a significant drawback: as the network size increases and the topology changes dynamically, the number of routing status messages that need to be exchanged between nodes expands dramatically, leading to excessive network control overhead, severely consuming limited wireless channel resources, and thus reducing the effective transmission efficiency of data packets.

[0026] This application provides a solution that, by constructing a network model and an attack model, obtains the optimal routing strategy through a Markov decision process combined with a reinforcement learning algorithm, thereby achieving rapid dynamic route reconstruction and stable communication of the network under disturbance conditions.

[0027] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or dynamic route reconfiguration device capable of performing the above functions. The following description uses a dynamic route reconfiguration device as an example to illustrate this embodiment and the subsequent embodiments.

[0028] Based on this, embodiments of this application provide a dynamic route reconstruction method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the dynamic route reconstruction method of this application.

[0029] In this embodiment, the dynamic route reconstruction method includes steps S11 to S14: Step S11: Systematically model the routing reconstruction problem based on the UAV swarm network, and generate a network topology model and a routing reconstruction model.

[0030] It should be noted that a drone swarm network refers to a cooperative communication network composed of multiple drone nodes (low-altitude aircraft). The routing reconstruction problem refers to the problem of replanning the data transmission path from the source node to the destination node after some nodes have failed due to an attack. System modeling is the process of mathematically abstracting the real physical network and its routing problem. A network topology model describes the connection relationships between nodes in the network, specifically represented as an undirected graph G=(U,E), where U is the set of nodes and E is the set of links. The routing reconstruction model formalizes the routing reconstruction objective (minimizing end-to-end latency) and constraints (such as communication distance constraints and latency constraints) into a mathematical optimization problem.

[0031] Understandably, the purpose of this step is to lay a precise mathematical model foundation for subsequent intelligent decision-making. By performing graph theory modeling on the physical network, nodes, links, and their spatial relationships can be formally described. By performing optimization modeling on the routing problem, it can be clearly stated that the goal of reconstruction is to find a path with the minimum end-to-end transmission latency under a specific network topology.

[0032] Specifically, network topology modeling refers to obtaining the three-dimensional coordinates of all UAV nodes, calculating the Euclidean distance between any two nodes, and if the calculated distance is within the communicable range, it indicates the existence of a valid link; otherwise, no valid link exists. The construction of the route reconfiguration model aims to minimize the total end-to-end delay and imposes constraints to build the route reconfiguration model.

[0033] Furthermore, the drone swarm network is an undirected graph consisting of N drones and M edges. The indicated, among which middle This represents the i-th node. middle A binary variable represents whether there is an active link between the aircraft. The communication range of each aircraft is limited to a maximum radius. and minimum radius is Inside the hollow sphere, Indicates the maximum communication distance. This represents the minimum distance between two aircraft that will not collide.

[0034] Meanwhile, the construction of the route reconstruction model defines This refers to a collection of low-altitude aircraft damaged by physical destruction or electromagnetic suppression, among which... Indicates aircraft Destroyed and corresponding and Set to 0. Aircraft The position is represented as At the beginning of the route, the source node This will produce a size of The data packets are routed through a predefined path (such as the transmission path in the previous section). Send it to the destination node Signal-to-noise ratio between aircraft and reachability The calculation formula is:

[0035]

[0036] in, This indicates the channel bandwidth between the two aircraft; This indicates the transmission power between the two aircraft; This represents the path loss between two aircraft; This indicates the noise power between two aircraft; This represents the signal-to-noise ratio between two aircraft. This indicates the reachability between two aircraft.

[0037] Furthermore, the end-to-end latency of an ad hoc aircraft swarm network mainly includes buffer latency and transmission latency. Specifically, the single-hop latency of the inter-aircraft link... The calculation formula is:

[0038] in, Represents the speed of light; Indicates the size of the data packet waiting to be transmitted; This indicates a one-hop link.

[0039] Furthermore, the transmission delay between ends for:

[0040] In summary, the routing reconstruction problem that minimizes end-to-end latency in an attacked drone swarm network can be modeled as follows:

[0041] St

[0042]

[0043]

[0044]

[0045] in, This represents the total delay in transmitting data packets along the routing path; Represented as a potential routing policy, where Potential routing paths; Represents the set of aircraft on the route; This represents the maximum tolerable delay.

[0046] For example, a cluster of 5 drones can be represented by an undirected graph containing 5 vertices (drones) and several edges (communication links satisfying distance constraints). When drone 1 (the source node) needs to send data to drone 5 (the destination node), the route reconstruction model searches for a path from vertex 1 to vertex 5 in this graph that minimizes the total latency of data transmission along that path.

[0047] Step S12: Based on the UAV swarm network, perform node importance assessment and establish a deliberate attack model corresponding to the UAV swarm network.

[0048] It should be noted that node importance assessment refers to the process of quantifying the impact of each drone node on network connectivity and performance. A deliberate attack model simulates an attacker selecting targets based on a certain strategy (such as prioritizing attacks on important nodes). When assessing node importance, the node degree used refers to the number of links directly connected to that node. Link importance measures the criticality of a link in the network and is related to the tightness of the connection between the nodes at both ends of the link.

[0049] Understandably, the purpose of this step is to establish a threat model that closely resembles a realistic adversarial environment. Attackers do not randomly damage nodes, but rather purposefully attack critical hubs of the network. By comprehensively considering the number of direct connections a node has (degree centrality) and the importance of its connected links, the extent of damage caused by node failure to the entire network can be assessed more accurately. An attack model is then built based on this assessment, allowing subsequent reinforcement learning training to be conducted in a more challenging dynamic failure environment that simulates intelligent attacks. This results in routing policies that are better able to cope with such targeted attacks, improving network resilience.

[0050] Specifically, firstly, node importance is assessed. For each node, its degree is calculated. For each associated link, the number of triangles in the network containing that link is counted, and the link importance and the node's contribution to that link are calculated. Finally, the overall importance of the nodes is calculated. Based on this, establishing a deliberate attack model means that at the beginning of each round of training or simulation, the top K nodes are selected and marked as "attacked and ineffective" (i.e., their overall importance is set to be ineffective) according to their overall importance values ​​from high to low. This involves removing related links from the topology model. By combining importance assessment and sequential attacks, the most destructive attack sequences to the network can be simulated, providing a high-pressure testing environment for training the robustness of routing strategies.

[0051] For example, in a drone swarm, a central node that connects multiple critical data stream relay links has a high node degree and the importance of the links it connects to is also high, so its overall importance value will be very large. In a deliberate attack model, this node will be prioritized for "destruction" in the early stages of the simulated attack, thus forcing the routing algorithm to learn to bypass this critical but vulnerable hub and find alternative paths.

[0052] Step S13: Based on the network topology model, the route reconstruction model, and the intentional attack model, perform Markov decision process transformation to generate the corresponding Markov decision process.

[0053] It should be noted that the Markov Decision Process (MDP) transformation refers to the process of transforming the aforementioned dynamic problems of route reconstruction and network attack into the standard reinforcement learning framework, consisting of a state space S, an action space A, a reward function r, and transition probabilities. and discount factor It consists of core elements, in addition, Representation strategy, A trajectory sequence in a round, defined as The state space is the agent's (e.g., an aircraft's) observation of its environment. Serving as input to a reinforcement learning model, it represents the environment the aircraft is in, including distances between nodes and node workloads to form the set of state parameters. The action space is the set of decisions that can be made given a state, i.e., which neighbor of the current node to choose as the next hop. The reward function is used to evaluate the quality of an action.

[0054] Understandably, the purpose of this step is to bridge the gap for reinforcement learning algorithms. Routing decision-making is a typical sequential decision problem: choosing which node as the next hop affects the subsequent network state and available actions. By transforming it into an MDP, the routing problem becomes a problem where an agent learns the optimal decision strategy (routing strategy) by trying different actions (choosing the next hop) and receiving rewards (negative latency) in interaction with the environment (a dynamically changing network). The state space design allows the agent to perceive the local network topology and congestion; the action space defines its decision range; and the reward function breaks down the ultimate goal of minimizing end-to-end latency into immediate feedback signals for each decision step, guiding the learning direction.

[0055] Specifically, constructing a Markov decision process first involves defining the state. This is a vector containing the distances from the current node to all its neighboring nodes, and the packet queue length of the current node. Defined in the state... The following action From the set of neighbor nodes Select a node index. Define the action to be performed. (i.e., from the node) Send to node The instant reward obtained after ) is Its calculation depends on the transmission delay of the hop. and purpose judgment .

[0056] For example, when a drone (intelligent agent) is in a certain state s t (Observing that its distances to its three neighbors are 100m, 150m, and 200m respectively, and that it has two data packets in its cache), it can choose one of three actions (jump to neighbor A, B, or C). If it chooses to jump to neighbor A, it calculates a negative reward value based on the link quality with A. This value reflects the "cost" (latency) of choosing this link. The agent's goal is to learn a policy (a mapping from state to action) through extensive trial and error, such that the cumulative negative reward (total latency) obtained from any source node is minimized.

[0057] Step S14: Iteratively solve the Markov decision process using a reinforcement learning algorithm to output the optimal routing strategy for the UAV swarm network.

[0058] It's important to note that reinforcement learning algorithms are machine learning methods that learn optimal behavioral policies through the interaction between an agent and its environment. The inverse Sarsa(λ) algorithm was employed, which combines temporal difference and qualification trace. Iterative solution refers to the algorithm continuously updating its internal action-value function (i.e., evaluating the long-term value of taking action a in state s) through multiple rounds of simulation until convergence. The optimal routing policy is the final learned result, which, for any network state s, provides the next hop selection that maximizes the long-term cumulative reward (i.e., minimizes end-to-end latency).

[0059] Understandably, due to the dynamic changes in network topology and the complexity of attack models, it is difficult to solve problems in real time using traditional optimization methods. Reinforcement learning algorithms, especially model-free methods like Sarsa(λ), do not rely on precise knowledge of environmental dynamics (transition probabilities) but instead learn the value function directly from experience through online or simulated interactions. The introduction of eligibility traces (parameter λ) can more efficiently allocate the contribution of future rewards to earlier actions, accelerating learning. Through extensive iterative training, the algorithm can eventually output a robust routing policy that can quickly and adaptively select low-latency paths when faced with random or intentional node failures, achieving intelligent network reconstruction.

[0060] Specifically, the action-value function Q-table and eligibility trace E-table are first initialized. At the start of each training round, the network topology is updated according to the intentional attack model, and the environment state is reset. Then, starting from the source node, the next hop is selected according to an ε-greedy policy (selecting the action with the highest current Q-value with a probability of 1-ε, and randomly exploring with a probability of ε). After executing the action, the immediate reward and the next state are obtained, and the temporal difference error δt is calculated. Next, the eligibility trace E(s,a) of the current state-action pair is updated, and the Q-values ​​of all state-action pairs are updated using the temporal difference error and E(s,a). This process is repeated until the destination node is reached, completing one round. After multiple rounds of iteration, the action-value function converges to maximize the cumulative expected reward, thus obtaining the optimal policy.

[0061] For example, in the early stages of training, due to the inaccuracy of the Q-value, the algorithm will engage in more random exploration (with a large ε), potentially trying some roundabout or low-quality paths and recording the high latency (negative reward) of these paths, thereby reducing the relevant Q-value. As training progresses, the algorithm gradually tends to utilize (ε decreases) the learned knowledge, that is, choosing paths with high historical rewards (large Q-values). When a critical node is attacked, the Q-value of the original optimal path will decrease due to the inability to connect, and the algorithm will quickly adjust, exploring and consolidating new alternative paths. Ultimately, the learned strategy can quickly guide data flow to the available path with the lowest latency in various attack scenarios.

[0062] This embodiment, through the aforementioned scheme, provides a clear problem definition and constraints for automated decision-making by accurately mathematically modeling the UAV swarm network and routing problem. Furthermore, by establishing a deliberate attack model based on node importance, it simulates the intelligent attack behavior of attackers targeting key network nodes in a real adversarial environment, making the training environment closer to high-threat combat scenarios. Based on this, the path selection problem in a dynamic environment is transformed into a Markov decision process, decomposing the complex global optimization problem into local decision problems that can be learned step-by-step by the agent. Finally, using reinforcement learning algorithms, especially the Sarsa(λ) algorithm with a qualification trace mechanism, the system autonomously learns the optimal routing strategy that can adapt to dynamic topology changes and adversarial attacks through continuous interaction and trial and error with the environment. The entire scheme forms a complete technical closed loop of "modeling-threat simulation-problem transformation-autonomous learning," enabling the UAV swarm network to quickly and adaptively reconstruct routes without relying on pre-set fixed alternative routes or cumbersome network-wide signaling exchange after some nodes fail. This effectively reduces end-to-end transmission latency and significantly enhances the network's dynamic routing reconstruction efficiency in adversarial environments.

[0063] Based on the above implementation scheme, in one feasible implementation, the step of systematically modeling the routing reconstruction problem based on the UAV swarm network and generating a network topology model and a routing reconstruction model includes S21~S22: Step S21: Obtain the node location information of each drone node in the drone swarm network, and construct a network topology model based on the node location information and node communication distance constraints.

[0064] It should be noted that node location information is data describing the precise coordinates of UAV nodes in three-dimensional space, and is the foundation for constructing physical connections in the network. Node communication distance constraints are physical rules that determine whether an effective link can be established between two nodes, including the minimum communication distance and the maximum communication distance (the range within which the signal can be effectively received).

[0065] Understandably, this step is the first step in digitally abstracting the physical network, aiming to establish a computable network skeleton that reflects the actual communication possibilities. By acquiring the real-time location of each node and determining the survivability of inter-node links based on the physical characteristics of radio wave propagation (maximum communication distance) and safe flight rules (minimum collision avoidance distance), the dynamically changing drone swarm network can be "snapshotted" into a static graph model. This graph model is the cornerstone of all subsequent routing calculations, attack simulations, and intelligent decision-making, ensuring that the "network" processed by the algorithm maintains consistency in connectivity with the real-world physical network.

[0066] Specifically, network topology modeling refers to obtaining the three-dimensional coordinates of all UAV nodes, calculating the Euclidean distance between any two nodes, and if the calculated distance is within the communicable range, it indicates that there is a valid link; otherwise, there is no valid link.

[0067] Furthermore, the drone swarm network is an undirected graph consisting of N drones and M edges. The indicated, among which middle This represents the i-th node. middle A binary variable represents whether there is an active link between the aircraft. The communication range of each aircraft (in Euclidean distance) is limited to a maximum radius of... and minimum radius is Inside the hollow sphere, Indicates the maximum communication distance. This represents the minimum distance between two aircraft that will not collide. At that time, establish connections in the topology model, and let Drones and Euclidean distance between The calculation formula is:

[0068] in, For drone nodes The coordinates; For drone nodes The coordinates.

[0069] Step S22: Model the routing reconstruction problem based on the network topology model to generate a routing reconstruction model.

[0070] It should be noted that the route reconstruction problem refers to the decision-making problem of finding an optimal data transmission path for a specified source-destination node pair under a given network topology (which may contain failed nodes).

[0071] Understandably, the purpose of this step is to transform the fuzzy objective of "finding a good path" into a precise and computable mathematical optimization problem. By decomposing the communication process, the end-to-end delay is modeled as the sum of the delays of each hop on the path, and the single-hop delay consists of signal propagation delay and data packet transmission delay. This quantitative model directly links link quality (distance, bandwidth, signal-to-noise ratio), data packet characteristics (size), and the final performance metric (delay). Imposing constraints on this optimization problem (such as upper limits on single-hop delay and communication distance ranges) ensures that the obtained path is not only theoretically optimal in terms of delay but also physically feasible.

[0072] Specifically, the construction of a route reconfiguration model refers to constructing a route reconfiguration model with the goal of minimizing the total end-to-end latency and by imposing constraints.

[0073] Furthermore, the route reconstruction model is defined in its construction. This refers to a collection of low-altitude aircraft damaged by physical destruction or electromagnetic suppression, among which... Indicates aircraft Destroyed and corresponding and Set to 0. Aircraft The position is represented as At the beginning of the route, the source node This will produce a size of The data packets are routed through a predefined path (such as the transmission path in the previous section). Send it to the destination node Signal-to-noise ratio between aircraft and reachability The calculation formula is:

[0074]

[0075] in, This indicates the channel bandwidth between the two aircraft; This indicates the transmission power between the two aircraft; This represents the path loss between two aircraft; This indicates the noise power between two aircraft; This represents the signal-to-noise ratio between two aircraft. This indicates the reachability between two aircraft.

[0076] Furthermore, the end-to-end latency of an ad hoc aircraft swarm network mainly includes buffer latency and transmission latency. Specifically, the single-hop latency of the inter-aircraft link... The calculation formula is:

[0077] in, Represents the speed of light; Indicates the size of the data packet waiting to be transmitted; This indicates a one-hop link.

[0078] Furthermore, the transmission delay between ends for:

[0079] In summary, the routing reconstruction problem that minimizes end-to-end latency in an attacked drone swarm network can be modeled as follows:

[0080] St

[0081]

[0082]

[0083]

[0084] in, This represents the total delay in transmitting data packets along the routing path; Represented as a potential routing policy, where Potential routing paths; Represents the set of aircraft on the route; This represents the maximum tolerable delay.

[0085] This embodiment, through the aforementioned scheme, constructs a network topology model using node locations and physical communication constraints, abstracting the dynamic and complex physical UAV network into a clearly structured graph theory model. This provides accurate network structure input for all subsequent analyses. Furthermore, based on this topology model, the engineering problem of "finding the optimal route" is formalized into a mathematical optimization model with minimizing end-to-end latency as its core objective, incorporating various physical and operational constraints. These two steps together constitute a systematic mathematical description of the entire routing reconstruction problem, transforming the original path selection problem, which relied on experience and trial and error, into a computable problem with clear inputs, outputs, and evaluation criteria. This provides an indispensable and precise problem definition foundation for subsequent applications of model-based optimization or model-free reinforcement learning methods.

[0086] Based on the above implementation scheme, in one feasible implementation, the step of evaluating the importance of nodes based on the drone swarm network and establishing a deliberate attack model corresponding to the drone swarm network includes S31~S33: Step S31: Obtain the node degree and link importance from the node location information of each UAV node.

[0087] It should be noted that node degree refers to the number of edges directly connected to a node, reflecting the breadth of node connectivity. Link importance is an indicator that measures the criticality of an edge in a network.

[0088] Understandably, the purpose of this step is to collect key features across two dimensions to assess node importance. Node degree is a local feature, considering only the number of direct neighbors, and can quickly identify hub nodes (high degree) in the network. However, in UAV networks, if one node has redundant, non-critical links while the other has critical bridges connecting different subnets, the failure of the latter will obviously have a greater impact on the entire network. Therefore, node degree alone is insufficient to comprehensively assess node importance. Introducing the feature of "link importance" can compensate for the shortcomings of node degree. It quantifies the contribution of each link to network connectivity from the perspective of the network's global or community structure.

[0089] Specifically, firstly, based on the constructed network topology model, the node degree is directly calculated by counting the number of communicable edges connected to each node. Secondly, for each valid link in the network, the importance value of that link is calculated.

[0090] Step S32: Weighted fusion of the node degree and the link importance is performed to obtain the comprehensive importance evaluation value of the UAV node.

[0091] It should be noted that weighted fusion adds the node degree and the importance contribution values ​​of all links related to that node to obtain the overall importance assessment value of that node.

[0092] Understandably, the goal of this step is to generate a more comprehensive and reasonable ranking of node importance. Simply using node degree ignores the structural attributes of links, while solely relying on link-based importance may be too global. The fusion method employed in this step cleverly combines the two: node degree serves as the base score, reflecting the scale of a node's direct connections; adding the contribution values ​​of all associated links introduces information about the criticality of those links. This additive fusion means that a node that is both a connection hub (high degree) and controls multiple critical links will have a very high overall importance.

[0093] Specifically, first, the node degree and the importance of all associated links of that node are obtained. Next, for each associated link, the importance contribution of the node to that link is calculated. Then, the contributions of all associated links are summed, and this sum is added to the node degree. This calculation is performed on all nodes in the network to obtain the overall importance score for each node.

[0094] Furthermore, the formula for calculating the overall importance score is as follows:

[0095] in, A comprehensive importance score; Represents the set of adjacent aircraft nodes; Indicates the degree of adjacent spacecraft nodes; Represents a node For the link Significant contributions.

[0096] Furthermore, The calculation formula is:

[0097] in, The formula for calculating link importance is:

[0098] in, Indicates the degree of connectivity of the link. This indicates that the aircraft network topology contains links. The number of triangles.

[0099] Step S33: Determine the attack priority based on the comprehensive importance assessment value, and generate a deliberate attack model for the drone swarm network.

[0100] It should be noted that attack priority refers to the order in which nodes are selected for attack during a simulated attack on the network. The intentional attack model is a model that simulates attacker behavior based on this priority rule. Attackers always prioritize attacking the node with the highest overall importance assessment value in the current network. In the simulation, the state of the attacked node is marked as "invalid" (or "failed"). ), and remove all its associated links from the topology model.

[0101] Understandably, the purpose of this step is to transform the static node importance indices calculated in the first two steps into a dynamic, sequential model of the network disruption process. Compared to random attack models, this deliberate attack model based on node importance ranking simulates "decapitation" or "hub-breaking" intelligent attack behavior, which is more destructive to the network and poses a greater realistic threat. During reinforcement learning training, using this attack model to dynamically change the environment (network topology) forces the agent to learn routing strategies that cannot rely on a few key nodes, but must instead explore more redundant paths and adaptability, thereby greatly improving the robustness of the learned strategies in real adversarial environments and the resilience of the network.

[0102] Specifically, the intentional attack model is generated by first calculating the comprehensive importance assessment value of all surviving nodes based on the current network topology before each round of training (or simulated adversarial training). These values ​​are then sorted in descending order. Next, based on the simulated attack intensity (e.g., attacking K nodes in this round), the top K nodes are selected sequentially from the top of the sorted list and marked as "destroyed" nodes, i.e., their importance is set... Finally, update the network topology model: traverse all edges related to these attacked nodes and assign their corresponding edges to the nodes. and The value is set to 0. This process simulates the behavior of an attacker precisely targeting the most critical nodes in the network in each round.

[0103] For example, in a network of 10 drones, the calculated L values ​​for nodes 3, 7, and 1 rank in the top three. In a deliberate attack model, if the simulated attack targets two nodes in this round, nodes 3 and 7 will be chosen for the "attack." In the updated network topology, nodes 3 and 7 are considered disabled, and their connections to all other nodes are broken. This simulates a successful attack on core network nodes, forcing the routing algorithm to re-pathfind in the absence of these two critical nodes.

[0104] This embodiment, through the aforementioned scheme, simultaneously acquires two types of features: node degree, reflecting the local connectivity of a node, and link importance, reflecting the global structural importance of the link. This provides a multi-dimensional data foundation for comprehensively evaluating node value. Furthermore, by adding the node degree to the contribution value of associated link importance after correction based on neighbor degree, a comprehensive importance evaluation value is generated that simultaneously captures the breadth of node connectivity and the criticality of the controlled links, making the evaluation results more accurate. Finally, attack priorities are established based on this evaluation value, generating a deliberate attack model simulating intelligent adversarial attacks. This series of steps works together to construct a highly realistic network threat environment simulator. It enables subsequent reinforcement learning training to optimize against the most destructive targeted attacks, rather than random failures, thereby ensuring that the finally learned routing reconstruction strategy possesses strong resilience and rapid path recovery capabilities in extreme situations.

[0105] Based on the above implementation scheme, in one feasible implementation, the step of transforming the network topology model, the routing reconstruction model, and the intentional attack model into a Markov decision process to generate the corresponding Markov decision process includes S41~S44: Step S41: Construct the state space based on the network topology model.

[0106] It should be noted that the state space S is the set of observations or perceptions of the environment by the UAV at a given moment. The state is the basis for the UAV to make action choices. The state contains two key elements: first, the distance between nodes, which determines link quality and propagation delay; and second, the node's workload, specifically the total length of data packets queued for forwarding. ,in It is the number of cached data packets. This refers to the size of a single data packet. A specific state typically includes the distance from the current decision node to all its neighboring nodes, as well as the queue length of the current node itself.

[0107] Understandably, the purpose of this step is to define the observation window for the reinforcement learning agent, enabling it to perceive the critical network information necessary for making routing decisions. Distance is included in the state because in ad hoc wireless networks, distance directly affects the signal-to-noise ratio and transmission rate, providing the most direct local information for judging link quality, which can be obtained without network-wide broadcasting. "Traffic load" is included in the state because the congestion level (queue length) of a node directly affects the buffering latency of data packets at that node, a key dynamic factor in determining whether a route should avoid that node. Through these two dimensions of information, the agent can comprehensively evaluate the physical transmission capacity of each potential next-hop link and the instantaneous processing capacity of the nodes, thereby making a better path selection.

[0108] Specifically, when the drone is located at the node At that time, the observed state s is a vector. The first part of this vector is the node... to each of its neighbor nodes ( The Euclidean distance of ). The second part is the nodes. The total amount of data Li in its own cache queue. The state space S is the set of all possible state vectors.

[0109] For example, suppose drone A needs to choose the next hop for a data packet. It observes that its distances to neighbors B, C, and D are 120 meters, 80 meters, and 200 meters, respectively, and it has cached three data packets of size 1KB. Then, the state s perceived by the agent (on A) at this moment can be represented as a vector [120, 80, 200, 3072] (distance in meters, payload in bits). This state encodes all the local environmental information that can be used for decision-making.

[0110] Step S42: Construct the action space based on the intentional attack model.

[0111] It should be noted that the action space A is the set of all possible operations that can be performed in a given state. An action is defined as selecting the next hop node.

[0112] Understandably, defining actions as selecting neighboring nodes directly maps the routing problem into a discrete selection problem, making reinforcement learning algorithms applicable. The design of the action space is closely related to the intentional attack model: when the network is attacked and some nodes fail, these nodes and their associated links are removed from the network topology model. Correspondingly, the neighbor set of the affected node shrinks, causing the agent's available action space at that node to also shrink dynamically and immediately. This simulates the situation in real networks where certain routing directions become unavailable due to node destruction.

[0113] Specifically, the action space is constructed by first maintaining an adjacency list generated based on the current network topology (updated according to the intentional attack model). The process then involves querying drone nodes. From the adjacency list, retrieve all pairs of elements that satisfy the condition. =1 node The set of neighbors Then, the identifier (such as ID) of each node in the neighbor set is mapped to a unique action number. For example, if the neighbor set... If the action space is {B, C, D}, then the action space can be defined as {Action 0: Select B, Action 1: Select C, Action 2: Select D}. Executing an action involves forwarding the data packet to the corresponding neighbor node.

[0114] Furthermore, the action space is represented as:

[0115] in, Indicates low-altitude aircraft The next low-altitude aircraft is .

[0116] Step S43: Construct a reward function based on the routing reconstruction model.

[0117] It should be noted that the reward function r is the immediate feedback signal given to the corresponding environment after an action is performed, used to evaluate the quality of that action. The reward function transforms the optimization objective (minimizing end-to-end latency) into a guiding signal for each step of the decision.

[0118] Understandably, the reward function acts as a bridge connecting the route reconstruction model (minimizing total latency) and the reinforcement learning process (maximizing cumulative reward). Its design principle is that each step of choosing the next hop incurs a latency cost. Using the negative value of the cost as the reward implies that the drone's long-term goal is to maximize the cumulative reward, i.e., minimize the cumulative latency (total latency).

[0119] Specifically, when the drone is in a state Next, execute the action. (Select Neighbors) After the next hop, calculate the distance between the two nodes, the signal-to-noise ratio, and the reachability; calculate the single-hop delay, and then determine whether the next node to jump to is the destination node. Calculate the instant reward based on the determination result.

[0120] Furthermore, the transitive probability is used This indicates that the state is... To action The probability of another state. Since precise modeling is difficult, a model-free approach is used. To minimize the total end-to-end delay, we assume... In the current state Execute action The instant reward is calculated using the following formula:

[0121] in, Defined as the marker indicating the end of the routing process, i.e.:

[0122] in, The destination node.

[0123] For example, drone A (not the destination node) chooses to hop to neighbor B, and the calculated hop delay is 0.05 seconds. Then, at this time, H... k =1, reward If drone A jumps directly to destination node D, the calculated delay is 0.08 seconds, but at this time H k =0, reward Although a single hop directly to the destination has higher latency, receiving zero reward means the task is successfully completed, and the agent will not receive any further negative rewards. The algorithm learns by accumulating rewards and will discover that although some relay hops have lower latency, they may increase the total number of hops, ultimately resulting in a larger cumulative negative reward; while sometimes choosing a path with slightly higher latency but that leads directly to the destination may yield a higher cumulative reward (i.e., a smaller total latency).

[0124] Step S44: Based on the state space, the action space, and the reward function, construct the Markov decision process corresponding to the UAV swarm network.

[0125] It should be noted that the state transition probability describes the probability distribution of transitioning to the next state after performing action a in state s. Due to the complexity of network dynamics, this method employs model-free reinforcement learning, eliminating the need to pre-define or learn precise state transition probabilities. The discount factor γ is a number between 0 and 1 used to calculate the current value of future rewards; the closer γ is to 1, the more the agent values ​​long-term rewards.

[0126] Specifically, constructing a Markov decision process first involves constructing a state space, an action space associated with the state, and a reward function. A discount factor is set, for example, γ=0.9. The state transition probability is unknown, but the environment can provide a deterministic or probabilistic next state based on the physical model (such as node movement, attack, channel changes) and the action execution results.

[0127] This embodiment, through the above-described scheme, constructs a state space that integrates link physical characteristics (distance) and node dynamic load, enabling the agent to perceive key local information affecting routing performance. By constructing an action space closely related to dynamic changes in network topology (especially changes due to attacks), it ensures the real-time effectiveness of decision options. By designing a reward function that decomposes the end-to-end latency optimization objective into immediate feedback at each step, it provides clear and continuous evaluation signals for the agent's learning. Finally, by integrating these elements with a discount factor, a complete Markov decision process is constructed. This series of steps successfully transforms a complex, global network routing optimization problem influenced by multiple factors into a standard sequential decision problem that can be handled by model-free reinforcement learning algorithms. This transformation allows the agent to autonomously learn, through trial-and-error interaction with a simulated environment, how to make local routing decisions at each step in adversarial environments with dynamic node failures and frequent topology changes, ultimately leading to globally optimized route reconstruction capabilities. This serves as a crucial bridge connecting the initial problem modeling with the subsequent intelligent algorithm solution.

[0128] Based on the above implementation scheme, in one feasible implementation, the step of iteratively solving the Markov decision process using a reinforcement learning algorithm to output the optimal routing strategy for the UAV swarm network includes S51~S55: Step S51: Initialize the action value function and qualification trace value.

[0129] It's important to note that the action-value function Q(s,a) is a core structure in reinforcement learning, also known as the Q-table. It stores estimates of the long-term cumulative reward obtained by taking action a in state s, serving as a knowledge base for the drone's decision-making. The eligibility trace tracks the most recently visited state-action pair to more efficiently allocate credit for earlier decisions to long-term rewards. Initialization involves assigning initial values ​​to the Q-table and E-table; typically, Q(s,a) is initialized to 0 or a small random number, and the eligibility trace E(s,a) is initialized to 0.

[0130] Understandably, this step is preparatory work for the reinforcement learning training process. Initializing the action-value function Q(s,a) to 0 or a random small value means that the agent has no knowledge or only preliminary guesses about the long-term value of all state-action pairs at the start of training, which will encourage it to conduct extensive exploration early on. Initializing the eligibility trace E(s,a) to 0 prepares for subsequent tracking of the access sequence of state-action pairs. A good initialization (such as small random numbers) can break symmetry and facilitate exploration; while all-zero initialization is also a common and simple starting point.

[0131] Specifically, first, all possible state-action pairs (s, a) in the state space S and action space A are enumerated. For each state-action pair, a storage unit Q(s, a) is allocated. Initially, all values ​​of Q(s, a) can be set to 0, indicating that the agent initially considers all actions worthless. Another feasible implementation is to set it to a random decimal close to 0 (e.g., sampled from a uniform distribution U(-0.01, 0.01)), which introduces small differences while keeping the initial expected value neutral (approximately equal to 0), encouraging the agent to try different actions initially. Simultaneously, an E table with the exact same structure as the Q table is created, and all values ​​of E(s, a) are initialized to 0.

[0132] For example, suppose a simplified network has 10 possible state spaces, with an average of 3 action options per state. A Q-table with approximately 30 entries needs to be initialized. At the start of training, all these entries are set to 0. Simultaneously, an E-table with the same structure is created, also with all values ​​set to 0.

[0133] Step S52: Update the node connection state in the network topology model according to the intentional attack model, and determine the current state in the state space for the current training round.

[0134] It should be noted that this step is the initial operation for each training round. Updating the node connection state in the network topology model according to the intentional attack model means simulating an attack event at the beginning of each new round of route learning: selecting nodes to be "destroyed" in this round according to the intentional attack model (such as an attack ranked by node importance), and removing these nodes and all their associated links from the current network topology graph G. Determining the current state in the state space for the current training round means determining the source node for this routing task under the updated topology. The observation information of the source node (distance to its neighbors, its own workload) is used as the starting state for the agent's decision-making. .

[0135] Understandably, if training is conducted within a fixed, intact network topology each time, the learned policy will be unable to cope with node failures. By randomly or sequentially removing some nodes at the beginning of each round based on the intentional attack model, the agent is essentially provided with a massive number of different, damaged network topology scenarios for learning. Determining the current state... This means that each routing task begins with a specific, locally viewed view of the attacked network, simulating the real-world scenario in adversarial scenarios where packets need to find their way from a specific point within a compromised network. This step ensures the diversity and challenge of the training data.

[0136] Specifically, at the beginning of each training round, firstly, the set F of nodes to be simulated as targets for attack in this round is selected according to the intentional attack model (e.g., sorted in descending order of node importance). Then, the network topology model is traversed; for each attacked node, it is disabled, and all edges connected to the attacked node are also disabled. Next, a source node is randomly selected or specified for this round. and destination node Finally, obtain the source node's state information: calculate its distance to all surviving neighbors, read its workload L, and combine this information into a vector as the agent's initial state. .

[0137] For example, a new round of training begins. The attack model, based on the node importance ranking at the end of the previous round, decides to attack nodes 3 and 7. Therefore, in the topology model, nodes 3 and 7 are marked as ineffective, and all their connections with other nodes are severed. The source node for this round is randomly selected as node 1, and the destination node is node 10. Node 1 observes its distances to its neighbors nodes 2, 4, and 5 as [50, 120, 80] meters, and its payload is 1 data packet. The initial state s0 is then set to the vector [50, 120, 80, 1024] (assuming a packet length of 1KB).

[0138] Step S53: In the current state, determine the current action corresponding to the current state based on the target greedy strategy, and obtain the immediate reward according to the reward function.

[0139] It should be noted that the objective-oriented greedy strategy refers to the ε-greedy strategy, which uses 1 as the objective. The strategy selects the action with the highest Q-value with probability ε (utilizing known optimal knowledge), and randomly selects another action with probability ε (exploring unknown possibilities). In the current state, this strategy selects an action, known as the "current action." Obtaining the immediate reward based on the reward function means that after receiving the current action, the environment simulates the hop transmission process and calculates the immediate reward for that action using the reward function formula.

[0140] Understandably, the ε-greedy strategy is an effective way to solve the explore-exploit dilemma: in the early stages of training, a large ε value is set to encourage the agent to explore the state space extensively, try various possible paths, and avoid getting trapped in local optima; as training progresses, ε can be gradually decreased, allowing the drone to make more use of the better strategies it has learned. The immediate reward obtained after performing an action is a direct, quantitative feedback on the quality of that decision. This negative reward value (corresponding to positive latency) tells the agent how much "cost" choosing this link / node will bring. By continuously receiving this feedback, the agent can gradually correct its estimation of the value of different state-action pairs (i.e., the Q-value).

[0141] Specifically, assuming a drone Current state First, query all legal actions for the current node (list of neighboring nodes). Then, generate a random number ρ between 0 and 1. If ρ < ε (e.g., ε = 0.1), randomly and uniformly select one legal action as the current action. (Explore). Otherwise, query the Q table to find the state. Next, which action? The corresponding Q( , If the value is the largest, select this action. (Utilization). Execution of actions. (For example, sending data packets to nodes) After the environment receives this action, it calculates from... arrive The single-hop delay, and determine Is it the destination node? To determine whether the event is over, the instant reward is calculated at the end.

[0142] Step S54: Calculate the temporal difference error in the current state based on the current state, the current action, and the immediate reward.

[0143] It should be noted that the timing difference error represents the current action value function. The difference between the estimated value and a better estimated target. In the Sarsa algorithm, the formula for calculating the time-series difference error is: .in, It's an instant reward. It is a discount factor. In the next state The next action will be selected based on the current strategy. Q-value estimation.

[0144] Specifically, after the action was performed And received a reward and the next state Afterwards, the drone needs to be in a state according to the current strategy. Select next action Then, query the Q table for the Q value. And the Q-value of the next state-action pair Finally, substitute the values ​​into the formula to perform the calculation. If If it is a terminated state (i.e., the destination node has been reached), then define =0.

[0145] Step S55: Update the action value function and the qualification trace value based on the time-series differential error, update the current state to the next state and perform iterative execution until the action value function converges, and obtain the optimal routing strategy of the UAV swarm network.

[0146] Understandably, the qualification trace acts like a short-term memory, recording which state-action pairs have been recently visited and the strength of those visits. By iteratively executing the loop of "perceiving state - selecting action - obtaining reward - calculating temporal difference error - updating Q-value and qualification trace - state transition," the Q-value function gradually converges, ultimately extracting the optimal routing strategy.

[0147] Specifically, in calculating the timing difference error Then, firstly, for all state-action pairs (s, a), eligibility trace decay and update are performed. The update rule for eligibility trace is as follows:

[0148] in, Indicates the deterioration parameter; For indicator functions, i.e. when When the action value is 1, the value is 1; otherwise, it is 0. Therefore, the action value function is updated as follows:

[0149] in, This is the learning rate.

[0150] Furthermore, in order to balance exploration and development during continuous interaction and avoid obtaining suboptimal solutions, Greedy strategies are typically used to select actions, i.e.:

[0151] in, This represents the probability of an exploratory action.

[0152] Finally, set the current state and action as the next state and action. If the new state 'st' is the terminating state (destination node), the round ends, the eligibility trace is reset to 0, and the next round begins. Repeat this process for thousands of rounds until the changes in the Q-table are negligible. At this point, extract the policy from the converged Q-table: for each state 's', select the action 'a' that maximizes the action-value function value, i.e.:

[0153] According to the policy optimization theorem, the optimal policy is... Equal to the optimal action value function Furthermore, if Given the optimal strategy Corresponding to the state Next action ,Right now:

[0154] in, In a given state Next, select whether the action value is... Represented by expected future rewards:

[0155] This embodiment, through the aforementioned scheme, introduces dynamic topology changes based on a deliberate attack model in each training round, ensuring a highly complex and adversarial training environment, making policy learning adaptable to the most demanding scenarios. By employing an ε-greedy policy for decision-making interaction, a balance is struck between exploring the unknown and utilizing the known, and immediate performance feedback (rewards) is obtained for each decision. The temporal difference error is calculated, accurately characterizing the deviation between the immediate result of a single-step decision and its long-term value expectation, serving as the core signal driving knowledge updates. Finally, using the Sarsa(λ) algorithm, the temporal difference error is efficiently distributed back to a series of previous decisions through a qualification trace mechanism, achieving progressive optimization of the action-value function and policy. Through iterative training across massive rounds, the agent can ultimately autonomously learn a highly adaptive optimal routing policy. This policy can quickly and autonomously make local routing decisions when network nodes dynamically fail due to intelligent attacks, thereby minimizing end-to-end latency at the global level and significantly improving the self-healing and reconstruction capabilities of the UAV swarm network in adversarial environments.

[0156] Based on the above implementation scheme, in one feasible implementation, the step of updating the action value function and the qualification trace value based on the temporal difference error includes S61~S62: Step S61: Determine the attenuation coefficient based on the discount factor and the qualification attenuation parameter, and update the qualification trace value corresponding to the current state and the current action using the attenuation coefficient.

[0157] It should be noted that the discount factor γ and the eligibility decay parameter λ are two hyperparameters of the algorithm. Their product γλ constitutes the decay coefficient, which is used to decay all entries in the eligibility trace table E(s,a). The eligibility trace value E(s,a) is a table with the same structure as the Q-table, used to track the trace strength of each state-action pair (s,a) in recent visits.

[0158] Understandably, the decay coefficient γλ must be less than 1 (since γ,λ∈(0,1)). Multiplicative decay of all trace values ​​at each step means that the "credit" of state-action pairs visited long ago will gradually be forgotten over time (trace values ​​approaching 0), which aligns with the principle that "recent decisions have a greater impact on the current outcome." Simultaneously, incrementing the trace value of the currently visited state-action pair by 1 is equivalent to making a significant entry in its "credit book," indicating that it has just been used. The λ parameter controls the length of the backtracking assignment; λ=0 results in a single-step update, while λ closer to 1 allows credit to be assigned further back.

[0159] Specifically, one feasible implementation is: at each time step t, an action is performed. After calculating the timing difference error, the eligibility trace table is updated: traversing all entries (s, a) in the eligibility trace table E, and performing the following for each entry... Calculation. This operation achieves global decay.

[0160] Step S62: Calculate and update the action value function based on the updated eligibility trace value and the temporal difference error.

[0161] Understandably, updating the formula This means that this timing difference error Not only used to correct the action that just caused this result The Q-value is also proportionally adjusted to the Q-values ​​of all recently visited state-action pairs, with the adjustment ratio proportional to their respective eligibility traces. A state-action pair with a larger eligibility trace means it is closer to the current time step or has been visited more frequently; therefore, it bears greater responsibility for the current outcome and should receive a larger adjustment. This mechanism is highly efficient, allowing feedback from a single interaction to simultaneously update multiple decisions along an entire trajectory, greatly accelerating the learning process, especially in scenarios with delayed rewards (such as positive rewards only upon reaching the endpoint). The learning rate α controls the magnitude of the updates, preventing learning instability caused by excessively large single updates.

[0162] Specifically, this involves updating the eligibility trace table and resolving temporal difference errors. After calculation, the Q-table is iterated and updated. For each state-action pair in the Q-table, its current Q-value and corresponding eligibility trace are read. Then, the Q-value is updated. This update is performed simultaneously on all entries in the Q-table. After this update is complete, the eligibility trace is typically used for the next decay, or reset to 0 at the end of the round.

[0163] This embodiment, through the above-described scheme, uses the product of the discount factor and the eligibility decay parameter as the decay coefficient to globally decay the eligibility trace table and enhance the trace of the current state-action pair, thereby dynamically and exponentially recording the "merit" or "responsibility" allocation weights of a series of historical decisions. Then, the calculated temporal difference error (i.e., the empirical bias of a single step) is multiplied by the eligibility trace value of each state-action pair to globally and weight-wise update the Q-table. These two steps together realize the core advantage of the Sarsa(λ) algorithm—efficient multi-step credit allocation. This greatly accelerates the learning convergence speed and allows for online updates at each step, making learning more timely. In dynamic route reconstruction problems, data packets often require multiple hops from source to destination, and the final end-to-end delay is the sum of the delays of each hop, with rewards being delayed. The update mechanism of this claim ensures that each hop decision in the path leading to high latency receives a corresponding negative update, enabling the agent to quickly learn to avoid "bad" relay nodes or link combinations that cause high latency, thereby more efficiently approaching the optimal routing strategy.

[0164] Based on the above implementation scheme, in one feasible implementation, the step of determining the current state in the state space of the current training round by updating the node connection state in the network topology model according to the intentional attack model includes S71~S73: Step S71: Determine the failed nodes that have been attacked and rendered ineffective in the current training round based on the intentional attack model.

[0165] It should be noted that the current training round refers to a complete route exploration process of the reinforcement learning algorithm from the initial state (source node) to the final state (destination node). Failed nodes refer to the drone nodes selected by the attacking model as being destroyed or suppressed in this round, represented by a set F in the model, where... This indicates that a node has failed. This step generates the set F of failed nodes for this round, based on the rules of the attack model.

[0166] Understandably, the purpose of this step is to introduce a deterministic or probabilistic network disruption event at the beginning of each training round to simulate the dynamics of node destruction in an adversarial environment. Unlike random failures, the deliberate attack model simulates targeted, highly destructive attack patterns. This forces the reinforcement learning agent not only to learn the optimal path in a healthy network, but also to learn alternative paths and detour strategies in the event of a critical node's absence. By (potentially) changing the set of failed nodes in each training round, the agent will face a large number of different, compromised network topologies, thus requiring the learned strategy to possess high generality and robustness, capable of handling various possible combinations of node failures.

[0167] Specifically, at the beginning of each training round, the overall importance evaluation value of all nodes is first calculated based on the network topology at the end of the previous round (or the initial complete topology). Then, according to the attack strength parameter (e.g., attacking K nodes per round), starting with the node with the highest importance, K nodes are selected sequentially and added to the set F of failed nodes for this round, and their failure flags are set. Another, more complex implementation involves introducing randomness, such as selecting nodes for attack with a probability proportional to their importance. Regardless of the approach, the output of this step is a well-defined list of nodes that should be considered unusable in this round.

[0168] Step S72: Overflow the failed node and the links connected to the failed node from the network topology model to obtain an updated network topology model.

[0169] It should be noted that removal means logically isolating the attacked, faulty node from the set of available nodes and invalidating all its associated communication links.

[0170] Understandably, it modifies the "environment" the agent interacts with in real time based on the determined attack results. Removing failed nodes and their links from the topology model directly alters the network's connectivity graph. Paths that previously passed through these nodes become unavailable, and the nodes' neighbors find that the number of possible next-hop actions has decreased. This dynamically changing topology is precisely the source of uncertainty that reinforcement learning agents need to learn and adapt to. By updating the topology based on the attack model in each round, the drone is exposed to countless possible network impairment scenarios, thereby driving it to learn a policy that does not rely on any fixed, vulnerable path, but rather makes optimal, adaptive routing decisions based on the current topology. This directly corresponds to the "fast, adaptive network reconstruction" problem that the patent aims to solve.

[0171] Specifically, after obtaining the set of failed nodes F for this round, the adjacency matrix (or adjacency list) E of the network is traversed. For each element in the adjacency matrix, if its row index or column index appears in the set of failed nodes F, the value of that element is set to 0. After completing the traversal, an updated adjacency matrix E' is obtained. This updated topology model will be used for all subsequent state awareness, action space construction, and reward calculation for this round.

[0172] Step S73: Based on the updated network topology model, determine the current initial node for the current training round, and determine the state information corresponding to the current initial node in the state space as the current state.

[0173] It should be noted that the current initial node in the current training round is a specified or randomly selected source node. The state information is environmental information observed from the perspective of the current UAV, according to the defined state representation method, and includes at least the distances from the source node to its neighboring nodes and the source node's own service load.

[0174] Understandably, after the environment (network topology) changes due to a simulated attack, the agent needs to begin its route exploration task from a specific location within a specific, compromised network. Determining the local observation information (distance, load) of the source node as the initial state provides the drone with all the information needed to make its first decision.

[0175] Specifically, after updating the network topology model, a source node and a destination node are randomly selected (or specified in a certain sequence) for this round. Then, based on the updated topology model E', the set of all surviving neighbor nodes of the source node is found. Next, the location coordinates of the source node and each of its neighbor nodes are obtained, and the Euclidean distance is calculated. Then, the number of data packets currently buffered by the source node and the data packet size are read (or randomly generated), and its traffic load is calculated. Finally, this information is combined into a state vector.

[0176] This embodiment, through the aforementioned scheme, simulates the attack behavior of an intelligent attacker targeting network vulnerabilities in an adversarial environment by generating a dynamic set of failed nodes for each training round based on a deliberate attack model. This injects the most challenging environmental dynamics into the training. By instantly removing failed nodes and their associated links from the network topology model, it achieves a realistic and computable update of the environmental state of the attack consequences, ensuring that the environment in which the agent interacts always reflects the current network connectivity status. Finally, it determines the starting point in the updated dynamic topology and constructs its local state observation, providing the agent with a unique and realistic starting point for each exploration task. These three steps are interconnected and together constitute the core content of the environment reset stage in reinforcement learning training. They ensure that each round of training begins in a new, possibly never-before-seen, network impairment scenario, thereby greatly enriching the agent's training experience. Through learning in this highly dynamic and random environment for a massive number of rounds, the optimal routing strategy that ultimately converges can deeply understand the inherent vulnerability and redundancy of the network structure. Thus, in the event of any node failure, it can quickly make a near-optimal next-hop decision based on the current local network state (distance, load), achieving truly intelligent dynamic route reconstruction with strong resilience and adaptability.

[0177] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the dynamic route reconstruction method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0178] This application also provides a dynamic route reconfiguration device; please refer to [reference needed]. Figure 2 The dynamic route reconstruction device includes: System modeling module 201 is used to perform system modeling of the routing reconstruction problem based on UAV swarm network, and generate network topology model and routing reconstruction model; Attack modeling module 202 is used to evaluate the importance of nodes based on the UAV swarm network and establish a deliberate attack model corresponding to the UAV swarm network. The decision process module 203 is used to perform Markov decision process transformation based on the network topology model, the routing reconstruction model and the intentional attack model, and generate the corresponding Markov decision process. The policy generation module 204 is used to iteratively solve the Markov decision process using a reinforcement learning algorithm and output the optimal routing policy of the UAV swarm network.

[0179] The dynamic route reconstruction apparatus provided in this application, employing the dynamic route reconstruction method in the above embodiments, can solve the technical problem of low efficiency in dynamic route reconstruction. Compared with the prior art, the beneficial effects of the dynamic route reconstruction apparatus provided in this application are the same as those of the dynamic route reconstruction method provided in the above embodiments, and other technical features in the dynamic route reconstruction apparatus are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0180] This application provides a dynamic route reconstruction device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the dynamic route reconstruction method in Embodiment 1 above.

[0181] The following is for reference. Figure 3 The diagram illustrates a structural schematic suitable for implementing the dynamic route reconfiguration device in the embodiments of this application. The dynamic route reconfiguration device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 3 The dynamic route reconfiguration device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0182] like Figure 3As shown, the dynamic routing reconstruction device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in read-only memory 1002 or a program loaded from storage device 1003 into random access memory 1004. Random access memory 1004 also stores various programs and data required for the operation of the dynamic routing reconstruction device. The processing unit 1001, read-only memory 1002, and random access memory 1004 are interconnected via bus 1005. Input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to input / output interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the dynamic route reconfiguration device to communicate wirelessly or wiredly with other devices to exchange data. While the figure shows dynamic route reconfiguration devices with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0183] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0184] The dynamic route reconstruction device provided in this application, employing the dynamic route reconstruction method in the above embodiments, can solve the technical problem of low efficiency in dynamic route reconstruction. Compared with the prior art, the beneficial effects of the dynamic route reconstruction device provided in this application are the same as those of the dynamic route reconstruction method provided in the above embodiments, and other technical features in this dynamic route reconstruction device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0185] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0186] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0187] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the dynamic route reconstruction method in the above embodiments.

[0188] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0189] The aforementioned computer-readable storage medium may be included in the dynamic route reconfiguration device; or it may exist independently and not be assembled into the dynamic route reconfiguration device.

[0190] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by the dynamic route reconstruction device, the dynamic route reconstruction device performs the following actions: It performs system modeling of the route reconstruction problem based on the UAV swarm network, generating a network topology model and a route reconstruction model; it performs node importance assessment based on the UAV swarm network, establishing a deliberate attack model corresponding to the UAV swarm network; it performs Markov decision process transformation based on the network topology model, the route reconstruction model, and the deliberate attack model, generating a corresponding Markov decision process; and iteratively solves the Markov decision process using a reinforcement learning algorithm, outputting the optimal routing strategy for the UAV swarm network.

[0191] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0192] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0193] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0194] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described dynamic route reconstruction method, thereby solving the technical problem of low efficiency in dynamic route reconstruction. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the dynamic route reconstruction method provided in the above embodiments, and will not be repeated here.

[0195] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the dynamic route reconfiguration method described above.

[0196] The computer program product provided in this application can solve the technical problem of low efficiency in dynamic route reconstruction. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the dynamic route reconstruction method provided in the above embodiments, and will not be repeated here.

[0197] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A dynamic route reconstruction method, characterized in that, The dynamic route reconstruction method includes: A systematic model of the routing reconstruction problem is performed based on the UAV swarm network, generating a network topology model and a routing reconstruction model; Based on the aforementioned drone swarm network, node importance is assessed, and a deliberate attack model corresponding to the drone swarm network is established. Based on the network topology model, the route reconstruction model, and the intentional attack model, a Markov decision process transformation is performed to generate the corresponding Markov decision process. The Markov decision process is iteratively solved using a reinforcement learning algorithm to output the optimal routing strategy for the UAV swarm network. The steps of systematically modeling the routing reconstruction problem based on the UAV swarm network and generating a network topology model and a routing reconstruction model include: Obtain the node location information of each drone node in the drone swarm network, and construct a network topology model based on the node location information and node communication distance constraints; The routing reconstruction problem is modeled based on the network topology model, and a routing reconstruction model is generated. The step of assessing node importance based on the drone swarm network and establishing a deliberate attack model corresponding to the drone swarm network includes: Obtain the node degree and link importance from the node location information of each drone node; The node degree and the link importance are weighted and fused to obtain the comprehensive importance evaluation value of the UAV node; Attack priorities are determined based on the comprehensive importance assessment value, and a deliberate attack model for the drone swarm network is generated.

2. The dynamic route reconstruction method as described in claim 1, characterized in that, The step of transforming the network topology model, the routing reconstruction model, and the intentional attack model into a Markov decision process to generate the corresponding Markov decision process includes: Based on the network topology model, a state space is constructed; Based on the aforementioned intentional attack model, an action space is constructed; Based on the aforementioned route reconstruction model, a reward function is constructed; Based on the state space, the action space, and the reward function, a Markov decision process corresponding to the UAV swarm network is constructed.

3. The dynamic route reconstruction method as described in claim 2, characterized in that, The step of iteratively solving the Markov decision process using a reinforcement learning algorithm to output the optimal routing strategy for the UAV swarm network includes: Initialize the action value function and qualification trace value; Update the node connection state in the network topology model according to the intentional attack model, and determine the current state of the current training round in the state space; In the current state, the current action corresponding to the current state is determined based on the target greedy strategy, and the immediate reward is obtained according to the reward function; Based on the current state, the current action, and the immediate reward, calculate the temporal difference error in the current state; The action value function and the qualification trace value are updated based on the time-series differential error. The current state is updated to the next state and iteratively executed until the action value function converges, thereby obtaining the optimal routing strategy of the UAV swarm network.

4. The dynamic route reconstruction method as described in claim 3, characterized in that, The step of updating the action value function and the qualification trace value based on the temporal difference error includes: A decay coefficient is determined based on the discount factor and the qualification decay parameter, and the qualification trace values ​​corresponding to the current state and the current action are updated by decaying the decay coefficient. The action value function is calculated and updated based on the updated eligibility trace value and the temporal difference error.

5. The dynamic route reconstruction method as described in claim 3, characterized in that, The step of updating the node connection states in the network topology model according to the intentional attack model and determining the current state in the state space for the current training round includes: The failure nodes that are attacked and rendered ineffective in the current training round are determined based on the intentional attack model. The failed node and the links connected to the failed node are overflowed from the network topology model to obtain an updated network topology model; Based on the updated network topology model, the current initial node of the current training round is determined, and the state information corresponding to the current initial node in the state space is determined as the current state.

6. A dynamic route reconstruction device, characterized in that, The dynamic route reconstruction device includes: The system modeling module is used to perform system modeling of the routing reconstruction problem based on the UAV swarm network, generating network topology models and routing reconstruction models; An attack modeling module is used to assess the importance of nodes based on the drone swarm network and establish a deliberate attack model corresponding to the drone swarm network. The decision process module is used to transform the Markov decision process based on the network topology model, the routing reconstruction model and the intentional attack model, and generate the corresponding Markov decision process. The strategy generation module is used to iteratively solve the Markov decision process using a reinforcement learning algorithm and output the optimal routing strategy for the UAV swarm network. The system modeling module is also used to obtain the node location information of each UAV node in the UAV swarm network, and to construct a network topology model based on the node location information and node communication distance constraints. The routing reconstruction problem is modeled based on the network topology model, and a routing reconstruction model is generated. The attack modeling module is also used to obtain the node degree and link importance in the node location information of each drone node. The node degree and the link importance are weighted and fused to obtain the comprehensive importance evaluation value of the UAV node; Attack priorities are determined based on the comprehensive importance assessment value, and a deliberate attack model for the drone swarm network is generated.

7. A dynamic route reconfiguration device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the dynamic route reconfiguration method as described in any one of claims 1 to 5.

8. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the dynamic route reconstruction method as described in any one of claims 1 to 5.

Images