A fault routing decision method for multi-UAV systems

Abstract

The present invention proposes a fault routing decision method for a multi-UAV system, comprising the following steps: Step 1, improving the traditional multi-agent reinforcement learning method by adding a collaborative reward revaluation mechanism to obtain an improved multi-agent reinforcement learning method; Step 2, setting the multi-UAV system to an idealized network environment, training and optimizing the improved multi-agent reinforcement learning method to obtain an optimized policy network; Step 3, setting a fault detection cycle and making routing decisions based on the fault detection results to complete the fault routing decision for the multi-UAV system. The present invention can adapt to the demand for efficient data transmission in complex dynamic environments and plays a key role in improving the mission execution capability and application efficiency of multi-UAV systems. Among them, the reward revaluation mechanism can improve the propagation efficiency of delayed rewards and accelerate the training convergence speed of the routing decision algorithm.

Description

Technical Field

[0001] The present invention relates to a fault routing decision method, in particular to a fault routing decision method for a multi-UAV system. Background Art

[0002] With the continuous innovation and intelligent development of UAV technology, the application scenarios of multi-UAV systems in emerging fields such as intelligent traffic scheduling and marine resource exploration are constantly expanding. Multi-UAV collaborative operations have greatly promoted the digital transformation process of various industries with their convenient deployment, rapid response, and flexible operation. However, in actual operation, multi-UAV systems are facing increasingly severe challenges in data transmission and routing decisions. On the one hand, when UAVs fly in complex airspace environments, they are interfered by factors such as atmospheric turbulence and urban canyon effects, causing the network topology to show frequent and irregular dynamic changes. At the same time, the communication links of multi-UAV systems are susceptible to co-frequency interference, multipath effects, and sudden obstacles, resulting in packet loss, delay jitter and other problems in data transmission, which seriously threaten the stability of the communication links. On the other hand, when performing tasks, multi-UAV systems must transmit massive amounts of heterogeneous information in real time, including high-precision positioning data, multispectral imagery, and complex environmental perception data. Traditional routing decision-making methods struggle to balance routing stability and data transmission efficiency when dealing with dynamic topology changes and the mixed transmission of heterogeneous data. This can easily lead to routing holes and path oscillation, significantly reducing the efficiency of multi-UAV systems when performing complex collaborative tasks. Furthermore, the traditional use of reinforcement learning to intelligently determine UAV data routing paths suffers from a serious "delayed reward problem." This poses a significant challenge to data routing decision-making algorithms, such as Multi-Agent Reinforcement Learning (MARL) algorithms. Agents struggle to determine the impact of current actions on future outcomes and to distinguish the contribution of specific actions to the final outcome. Consequently, agents must spend more time and resources trying different action combinations to find strategies that yield long-term benefits. Summary of the Invention

[0003] Purpose of the invention: The technical problem to be solved by the present invention is to provide a fault routing decision method for a multi-UAV system in response to the shortcomings of the existing technology.

[0004] In order to solve the above technical problems, a fault routing decision method for a multi-UAV system is proposed, which includes the following steps:

[0005] Step 1: Improve the traditional multi-agent reinforcement learning method by adding a collaborative reward revaluation mechanism to obtain an improved multi-agent reinforcement learning method;

[0006] Step 2: Set the multi-UAV system as an idealized network environment, train and optimize the improved multi-agent reinforcement learning method, and obtain the optimized policy network;

[0007] Step 3: Set the fault detection cycle and make routing decisions based on the fault detection results to complete the fault routing decision for the multi-UAV system.

[0008] Furthermore, the routing decision is made based on the fault detection result as described in step 1, namely:

[0009] When a fault is detected, an offline fault-tolerant interruption mechanism is used to replace the optimized policy network for decision-making. Otherwise, the optimized policy network is used for decision-making.

[0010] Furthermore, the traditional multi-agent reinforcement learning method described in step 1 is improved, namely:

[0011] When a data packet in a multi-UAV system successfully reaches the sink node, the cross-agent collaborative reward revaluation mechanism is triggered.

[0012] Furthermore, the traditional multi-agent reinforcement learning method described in step 1 is improved, specifically including the following steps:

[0013] Step 1-1: The aggregation node analyzes the transmission path of the data packet , which is expressed as follows:

[0014] ;

[0015] in, is the sink node, is the source node, Number the node;

[0016] Step 1-2: Calculate the three core indicators that reflect transmission efficiency on the transmission path: hop count , the total delay from source node to sink node and the probability of successful transmission from the source node to the sink node ;

[0017] Steps 1-3: Calculate revaluation rewards based on core indicators , as follows:

[0018] ;

[0019] in, 、 and are the optimization weights for controlling hop count, delay, and reliability respectively;

[0020] Steps 1-4, based on the revaluation reward, correct the immediate reward in the traditional multi-agent reinforcement learning method to the revaluated collaborative reward.

[0021] Furthermore, the three core indicators reflecting the transmission efficiency on the transmission path are calculated as described in steps 1-2. The specific method is as follows:

[0022] Step 1-2-1, calculate the number of hops , which is expressed as follows:

[0023] ;

[0024] Among them, the hop count refers to the number of nodes that the data packet passes through except the source node;

[0025] Step 1-2-2, calculate the total delay from the source node to the sink node , which is expressed as follows:

[0026] ;

[0027] in, For nodes To Node The link transmission delay, is the node number;

[0028] Step 1-2-3, calculate the number of packets successfully received from the source node Transmit to the sink node Probability , which is expressed as follows:

[0029] ;

[0030] in, is a node To Node Packet loss rate.

[0031] Furthermore, the immediate rewards in the traditional multi-agent reinforcement learning method are modified to the revalued collaborative rewards based on the revalued rewards described in steps 1-4, as follows:

[0032] ;

[0033] in, For instant rewards, is the revalued synergy reward, is the backtracking gain coefficient, which is used to control the global reward injection intensity; is the time attenuation factor, used to suppress overcompensation of the end node;

[0034] The path Corresponding decision chain The reward is updated to ,in, Indicates that the status Select Action decision.

[0035] Furthermore, the training and optimization of the improved multi-agent reinforcement learning method described in step 2 includes the following steps:

[0036] Step 2-1, recalculate the advantage function estimate ;

[0037] Step 2-2: Update the parameters of the policy network of the drone node according to the new advantage function estimate to obtain the optimized policy network.

[0038] Furthermore, the advantage function estimate is recalculated as described in step 2-1 , as follows:

[0039] ;

[0040] in, is the value network pair state of the current drone node valuation, Represents the new advantage function estimate, which is used to measure the time step Inner State Select Action How good or bad is relative to the average situation? represents the discount factor, Indicates the total number of time steps.

[0041] Furthermore, the parameters of the policy network of the updated drone node described in step 2-2 are as follows:

[0042] ;

[0043] in, Represents the loss function of the policy network About policy network parameters The gradient, Represents the expected value, which is used to calculate the policy network parameters The average gradient under all possible state and action combinations, represents the gradient, Indicates that the status Select Action probability.

[0044] Furthermore, setting the fault detection period and making routing decisions based on the fault detection results described in step 3 include the following steps:

[0045] Step 3-1, set the fault detection period to ;

[0046] Step 3-2: perform fault detection on each adjacent node in each cycle. Let the set of neighboring nodes that detect faults be ;

[0047] Step 3-3, construct a modified reward function based on the set of neighbor nodes , correct the reward function Used to determine the fault neighbor node set To adjust the reward value, obtain a new reward by modifying the reward function , which is used to replace the optimized policy network with an offline fault-tolerant interruption mechanism for decision-making, as follows:

[0048] ;

[0049] in, The reward obtained by the node under normal operating conditions of the policy network, that is, when no faults or anomalies are detected;

[0050] In steps 3-4, each node monitors the status of its adjacent links in real time and makes a switching decision, as shown below:

[0051] ;

[0052] in, Indicates that the status Next, according to the strategy Select Action , Representation node and The link loss between The next-hop node selection of the traditional method is to use the optimized policy network to make decisions.

[0053] Beneficial effects:

[0054] 1. The proposed fault routing decision-making scheme for multi-UAV systems can adapt to the demand for efficient data transmission in complex dynamic environments and plays a key role in improving the mission execution capability and application efficiency of multi-UAV systems.

[0055] 2. The reward revaluation mechanism designed by the present invention can improve the propagation efficiency of delayed rewards and accelerate the training convergence speed of the routing decision algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more apparent.

[0057] Figure 1 This is a diagram of a reward value revaluation mechanism of the present invention.

[0058] Figure 2a This is a diagram of the reward value distribution of the RR-MADDPG algorithm after routing faults.

[0059] Figure 2b This is a diagram of the reward value distribution of the RR-MADDPG algorithm under normal circumstances.

[0060] Figure 3 The invention discloses a UAV routing decision diagram based on an offline fault-tolerant interruption mechanism OFRT.

[0061] Figure 4 This is an example diagram of a fault routing decision method based on an interruption perception mechanism of the present invention. DETAILED DESCRIPTION

[0062] In order to provide accurate, reliable and stable data routing and sharing solutions for multi-UAV collaborative systems based on multi-agent reinforcement learning (MARL) decision-making algorithm in complex airspace environments, the present invention proposes a fault routing decision-making solution for multi-UAV systems.

[0063] The technical solutions of the present invention are as follows:

[0064] Firstly, a multi-agent reinforcement learning algorithm based on the reward value revaluation mechanism is constructed. Based on this, a fault routing decision algorithm based on the interruption perception mechanism is proposed to solve the multi-UAV fault data routing problem.

[0065] 1. Multi-agent reinforcement learning algorithm based on reward value revaluation mechanism

[0066] Traditional multi-agent reinforcement learning (MARL) algorithms typically use immediate rewards to guide the learning process (see: Zhu Jiahui, Ning Jin, Zhou Yanhong, et al. Dynamic SPMA Protocol for UAV Self-Organizing Networks Based on Q-Learning [J / OL]. Computer Applications and Software, 1-8 [2025-05-30]). This means that after an agent takes an action, it immediately receives a reward. This reward mechanism helps the algorithm quickly adjust and optimize its strategy, allowing the agent to promptly understand the effects of its actions.

[0067] In UAV routing scenarios, data must be forwarded multiple times from the source node to the sink node, often resulting in long transmission delays. After the UAV selects an action, it can only observe the environmental state of the data packet at the next hop and receive an immediate reward. When the data is transmitted to the sink node, a result is generated to verify the rationality of the previous action. Although traditional multi-agent reinforcement learning algorithms consider backpropagating reward signals for future time steps, they do not fully utilize routing information to strengthen the rationality of reward distribution, resulting in a serious delayed reward problem.

[0068] The emergence of delayed rewards poses a significant challenge to multi-agent reinforcement learning (MARL) algorithms. Unmanned aerial vehicles (UAVs) struggle to determine the impact of current actions on future outcomes and to distinguish the contribution of a specific action to the final outcome. Consequently, UAVs must expend significant time and resources experimenting with different action combinations to find strategies that yield long-term benefits. This reliance on long-term rewards makes it difficult for agents to balance exploration and exploration, ultimately impacting the convergence of algorithm training.

[0069] In the standard MADDPG (Multi-Agent Deep Deterministic Policy Gradient) algorithm, the processing of reward signal backpropagation events mainly relies on the following mechanisms:

[0070] Mechanism 1: Centralized Policy Network Critic Network

[0071] The agent's policy network receives global state information s, the actions of all agents during training The output of the policy network is the Q-value estimate of the agent's action. , backpropagation is achieved by minimizing the following Bellman error:

[0072]

[0073] in, To represent an agent The loss function is used to measure the prediction Values ​​and Goals The difference between the values, Indicates that the status Take action Transition to a new state and get rewarded expectations, It is an intelligent agent The reward signal, It is an intelligent agent In state Take the following action sequence The Q value of , which is the predicted total reward for this action sequence, is the output of the target policy network, is the action generated by the target policy network. When the policy gradient is updated, the policy network Partial guidance of action It is passed to its policy network through the chain rule, as shown in formula (2). This implicit propagation is inefficient, especially in long path scenarios, making it difficult for the agent to quickly learn the optimal path strategy.

[0074]

[0075] in, Represents policy network parameters About loss function The gradient, Representation Policy Network Parameters About Observations gradient.

[0076] Mechanism 2: Experience Replay

[0077] By storing historical experience data, the agent is allowed to repeatedly learn from past events. After the reward finally arrives, the intermediate state-action pairs stored in the experience pool can be gradually associated with the final reward through multiple samplings. However, the independent decision fragments stored in the experience replay pool are difficult to reflect the global causal relationship of multi-hop collaboration. Figure 1 As shown, the local value network converts the current observation value , Current Action , Current Rewards , the observation value of the next state of the current node , put it into the experience buffer pool.

[0078] The common problem of the above two mechanisms is that the efficiency of reward signal propagation to early states is relatively low, the final result cannot be explicitly associated with intermediate decisions, and they rely on global reward information.

[0079] Therefore, the present invention designs a reward revaluation mechanism to improve the propagation efficiency of delayed rewards and accelerate the training convergence speed of the routing decision algorithm.

[0080] 1.1 Reward Revaluation Mechanism Design

[0081] This paper proposes a distributed routing optimization method based on delayed global evaluation. Its core innovation lies in: through the offline reward estimation and recalibration mechanism (RERM) of the aggregation node, the real-time local feedback of traditional reinforcement learning is converted into a global decision correction with a time span, thereby enhancing the long-term decision-making ability of the intelligent agent in complex environments.

[0082] When a data packet arrives at the sink node, it comprehensively analyzes historical decision sequences, including the number of hops along the data transmission path, link quality, and total transmission latency. Using a pre-established lightweight non-reinforcement learning model, the sink node reassesses the quality of the UAV node's historical decisions and generates a set of compensatory rewards, indirectly modifying the current decision model and influencing the agent's strategy update.

[0083] At the implementation level, this invention leverages the network's existing heartbeat packet channel to transmit these patch rewards, avoiding additional communication overhead. Because these rewards are not time-sensitive, they are well adapted to various network environments. Upon arrival, patch reward data is used to dynamically update the experience replay pool, revising the long-term value estimates of historical state-action pairs. This mechanism enables the agent to identify the potential impact of earlier decisions.

[0084] This approach is particularly well-suited to handling rapidly changing network environments, ensuring the algorithm's adaptability. When changes such as node movement or link interruption are detected, the system can reflect these changes in the experience replay pool through reward revaluation, thereby quickly guiding the agent to adapt to new decision-making plans. Targeted corrections can be made in rapidly changing network environments. For example, increasing the delay penalty coefficient during reward revaluation can encourage the agent to more quickly avoid congested paths.

[0085] In terms of effectiveness, the method of the present invention demonstrates significant advantages in multiple aspects. First, by introducing a reward revaluation mechanism, the algorithm can more accurately capture the long-term impact of actions, addressing the short-term field of view limitations of traditional reinforcement learning. Second, the patching reward acts as a soft intervention of expert experience, effectively reducing ineffective exploration and significantly improving training efficiency. This reward revaluation mechanism is also universal and flexible. It is not only applicable to various network topologies, but can also work in conjunction with other optimization algorithms.

[0086] 1.2 Reward Function Design

[0087] In the Markov Decision Process (MDP) model of this application, the reward function The definition is as follows:

[0088]

[0089] in, All are adjustable parameters. is the reward for forward propagation of data packets, Reward for the environment, is the minimum hop reward, For delayed rewards, For link transmission quality rewards, the specific definition of the reward function is as follows:

[0090] 1. Data packet forward propagation reward :

[0091] It is used to indicate whether the data has been delivered to the sink node and to impose step penalties to limit the behavior of returning data between nodes. The formula is as follows:

[0092]

[0093] in, is the positive reward for successful data arrival, is a time-step-based penalty.

[0094] 2. Environmental rewards :

[0095] Environmental Rewards The actual airspace environmental factors (such as air speed and ambient noise intensity) are taken into account. These factors will affect the routing performance of the nodes. Therefore, the routing quality can be optimized by encouraging UAV nodes to adapt to changes in the dynamic environment. The environmental reward is defined as follows:

[0096]

[0097] in, represents the air velocity at node i, represents the ambient noise intensity in the area where the node is located, and and are weight factors that control the impact of air velocity and ambient noise intensity on stability.

[0098] 3. Minimum hop reward :

[0099] The minimum number of hops in the routing path from the sink node directly guides the convergence direction of the reinforcement learning algorithm. The smaller the minimum hop number, the higher the routing reliability. as follows:

[0100]

[0101] in, is the next hop node from the decision To the sink node The shortest path length.

[0102] 4. Delay Rewards :

[0103] Delay Rewards Considering only propagation delay, the propagation delay is usually large due to the long distance between UAVs, so standardization is required. The standardization process is as follows:

[0104]

[0105] in, and They represent the maximum and minimum propagation delays between nodes, is the normalized propagation delay, ranging from [0,1]. The definition is as follows:

[0106]

[0107] in, is an adjustable parameter, The larger it is, the greater the impact of propagation delay on the reward function.

[0108] 5. Link transmission quality rewards :

[0109] When a packet loss event occurs during data transmission, a penalty is given to the agent. The formula is as follows:

[0110]

[0111] in, is an adjustable parameter, is the next hop node for action decision, is a node The packet loss indicator function between follows the Bernoulli distribution:

[0112]

[0113] in, It is at the node Select an action The link packet loss rate.

[0114] 1.3, MADDPG algorithm based on reward revaluation mechanism

[0115] To address the credit allocation shortcomings of traditional MADDPG in multi-hop routing scenarios, this paper proposes a reward revaluation-based multi-agent deep deterministic policy gradient algorithm RR-MADDPG (Reward Revaluation Multi-Agent Deep Deterministic Policy Gradient). The key design of the algorithm is as follows:

[0116] Whenever a data packet successfully reaches the sink node (data packet destination), the cross-agent collaborative reward revaluation mechanism is triggered. Figure 1 As shown:

[0117] First, the aggregation node analyzes the transmission path of this data packet (in The three core indicators of path transmission efficiency are calculated:

[0118] 1. Hop count :

[0119]

[0120] The hop count is the number of nodes that a data packet passes through except the source node.

[0121] 2. Total delay from data source node to sink node :

[0122]

[0123] in, For nodes arrive link transmission delay.

[0124] 3. The data packet is successfully transmitted from the source node Transmit to the sink node The probability R is:

[0125]

[0126] in, is a node arrive Packet loss rate Therefore, revaluation rewards are generated based on path indicators :

[0127]

[0128] in Control the optimization weights of hop count, delay, and reliability respectively.

[0129] Next, node-level reward distribution and experience reassessment are carried out, and nodes in the path are weighted according to their contribution. , the modified rewards are distributed by position decay as follows:

[0130]

[0131] in, is the revalued synergy reward, This is the instant reward before the correction. (Default 0.8) is the backtracking gain coefficient, which is used to control the global reward injection intensity; (Default 1.2) is the time decay factor used to suppress overcompensation of the end nodes.

[0132] Finally, for the path Corresponding decision chain Link the experience data of all participating nodes and update the experience data rewards according to the global effectiveness of the path , rewrite the reward signal of historical experience so that it carries posterior knowledge.

[0133] Reward correction cannot directly drive policy network optimization, and the direction of policy gradient update is determined by the advantage function estimate. Therefore, the present invention will recalculate the advantage function estimate during training. , so that it satisfies:

[0134]

[0135] in, is the current value network state valuation, Represents the new advantage function estimate, which is used to measure the time step Inner State Select Action How good or bad is relative to the average situation? represents the discount factor, Indicates the total number of time steps.

[0136] The parameter update of the policy network depends on the above gradient estimation formula as follows:

[0137]

[0138] in, Represents the loss function of the policy network About policy network parameters The gradient, Represents the expected value, which is used to calculate the policy network parameters The average gradient under all possible state and action combinations, represents the gradient, Indicates that the status Select Action probability.

[0139] The pseudo code of the RR-MADDPG algorithm is shown in Table 1:

[0140] Table 1 Pseudo code table of RR-MADDPG algorithm

[0141]

[0142] 2. Fault Routing Decision Algorithm Based on Interruption Awareness Mechanism

[0143] When a node in a UAV network fails or is damaged, preventing it from performing functions such as data collection, forwarding, or communication, a faulty routing phenomenon occurs. This phenomenon impacts data transmission and routing strategies within the network. In multi-hop routing scenarios, this may prevent data from being forwarded through the node. Other healthy nodes (i.e., nodes that can normally perform routing functions) need to quickly detect the failure and readjust routing decisions. During the training of a multi-agent reinforcement learning algorithm, such as the RR-MADDPG algorithm of the present invention, when an agent selects a faulty node, it immediately receives a sudden negative reward. The agent may face a long period of ineffective feedback. This unstable reward-feedback mechanism makes the training process more difficult, slowing convergence. This phenomenon is similar to the traditional sparse reward problem. Furthermore, this situation increases the complexity of reward design. When the faulty node changes frequently, the reward function cannot accurately reflect the state of the faulty node. The agent may learn a suboptimal strategy, significantly affecting the algorithm's training efficiency and policy stability. Figure 2a and Figure 2b The performance of the reward value distribution of the RR-MADDPG algorithm discovered by this invention when facing routing faults is demonstrated. Compared with the reward value distribution when there are no routing faults, there is obvious unevenness, which greatly affects the normal convergence of the training process.

[0144] Aiming at the problem of convergence difficulty of multi-agent reinforcement learning training caused by routing faults, this paper proposes an "offline fault-tolerant interruption mechanism" to enhance the adaptability of the RR-MADDPG algorithm and the robustness of the UAV routing decision algorithm.

[0145] 2.1 Principle and Design of Interrupt Awareness Mechanism

[0146] Interrupts are a crucial technology in computer systems, used to respond to external or internal events during program execution, pausing the current task and transitioning to a higher-priority task. Interrupts can be triggered by hardware or software and are typically used to respond to requests from external devices (such as input devices and network interfaces) or system anomalies (such as errors and overflows).

[0147] The present invention draws on the idea of ​​the interruption mechanism in multi-agent reinforcement learning. The interruption mechanism can be regarded as a means of dynamic adjustment. When an "abnormal situation" occurs (such as node damage or certain environmental changes), the node failure event is temporarily blocked, or the current training strategy is temporarily suspended, and other traditional routing mechanisms are adopted to deal with it. This can greatly improve the convergence efficiency of the multi-agent reinforcement learning algorithm while saving computing resources.

[0148] The core idea of ​​Offline Fault-Resilient Training (OFRT) is to completely block node failure events during the training phase, maintain the original optimization goal of the MARL strategy, and build an ideal network environment. , ignoring routing fault events such as node failure and link interruption, the UAV node agent is based on Learning the optimal strategy In the actual deployment phase, if a link is detected to be damaged, the traditional routing method is dynamically switched to make the path decision at the current moment, avoiding the problem of difficult convergence of strategy training caused by the dynamic environment. Figure 3 As shown in the figure, during the training phase, each UAV only considers sending packets to other UAVs, ensuring successful packet transmission. Routing faults are not considered. At this point, the UAV's value network is solely responsible for determining which nodes the current UAV selects to transmit packets, and is not responsible for other decisions. During the execution phase, normal UAVs use the value network to make decisions, while abnormal UAVs, upon discovering routing faults, use specific routing algorithms to make decisions.

[0149] 2.2 Fault Routing Decision Algorithm Based on Interrupt Awareness Mechanism

[0150] After enabling the offline fault-tolerant interruption mechanism OFRT, define the fault detection cycle , each node collects the set of faulty neighbor nodes in each period , correct the reward function , correct the reward function Used to determine the fault neighbor node set To adjust the reward value, a more effective routing decision is made in the multi-UAV system, in order to ignore the packet loss penalty caused by node failure and construct an idealized observation state, as shown in formula (18).

[0151]

[0152] in, is the reward obtained by the policy network under normal operating conditions, that is, when no faults or anomalies are detected; The function's design takes into account the impact of faulty nodes on network performance. By reducing the reward value for faulty paths, the policy network will tend to choose paths that do not pass through faulty nodes. This way, even in the event of partial node failure, the policy network can optimize path selection by adjusting the reward function, thereby improving the fault tolerance and stability of the entire system.

[0153] During the execution phase, the offline fault-tolerant interruption mechanism OFRT monitors the link status in real time. If the packet loss rate is greater than the interruption judgment threshold, , then switch to the traditional method for decision making, the logic is shown in formula (19).

[0154]

[0155] in, Indicates that the status Select Action The probability of Representation node and The link loss between Indicates the next hop node in the traditional method.

[0156] The pseudo code of the RROF-MADDPG algorithm based on OFRT and RR-MADDPG is as follows:

[0157] Table 2 Pseudo code table of the RROF-MADDPG algorithm

[0158]

[0159] Example

[0160] This paper proposes a fault routing decision method for a multi-UAV system, aiming to improve the autonomous data routing capability of the multi-UAV system in the presence of faulty nodes. This method is based on the improved multi-agent reinforcement learning algorithm (MADDPG) algorithm. By setting a fault detection cycle and making routing decisions based on the fault detection results, it can achieve efficient collaborative decision-making among multiple UAVs. This paper will use a multi-UAV system with 10 nodes as an example to illustrate the operation and implementation effect. Figure 4 Node 3 is the source node (the starting point for sending data packets), and node 7 is the sink node (the end point for sending data packets).

[0161] like Figure 4 As shown, first set the fault detection cycle for the UAV node. In this example, the initial setting Each UAV node collects the set of faulty neighbor nodes during the fault detection cycle ,like Figure 4 In the multi-UAV system with 10 nodes shown in the figure, two faulty nodes (nodes 4 and 5) are pre-set as the set of faulty neighbor nodes in the local UAV network. In this way, the system can detect faulty nodes in the network and update the set of faulty nodes in a timely manner.

[0162] During the fault-tolerant interruption training process, the policy network will completely shield node failures, link interruptions and other failures, and learn the optimal policy based on an idealized network environment. Specifically, the present invention constructs an idealized network environment. In this environment, all nodes and links are in normal state and no faults occur. By training in an idealized environment, the policy network can learn the optimal routing strategy in the absence of faults, such as Figure 4 In the training phase, the optimal strategy (3-2-5-8-7) selected a faulty node (node ​​5) because the faulty node was not accessible during training. In this process, the reward function was modified. .

[0163]

[0164] in, The reward earned by the policy network under normal operating conditions—that is, when no faults or anomalies are detected. By modifying the reward function, the system can promptly adjust the reward value when a link failure is detected, guiding the drone to choose a more reliable path and avoiding communication interruptions or delays caused by link failures.

[0165] During fault routing, drone nodes first execute the routing path according to the optimal strategy. Upon detecting a fault event such as a broken link or node failure, the system dynamically switches to traditional routing methods (for example, forwarding according to the routing table). This dynamic switching mechanism ensures that the system can quickly revert to traditional routing methods in the event of a fault, thereby ensuring the continuity and reliability of data transmission.

[0166] like Figure 4As shown in the figure, the data routing and forwarding process under 10 UAV nodes is demonstrated, in which two faulty nodes are set, namely nodes 4 and 5. Since these two nodes cannot complete the related functions of data routing normally, other nodes are required to forward data. Specifically, when node 2 forwards to node 5, due to the failure of node 5, node 2 needs to forward to other neighboring nodes, nodes 1 or 6. After selecting node 6 for routing according to the traditional method, training is performed based on the current routing situation to execute the optimal routing strategy under the current situation. After that, according to the above process, if a faulty node occurs, the interruption mechanism will be enabled, and then the optimal routing strategy will be restored. According to this process, the data packet is sent. Figure 4 The actual routing path shown is 1-2-6-9-8-7.

[0167] In summary, the RROF-MADDPG algorithm can realize autonomous data routing among multiple UAVs in the presence of faulty nodes based on the multi-UAV routing collaborative decision-making strategy.

[0168] In a specific implementation, this application provides a computer storage medium and a corresponding data processing unit. The computer storage medium is capable of storing a computer program that, when executed by the data processing unit, executes the invention disclosure of a fault routing decision method for a multi-UAV system, as well as some or all of the steps in each embodiment. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0169] Those skilled in the art will clearly understand that the technical solutions in the embodiments of the present invention can be implemented using computer programs and their corresponding general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the portion that contributes to the prior art, can be embodied in the form of a computer program, i.e., a software product. This computer program software product can be stored in a storage medium and includes instructions for enabling a device including a data processing unit (such as a personal computer, server, single-chip microcomputer, MCU, or network device) to execute the methods described in various embodiments of the present invention or certain portions of these embodiments.

[0170] This invention provides a method and approach for fault routing decision-making in a multi-UAV system. While there are numerous methods and approaches for implementing this technical solution, the aforementioned are merely preferred embodiments of the invention. It should be noted that those skilled in the art may make improvements and modifications without departing from the principles of the invention, and such improvements and modifications are considered within the scope of protection of this invention. Any components not specified in this embodiment may be implemented using existing technologies.

Claims

1. A fault routing decision method for a multi-UAV system, characterized in that: The following steps are involved: Step 1: Improve the traditional multi-agent reinforcement learning method by adding a collaborative reward revaluation mechanism to obtain an improved multi-agent reinforcement learning method; Step 2: Set the multi-UAV system as an idealized network environment, train and optimize the improved multi-agent reinforcement learning method, and obtain the optimized policy network; Step 3: Set the fault detection cycle and make routing decisions based on the fault detection results to complete the fault routing decision for the multi-UAV system; The improvement based on the traditional multi-agent reinforcement learning method described in step 1 specifically includes the following steps: Step 1-1: The aggregation node analyzes the transmission path of the data packet , which is expressed as follows: ； in, is the sink node, is the source node, Number the node; Step 1-2: Calculate the three core indicators that reflect transmission efficiency on the transmission path: hop count , the total delay from source node to sink node and the probability of successful transmission from the source node to the sink node ; Steps 1-3: Calculate revaluation rewards based on core indicators , as follows: ； in, 、 and are the optimization weights for controlling hop count, delay, and reliability respectively; Steps 1-4, based on the revaluation reward, correct the immediate reward in the traditional multi-agent reinforcement learning method to the revaluated collaborative reward.

2. The fault routing decision method for a multi-UAV system according to claim 1, characterized in that: The routing decision is made based on the fault detection results described in step 3, namely: When a fault is detected, an offline fault-tolerant interruption mechanism is used to replace the optimized policy network for decision-making. Otherwise, the optimized policy network is used for decision-making.

3. The fault routing decision method for a multi-UAV system according to claim 2, characterized in that: The traditional multi-agent reinforcement learning method described in step 1 is improved, namely: When a data packet in a multi-UAV system successfully reaches the sink node, the cross-agent collaborative reward revaluation mechanism is triggered.

4. The fault routing decision method for a multi-UAV system according to claim 3, characterized in that: The three core indicators reflecting transmission efficiency on the transmission path are calculated as described in steps 1-2. The specific method is as follows: Step 1-2-1, calculate the number of hops , which is expressed as follows: ； Among them, the hop count refers to the number of nodes that the data packet passes through except the source node; Step 1-2-2, calculate the total delay from the source node to the sink node , which is expressed as follows: ； in, For nodes To Node The link transmission delay, is the node number; Step 1-2-3, calculate the number of packets successfully received from the source node Transmit to the sink node Probability , which is expressed as follows: ； in, is a node To Node Packet loss rate.

5. The fault routing decision method for a multi-UAV system according to claim 4, characterized in that: The revaluation reward described in steps 1-4 corrects the immediate reward in the traditional multi-agent reinforcement learning method to the revalued collaborative reward as follows: ； in, For instant rewards, is the revalued synergy reward, is the backtracking gain coefficient, which is used to control the global reward injection intensity; is the time attenuation factor, used to suppress overcompensation of the end node; The path Corresponding decision chain The reward is updated to ,in, Indicates that the status Select Action decision.

6. The fault routing decision method for a multi-UAV system according to claim 5, characterized in that: Training and optimizing the improved multi-agent reinforcement learning method described in step 2 includes the following steps: Step 2-1, recalculate the advantage function estimate ; Step 2-2: Update the parameters of the policy network of the drone node according to the new advantage function estimate to obtain the optimized policy network.

7. The fault routing decision method for a multi-UAV system according to claim 6, characterized in that: Recalculate the advantage function estimate as described in step 2-1 , as follows: ； in, is the value network pair state of the current drone node valuation, Represents the new advantage function estimate, which is used to measure the time step In-state Select Action The difference between the return and the average return is represents the discount factor, Indicates the total number of time steps.

8. The fault routing decision method for a multi-UAV system according to claim 7, characterized in that: Update the parameters of the drone node's policy network described in step 2-2 as follows: ； in, Represents the loss function of the policy network About policy network parameters The gradient, Represents the expected value, which is used to calculate the policy network parameters The average gradient of all state and action combinations under the gradient, represents the gradient, Indicates that the status Select Action probability.

9. The fault routing decision method for a multi-UAV system according to claim 8, characterized in that: Setting the fault detection period and making routing decisions based on the fault detection results described in step 3 includes the following steps: Step 3-1, set the fault detection period to ; Step 3-2: perform fault detection on each adjacent node in each cycle. Let the set of neighboring nodes that detect faults be ; Step 3-3, construct a modified reward function based on the set of neighbor nodes , correct the reward function Used to determine the fault neighbor node set To adjust the reward value, obtain a new reward by modifying the reward function , which is used to replace the optimized policy network with an offline fault-tolerant interruption mechanism for decision-making, as follows: ； in, The reward obtained by the node under normal operating conditions of the policy network, that is, when no faults or anomalies are detected; In steps 3-4, each node monitors the status of its adjacent links in real time and makes a switching decision, as shown below: ； in, Indicates that the status Next, according to the strategy Select Action , Representation node and The link loss between The next-hop node selection of the traditional method is to use the optimized policy network to make decisions.

Images