Fault routing decision-making method for multi-unmanned aerial vehicle system

Abstract

The invention provides a fault routing decision-making method for a multi-unmanned aerial vehicle system, and the method comprises the following steps: 1, carrying out the improvement based on a conventional multi-agent reinforcement learning method, adding a collaborative reward revaluation mechanism, and obtaining an improved multi-agent reinforcement learning method; 2, setting a multi-unmanned aerial vehicle system as an ideal network environment, and training and optimizing the improved multi-agent reinforcement learning method to obtain an optimized strategy network; and step 3, setting a fault detection period and carrying out routing decision according to a fault detection result so as to complete fault routing decision oriented to the multi-unmanned aerial vehicle system. The method can adapt to efficient data transmission requirements in a complex dynamic environment, and plays a key role in improving the task execution capability and the application efficiency of a multi-UAV system; according to the reward reestimation mechanism, the propagation efficiency of delayed rewards can be improved, and the training convergence speed of a routing decision algorithm is increased.

Description

Technical Field

[0001] The present invention relates to a tomographic routing decision method, in particular to a tomographic routing decision method for a multi-UAV system. Background Art

[0002] With the continuous innovation and intelligent development of UAV technology, the application scenarios of multi-UAV systems in emerging fields such as intelligent transportation scheduling and marine resource exploration have been continuously expanding. The collaborative operation of multi-UAVs, with its characteristics of convenient deployment, rapid response, and flexible operation, has greatly promoted the digital transformation process of various industries. However, in the actual operation of multi-UAV systems, increasingly severe challenges in data transmission and routing decision-making are being encountered. On the one hand, when UAVs fly in complex airspace environments, they are interfered by factors such as atmospheric turbulence and urban canyon effects, resulting in frequent and irregular dynamic changes in the network topology structure; at the same time, the communication links of multi-UAV systems are vulnerable to co-frequency interference, multipath effects, and sudden obstacle occlusions, leading to problems such as packet loss and delay jitter in data transmission, seriously threatening the stability of communication links. On the other hand, when multi-UAV systems execute tasks, they need to transmit a large amount of heterogeneous information such as high-precision positioning data, multi-spectral image data, and complex environment perception data in real time. Traditional routing decision methods are difficult to balance routing stability and data transmission efficiency when dealing with dynamic topology changes and mixed transmission of heterogeneous data, and are prone to problems such as routing holes and path oscillations, greatly reducing the efficiency of multi-UAV systems when performing complex collaborative tasks. In addition, in the traditional process of using reinforcement learning to intelligently decide the data routing path of UAVs, there is a serious "delayed reward problem". The emergence of the delayed reward problem has brought huge challenges to data routing decision algorithms represented by the multi-agent reinforcement learning (MARL) algorithm. It is difficult for agents to judge the impact of current actions on future results, and it is difficult to distinguish the contribution of a specific action to the final result. Agents need to spend more time and resources trying different action combinations to find those strategies that can bring long-term benefits. Summary of the Invention

[0003] Object of the Invention: The technical problem to be solved by the present invention is to provide a tomographic routing decision method for a multi-UAV system in view of the deficiencies of the prior art.

[0004] To solve the above technical problem, a tomographic routing decision method for a multi-UAV system includes the following steps: Step 1, based on the improvement of the traditional multi-agent reinforcement learning method, adding a collaborative reward re-evaluation mechanism to obtain an improved multi-agent reinforcement learning method; Step 2: Set the multi-UAV system as an ideal network environment, train and optimize the improved multi-agent reinforcement learning method to obtain an optimized policy network. Step 3: Set the fault detection period and make routing decisions according to the fault detection results to complete the tomographic routing decision for the multi-UAV system.

[0005] Furthermore, the routing decision-making according to the fault detection results in Step 1 is as follows: When a fault is detected, an offline fault-tolerant interruption mechanism is used to replace the optimized policy network for decision-making; otherwise, the optimized policy network is used for decision-making.

[0006] Furthermore, the improvement based on the traditional multi-agent reinforcement learning method in Step 1 is as follows: When the data packet in the multi-UAV system successfully reaches the sink node, a cross-agent collaborative reward re-evaluation mechanism is triggered.

[0007] Furthermore, the improvement based on the traditional multi-agent reinforcement learning method in Step 1 specifically includes the following steps: Step 1-1: The sink node parses the transmission path of the data packet , which is expressed as follows: ; Among them, is the sink node, is the source node, is the node number; Step 1-2: Calculate three core indicators reflecting the transmission efficiency on the transmission path: the number of hops , the total delay from the source node to the sink node and the probability of successfully transmitting from the source node to the sink node ; Step 1-3: Calculate the re-evaluated reward based on the core indicators, specifically as follows: ; Among them,, , and are the optimization weights for controlling the number of hops, delay, and reliability respectively; Step 1-4: According to the re-evaluated reward, correct the immediate reward in the traditional multi-agent reinforcement learning method to the re-evaluated collaborative reward.

[0008] Furthermore, the method for calculating the three core indicators reflecting the transmission efficiency on the transmission path in Step 1-2 is as follows: Step 1-2-1: Calculate the number of hops , which is expressed as follows: ; Among them, the hop count represents the number of nodes that the data packet passes through except the source node; Step 1-2-2, calculate the total delay from the source node to the sink node , which is expressed as follows: ; Among them, is the link transmission delay from node to node , is the node number; Step 1-2-3, calculate the probability that the data packet is successfully transmitted from the source node to the sink node , which is expressed as follows: ; Among them, is the packet loss rate from node to node .

[0009] Furthermore, in step 1-4, the immediate reward in the traditional multi-agent reinforcement learning method is corrected to the re-evaluated collaborative reward according to the re-evaluated reward, as follows: ; Among them, is the immediate reward, is the re-evaluated collaborative reward, is the backtracking gain coefficient, which is used to control the global reward injection intensity; is the time decay factor, which is used to suppress the over-compensation of the end node; Update the reward of the decision chain corresponding to the path to , among which, represents the decision to select action in state .

[0010] Furthermore, the training and optimization of the improved multi-agent reinforcement learning method described in step 2 includes the following steps: Step 2-1, recalculate the advantage function estimate ; Step 2-2, update the parameters of the policy network of the UAV node according to the new advantage function estimate value to obtain an optimized policy network.

[0011] Furthermore, the recalculation of the advantage function estimate described in step 2-1​ , which is as follows: ; Among them, is the valuation of the state by the value network of the current UAV node of. represents the estimated value of the new advantage function, which is used to measure the time step internal state when selecting an action relative to the average situation, represents the discount factor, represents the total number of time steps.

[0012] Further, the parameters of the policy network for updating the UAV node described in step 2-2 are as follows: ; Among them, represents the loss function of the policy network with respect to the parameters of the policy network of the gradient, represents the expected value, which is used to calculate the average gradient of all possible state and action combinations under the gradient of the policy network parameters , represents the gradient, represents the probability of selecting an action in the state .

[0013] Further, setting the fault detection period and making routing decisions according to the fault detection results described in step 3 includes the following steps: Step 3-1, set the fault detection period to ; Step 3-2, perform fault detection on each adjacent node within each period, and set the set of neighbor nodes detected with faults to ; Step 3-3, construct a modified reward function according to the set of neighbor nodes, and the modified reward function is used to adjust the reward value according to the set of faulty neighbor nodes , and obtain a new reward through the modified reward function, which is used to make decisions by adopting an offline fault-tolerant interruption mechanism instead of the optimized policy network, specifically as follows: ; Among them, is the reward obtained by the node under the normal operating conditions of the policy network, that is, when no faults or abnormalities are detected; Step 3-4: Each node monitors the status of the links adjacent to it in real time and makes a decision to switch, which is expressed as follows: ; Among them, indicates that in the state according to the policy select the action , indicates the link loss between nodes and . indicates that the next-hop node selection of the traditional method, that is, uses the optimized policy network for decision-making.

[0014] Beneficial effects: 1. A tomographic routing decision scheme for a multi-UAV system proposed by the present invention can meet the high-efficiency data transmission requirements in a complex dynamic environment and plays a key role in improving the task execution ability and application efficiency of the multi-UAV system.

[0015] 2. The reward re-evaluation mechanism designed by the present invention can improve the propagation efficiency of delayed rewards and accelerate the training convergence speed of the routing decision algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The following further specifically describes the present invention in conjunction with the drawings and specific embodiments, and the above and / or other advantages of the present invention will become clearer.

[0017] Figure 1 FIG. is a diagram of a reward value re-evaluation mechanism of the present invention.

[0018] Figure 2a FIG. is a schematic diagram of the reward value distribution of the RR-MADDPG algorithm after the routing fault.

[0019] Figure 2b FIG. is a schematic diagram of the reward value distribution of the RR-MADDPG algorithm under normal conditions.

[0020] Figure 3 FIG. is a UAV routing decision diagram of the present invention based on the offline fault tolerance interruption mechanism OFRT.

[0021] Figure 4 FIG. is an example diagram of a tomographic routing decision method based on the interruption perception mechanism of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0022] In order to provide an accurate, reliable and stable data routing and sharing scheme for a multi-UAV cooperative system based on a multi-agent reinforcement learning MARL decision algorithm in a complex airspace environment, the present invention proposes a tomographic routing decision scheme for a multi-UAV system.

[0023] The technical solution of the present invention is as follows: First, a multi-agent reinforcement learning algorithm based on a reward value re-estimation mechanism is constructed. Based on this, for the multi-UAV tomographic data routing problem, a tomographic routing decision algorithm based on an interruption awareness mechanism is proposed.

[0024] I. Multi-agent reinforcement learning algorithm based on a reward value re-estimation mechanism Traditional multi-agent reinforcement learning (MARL) algorithms usually use immediate rewards to guide the learning process of agents (Reference: Zhu Jiahui, Ning Jin, Zhou Yanhong, etc. Dynamic SPMA protocol for UAV ad hoc networks based on Q-learning [J / OL]. Computer Applications and Software, 1-8 [2025-05-30]). That is, after an agent selects an action, it immediately receives a feedback reward. This reward mechanism can help the algorithm quickly adjust and optimize the strategy, enabling the agent to promptly understand the effects of its actions.

[0025] In the scenario of UAV routing, data needs to be forwarded multiple times from the source node to the sink node, often resulting in a long transmission delay. After the UAV selects an action, it can only observe the environmental state where the data packet reaches the next-hop node and obtain an immediate reward. When the data is transmitted to the sink node, a result is generated to verify whether the previous action is reasonable. Although traditional multi-agent reinforcement learning algorithms consider backpropagating the reward signals of future time steps, they do not fully utilize the routing path information to strengthen the rationality of reward distribution, that is, there is a serious delayed reward problem.

[0026] The emergence of the delayed reward problem poses a huge challenge to the multi-agent reinforcement learning (MARL) algorithm. It is difficult for the UAV to judge the impact of the current action on future results and distinguish the contribution of a specific action to the final result. The UAV needs to spend more time and resources trying different action combinations to find those strategies that can bring long-term benefits. This dependence on long-term rewards makes it difficult for the agent to balance the exploration or not, ultimately affecting the convergence of algorithm training.

[0027] In the standard MADDPG (Multi-Agent Deep Deterministic Policy Gradient) algorithm, the handling of reward signal backpropagation events mainly relies on the following mechanisms: Mechanism 1: Centralized policy network Critic network The policy network of the agent receives the global state information s and the actions of all agents during training . The output of the policy network is the Q-value estimate of the agent's action , backpropagation is achieved by minimizing the following Bellman error:

[0028] where is the loss function representing the agent , which is used to measure the difference between the predicted value and the target value, represents the expectation of taking action in state to transfer to a new state and obtaining a reward , is the reward signal of the agent , is the Q-value of the agent taking the action sequence in state , that is, the predicted total return of this action sequence, is the output of the target policy network, is the action generated by the target policy network. When updating the policy gradient, the partial derivative of the policy network with respect to the action of the agent is passed to its policy network through the chain rule, as shown in Equation (2). This implicit propagation is inefficient, especially in long-path scenarios, resulting in the agent having difficulty quickly learning the optimal path policy. where

[0029]

[0030] where represents the gradient of the policy network parameters with respect to the loss function , represents the gradient of the policy network parameters with respect to the observation .

[0031] Mechanism 2: Experience Replay By storing historical experience data, it allows the agent to repeatedly learn past events. After the reward finally arrives, the intermediate state-action pairs stored in the experience pool can be gradually associated with the final reward through multiple samplings. However, the independent decision fragments stored in the experience replay pool are difficult to reflect the global causal relationship of multi-hop cooperation. As Figure 1 shown, the local value network puts the current observation , the current action , the current reward , and the observation of the next state of the current node into the experience buffer pool.

[0032] The common problems of the above two mechanisms are that the reward signal propagates to the early states with low efficiency, cannot explicitly correlate the final result with the intermediate decisions, and relies on global reward information, etc.

[0033] Therefore, the present invention designs a reward re - evaluation mechanism to improve the propagation efficiency of delayed rewards and accelerate the training convergence speed of the routing decision algorithm.

[0034] 1.1 Design of the reward re - evaluation mechanism The present invention proposes a distributed routing optimization method based on delayed global evaluation. Its core innovation lies in: through the offline value re - evaluation mechanism (Reward Estimation and Recalibration Mechanism, RERM) of the sink node, it transforms the real - time local feedback of traditional reinforcement learning into a global decision correction with a time span, thereby enhancing the long - term decision - making ability of the agent in a complex environment.

[0035] When a data packet arrives at the sink node, the sink node comprehensively analyzes multi - dimensional information such as the historical decision sequence, the number of hops of the data transmission path, the link quality, and the total transmission delay. The sink node uses a pre - established lightweight non - reinforcement learning model to re - evaluate the historical decision quality of the UAV node, generates a set of compensatory rewards, forms an indirect correction to the current decision model, and affects the policy update of the agent.

[0036] At the implementation level, the present invention uses the existing heartbeat packet channel in the network to transmit these patched rewards, avoiding additional communication overhead. Since these rewards have low requirements for timeliness, they can well adapt to various network environments. After the patched reward data arrives, it is used to dynamically update the experience replay pool and correct the long - term value estimation of the historical state - action pairs. This mechanism enables the agent to identify the potential impact of early decisions.

[0037] This method is particularly suitable for dealing with the situation where the network environment changes rapidly, ensuring the adaptability of the algorithm to the environment. When changes such as node movement or link interruption are detected, the system can reflect these changes to the experience replay pool through reward re - evaluation, thereby quickly guiding the agent to adapt to the new decision scheme. Some targeted corrections can be made in the scenario of rapid network environment changes. For example, increasing the delay penalty coefficient during the reward re - evaluation process can prompt the agent to avoid congested paths faster.

[0038] In terms of the effects, the method of the present invention demonstrates significant advantages in multiple aspects. First, by introducing the reward re - evaluation mechanism, the algorithm can more accurately capture the long - term impact of actions, solving the problem of the limited short - term vision of traditional reinforcement learning. Second, as a soft intervention of expert experience, the patched reward effectively reduces ineffective exploration and greatly improves the training efficiency. This reward re - evaluation mechanism also has universality and flexibility. It is not only applicable to various network topologies but can also work in cooperation with other optimization algorithms.

[0039] 1.2 Reward Function Design In the Markov decision process (MDP) model of this application, the reward function is defined as follows:

[0040] where are all adjustable parameters, is the forward propagation reward of the data packet, is the environment reward, is the minimum hop count reward, is the delay reward, is the link propagation quality reward, and the specific definition of the reward function is as follows: 1. Forward Propagation Reward of Data Packet : It is used to indicate whether the data has reached the sink node and at the same time imposes a step penalty to limit the behavior of the nodes to send data back and forth between nodes. The formula is as follows:

[0041] where is the positive reward for the successful arrival of the data, is the penalty based on the time step.

[0042] 2. Environment Reward : The environment reward considers the actual airspace environment factors (such as airspeed, environmental noise intensity), which will affect the routing performance of the nodes. Therefore, the routing quality can be optimized by encouraging the UAV nodes to adapt to the changes in the dynamic environment. The definition of the environment reward is as follows:

[0043] where represents the magnitude of the airspeed at node i, represents the environmental noise intensity in the area where the node is located, and and represent the weight factors controlling the influence of airspeed and environmental noise intensity on stability, respectively.

[0044] 3. Minimum hop count reward : The minimum hop count of the routing path to the sink node directly guides the convergence direction of the reinforcement learning algorithm. The smaller the minimum hop count, the higher the routing reliability. It is defined as follows:

[0045] where is the shortest path length from the next-hop node of the decision to the sink node

[0046] 4. Delay reward : Delay reward Only considers the propagation delay. Since the distance between UAVs is relatively far, the propagation delay is usually large and needs to be standardized. The standardization process is as follows:

[0047] where and respectively represent the maximum and minimum propagation delays between nodes, is the standardized propagation delay, and its value range is [0, 1]. The delay reward is defined as follows:

[0048] where is an adjustable parameter, the larger it is, the greater the impact of the propagation delay on the reward function.

[0049] 5. Link transmission quality reward : When a packet loss event occurs during data packet transmission, a penalty is given to the agent, and the formula for

[0050] where is an adjustable parameter, is the next-hop node of the action decision, is the node and the packet loss indication function between nodes

[0051] where is the link packet loss rate when choosing the action at the node ​​

[0052] 1.3, MADDPG Algorithm Based on Reward Re-evaluation Mechanism Aiming at the credit assignment defect of traditional MADDPG in multi-hop routing scenarios, the present invention proposes a multi-agent deep deterministic policy gradient algorithm RR-MADDPG (Reward Revaluation Multi-Agent Deep Deterministic Policy Gradient) based on reward re-evaluation. The key design of the algorithm is as follows: Whenever a data packet successfully reaches the sink node (the end point of the data packet), a cross-agent collaborative reward re-evaluation mechanism is triggered. As Figure 1 shown: First, the sink node parses the transmission path of the current data packet (where is the sink node), and calculates three core indicators of the path transmission efficiency: 1. Hop count :

[0053] Among them, the hop count represents the number of nodes passed by the data packet except the source node.

[0054] 2. Total delay from the data source node to the sink node :

[0055] Among them, is the link transmission delay from node to .

[0056] 3. Probability R that the data packet is successfully transmitted from the source node to the sink node :

[0057] Among them, is the packet loss rate from node to . Therefore, the re-evaluated reward is generated based on the path metrics : :

[0058] Among them control the optimization weights of hop count, delay, and reliability respectively.

[0059] Next, node-level reward allocation and experience re-evaluation are performed. Weighted by contribution degree, for the nodes in the path , the corrected reward is allocated according to the position attenuation as follows:

[0060] Among them, is the re-evaluated collaborative reward, is the immediate reward before correction, (default 0.8) is the backtracking gain coefficient, which is used to control the intensity of global reward injection; (default 1.2) is the time decay factor, which is used to suppress the overcompensation of the end nodes.

[0061] Finally, for the path corresponding decision chain associate the experience data of all participating nodes, and update the reward of each experience data according to the global efficiency of the path , rewrite the reward signal of historical experience so that it carries posterior knowledge.

[0062] The reward correction cannot directly drive the optimization of the policy network. The direction of policy gradient update is determined by the advantage function estimation. Therefore, the present invention will recalculate the advantage function estimation during training , so that it satisfies:

[0063] Among them, is the valuation of the current value network for the state , represents the new advantage function estimation value, which is used to measure the goodness or badness of the action selected in the internal state relative to the average situation, represents the discount factor, represents the total number of time steps.

[0064] The parameter update of the policy network depends on the above gradient estimation as follows:

[0065] Among them, represents the loss function of the policy network with respect to the policy network parameter gradient, represents the expected value, which is used to calculate the average gradient under all possible state and action combinations for the gradient of the policy network parameter , represents the gradient, represents at the state select the action probability.

[0066] The pseudo-code of the RR-MADDPG algorithm is shown in Table 1: Table 1 Pseudo-code table of the RR-MADDPG algorithm

[0067] II. Fault Routing Decision Algorithm Based on Interruption Awareness Mechanism When a node in the UAV network fails or is damaged and can no longer perform functions such as data collection, forwarding, or communication, a fault routing phenomenon occurs. This phenomenon will affect data transmission and routing strategies in the network. In the scenario of multi-hop routing, it may cause data not to be forwarded through this node, and other healthy nodes (i.e., nodes that can normally perform routing functions) need to quickly detect the fault and readjust the routing decision. During the training process of multi-agent reinforcement learning algorithms, such as the RR-MADDPG algorithm of the present invention, when an agent selects a faulty node, it will immediately obtain a sudden negative reward. The agent may face ineffective feedback for a long time. This unstable reward feedback mechanism makes the training process more difficult, thus slowing down the convergence speed. This phenomenon is similar to the traditional sparse reward problem; at the same time, this situation increases the complexity of reward design. When the faulty nodes change dynamically and frequently, the reward function cannot accurately reflect the state of the faulty nodes, and the agent may learn sub-optimal strategies, greatly affecting the training efficiency and policy stability of the algorithm. Figure 2a and Figure 2b shows the performance of the reward value distribution of the RR-MADDPG algorithm discovered by the present invention when facing routing faults. Compared with the reward value distribution without routing faults, there is obvious non-uniformity, which greatly affects the normal convergence of the training process.

[0068] The present invention proposes an "offline fault-tolerant interruption mechanism" to enhance the adaptability of the RR-MADDPG algorithm and the robustness of the UAV routing decision algorithm for the problem of difficult convergence in multi-agent reinforcement learning training caused by routing faults.

[0069] 2.1 Principle and Design of the Interruption Awareness Mechanism The interruption mechanism is an important technology in computer systems, used to respond to the occurrence of external or internal events during program execution and timely suspend the execution of the current task, and then process tasks with higher priorities. Interrupts can be triggered by hardware or software, and are usually used to respond to requests from external devices (such as input devices, network interfaces, etc.) or system exceptions (such as errors, overflows, etc.).

[0070] In multi-agent reinforcement learning, the present invention draws on the idea of an interruption mechanism, which can be regarded as a means of dynamic adjustment. When an "abnormal situation" occurs (such as node damage or certain environmental changes), the node failure event can be temporarily masked, or the current training strategy can be temporarily aborted, and other traditional routing and pathfinding mechanisms can be adopted to deal with it. This can greatly improve the convergence efficiency of the multi-agent reinforcement learning algorithm and save computing resources at the same time.

[0071] The core idea of the Offline Fault-Resilient Training (OFRT) mechanism is to completely mask node failure events during the training phase, maintain the original optimization goal of the MARL strategy, and construct an idealized network environment. Ignoring routing fault events such as node failures and link interruptions, the UAV node agent learns the optimal strategy based on learning the optimal strategy . During the actual deployment phase, if a link failure is detected, it will dynamically switch to a traditional routing method for path decision-making at the current moment, avoiding the problem that the policy training oscillation caused by environmental dynamics is difficult to converge. A specific example is as Figure 3 shown. During the training phase, only consider that each UAV sends data packets to other UAVs during the training phase, and the data packet sending must be successful, without considering routing fault events. At this time, the value network of the UAV only responsible for deciding which nodes the current UAV selects to transmit data packets, not for other decisions. During the execution phase, normal UAVs use the value network for decision-making, and abnormal UAVs, when discovering routing fault problems, use specific routing algorithms for decision-making.

[0072] 2.2 Fault Routing Decision Algorithm Based on Interruption Awareness Mechanism After enabling the offline fault-tolerant interruption mechanism OFRT, define the fault detection period , and each node collects the set of faulty neighbor nodes per period , and correct the reward function , and correct the reward function used to adjust the reward value according to the set of faulty neighbor nodes in order to make more effective routing decisions in a multi-UAV system, ignoring the packet loss penalty caused by node failures, and constructing an idealized observation state, as shown in formula (18).

[0073]

[0074] Among them, is the reward obtained by the policy network under normal operating conditions, that is, when no faults or anomalies are detected. The design of the function takes into account the impact of faulty nodes on network performance. By reducing the reward value of faulty paths, the policy network will tend to select paths that do not pass through faulty nodes. In this way, even in the case of partial node failures, the policy network can optimize path selection by adjusting the reward function, thereby improving the fault tolerance and stability of the entire system.

[0075] During the execution phase, the offline fault-tolerant interruption mechanism OFRT monitors the link status in real time. If the detected packet loss rate is greater than the interruption determination threshold , it switches to the traditional method for decision-making, and the logic is shown in formula (19).

[0076]

[0077] Among them, represents the probability of selecting action in state , represents the link loss between nodes and , represents the next-hop node of the traditional method.

[0078] The pseudo-code of the RROF-MADDPG algorithm based on OFRT and RR-MADDPG is as follows: Table 2 Pseudo-code table of the RROF-MADDPG algorithm

[0079] Embodiment

[0080] The present invention proposes a tomographic routing decision method for a multi-unmanned aerial vehicle (UAV) system, aiming to improve the autonomous data routing ability of the multi-UAV system in the presence of faulty nodes. This method is based on an improved multi-agent reinforcement learning algorithm (MADDPG). By setting a fault detection period and making routing decisions according to the fault detection results, efficient collaborative decision-making of multiple UAVs is achieved. The present invention will take a multi-UAV system with 10 nodes as an example for example operation and implementation effect description. Set Figure 4 Node 3 in it as the source node (starting point of data packet transmission), and set Node 7 as the sink node (ending point of data packet transmission).

[0081] As Figure 4 shown, first set a fault detection period for the UAV nodes. In this example, the initial setting is . Each UAV node collects the set of faulty neighbor nodes within the fault detection period, as Figure 4In the multi-UAV system with 10 nodes shown, 2 faulty nodes (node 4 and node 5) are preset as the set of faulty neighbor nodes in the local UAV network. . In this way, the system detects faulty nodes in the network and updates the set of faulty nodes in a timely manner.

[0082] During the fault-tolerant interruption training process, the policy network completely shields faults such as node failures and link interruptions, and learns the optimal policy based on an idealized network environment. Specifically, the present invention constructs an idealized network environment , in which all nodes and links are in a normal state and no faults occur. By training in the idealized environment, the policy network can learn the optimal routing policy under fault-free conditions. For example, Figure 4 in, since the situation of unreachable faulty nodes is ignored during the training phase, the optimal policy (3-2-5-8-7) selects a faulty node (node 5). During this process, the modified reward function .

[0083]

[0084] Among them, is the reward obtained by the policy network under normal operating conditions, that is, when no faults or anomalies are detected. By modifying the reward function, the system can adjust the reward value in a timely manner when a link fault is detected, thereby guiding the UAV to select a more reliable path and avoiding communication interruptions or delays caused by link faults.

[0085] During the execution of tomographic routing, the UAV node first executes the routing path according to the optimal policy. After detecting fault events such as link damage and node failure, the system will dynamically switch to the traditional routing method (such as forwarding according to the routing table) for routing. This dynamic switching mechanism ensures that when a fault occurs, the system can quickly revert to the traditional routing method, thereby ensuring the continuity and reliability of data transmission.

[0086] As Figure 4 shown, it shows the data routing and forwarding process under 10 UAV nodes, in which two faulty nodes are set, namely node 4 and node 5. Since these two nodes cannot normally complete the relevant functions of data routing, other nodes are required to forward the data. Specifically, when node 2 forwards data to node 5, due to the failure of node 5, node 2 needs to forward the data to other neighbor nodes, node 1 or node 6. After selecting node 6 for routing according to the traditional method, training is carried out according to the current routing situation, and the optimal routing policy in the current situation is executed. After that, in case of a faulty node situation as above, the interruption mechanism will be enabled, and then the optimal routing policy will be restored, and the data packet will be sent according to this process, as Figure 4The actual routing path shown is 1-2-6-9-8-7. In summary, by using the RROF-MADDPG algorithm and based on the multi-UAV routing collaborative decision-making strategy, autonomous data routing between multiple UAVs in the presence of faulty nodes can be achieved.

[0087] In specific implementation, the present application provides a computer storage medium and a corresponding data processing unit. Among them, the computer storage medium can store a computer program, and when the computer program is executed by the data processing unit, it can run the content of the invention and some or all of the steps in each embodiment of a tomographic routing decision method for a multi-UAV system provided by the present invention. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM), etc.

[0088] Those skilled in the art can clearly understand that the technical solutions in the embodiments of the present invention can be implemented by means of a computer program and its corresponding general hardware platform. Based on such an understanding, the essence of the technical solutions in the embodiments of the present invention, or the part that contributes to the prior art, can be embodied in the form of a computer program, that is, a software product. The computer program software product can be stored in a storage medium and includes several instructions to enable a device (which can be a personal computer, a server, a single-chip microcomputer, an MCU, or a network device, etc.) including a data processing unit to execute the methods described in each embodiment or some parts of the embodiments of the present invention.

[0089] The present invention provides an idea and method for a tomographic routing decision method for a multi-UAV system. There are many methods and ways to specifically implement this technical solution. The above is only the preferred implementation manner of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention. Each component not clearly defined in this embodiment can be implemented by the prior art.

Claims

1. A tomographic routing decision method for a multi-UAV system, characterized in that, It includes the following steps: Step 1: Improve based on the traditional multi-agent reinforcement learning method, add a collaborative reward re-evaluation mechanism, and obtain the improved multi-agent reinforcement learning method; Step 2: Set the multi-UAV system as an ideal network environment, train and optimize the improved multi-agent reinforcement learning method, and obtain the optimized policy network; Step 3: Set a fault detection period and make a routing decision according to the fault detection result to complete the tomographic routing decision for the multi-UAV system.

2. The fault routing decision method for a multi-UAV system according to claim 1, wherein, The routing decision according to the fault detection result described in Step 3 is as follows: When a fault is detected, use the offline fault-tolerant interruption mechanism to replace the optimized policy network for decision-making, otherwise use the optimized policy network for decision-making.

3. The tomographic routing decision method for a multi-UAV system according to claim 2, wherein, The improvement based on the traditional multi-agent reinforcement learning method described in Step 1 is as follows: When the data packet in the multi-UAV system successfully reaches the sink node, trigger the cross-agent collaborative reward re-evaluation mechanism.

4. The tomographic routing decision method for a multi-UAV system according to claim 3, characterized in that The improvement based on the traditional multi-agent reinforcement learning method described in Step 1 specifically includes the following steps: Step 1-1, the aggregation node analyzes the transmission path of the data packet , which is expressed as follows: ； Among them, is the aggregation node, is the source node, is the node number; Step 1-2, calculate three core indicators reflecting the transmission efficiency on the transmission path: the number of hops , the total delay from the source node to the sink node , and the probability of successfully transmitting from the source node to the sink node ; Step 1-3, calculate the revaluation reward based on the core metrics , specifically as follows: ； Among them, , and are the optimization weights for controlling the hop count, delay, and reliability, respectively; Step 1-4: According to the re-evaluated reward, correct the immediate reward in the traditional multi-agent reinforcement learning method to the re-evaluated collaborative reward.

5. A tomographic routing decision method for a multi-UAV system according to claim 4, characterized in that The specific method for calculating the three core indicators reflecting the transmission efficiency on the transmission path described in Step 1-2 is as follows: Step 1-2-1, calculate the hop count , which is expressed as follows: ； Among them, the hop count represents the number of nodes that the data packet passes through except the source node; Step 1-2-2, calculate the total delay from the source node to the sink node , which is expressed as follows: ； Among them, is the link transmission delay from node to node , where is the node number; Step 1-2-3, calculate the probability that the data packet is successfully transmitted from the source node to the sink node as follows: , which is expressed as follows: ； Among them, is the packet loss rate from node to node 6. The tomographic routing decision method for a multi-UAV system according to claim 5, characterized in that The correction of the immediate reward in the traditional multi-agent reinforcement learning method to the re-evaluated collaborative reward according to the re-evaluated reward described in Step 1-4 is as follows: ； Among them, is the immediate reward, is the re-evaluated collaborative reward, is the backtracking gain coefficient, which is used to control the intensity of global reward injection; is the time decay factor, which is used to suppress the over-compensation of terminal nodes; Update the reward of the decision chain corresponding to the path to , where indicates the decision of selecting action in state . ​ 7. A tomographic routing decision method for a multi-UAV system according to claim 6, characterized in that, The training and optimization of the improved multi-agent reinforcement learning method described in Step 2 include the following steps: Step 2-1, recalculate the advantage function estimate ; Step 2-2: According to the new estimated value of the advantage function, update the parameters of the policy network of the UAV node to obtain the optimized policy network.

8. The tomographic routing decision method for a multi-UAV system according to claim 7, wherein The recalculated advantage function estimation described in Step 2-1 , which is specifically as follows: ； Among them, is the valuation of the state by the value network of the current UAV node . represents the estimated value of the new advantage function, which is used to measure the time step internal state for selecting an action relative to the average case. represents the discount factor, represents the total number of time steps.

9. The tomographic routing decision method for a multi-UAV system according to claim 8, wherein, The update of the parameters of the policy network of the UAV node described in Step 2-2 is as follows: ； Among them, represents the loss function of the policy network with respect to the policy network parameters gradient, represents the expected value used to calculate the average gradient under all state and action combinations under the gradient of the policy network parameters where represents the gradient represents the probability of selecting action in state ​ 10. The tomographic routing decision method for a multi-UAV system according to claim 9, characterized in that, The setting of the fault detection period and making a routing decision according to the fault detection result described in Step 3 include the following steps: Step 3-1, set the fault detection period to ; Step 3-2, perform fault detection on each adjacent node within each cycle, and let the set of neighbor nodes detected with faults be ; Step 3-3: Construct a corrected reward function based on the set of neighbor nodes , the corrected reward function is used to adjust the reward value according to the set of faulty neighbor nodes and obtain a new reward through the corrected reward function It is used to replace the optimized policy network with an offline fault-tolerant interruption mechanism for decision-making, as follows: ； Among them, is the reward obtained when the node is under normal operating conditions of the policy network, that is, when no faults or anomalies are detected. Step 3-4: Each node monitors the link status adjacent to it in real time and makes a decision switch, which is expressed as follows: ； Among them, indicates that in state under the policy select the action , represents the node and the link loss between them, indicates that the next-hop node selection of the traditional method, that is, using the optimized policy network for decision-making.

Images