Airborne network path planning method based on multi-target constraint and deep reinforcement learning

Abstract

The application discloses an airborne network path planning method based on multi-target constraints and deep reinforcement learning, which comprises the following steps: creating a task set Q containing a plurality of tasks q in an airborne network model; constructing a path optimization fusion model which integrates a multi-target collaborative optimization model and a deep reinforcement learning algorithm network; constructing a task path set based on the start and end nodes of the tasks in the task set Q by the deep reinforcement learning algorithm network, and screening a task path subset which meets the path risk constraints and the task isolation constraints; obtaining an optimal task path combination from the task path subset by a path link balancing module; and taking the optimal task path combination as the path planning result of the task set Q. The application plans a path which meets the isolation demand and realizes the risk balance of each link of the network by combining the service attributes of the tasks under the premise of considering the reliable transmission of data flow and the network load, thereby improving the overall network transmission safety of the aircraft avionics system.

Description

Technical Field

[0001] This invention relates to the field of airborne network path planning, and in particular to an airborne network path planning method based on multi-objective constraints and deep reinforcement learning. Background Technology

[0002] With the increasing integration of avionics systems, the types and quantities of data streams carried by airborne networks have surged, posing a severe challenge to airborne network path planning. Taking avionics full-duplex switched Ethernet airborne networks as an example, how to achieve multi-objective collaborative optimization of bandwidth utilization, risk distribution, and isolation requirements is a technical problem that urgently needs to be solved. Airborne networks consist of nodes and links. Nodes are typically switches, which serve as communication nodes for various sensors, embedded electronic devices, and actuators, transmitting data according to the task path within the airborne network. The technology needs to address the following challenges: First, how to construct an airborne network model that includes node safety criticality, link risk accumulation, and task isolation constraints, transforming the path planning problem into a multi-objective combinatorial optimization problem; second, the concurrent transmission of multiple priorities of information in the airborne network, such as non-critical maintenance data, abnormal event alarms, normal communication messages, and application service messages (see reference: Hotescu OA, Jaffrès-Runser K, Scharbarg JL, et al. Towards Quality of Service Provision with Avionics Full Duplex Switching[C] / / 29th Euromicro Conference on Real-Time Systems (ECRTS 2017). Dubrov-nik, Croatia, 2017: (1-3) As the complexity of aircraft functions increases, the interaction and coupling between devices are significantly enhanced, necessitating the configuration of numerous virtual links. Simultaneously, to utilize remaining network bandwidth, aircraft manufacturers envision carrying other types of traffic in next-generation airborne networks, such as audio and video information related to aircraft monitoring and maintenance (see: Hotescu O, Jaffrès-Runser K, Scharbarg J L. Time-triggered scheduling of mixed-critical flows at end-system in asynchronous AFDX avionic network[J]. ComputerNetworks, 2024, 246: 110427), to reduce the need for additional dedicated transmission lines. If only the physical performance of the network is considered while ignoring the service attributes of the data, it may result in a single link in the network carrying multiple high-security-critical task transmissions, causing risk concentration and potentially triggering multiple critical data flow common-mode failures in a single-point-of-failure scenario.Second, a reinforcement learning framework based on Deep Q-Network (DQN) is designed. This framework represents the network topology and dynamic load characteristics through a multi-dimensional state space, and integrates risk quantification indicators and bandwidth constraints to construct a composite reward function. The stability of the algorithm is improved by combining an experience replay mechanism and a dual Q-network structure. Deep reinforcement learning combines deep learning and reinforcement learning techniques, transforming the optimization objective in COP into a reward function in strength learning. It utilizes the approximation function of a deep neural network (see reference: Ye. Algorithm Design for Multi-Objective Combinatorial Optimization Problem Based on Deep Reinforcement Learning [D]. Sun Yat-sen University, 2024) to learn the optimal decision-making strategy through interaction with the environment. Third, the effectiveness of the algorithm is verified in small-scale pre-defined tasks and large-scale random tasks. Experimental results show that compared with traditional shortest path, load balancing, and risk balancing algorithms, the proposed method achieves a Pareto front ratio of 85.7% in typical industrial AFDX networks, improves the hypervolume index by 18.6%, reduces the back-iteration distance by 32.8%, and strictly meets the 100% task isolation requirement. This method provides a decision-making framework for path planning in high-security airborne networks that balances resource efficiency and risk constraints.

[0003] As mechanical control surfaces in civil aircraft are gradually replaced by automated systems, the number of embedded electronic devices, sensors, and actuators is rapidly increasing, providing new functions for aircraft and improving operational performance and maintainability (see reference: Amari A, Mifdaoui A. Specification and performance indicators of AeroRing—a multiple-ring ethernet network for avionics embedded systems[J]. Sensors,2018, 18(11): 3871). Against this backdrop, the transmission mode of airborne bus networks has evolved from isolated point-to-point transmission to a shared interconnected network architecture. The Ethernet-based AFDX protocol, with its Virtual Link (VL) technology, ensures the determinism of information transmission through bandwidth allocation and traffic isolation, and has become the most widely used airborne bus standard (see reference: Vera-Soto P, Villegas J, Fortes S, et al. An event-driven link-levelsimulator for the validation of AFDX and ethernet avionics networks[J]. Aerospace, 2024, 11(4): 247). Currently, terminal equipment manufacturers are integrating end system (ES) modules into devices such as sensors, actuators, and controllers. The types and quantities of data streams carried by airborne networks are constantly increasing. This trend will lead to the use of tens of thousands of nodes and edges in ARINC 664 AFDX networks (see reference: Bayram OB, Ozcan A. Determining optimal paths of virtual links in Avionics Full-DuplexSwitched Ethernet networks using modified ant colony optimization algorithm[J]. EXPERT SYSTEMS WITH APPLICATIONS, 2023, 229: 120433).On the other hand, in integrated modular avionics (IMA), multiple tasks serving the same function may reside in different hardware modules (see reference: Zhou X, He F, Zhao L, et al. Hybrid scheduling of tasks and messages for TSN-based avionics systems[J]. IEEE Transactions on Industrial Informatics, 2024, 20(2): 1081-1092). The interaction between systems is becoming increasingly dense, and it is necessary to ensure that the effectiveness of the isolation mechanism between safety-critical systems extends to the network level, which poses new challenges to the optimization of virtual links in airborne networks. Airborne data buses provide standardized inter-module communication services in avionics systems, ensuring configured data exchange between software modules (see: Pérez Tijero H, Gutiérrez García JJ. On the integration of DDS and AFDX standards[M] / / Proceedings of the 2024 IEEE 30th International Conference on Embedded and Real-Time Computing Systems and Applications (RTCSA), Sokcho, South Korea, 73-84. Institute of Electrical and Electronics Engineers, Inc., 2024). Avionics full-duplex switched Ethernet enhances these capabilities by providing a deterministic Ethernet-based network that uses static exchange tables to pre-configure multicast VL paths, enabling high-speed data transmission between devices. Specified input / output paths ensure redundant information flow and eliminate bidirectional collisions that can occur in other network technologies such as standard Ethernet (see: [reference missing]). Ahlbrecht A, Friedrich S, et al. State-of-the-art technologies for integrated modular avionics and the way ahead [C] / / 2023 IEEE / AIAA 42nd Digital Avionics Systems Conference (DASC). 2023: 1-10). Although the resource partitioning management mechanism in the IMA system and the virtual link technology of the AFDX network both have functional isolation characteristics, they can ensure that different avionics functions within the same hardware platform maintain independent operation. However, during the integration process, multiple system applications are executed on the same computer, and their data communication needs to be multiplexed through a shared airborne network transmission link (see reference: Yang H, Sun Y. A combination method for integrated modular avionics safety analysis[J]. Aircraft Engineering and Aerospace Technology, 2022, 95(2): 345-357). An unreasonable VL configuration may affect the original partition isolation effect. The avionics functions residing on the end system ES4 can be realized by receiving data processed by ES1 and ES2 respectively to meet the redundancy mechanism of high safety critical systems. However, at the network topology level, the two equipment components share a physical link and forward it to SW4 via switch SW3. This VL planning violates the isolation requirements. Under extreme conditions, a single point of failure in the shared physical link may cause the function to be completely lost, thereby destroying the functional redundancy design and endangering flight safety.

[0004] A virtual link is defined as a unidirectional statically configured logical path in an airborne network, enabling data transmission from one terminal system to one or more terminal systems. Airborne networks are highly specialized communication networks that manage data flows from multiple systems in real time. Therefore, path planning requires consideration of multiple factors such as end-to-end delay, bandwidth utilization, scalability, and harness weight. Essentially, it is a combinatorial optimization problem (COP). Researchers have proposed numerous feasible solutions for different optimization directions, applying graph theory. For example, the literature: Liu Cheng, Li Hang, He Feng, et al. AFDX Network Routing Configuration Algorithm Based on Trajectory Method [J]. Journal of Beijing University of Aeronautics and Astronautics, 2012, 38(12): 1587-1590+1595, publishes a static path configuration method based on the trajectory method. During the planning process, the end-to-end delay of the virtual link (VL) is limited to not exceeding the delay constraint. The cost is that choosing a path that will not exceed the timeout results in the consumption of more network resources. For example, the literature: Liu Cheng, He Feng, Wang Tong, et al. A routing configuration algorithm for virtual links in AFDX network [J]. Electro-Optics and Control, 2010, 17(12): 20-23+52, embeds bandwidth utilization and hop count weighted into the Floyd-Warshall algorithm, prioritizes the configuration of high-volume VLs, significantly reduces the average network latency, and balances the link load. However, the static weight coefficients need to be manually adjusted and are not suitable for dynamic task sets. To address this issue, the literature: Dai Zhen, He Feng, Zhang Yujing, et al. Real-time optimization algorithm for AFDX virtual link paths [J]. Acta Aeronautica Sinica, 2015, 36(6): 1924-1932, applies a genetic algorithm to use the delay results of VL messages as feedback information, dynamically optimizes the original VL path configuration, and searches for VL path configuration information that maximizes the real-time transmission performance of the network.At the level of network scalability and determinism, the literature Zheng X, Huang N, Zhang Y, et al. Performability optimization design of virtual links in AFDX networks[C] / / 2016 Annual Reliability and Maintainability Symposium (RAMS). 2016: 1-6, optimizes the global path from the perspective of interface control documents, and handles large-scale tasks in a layered manner, which improves the scalability of the network while balancing the load. The literature Al Sheikh A, Brun O, Chéramy M, et al. Optimal design of virtual links in AFDX networks[J]. Real-Time Systems, 2013, 49(3): 308-336, proposes an integer-linear programming formula for the VL planning problem, which allows maximizing the remaining capacity of network links, thereby providing greater flexibility for new VLs or modifying existing VLs; through frame insertion mechanism and sub-virtual link aggregation strategy, it balances bandwidth utilization and latency increase, and the terminal system detects frame loss at fixed intervals, thereby improving the determinism of message transmission.

[0005] Existing research focuses on pre-configured network performance parameters. Two key parameters for airborne network path planning include the bandwidth allocation gap (BAG) and the maximum frame size (Lmax). Assuming the same BAG constraint (reference: Optimal Scheduling and Delay Analysis for AFDX End-Systems[J]), network calculus is applied to determine the upper bound of delay through the service curve (rate-delay model) and the maximum horizontal distance of the arrival curve. The priority scheduling strategy of VL is determined to minimize the aggregate delay and meet the AFDX requirement for deterministic delay. Based on the same assumption, the optimal parameters of VL in the network are demonstrated by combining the node-link formula and the two-pole VL routing algorithm (reference: Optimal scheduling policy for AFDX End-Systems with virtual links of identical bandwidth allocation gap size | IEEE Conference Publication | IEEEXplore[EB / OL]. [2023-11-28]), so as to minimize bandwidth consumption. Some literature proposes a branch-and-bound algorithm to precisely solve BAG and Lmax (Reference: An D, Kim KH, Kim K I. Optimal Configuration of Virtual Links for Avionics Network Systems[J]. International Journal of Aerospace Engineering, 2015, 2015: e329820), which transforms the optimization of the two parameters into a combinatorial mathematical problem that satisfies real-time, bandwidth and jitter constraints, and combines heuristic algorithms to quickly generate feasible solutions, and minimizes the total bandwidth by aggregating multiple messages. Some literature has constructed a switching network with minimal complexity required for real-time transmission of a set of periodic messages and performed VL path planning (Reference: Building AFDX Networks of Minimal Complexity for Real-Time Systems | SpringerLink[EB / OL]. [2023-11-28].https: / / link.springer.com / chapter / 10.1007 / 978-3-030-76773-0_29). By minimizing cable length, electromagnetic interference between cables is reduced, while the real-time transmission requirements of periodic messages are met.In response to the increasing communication demands in networks, some literature has proposed a new algorithm to improve ant colony optimization (e.g., Bayram OB, Ozcan A. Determining optimal paths of virtual links in Avionics Full-DuplexSwitched Ethernet networks using modified ant colony optimization algorithm[J]. EXPERT SYSTEMS WITH APPLICATIONS, 2023, 229: 120433). This method routes each virtual link (VL) separately and then uses an ant colony to determine the entire path. Due to the parallelism of the ant colony algorithm, the impact of the VL lines on the entire network is permanent, thus eliminating the merging and integration problems that occur when planning routes individually. This method takes into account bandwidth utilization, latency, and cable weight when optimizing paths, making up for the shortcomings of traditional algorithms that are inefficient or unable to handle redundancy in complex topologies.

[0006] However, existing algorithms mainly focus on the AFDX network itself and primarily consider the physical performance of network information, neglecting the importance of avionics mission data streams and the criticality of each VL during path planning. If multiple safety-critical VLs are transmitted on the same physical link, failure of that link may affect the overall safety of the aircraft (see: Zhao C, Yan F, Tian Y, et al. Safety issues caused by the integration of the IMAplatform and AFDX[C] / / 2017 IEEE / AIAA 36th Digital Avionics Systems Conference(DASC). 2017: 1-5). To address the issue of concentrated risk, some literature has proposed risk models based on avionics functions, zones, end systems, and VL (see: Zhao Changxiao, He Feng, Yan Fang, et al. AFDX Virtual Link Path Optimization Algorithm Oriented to Risk Balancing [J]. Acta Aeronautica Sinica, 2018, 39(1): 261-272). These models aim to balance the risks of each physical link and meet system-level isolation requirements. However, the quantification method at the aircraft functional level has poor accuracy, and in complex network communication scenarios, the quantification results cannot fully and accurately reflect the network risk relationships.

[0007] After confirming top-level safety requirements during the aircraft development phase, a top-down safety requirement allocation process is initiated based on systems engineering, decomposing safety requirements from aircraft-level functions to equipment-level according to function. In the currently effective FAA advisory circular AC 25.1309-1B, qualitative and quantitative methods have been revised for different functional failure impacts. SAE ARP 4754A, "Guideline for Civil Aircraft and Systems Development" (hereinafter referred to as 4754A), describes the allocation method for functional / project development assurance levels from a systems development perspective. However, this technical specification explicitly states in its technical notes that while safety objectives and failure condition classifications can meet the needs of specific functional development assurance levels and numerical analysis methods, there is no direct connection between the two, and they cannot complement each other. Summary of the Invention

[0008] The purpose of this invention is to provide an airborne network path planning method based on multi-objective constraints and deep reinforcement learning. The deep reinforcement learning algorithm network constructs a task path set based on the start and end nodes of the task set Q and selects a subset of task paths that satisfy path risk constraints and task isolation constraints. The optimal task path combination is obtained from the task path subset through the path link balancing module, and the optimal task path combination is used as the path planning result of the task set Q.

[0009] The objective of this invention is achieved through the following technical solution:

[0010] An airborne network path planning method based on multi-objective constraints and deep reinforcement learning, the method comprising:

[0011] S1. Build an airborne network model containing N network nodes and M links. In the airborne network model, create a task set Q containing several tasks q. Task q includes the task start and end nodes, task risk threshold, task isolation preset value, and task data size. The task start and end nodes include network nodes of source node and destination node.

[0012] S2. Construct a path optimization fusion model that integrates a multi-objective collaborative optimization model and a deep reinforcement learning algorithm network. The multi-objective collaborative optimization model includes a task path constraint module and a path link balancing module. The task path constraint module includes path risk constraints and task isolation constraints, and the path link balancing module includes link risk balancing constraints and link load balancing constraints. The deep reinforcement learning algorithm network constructs a task path set based on the start and end nodes of the tasks in the task set Q and selects a subset of task paths that satisfy the path risk constraints and task isolation constraints. Then, the path link balancing module solves for the optimal task path combination from the task path subset, and the optimal task path combination is used as the path planning result of the task set Q.

[0013] To better implement this invention, the path risk constraint in the task path constraint module is that the cumulative risk of the task path is less than the task risk threshold in the task set Q. The cumulative risk expression for the task path is as follows: , ,in The risk accumulation for task path p in task q. The risk to network node n in task path p of task q. This represents the total number of network nodes contained in task path p within task q. The risk of link m in task path p in task q. This represents the total number of links contained in task path p within task q.

[0014] Preferably, the task isolation constraint of the task path constraint module is that all overlapping network nodes and / or overlapping links of all task paths in the task path subset satisfy the task isolation preset value in the task set Q.

[0015] Preferably, the link risk balancing constraint of the path link balancing module adopts the minimum variance of the risk accumulation of task paths in the task path subset as the risk balancing objective function.

[0016] Preferably, the link load balancing constraint of the path link balancing module adopts the minimum variance of bandwidth utilization of task paths in the task path subset as the load balancing objective function.

[0017] Preferably, the path link balancing module uses the risk balancing objective function and the load balancing objective function to solve for the optimal task path combination from the task path subset.

[0018] Preferably, the task path combination is all task paths that adapt to the task set Q.

[0019] Preferably, the airborne network model is configured with the bandwidth capacity corresponding to each link, and the task path constraint module further includes a bandwidth constraint, wherein the bandwidth constraint of the task path constraint module is that the data traffic of the task path combination on the link does not exceed the bandwidth capacity.

[0020] Preferably, each network node and each link in the airborne network model is assigned a risk attribute through an algorithm.

[0021] Preferably, the airborne network model is constructed using avionics full-duplex switched Ethernet (AFDX) as the network architecture, and the network nodes of the airborne network model are AFDX switches.

[0022] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0023] This invention constructs a path optimization fusion model that integrates a multi-objective collaborative optimization model and a deep reinforcement learning algorithm network. The deep reinforcement learning algorithm network constructs a task path set based on the start and end nodes of the task set Q and selects a subset of task paths that meet the path risk constraints and task isolation constraints. The optimal task path combination is obtained from the task path subset through the path link balancing module. The optimal task path combination is used as the path planning result of the task set Q. Under the premise of considering reliable data flow transmission and network load, the business attributes of the tasks are combined to plan a path that achieves risk balancing of each link in the network and meets the isolation requirements, thereby improving the overall network transmission security of the aircraft avionics system. Attached Figure Description

[0024] Figure 1 This is a flowchart of the airborne network path planning method of the present invention;

[0025] Figure 2 This is a schematic diagram illustrating the principle of the airborne network path planning method in the embodiment.

[0026] Figure 3 This is a network diagram of a typical industrial AFDX network in Example 2;

[0027] Figure 4 This is a network diagram of the AFDX network on the A380 in Example 2;

[0028] Figure 5 This is a graph showing the trend of network bandwidth utilization variance changing with rounds in Example 2;

[0029] Figure 6 This is a graph showing the trend of network risk variance changing with training rounds in Example 2;

[0030] Figure 7 This is a cumulative distribution map of the security criticality of the task path in Example 2;

[0031] Figure 8 This is a network topology diagram containing 26 network nodes in Example 3;

[0032] Figure 9 This is a trend chart of network variance parameters in the example task of Example 3. Detailed Implementation

[0033] The present invention will be further described in detail below with reference to embodiments:

[0034] Example 1

[0035] like Figure 1 As shown, an airborne network path planning method based on multi-objective constraints and deep reinforcement learning is proposed, the method comprising:

[0036] S1. Construct an airborne network model containing N network nodes and M links. Each network node and link in the airborne network model can be assigned a risk attribute through an algorithm. Preferably, the airborne network model is constructed using avionics full-duplex switched Ethernet (AFDX) as the network architecture, and the network nodes of the airborne network model are AFDX switches. In some embodiments, the airborne network model consists of N switch nodes and M links, which can be represented as follows: ,in , Let E represent the set of switch nodes, and E be the set of links connecting the switches. The connectivity of switch nodes in a network can be represented by a weighted adjacency matrix A: element v i The weight of node i (security criticality, indicating the importance of the devices connected to the node and its importance in the network structure), e ij This represents the weight (or security importance) connecting node i and node j.

[0037] In the airborne network model, a task set Q is created, containing several tasks q. Each task q corresponds to a task start and end node, a task risk threshold, a task isolation preset value, and a task data size. The task start and end nodes include network nodes including the source node and the destination node, and the task start and end nodes represent the process from the source node to the destination node. Each task q has a preset task risk threshold, a preset task isolation preset value, and a preset task data size.

[0038] S2. Construct a path optimization fusion model that integrates a multi-objective collaborative optimization model and a deep reinforcement learning algorithm network. The multi-objective collaborative optimization model includes a task path constraint module and a path link balancing module. The task path constraint module includes path risk constraints and task isolation constraints.

[0039] In some embodiments, such as Figure 2 As shown, the path risk constraint in the task path constraint module is that the cumulative risk of the task path is less than the task risk threshold in the task set Q. The cumulative risk expression for the task path is as follows: , ,in The risk accumulation for task path p in task q. The risk to network node n in task path p of task q. This represents the total number of network nodes contained in task path p within task q. The risk of link m in task path p in task q. This represents the total number of links contained in task path p within task q.

[0040] In some embodiments, such as Figure 2 As shown, the task isolation constraint of the task path constraint module is that all overlapping network nodes and / or overlapping links of all task paths in the task path subset satisfy the task isolation preset value in task set Q. The task isolation preset value of task q in task set Q is... The sum of the risks of task q and the network nodes that overlap with other task paths in the task path subset is The sum of the risks of overlapping links between task q and other task paths in the task path subset is , .

[0041] In some embodiments, the airborne network model is configured with bandwidth capacity corresponding to each link, and the task path constraint module further includes bandwidth constraints. The bandwidth constraint of the task path constraint module is that the data traffic of the task path combination on the link does not exceed the bandwidth capacity. In the case of the airborne network model with avionics full-duplex switched Ethernet (AFDX) as the network architecture, there are two key traffic parameters for data flow transmission in the AFDX airborne network model: the maximum frame size L. MAX The bandwidth allocation interval (BAG) is used. AFDX payload sizes range from 64 to 1518 bytes and are predefined, fixed segments that can be transmitted even if the information source does not fill in the relevant fields of the payload structure; the bandwidth allocation interval is generally defined as 2. k ms, where k is between 0 and 7, determines the Ethernet frame length L. MAX (i.e., the effective payload size of the AFDX Ethernet frame) and T BAG (That is, the bandwidth partitioning interval), then bandwidth S is allocated. When planning a route, it is essential to ensure that the data traffic passing through the physical link does not exceed its maximum bandwidth capacity at any given time.

[0042] like Figure 2 As shown, the path link balancing module includes link risk balancing constraints and link load balancing constraints. The link risk balancing constraint uses the minimum variance of risk accumulation of task paths within a subset of task paths as the risk balancing objective function. Minimizing network risk variance avoids the clustering of high-security-critical tasks. The link load balancing constraint uses the minimum variance of bandwidth utilization of task paths within a subset of task paths as the load balancing objective function. Minimizing link transmission volume variance ensures network load balance and avoids local link congestion. The path link balancing module (including link risk balancing constraints and link load balancing constraints) generates dynamic coupling through the path planning scheme P∈Ω (Ω is the solution space), exhibiting typical Pareto optimal boundary characteristics.

[0043] In some embodiments, in a multi-objective collaborative optimization model, the Pareto optimal solution in a multi-objective optimization problem is a solution that cannot improve any one objective without harming the others. Specifically, if It is not dominated by other solutions, that is, it satisfies Then X This is called a Pareto optimal solution. The set of all Pareto optimal solutions is called the Pareto Set (PS). Solutions in the PS are independent of each other and represent trade-offs among multiple objectives, satisfying the following definition: The Pareto front is the set of objective values ​​corresponding to PS, which can be defined as follows. Since there are often conflicts between multiple objectives, Pareto optimization is needed to find the non-dominated solution set. Hypervolume (HV) and Inverted Generational Distance (IGD) are two widely used hybrid metrics. Hypervolume represents the scope of the non-dominated solution set relative to a reference point and is often used to evaluate the convergence and distribution of approximate solution sets. The hypervolume calculation formula is as follows: In the formula, It is the Lebesgue measure. quilt The solution set is dominated by all solutions in the solution set, and a larger HV indicates better diversity and convergence. This invention can employ a dynamic reference point selection strategy, specifically, using the maximum value in each objective dimension of each solution set as the coordinate component of the reference point r. The back-iteration distance (IGD) is a performance metric commonly used to approximate the distribution and convergence of the Power Process (PF). It measures the average shortest distance between the solutions in the solution set and the true PF. The IGD definition is: ;in, Represents the approximate Pareto solution set For the real PF The reverse iteration distance value. This represents the number of solutions in the true Power Process (PF). z is a solution in the true PF. Representing the solution z and the solution set The shortest distance between the solution set and the true power factor (PF) is used. The shortest distance is typically Euclidean distance, with a smaller IGD indicating a closer distance between the solution set and the true power factor. To balance the optimization objective, this embodiment uses a deep reinforcement learning algorithm to encode the optimization objective as a composite reward function, and utilizes a deep neural network to achieve an accurate approximation of the value function. It learns the optimal policy through interaction with the environment, ultimately generating the optimal path planning scheme while satisfying real-time constraints.

[0044] The deep reinforcement learning algorithm network constructs a task path set based on the start and end nodes of tasks in the task set Q, and filters out a subset of task paths that satisfy path risk constraints and task isolation constraints. This subset contains several combinations of task paths that fit the task set Q and satisfy both constraints. Then, a path link balancing module solves for the optimal task path combination from this subset. This optimal combination represents all task paths that fit the task set Q. Preferably, the path link balancing module uses a risk balancing objective function and a load balancing objective function to solve for the optimal task path combination from the subset. This invention uses the optimal task path combination as the path planning result for the task set Q. The deep reinforcement learning algorithm network improves the q-value function that guides the agent's behavior selection strategy through iteration. In the formula, s t This represents the agent's state at time t. S is a finite set of states of the agent; a t Let t be the action taken by the agent. A q Represents a finite set of actions for the agent; α is the learning rate. ;r t+1 γ represents the reward for the agent to choose action a in state s; γ represents the discount factor for future rewards.

[0045] Deep reinforcement learning algorithms incorporate the Q-Learning algorithm, which uses a Q-table to store environmental information gained from the agent's interactions with the environment. However, the agent's learning performance is less than ideal when dealing with complex environments or requiring continuous action spaces. In the path planning environment of airborne networks, the agent not only needs to select the optimal action based on the current state but also needs to consider the overall network planning objectives, such as overall bandwidth utilization and network risk variance. DQN is an upgraded version of Q-Learning, combining high-dimensional feature information with convolutional neural networks in deep learning to transform it into low-dimensional feature information. The ultimate goal is to use deep convolutional neural networks to select the action function that maximizes the accumulation of future rewards. That is, under state s and action a, through action policy It is possible to achieve the maximum total reward r.

[0046] Deep reinforcement learning algorithms combine low-level features through multi-layered network structures and nonlinear transformations to form abstract, easily distinguishable high-level representations. They then use a target network and empirical replay to address pre-existing instabilities and overfitting. The target network is a copy of the Q-value network, but with a lower update frequency. Its role is to store the most recent optimal Q-value during training to guide the update of network parameters. The DQN algorithm transforms the calculation of Q-values ​​into the calculation of network parameters, and abstracts the updating of the Q-value table into the training of a deep network model. In the formula, θ represents the network parameters, and the update process can also be expressed as: In the formula, The parameters of the Q-value network during the i-th training iteration are... The Q-value parameters of the target network are given at the i-th training iteration, and the Q-value network parameters are updated every n time steps. And it remains fixed between each update.

[0047] The experience replay mechanism randomizes the data to eliminate correlations in the observation sequence, thereby balancing variations in the data distribution. To apply experience replay, the experience generated by the agent at each time step t needs to be processed. Stored in dataset During the learning process, experiences (s, a, r, s') are uniformly and randomly drawn from the stored experience pool. In the i-th training iteration, the Q-value network is updated using the objective value loss function shown in the equation: .

[0048] State Space: In this embodiment, the attributes of switch nodes and physical links are selected to represent the state of the agent. The reinforcement learning model adopts a multi-dimensional state space design to comprehensively represent the network state. The state vector consists of node location encoding and global network indicators: node location information is obtained through... The (number of switch nodes in the network) dimensional vector represents the current node, enabling a discretized representation of the network topology. Global state monitoring includes two key indicators: link bandwidth utilization variance and security cumulative distribution variance, reflecting the network traffic balance and risk clustering distribution, respectively. This hybrid state coding mechanism achieves a compact representation of the network state through spatial vectors, preserving the local topology information required for node-level path decision-making while integrating dynamic characteristics reflecting the overall network optimization goals. This provides agents with sufficient state awareness to simultaneously optimize load balancing and risk balancing.

[0049] Action Space: In each training round, the agent's state represents the current switch node. Action selection requires searching the network adjacency matrix for all switch nodes connected to the current node, and then selecting the next hop based on different strategies. An ε-greedy strategy is used for action selection. ;

[0050] In the formula, Q() represents the Q-value network in DQN; s is the current state of the task; a is the current action selection; c is a random number in (0, 1); and ε gradually decreases during training according to the following expression: In the formula, f represents the minimum non-zero value that ε can be reduced to, ensuring that the algorithm has random performance to escape local optima during the early training process; t is the number of training rounds; and k is the rate of decrease of the adjustment discount factor. As ε gradually decreases, the action selection strategy changes from random selection in the exploration phase to selecting the index with the largest Q value based on the neural network output.

[0051] Reward and Penalty Function: In reinforcement learning environments, the agent's behavior is largely influenced by the reward function. This embodiment designs a reward and penalty function that integrates the core constraints of the airborne network communication task transmission model, constructing a reward function with network load balancing and risk balancing coefficients as optimization objectives. A multi-objective collaborative optimization mechanism is used to solve the dynamic path planning problem for batch tasks. This hybrid reward mechanism guides the agent to achieve incremental policy optimization within the path search space, ultimately converging to the globally optimal solution that satisfies multiple constraints. After each action selection by the agent, network attributes are determined based on task information and network status. The first step is the accumulation of risk on the current physical link, i.e., the accumulated task safety criticality E on the physical link. ca The safety criticality threshold E of this side cannot be exceeded. t :

[0052] For tasks requiring isolation, the isolation requirements must not be violated in the path planning scheme. In the airborne network topology, multiple heterogeneous paths can be derived based on graph theory connectivity models. This embodiment adds a path safety criticality threshold for tasks supporting different avionics system functions, requiring that the cumulative path safety criticality of different tasks not exceed this threshold. A reward is given when the agent's action sequence successfully reaches the destination node; simultaneously, to accelerate path planning convergence, this embodiment introduces a step penalty, applying a negative incentive to each decision action to drive the agent to converge quickly to the destination node and simultaneously suppress the tendency to generate loops.

[0053] r step With mission safety level T c Relatedly, following the hierarchical planning logic described above, high-level tasks tend to favor the shortest path, while low-level tasks tend to balance load and risk; when the agent's state is the destination node, it marks the end of this round of training. The algorithm records all action selections and rewards in each round of training, and then adjusts the round rewards for all actions according to the path planning goal.

[0054] Example 2

[0055] This embodiment follows the method in Embodiment 1 to verify the effectiveness of the path optimization algorithm using a small-scale task, selecting node 6 from two networks as the destination node. Figure 3The diagram shows a small-scale task network based on a typical AFDX network with node 6 as the destination node, corresponding to the typical AFDX network in Table 2. Figure 4 The diagram illustrates a small-scale task network based on the A380 airborne network, with node 6 as the destination node (corresponding to the A380 airborne network in Table 2). A risk assessment model is used to calculate the safety criticality of each node and link in the network. Each node is pre-set to send one message to this node, and VL2 is completely isolated from VL4, and VL6 from VL7. The parameters and reward / penalty parameters of the deep reinforcement learning algorithm network are shown in Table 1.

[0056]

[0057] To ensure the reproducibility of the experiment, the sources of randomness were uniformly controlled: the random seed for the Python standard library, NumPy, and PyTorch framework (CPU / GPU) was fixed at 42, and PyTorch's deterministic computation mode was enabled.

[0058] Run the program and export the planning process data for Task 1 (i.e., a small-scale task with the airborne network execution node 6 as the destination node). The path planning results are shown in Table 2. It can be seen that the algorithm meets the preset requirements in terms of key isolation indicators.

[0059]

[0060] This embodiment directly represents load distribution using network data transmission volume. Bandwidth utilization is easily affected by differences in task set size, causing fluctuations, while data transmission volume more directly reflects the load balance of the network's physical links. Figures 5-6 It can be seen that the network bandwidth utilization variance of Task 1 (i.e., the small-scale task with airborne network execution node 6 as the destination node) changes with the rounds as follows: Figure 5 As shown, the network risk variance of Task 1 (i.e., a small-scale task with airborne network execution node 6 as the destination node) changes with the training rounds as follows: Figure 6 As shown, the agent begins to converge around 1500 rounds. Under the ε-greedy policy framework, a decay mechanism is defined to ensure that the exploration rate ε eventually converges to the theoretical lower bound of 0.01. When ε≥0.01, the policy space still retains approximately 1% random exploration probability, preventing the Q-table from prematurely converging to a local optimum in sparse reward scenarios.

[0061] To comprehensively evaluate the overall performance of the algorithms in complex environments, this study constructs a theoretically complete comparative experimental framework. Based on the same network topology and task, the classic Shortest Path (SP), Load Balance (LB), and Risk Balance (RB) algorithms are tested. Enhanced variants of these three algorithms—SP&Isolation, LB&Isolation, and RB&Isolation—are generated by introducing isolation constraints. Running the programs and analyzing the network topology parameters generates corresponding performance comparison histograms, and the cumulative distribution of path safety criticality is derived as follows: Figure 7 As shown in the diagram. According to the constructed reward and punishment model, the step penalty is positively correlated with the task level. Simulation results show that high-security-critical tasks prioritize the path with the fewest hops to optimize cumulative rewards, while low-security-critical tasks, through a dynamic reward adjustment mechanism, simultaneously achieve network load and risk balance in path planning, reaching a multi-objective Pareto optimal solution. Specifically, for the seven tasks, four evaluation indicators—network bandwidth utilization variance, risk variance, cumulative path security criticality, and isolation—were included in the solution set. The DQN algorithm was located on the Pareto front in 85.7% of the tasks, covering six tasks, which is higher than the suboptimal comparison algorithm LB (71.4%), indicating that its solution set has a better distribution breadth and convergence accuracy in the objective space. In the Pareto distributions of tasks 3, 4, and 6, the DQN algorithm simultaneously optimized all three indicators, while the comparison algorithm, considering only a single objective, exhibited a seesaw effect between the two objectives. Through IGB indicator analysis, the average distance between the DQN algorithm and the true front (0.552) was reduced by 17.7% compared to the suboptimal algorithm LB. In small-scale pre-defined tasks, the DQN algorithm is located at the Pareto front in 85.7% of the tasks, covering 6 / 7 of the tasks, and meets the isolation requirements between tasks. Compared with the suboptimal algorithm, the average distance between the DQN algorithm and the true front is reduced by 17.7%, enabling risk-constrained airborne network path optimization.

[0062] Example 3

[0063] This embodiment constructs a 26-node distributed integrated modular avionics system switch network model based on the Gephi platform, following the method in Embodiment 1. Figure 8 As shown, the overall communication architecture of a civil aviation avionics system is simulated. The experiment randomly generates 1000 task flows (including 50 pairs of tasks requiring strict isolation) based on the AFDX protocol data frame format, each containing risk constraints, bandwidth constraints, and isolation requirements. The algorithm's performance in planning large-scale tasks is verified by comparing it with other algorithms.

[0064] For large search spaces, the DQN algorithm exhibits a significant increase in computation time. This embodiment proposes a hybrid decision-making mechanism based on shortest path prior knowledge, and the pseudocode is shown in Algorithm 2. The core improvement of the algorithm is to combine the traditional shortest path algorithm with deep reinforcement learning, compressing the action search space through prior knowledge of topological structure. Running the program yields the following results: Figure 9 The network parameter changes shown in Table 3 were analyzed using evaluation metrics for this task set. The DQN algorithm had 933 tasks at the Pareto front, higher than all the comparison algorithms. Its average HV value was 18.67% higher than the suboptimal LB algorithm, and its average IGD was 32.76% lower, demonstrating better global Pareto optimality approximation.

[0065]

[0066] Statistical analysis was conducted on the highest HV (Highest Visibility) and lowest IGD (Lowest Indicator Discrimination) counts obtained by different algorithms. DQN demonstrated a significant advantage in both evaluation metrics. Network link throughput variance, risk variance, and path safety criticality were accumulated as coupling parameters. Data from 10 tasks were sampled and analyzed according to the planning order. Compared to baseline algorithms, the distribution of DQN's solution set in the target space is closer to the theoretical optimal boundary. It should be noted that the Pareto advantage exhibited by DQN mainly stems from the targeted design of the multi-objective collaborative mechanism in this study, rather than the inherent advantage of the algorithm itself. Specifically, in terms of state representation, it allows the agent's action selection to simultaneously consider multiple optimization objectives. In contrast, suboptimal algorithms only consider a single objective, resulting in poor evaluation metrics. Overall, the design approach proposed in this embodiment provides a multi-objective reinforcement learning architectural paradigm for path planning in airborne networks. Its core value lies in constructing a multi-objective collaborative decision-making framework rather than simply highlighting algorithmic advantages. In large-scale random tasks, the hybrid heuristic search algorithm has an average hypervolume value that is 18.67% higher than that of the suboptimal algorithm, an average IGD that is 32.76% lower than that of the suboptimal algorithm, and 93.3% of the tasks are at the Pareto front, proving that the algorithm has better global approximation.

[0067] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An airborne network path planning method based on multi-objective constraints and deep reinforcement learning, characterized in that: The methods include: S1. Build an airborne network model containing N network nodes and M links. Create a task set Q containing several tasks q in the airborne network model. Task q includes task start and end nodes, task risk threshold, task isolation preset value, and task data size. The task start and end nodes are network nodes including source node and destination node. S2. Construct a path optimization fusion model that integrates a multi-objective collaborative optimization model and a deep reinforcement learning algorithm network. The multi-objective collaborative optimization model includes a task path constraint module and a path link balancing module. The task path constraint module includes path risk constraints and task isolation constraints. The path risk constraint in the task path constraint module is that the cumulative risk of the task path is less than the task risk threshold in the task set Q. The cumulative risk expression for the task path is as follows: , ,in The risk accumulation for task path p in task q. The risk to network node n in task path p of task q. This represents the total number of network nodes contained in task path p within task q. The risk of link m in task path p in task q. Let be the total number of links contained in task path p within task q. The path link balancing module includes link risk balancing constraints and link load balancing constraints. The link risk balancing constraint of the path link balancing module uses the minimum variance of the cumulative risk of task paths in the task path subset as the risk balancing objective function. The link load balancing constraint of the path link balancing module uses the minimum variance of the bandwidth utilization of task paths in the task path subset as the load balancing objective function. The deep reinforcement learning algorithm network constructs a task path set based on the start and end nodes of tasks in the task set Q and selects a subset of task paths that satisfy the path risk constraints and task isolation constraints. Then, the path link balancing module solves for the optimal combination of task paths from the task path subset. The deep reinforcement learning algorithm network improves the q-value function that guides the agent's behavior selection strategy through iteration. In the formula, s t This represents the agent's state at time t. S is a finite set of states of the agent, a t Let t be the action taken by the agent. A q Represents a finite set of actions for an intelligent agent; For learning rate, ;r t+1 γ represents the reward for the agent to choose action a in state s; γ represents the discount factor for future rewards; and the optimal combination of task paths corresponds to the path planning result of task set Q.

2. The airborne network path planning method based on multi-objective constraints and deep reinforcement learning according to claim 1, characterized in that: The task isolation constraint of the task path constraint module is that all overlapping network nodes and / or overlapping links of all task paths in the task path subset satisfy the task isolation preset value in the task set Q.

3. The airborne network path planning method based on multi-objective constraints and deep reinforcement learning according to claim 2, characterized in that: The path link balancing module uses the risk balancing objective function and the load balancing objective function to solve for the optimal combination of task paths from the task path subset.

4. The airborne network path planning method based on multi-objective constraints and deep reinforcement learning according to claim 1, characterized in that: The task path combination is all task paths that fit the task set Q.

5. The airborne network path planning method based on multi-objective constraints and deep reinforcement learning according to claim 1, characterized in that: The airborne network model is configured with the bandwidth capacity corresponding to each link. The task path constraint module also includes bandwidth constraints, wherein the bandwidth constraint of the task path constraint module is that the data traffic of the task path combination on the link does not exceed the bandwidth capacity.

6. The airborne network path planning method based on multi-objective constraints and deep reinforcement learning according to claim 1, characterized in that: Each network node and each link in the airborne network model is assigned a risk attribute through an algorithm.

7. The airborne network path planning method based on multi-objective constraints and deep reinforcement learning according to claim 1, characterized in that: The airborne network model is constructed using avionics full-duplex switched Ethernet (AFDX) as its network architecture, and the network nodes of the airborne network model are AFDX switches.

Images