Dynamic routing collaborative optimization method for heterogeneous network based on reinforcement learning

Abstract

The invention relates to the technical field of heterogeneous network optimization, and discloses a heterogeneous network dynamic routing collaborative optimization method based on reinforcement learning, which comprises the following steps: S1, deploying an agent at each node of a heterogeneous network, and initializing a state sampling period and an action space; s2, sensing the state of the heterogeneous network in real time and obtaining sensing data; s3, encoding the sensing data into a state vector; s4, designing a layered dynamic reward function, including short-term reward and long-term reward; dynamically setting a weight ratio according to the scene type and outputting the weight ratio as a comprehensive reward; s5, carrying out multi-agent collaborative decision-making, which comprises the following steps: constructing a local topological graph based on neighbor state information broadcasted by nodes; each agent selects an action based on the state vector through the DQN; and updating the network weight of the DQN by using the comprehensive reward. According to the invention, transmission real-time performance and sustainability can be considered, transmission delay and transmission cost are reduced, and the method is suitable for film and television production and industrial Internet of Things scenes with variable topology and complex interference.

Description

Technical Field

[0001] The present invention relates to the technical field of heterogeneous network optimization, and in particular to a heterogeneous network dynamic routing collaborative optimization method based on reinforcement learning. Background Art

[0002] Currently, the demand for heterogeneous networks in dynamic scenarios such as film and television productions and large-scale events is exploding. These scenarios typically involve a variety of heterogeneous devices, including drones, wireless cameras, and mobile control consoles, requiring self-organizing networks for real-time transmission of high-definition video streams and command control. However, densely deployed lighting systems, wireless microphones, and other devices on-site generate strong intermittent interference in the 2.4 / 5.8 GHz frequency bands. Furthermore, the frequent movement of devices leads to constant changes in network topology. Traditional communication solutions based on static routing tables are no longer able to meet the low-latency, highly reliable transmission requirements.

[0003] Existing commercial routing protocols (such as OLSR and AODV) rely on periodic broadcast probe messages to maintain routes. Their fixed probe intervals cannot adapt to millisecond-level interference fluctuations, which can easily cause video freezes or even interruptions, especially in multi-hop transmission scenarios. A report from the film and television industry indicates that professional-grade 4K live streaming requires end-to-end latency of less than 200ms, while existing solutions often experience latency exceeding 500ms in complex live environments. This gap severely restricts real-time production efficiency.

[0004] In recent years, researchers have attempted to improve anti-interference capabilities through cognitive radio technology. For example, the dynamic spectrum access (DSA) mechanism can identify idle frequency bands to avoid interference. However, these approaches primarily focus on physical layer optimization and fail to form collaborative decision-making with the routing layer. In the field of routing optimization, improved protocols based on the expected transmission time (ETX) can assess link quality, but their metric calculation relies on historical statistical information and cannot respond to sudden load fluctuations in real time. Some researchers have introduced machine learning algorithms to predict network status, such as using LSTM models to estimate link latency. However, supervised learning requires the pre-collection of large amounts of labeled data, making it difficult to implement in temporary and active networks where devices enter and exit at will.

[0005] Furthermore, the core difficulty of dynamic routing in heterogeneous networks lies in the contradiction of multi-objective collaborative optimization, which is specifically manifested in three aspects: First, state awareness is fragmented. Parameters like network load, link quality, and node energy consumption belong to different protocol layers. Existing solutions lack a unified quantitative model, leading to one-sided routing decisions. For example, a node with high battery life may be on a congested link, while a node connected by a low-latency link may be energy-starved.

[0006] Second, there are conflicting optimization objectives. Live streaming requires maximizing instantaneous bandwidth, while long-term monitoring networks prioritize energy balance. Traditional fixed-weight reward mechanisms are unable to adapt to these changing scenarios.

[0007] Third, distributed decision-making coordination is difficult. Local node optimization can easily lead to global policy oscillation. For example, multiple transmitters simultaneously switching to the same idle frequency band can trigger new congestion. Typical multi-agent systems, such as Q-learning-based distributed routing, lack effective global information coupling mechanisms, often lead to multiple nodes simultaneously seizing the same idle link, causing new congestion. Rerouting convergence times often exceed 10 seconds. Although multi-agent reinforcement learning (MARL) has been proposed for distributed decision-making, it faces engineering bottlenecks such as slow model convergence and high rerouting latency on resource-constrained embedded devices (such as wireless cameras). Summary of the Invention

[0008] The present invention aims to provide a dynamic routing collaborative optimization method for heterogeneous networks based on reinforcement learning, which can take into account both real-time and sustainability of transmission, reduce transmission delay and transmission cost, and is suitable for film and television production and industrial Internet of Things scenarios with changeable topology and complex interference.

[0009] The basic solution provided by the present invention is: a method for collaborative optimization of dynamic routing in heterogeneous networks based on reinforcement learning, comprising the following steps: S1, deploy the agent at each node of the heterogeneous network and initialize the state sampling period and action space; S2, real-time perception of heterogeneous network status and acquisition of perception data, including: monitoring the percentage of the data queue length of this node to the cache capacity and the percentage of actual transmission rate to nominal bandwidth ;Measure the delay value of adjacent links and packet loss rate ; Read the remaining power percentage of this node and neighboring nodes ; S3, encodes the sensory data into a state vector; S4, design a hierarchical dynamic reward function, including short-term rewards and long-term rewards; dynamically set the weight ratio of short-term rewards to long-term rewards based on the scenario type, and output it as a comprehensive reward; S5, multi-agent collaborative decision-making, including: building a local topology map based on the neighbor status information broadcast by the node; each agent selects an action based on the state vector through the DQN network; and using the comprehensive reward to update the DQN network weight.

[0010] The working principle and advantages of the present invention are: This invention uses a reinforcement learning-based collaborative routing optimization method for dynamic heterogeneous networks. Through a collaborative routing mechanism driven by reinforcement learning, it effectively solves the problem of insufficient adaptability of traditional protocols under multi-dimensional optimization goal conflicts and frequent topology changes. It can balance real-time transmission with sustainability, reduce transmission delays and transmission costs, and is suitable for film and television production and industrial Internet of Things scenarios with variable topologies and complex interference. The key points are: First, this solution can achieve a balance between millisecond-level interference avoidance and scenario adaptation through the integration of multi-dimensional state perception and a hierarchical reward mechanism. Specifically, this solution encodes multiple layers of parameters such as network load, link quality, and node energy into a unified state vector, solving the state information island problem in existing technologies and overcoming the one-sidedness of decision-making caused by information fragmentation in traditional solutions. By setting a hierarchical dynamic reward function and a dynamic weight adjustment strategy, the short-term reward weight and long-term reward weight are dynamically adjusted according to different scenario types, breaking through the rigid constraints of the fixed optimization targets of traditional protocols, and enabling high-real-time scenarios such as live film and television broadcasts and high-energy-efficiency scenarios such as long-term monitoring to share the same decision-making model, avoiding the overhead of customizing dedicated protocols for different applications and helping to reduce costs.

[0011] Second, this solution proposes a multi-agent collaborative decision-making mechanism that can adapt to the dynamics of heterogeneous networks. Specifically, in this solution, agents continuously update the DQN network weights (S5) through state perception (S2) and reward feedback (S4). This allows them to autonomously learn the optimal routing strategy under different network conditions and adapt to dynamic changes without human intervention, significantly improving the solution's robustness in the complex environments of heterogeneous networks. Furthermore, multi-agent collaborative decision-making allows different types of nodes (such as high-computing edge nodes and low-computing sensors) to participate in decision-making based on their own capabilities (for example, complex nodes undertake more local topology calculations, while simple nodes focus on basic state broadcasts), thus compatibility with node heterogeneity. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 The figure is a flow chart of a method embodiment of a method for collaborative optimization of dynamic routing in heterogeneous networks based on reinforcement learning according to the present invention. DETAILED DESCRIPTION

[0013] The following is a further detailed description through specific implementation methods: The embodiment is basically as shown in the attached Figure 1 As shown: A dynamic routing collaborative optimization method for heterogeneous networks based on reinforcement learning includes the following steps: S1, deploy the agent at each node of the heterogeneous network and initialize the state sampling period and action space.

[0014] A heterogeneous network is a hybrid network composed of different types of network nodes, communication protocols, hardware devices, or technical standards. Node types may include sensors, routers, servers, and mobile terminals. Communication methods may include Wi-Fi, Bluetooth, cellular networks (such as 5G), and satellite communications. Hardware performance can vary significantly.

[0015] The agent is an embedded software module deployed at each network node, including wireless cameras, drone image transmission equipment, and relay routers. Each agent performs the following actions: collecting local and neighboring node status, calculating the optimal action through the DQN network, broadcasting HELLO messages, and responding to ALERT alarms.

[0016] Deploying intelligent agents at each node in a heterogeneous network can form a distributed intelligent architecture, enabling each node in the network to have perception, decision-making and collaboration capabilities, which helps to achieve adaptive, efficient and robust network operation.

[0017] The action space is defined as the set of next-hop neighbors that the node can choose (i.e., the set of directly connectable adjacent nodes within the communication range of the current node) and the communication frequency band combination (i.e., the frequency range that the current node can choose when communicating with its next-hop neighbors). Energy constraints are enforced during action selection: nodes with low energy levels are prohibited from being used as next hops unless there are no other available paths.

[0018] By defining the "action space," all optional operations for a node can be clarified (which neighbor to choose, which frequency band to use). Then, by filtering the operations using the "energy constraint rules," low-energy nodes can be prioritized for protection and only allowed to be used when there are no other options. Ultimately, a balance between the efficiency and stability of network paths can be achieved (e.g., extending network life and reducing premature node failure).

[0019] In this embodiment, the initialization state sampling period is 0.5 seconds, and the action space defined is {optional next-hop node ID, communication band number}, which is a pairing of neighbor nodes and bands. For example, the action space of node A might include all combinations such as (B, band 1), (B, band 2), and (C, band 1).

[0020] S2, real-time perception of heterogeneous network status and acquisition of perception data, including: monitoring the percentage of the data queue length of this node to the cache capacity and the percentage of actual transmission rate to nominal bandwidth ;Measure the delay value of adjacent links and packet loss rate ; Read the remaining power percentage of this node and neighboring nodes .

[0021] S3, encodes the perception data into a state vector.

[0022] The state vector ; in, Based on and Metered load level; Based on and The generated link quality vector; Based on Divided energy levels.

[0023] when and hour, ;when and hour, ;when and hour, ; The link quality vector is generated as follows: For each adjacent link, calculate the quality score : ; and mapping the quality scores into discrete levels; When , the corresponding level value is 0; , the corresponding level value is 1; When , the corresponding level value is 2; The energy levels include low, medium and high, and the corresponding levels are 0, 1 and 2.

[0024] In this embodiment, the energy level is divided based on the remaining power of each node corresponding to the intelligent body. When , the energy level is high; when Remaining power When the remaining power is , the energy level is low.

[0025] S4, design a hierarchical dynamic reward function, including short-term rewards and long-term rewards; and dynamically set the weight ratio of short-term rewards to long-term rewards according to the scenario type, and output it as a comprehensive reward.

[0026] The short-term rewards and long-term rewards are calculated according to the following formula: ; ; in, For short-term rewards, 、 is the benchmark delay and benchmark bandwidth; , ; For long-term rewards, is the number of node switching times in one cycle, is the standard deviation of the load of the entire network; , ; The dynamic setting of the weight ratio of short-term rewards to long-term rewards based on the scenario type includes: If the scene type is a live video scene, ; If the scene type is a data collection scene, ; For comprehensive rewards.

[0027] The scene type is automatically identified by: When the network is detected video streams and single-channel bandwidth requirements When the video is displayed, it is determined to be a live broadcast scene; When more than 80% of the node transmission interval seconds, it is determined to be a data collection scenario.

[0028] Under these constraints, when multiple HD video streams are detected, this solution automatically increases the short-term reward weight, driving the system to prioritize low-latency links. This dynamic target switching mechanism, based on traffic feature recognition, breaks through the rigid constraints of fixed optimization targets in traditional protocols and can achieve over 25% latency reduction in time-sensitive scenarios such as live streaming, far exceeding conventional parameter optimization solutions.

[0029] S5, multi-agent collaborative decision-making, including: building a local topology map based on the neighbor status information broadcast by the node; each agent selects an action based on the state vector through the DQN network; and using the comprehensive reward to update the DQN network weight.

[0030] The neighbor status information refers to a HELLO message containing [available bandwidth, remaining hops, energy level]; In this step, when constructing the local topology map, the local topology map is updated based on the HELLO message. , where vertex V is the set of neighbor nodes and edge E is the available link; The state vector is then input into the DQN network, which contains three fully connected layers (128-64-32 neurons). The DQN network outputs the Q value of each action. The action with the largest Q value is selected. Execute routing.

[0031] The DQN network is trained using the following method: The experience tuple is stored in the experience replay pool; the experience tuple is ; After receiving every 50 new tuples, randomly sample 32 tuples to calculate the loss function: ; ; The network weights are updated by gradient descent with a learning rate of 0.001.

[0032] Furthermore, in this step, a dynamic rerouting trigger mechanism is also provided; when the link packet loss rate in three consecutive sampling periods is and When the dynamic rerouting trigger mechanism is triggered, the following actions are performed after the trigger: The local node sends an ALERT message containing [failed link ID, current timestamp]; After receiving the ALERT message, the neighboring node removes the failed link and updates the ; Based on the updated Recalculate the state vector and output the new action.

[0033] Based on the above operations, this method can effectively achieve collaborative optimization. First, this solution adopts a local topology broadcast mechanism, which can reduce the communication overhead to less than 30% of the traditional OLSR protocol; second, it designs a counterfactual baseline credit allocation based on global rewards, so that each node can still evaluate the contribution of its own actions to the performance of the entire network in a distributed execution environment, avoiding selfish decisions; third, it introduces the collaborative rerouting trigger condition for link failure, and through the ALERT message, it can directly notify the associated nodes to achieve rapid path reconstruction. This closed-loop design of "local perception-global reward-directional collaboration" can achieve optimization efficiency close to centralized control while ensuring the reliability of the distributed architecture, effectively improving the collaboration speed.

[0034] S6, perform global policy synchronization: upload the local DQN network weights of each node to the central coordinator on a periodic basis (in this embodiment, a period of five minutes); the central coordinator generates a lightweight baseline model through policy distillation; and distributes the baseline model to all nodes in the network to replace the local model to achieve policy synchronization.

[0035] Specifically, when generating a lightweight benchmark model, the KL divergence loss function is used to compress the number of neurons in the output layer to 50% of the original model, thereby achieving lightweightness.

[0036] In this embodiment, after receiving the local DQN network weights of each node, the central coordinator first calculates the mean of the local DQN network weights of each node. and standard deviation , and remove the deviations from the mean More than 3 The abnormal weight (which may be caused by node failure) is taken; the remaining weights are weighted averaged, and the weight coefficient is the proportion of node computing resources.

[0037] Integrate the weights uploaded by all nodes, combine them with the trust coefficient for weighted aggregation, and build an integrated teacher model (the parameter scale is consistent with the node local model): ; is the weight matrix of the integrated teacher model; is the trust coefficient of node i, which is dynamically adjusted based on the node's historical upload quality (positively correlated with punctuality and weight stability), and is positively correlated. The weight uploaded for the node.

[0038] Based on the teacher model, a student model is constructed as a lightweight baseline model: a progressive compression strategy is adopted, the number of hidden layers remains unchanged (that is, the number of layers is kept the same as the teacher model), but the number of neurons in each layer is reduced by 50%; and channel pruning technology is used to retain important feature channels (such as sorting based on the weight L1 norm and removing the channel with the smallest L1 norm), and the output layer neurons are compressed to 50% of the original model. A clustering algorithm is used to merge similar action categories (such as merging "Select Neighbor A + Band 1" and "Select Neighbor A + Band 2" into "Select Neighbor A").

[0039] During the training process of the student model, the decision distribution of the teacher model is learned through the KL divergence loss function, and the parameters of the student model are adjusted through the backpropagation algorithm (such as the Adam optimizer) until the KL divergence loss value is reduced to a preset threshold (such as 0.01).

[0040] Based on the above operations, this solution achieves incremental global optimization: by periodically distilling the distributed experience into a lightweight baseline model (with half the number of neurons), new nodes can be fine-tuned and integrated into the routing system within 5 minutes of receiving the baseline model, achieving an 80% convergence speed improvement compared to traditional federated learning solutions. More importantly, this mechanism innovatively freezes the underlying feature extraction layer of the DQN network (accounting for 80% of the parameters) and fine-tunes only the top, fully connected layer. This preserves the ability to recognize common network features while avoiding overfitting on new devices. This hierarchical parameter update strategy significantly reduces computational overhead, with peak memory usage on embedded NPU devices exceeding 50MB, fully meeting the resource constraints of mobile terminals.

[0041] This embodiment provides a reinforcement learning-based collaborative optimization method for dynamic routing in heterogeneous networks, which can balance real-time transmission and sustainability, reduce transmission delay and transmission costs, and is suitable for film and television production and industrial Internet of Things scenarios with variable topologies and complex interference.

[0042] In order to verify the application effect of this solution, a network optimization test was carried out in the same application scenario using both the traditional solution and this solution.

[0043] Traditional methods include optimization solutions based on the traditional AODV protocol. Traditional AODV (Ad-hoc On-Demand Distance Vector) is a routing protocol suitable for mobile ad hoc networks (MANETs). Its core features are "on-demand route generation" and "dynamic path maintenance". It is designed for distributed networks without fixed infrastructure and where nodes can move freely.

[0044] The specific application scenario is a live broadcast of a large-scale variety show. The equipment consists of 12 4K wireless cameras (transmitters), three drones (mobile relays), five ground receiving base stations, and one central broadcast vehicle (global coordinator). Interference sources include 32 wireless microphones (2.4GHz band) and eight stage lighting control systems (5.8GHz band). Service requirements include real-time backhaul of 12 4K video streams (each with 25Mbps bandwidth and end-to-end latency of ≤200ms).

[0045] The obtained test data are shown in Table 1: Table 1 Test comparison results

[0046] As shown in the table above, compared to traditional solutions, this solution significantly reduces network latency, exponentially reduces 4K streaming interruptions, and exponentially reduces maximum path reconstruction time. Furthermore, the battery life of network node devices is significantly improved. This solution also demonstrates strong real-time status awareness, anti-interference capabilities, fault recovery capabilities, and energy optimization capabilities.

[0047] The above results fully verify the effectiveness of the design steps in this solution. The fundamental advantage of this solution over traditional routing protocols lies in the closed-loop adaptive optimization of environment perception, decision-making, and execution through a reinforcement learning framework. This is specifically reflected in three aspects: First, at the perception level, traditional solutions (such as AODV) only rely on binary link on / off status and fixed hop count indicators, while this solution uses a multi-dimensional state perception model (such as load classification formula and link quality score) to quantify the continuous changes in network load, link quality and node energy consumption in real time, constructing a high-precision environmental portrait, enabling the system to detect packet loss rates. or delay Predict link degradation before reaching the critical threshold.

[0048] Second, at the decision-making level, traditional protocols select paths based on static rules (such as the minimum number of hops) and often fall into local optimality (such as choosing a high-load, short path). This solution, on the other hand, transforms scenario requirements into mathematical optimization objectives through a hierarchical reward function. The DQN model dynamically generates routing strategies. For example, in live broadcast scenarios, it automatically avoids low-power nodes (energy level ≤ 1) and prioritizes high-bandwidth links, achieving the core effect of reducing transmission latency.

[0049] Third, at the execution level, traditional flood routing reconstruction requires network-wide broadcast consumption. In terms of communication resources, this solution uses multi-agent collaborative mechanisms (such as targeted ALERT broadcasts and local topology updates) to reduce rerouting communication overhead. At the same time, it periodically eliminates distributed policy divergences through incremental global optimization (strategy distillation), ensuring that new nodes can be quickly integrated into the network. Ultimately, in dynamic scenarios such as filming, it achieves collaborative optimization of path reconstruction and energy consumption reduction within 3 seconds, which is difficult to achieve with traditional protocols.

[0050] The above is only an embodiment of the present invention. Common knowledge such as the specific structure and characteristics of the scheme is not described in detail here. Ordinary technicians in the relevant field are aware of all common technical knowledge in the technical field of the invention before the application date or priority date, can obtain all existing technologies in the field, and have the ability to apply conventional experimental means before that date. Ordinary technicians in the relevant field can improve and implement this scheme in combination with their own abilities under the guidance of this application. Some typical well-known structures or well-known methods should not become obstacles for ordinary technicians in the relevant field to implement this application. It should be pointed out that for those skilled in the art, without departing from the structure of the present invention, several variations and improvements can be made, which should also be regarded as the scope of protection of the present invention. These will not affect the effect of the implementation of the present invention and the practicality of the patent.

Claims

1. A collaborative optimization method for dynamic routing in heterogeneous networks based on reinforcement learning, characterized by: The following steps are involved: S1, deploy the agent at each node of the heterogeneous network and initialize the state sampling period and action space; S2, real-time perception of heterogeneous network status and acquisition of perception data, including: monitoring the percentage of the data queue length of this node to the cache capacity and the percentage of actual transmission rate to nominal bandwidth ;Measure the delay value of adjacent links and packet loss rate ; Read the remaining power percentage of this node and neighboring nodes ; S3, encodes the sensory data into a state vector; S4, design a hierarchical dynamic reward function, including short-term rewards and long-term rewards; dynamically set the weight ratio of short-term rewards to long-term rewards based on the scenario type, and output it as a comprehensive reward; S5, multi-agent collaborative decision-making, including: building a local topology map based on the neighbor status information broadcast by the node; each agent selects an action based on the state vector through the DQN network; and using the comprehensive reward to update the DQN network weight.

2. The method for collaborative optimization of dynamic routing in heterogeneous networks based on reinforcement learning according to claim 1, characterized in that: It also includes S6, which performs global policy synchronization: the local DQN network weights of each node are uploaded to the central coordinator on a periodic basis; the central coordinator generates a lightweight baseline model through policy distillation; and the baseline model is distributed to all nodes in the network to replace the local model to achieve policy synchronization.

3. The method for collaborative optimization of dynamic routing in heterogeneous networks based on reinforcement learning according to claim 1, characterized in that: The agent is an embedded software module deployed at each network node, including wireless cameras, drone image transmission equipment, and relay routers. Each agent performs the following actions: collecting local and neighboring node status, calculating the optimal action through the DQN network, broadcasting HELLO messages, and responding to ALERT alarms.

4. The method for collaborative optimization of dynamic routing in heterogeneous networks based on reinforcement learning according to claim 1, characterized in that: The state vector ; in, Based on and Metered load level; Based on and The generated link quality vector; Based on Divided energy levels.

5. The method for collaborative optimization of dynamic routing in heterogeneous networks based on reinforcement learning according to claim 4, characterized in that: when and hour, ;when and hour, ;when and hour, ; The link quality vector is generated as follows: For each adjacent link, calculate the quality score : ； and mapping the quality scores into discrete levels; When , the corresponding level value is 0; , the corresponding level value is 1; When , the corresponding level value is 2; The energy levels include low, medium and high, and the corresponding levels are 0, 1 and 2.

6. The method for collaborative optimization of dynamic routing in heterogeneous networks based on reinforcement learning according to claim 5, characterized in that: The action space is defined as a set of next-hop neighbors and communication frequency band combinations that a node can select, and an energy constraint rule is implemented during action selection: nodes with low energy levels are prohibited from being used as next hops unless there are no other available paths.

7. The method for collaborative optimization of dynamic routing in heterogeneous networks based on reinforcement learning according to claim 1, characterized in that: The short-term rewards and long-term rewards are calculated according to the following formula: ; ; in, For short-term rewards, 、 is the benchmark delay and benchmark bandwidth; , ; For long-term rewards, is the number of node switching times in one cycle, is the standard deviation of the load of the entire network; , ; The dynamic setting of the weight ratio of short-term rewards to long-term rewards based on the scenario type includes: If the scene type is a live video scene, ; If the scene type is a data collection scene, ; For comprehensive rewards.

8. The method for collaborative optimization of dynamic routing in heterogeneous networks based on reinforcement learning according to claim 1, characterized in that: The scene type is automatically identified by: If the network detects ≥3 video streams and the bandwidth requirement for a single stream is >5Mbps, it is considered a live broadcast scenario. When the transmission interval of more than 80% of the nodes is greater than 10 seconds, it is determined to be a data collection scenario; otherwise, it is determined to be a general scenario. In the general scenario, the weight ratio of short-term rewards to long-term rewards is set to 0.5, that is, .

9. The method for collaborative optimization of dynamic routing in heterogeneous networks based on reinforcement learning according to claim 1, characterized in that: The neighbor status information refers to a HELLO message including available bandwidth, remaining hops, and energy level; In S5, when building a local topology map, the local topology map is updated based on the HELLO message. , where vertex V is the set of neighbor nodes and edge E is the available link; Then the state vector is input into the DQN network, which contains 3 fully connected layers; the DQN network outputs the Q value of each action; the action with the largest Q value is selected Execute routing.

10. The method for collaborative optimization of dynamic routing in heterogeneous networks based on reinforcement learning according to claim 9, characterized in that: In S5, there is also a dynamic rerouting trigger mechanism; when the link packet loss rate is and When the dynamic rerouting trigger mechanism is triggered, the following actions are performed after the trigger: The local node sends an ALERT message containing the failed link ID and the current timestamp; After receiving the ALERT message, the neighboring node removes the failed link and updates the ; Based on the updated Recalculate the state vector and output the new action.

Images