Method for dynamic routing and cooperative optimization of heterogeneous networks based on reinforcement learning

Abstract

The application relates to the technical field of heterogeneous network optimization, and discloses a heterogeneous network dynamic routing cooperative optimization method based on reinforcement learning, which comprises the following steps: S1, deploying an agent at each node of the heterogeneous network, initializing a state sampling period and an action space; S2, perceiving the heterogeneous network state in real time and obtaining sensing data; S3, encoding the sensing data into a state vector; S4, designing a hierarchical dynamic reward function, including a short-term reward and a long-term reward; dynamically setting a weight ratio according to a scene type and outputting the weight ratio as a comprehensive reward; S5, performing multi-agent cooperative decision-making, including: constructing a local topology graph based on neighbor state information broadcast by nodes; selecting actions by the agents through a DQN network based on the state vector; and updating the DQN network weight by using the comprehensive reward. The application can consider both transmission real-time performance and sustainability, reduce transmission delay and transmission cost, and is suitable for film and television production and industrial Internet of Things scenes with variable topologies and complex interference.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of heterogeneous network optimization, and in particular to a heterogeneous network dynamic routing cooperative optimization method based on reinforcement learning. BACKGROUND

[0002] Currently, the communication demand of heterogeneous networks in dynamic scenes such as film and television production and large-scale activities is showing explosive growth. Such scenes usually contain various heterogeneous terminals such as unmanned aerial vehicle aerial photography equipment, wireless cameras, and mobile control consoles, which need to realize real-time high-definition video stream return and instruction control through self-organizing networks. However, the intensive deployment of on-site lighting systems, wireless microphones and other equipment will produce strong intermittent interference in the 2.4 / 5.8GHz frequency band, and the frequent movement of equipment will cause the network topology to change continuously, so the traditional communication scheme based on static routing table has been difficult to meet the low-delay and high-reliability transmission requirements.

[0003] Existing commercial routing protocols (such as OLSR, AODV, etc.) rely on periodic broadcast probe packets to maintain routing, and their fixed probe intervals cannot adapt to millisecond-level interference changes, especially in multi-hop transmission scenarios, which can easily cause video stuttering or even interruption. Film and television industry reports show that professional 4k live streaming requires end-to-end delay to be less than 200ms, while the existing scheme often has a delay of more than 500ms in complex on-site environments, which seriously hinders real-time production efficiency.

[0004] In recent years, researchers have tried to improve anti-interference capability through cognitive radio technology, such as dynamic spectrum access (DSA) mechanism that can identify idle frequency bands to avoid interference. However, such methods focus on physical layer optimization and cannot form a cooperative decision with the routing layer. In the field of routing optimization, improved protocols based on expected transmission times (ETX) can assess link quality, but their index calculation relies on historical statistical information and cannot respond to sudden load fluctuations in real time. Some scholars have introduced machine learning algorithms to predict network status, such as using LSTM models to estimate link delay, but supervised learning requires pre-collection of a large amount of labeled data, which is difficult to implement in temporary activity networks where devices enter and leave at any time.

[0005] Further, the core difficulty of dynamic routing in heterogeneous networks lies in the contradiction of multi-objective cooperative optimization, which is manifested in three aspects:

[0006] First, the state perception dimension is fragmented. Network load, link quality, node energy consumption and other parameters belong to different protocol layers, and existing solutions lack a unified quantitative model, resulting in one-sided routing decisions. For example, a high-energy node may be in a congested link, while a node connected by a low-delay link may be energy-starved.

[0007] Second, the optimization target conflicts. Film and television live broadcast requires instantaneous bandwidth maximization, while the long-running monitoring network needs to prioritize energy consumption balance. The traditional fixed weight reward mechanism cannot adapt to scenario switching.

[0008] Third, distributed decision-making coordination is difficult. Local optimization of nodes can easily lead to global strategy shock, such as multiple transmitters simultaneously switching to the same free frequency band, which in turn causes new congestion. Typical multi-agent systems such as decentralized routing based on Q-learning often lead to multiple nodes simultaneously occupying the same free link, causing new congestion, and the re-routing convergence time is generally more than 10 seconds. Although multi-agent reinforcement learning (MARL) is proposed for distributed decision-making, on resource-constrained embedded devices such as wireless cameras, there are engineering bottlenecks such as slow model convergence and long re-routing delay. SUMMARY

[0009] The present application aims to provide a heterogeneous network dynamic routing collaborative optimization method based on reinforcement learning, which can balance transmission real-time and sustainability, reduce transmission delay and transmission cost, and is suitable for film and television production and industrial Internet of Things scenes with variable topology and complex interference.

[0010] The basic scheme provided by the present application is: a heterogeneous network dynamic routing collaborative optimization method based on reinforcement learning, comprising the following steps:

[0011] S1, deploying an agent at each node of the heterogeneous network, initializing the state sampling period and the action space;

[0012] S2, real-time sensing of the heterogeneous network state and obtaining sensing data, including: monitoring the percentage of the data queue length of the node to the cache capacity And the percentage of the actual transmission rate to the nominal bandwidth ; measuring the adjacent link delay value And the packet loss rate ; reading the remaining battery percentage of the node and the neighbor node ;

[0013] S3, encoding the sensing data into a state vector;

[0014] S4, designing a hierarchical dynamic reward function, including short-term reward and long-term reward; and dynamically setting the weight ratio of short-term reward and long-term reward according to the scene type, and outputting as a comprehensive reward;

[0015] S5, multi-agent collaborative decision-making, including: constructing a local topology graph based on the neighbor state information broadcast by the node; each agent selects an action based on the state vector through the DQN network; and updating the DQN network weight using the comprehensive reward.

[0016] The working principle and advantages of this invention are as follows:

[0017] This invention presents a dynamic routing cooperative optimization method for heterogeneous networks based on reinforcement learning. Through a reinforcement learning-driven cooperative routing mechanism, it effectively addresses the insufficient adaptability of traditional protocols under conditions of multi-dimensional optimization objective conflicts and frequent topology changes. It balances real-time transmission performance with sustainability, reduces transmission latency and costs, and is suitable for scenarios involving diverse topologies and complex interference, such as film and television production and industrial IoT. The key points are:

[0018] First, this solution achieves a millisecond-level balance between interference avoidance and scenario-adaptive objectives by integrating multi-dimensional state perception with a hierarchical reward mechanism. Specifically, this solution encodes multi-layered parameters such as network load, link quality, and node energy into a unified state vector, solving the problem of state information silos in existing technologies and overcoming the decision-making bias caused by information fragmentation in traditional solutions. Furthermore, by setting a hierarchical dynamic reward function and a dynamic weight adjustment strategy, the short-term and long-term reward weights are dynamically adjusted according to different scenario types. This breaks through the rigid constraints of fixed optimization objectives in traditional protocols, enabling high-real-time scenarios such as live streaming to share the same decision model with high-energy-efficiency scenarios such as long-term monitoring. This avoids the overhead of customizing dedicated protocols for different applications and helps reduce costs.

[0019] Second, this scheme proposes a multi-agent collaborative decision-making mechanism that can adapt to the dynamics of heterogeneous networks. Specifically, in this scheme, agents continuously update the DQN network weights (S5) through state awareness (S2) and reward feedback (S4), and can autonomously learn the optimal routing strategy under different network states. This allows them to adapt to dynamic changes without human intervention, significantly improving the robustness of the scheme in complex heterogeneous network environments. Furthermore, multi-agent collaborative decision-making allows different types of nodes (such as high-computing-power edge nodes and low-computing-power sensors) to participate in decision-making based on their own capabilities (e.g., complex nodes undertake more local topology calculations, while simple nodes focus on basic state broadcasting), thus ensuring compatibility with node heterogeneity. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the method flow of an embodiment of the dynamic routing cooperative optimization method for heterogeneous networks based on reinforcement learning of the present invention. Detailed Implementation

[0021] The following detailed explanation illustrates the specific implementation methods:

[0022] The basic implementation examples are as follows: Figure 1 As shown: A cooperative optimization method for dynamic routing in heterogeneous networks based on reinforcement learning includes the following steps:

[0023] S1, deploy agents on each node of the heterogeneous network, and initialize the state sampling period and action space.

[0024] The heterogeneous network refers to a hybrid network composed of different types of network nodes, communication protocols, hardware devices or technical standards. The node types can include sensors, routers, servers, mobile terminals, etc.; the communication methods can mix WiFi, Bluetooth, cellular networks (such as 5G), satellite communication, etc.; and the hardware performance can be significantly different.

[0025] The agent is an embedded software module deployed in each network node, including wireless cameras, unmanned aerial vehicle image transmission devices, and relay routers. Each agent performs the following actions: collects local and neighbor node states, calculates the optimal action through a DQN network, and broadcasts a HELLO packet and responds to an ALERT alarm.

[0026] Deploying an agent at each node of the heterogeneous network can constitute a distributed intelligent architecture, enabling each node in the network to have perception, decision-making, and collaboration capabilities, which helps to achieve adaptive, efficient, and robust network operation.

[0027] The action space is defined as the set of next-hop neighbors that the node can choose (i.e., the set of adjacent nodes that the current node can directly connect within the communication range) and the communication frequency band combination (i.e., the frequency range that the current node can choose when communicating with the next-hop neighbor), and the energy constraint rule is executed when selecting actions: it is prohibited to use a node with a low energy level as the next hop unless there is no other available path.

[0028] By defining the "action space", all available operations of the node (which neighbor to choose, which frequency band to use) can be clearly defined, and the "energy constraint rule" can be used to filter the operations, which can prioritize the protection of low-energy nodes and only allow their use when there is no other choice, ultimately achieving a balance between the efficiency and stability of the network path (such as extending the network lifetime and reducing premature node failure).

[0029] In this embodiment, the initialization state sampling period is 0.5s, and the defined action space is {optional next-hop node ID, communication frequency band number}, i.e., the paired combination of neighbor nodes and frequency bands. For example, the action space of node A can include (B, frequency band 1), (B, frequency band 2), (C, frequency band 1), etc.

[0030] S2, real-time perception of the heterogeneous network state and obtaining perception data, including: monitoring the percentage of the data queue length of the node to the cache capacity and the percentage of the actual transmission rate to the nominal bandwidth ; measuring the adjacent link delay value and the packet loss rate ; reading the remaining battery percentage of the node and the neighbor nodes .

[0031] S3, encode the perception data into a state vector.

[0032] The state vector ;

[0033] wherein, is based on the load level measured; is based on and the link quality vector generated; is the energy level divided according to the energy level.

[0034] When and , ; when and , ; when and , ;

[0035] The link quality vector is generated in the following manner:

[0036] For each adjacent link, a quality score is calculated:

[0037] ;

[0038] and the quality score is mapped to a discrete level; When , the corresponding level is valued as 1; When , the corresponding level is valued as 2;

[0039] The energy level includes low, medium and high levels, and the corresponding levels are valued as 0, 1 and 2.

[0040] In this embodiment, the energy level is divided based on the remaining energy of the agent corresponding to each node. When the remaining energy is high, the energy level is high; when the remaining energy is medium, the energy level is medium; and when the remaining energy is low, the energy level is low.

[0041] S4, design a hierarchical dynamic reward function, including short-term reward and long-term reward; and dynamically set the weight ratio of short-term reward and long-term reward according to the scene type, and output as a comprehensive reward.

[0042] The short-term reward and long-term reward are calculated according to the following formula:

[0043] ;

[0044] ;

[0045] wherein, is a short-term reward, , is a reference delay and a reference bandwidth; , ;

[0046] is a long-term reward, is the number of node switching in a period, is the standard deviation of the network load; , ;

[0047] The weight ratio of the short-term reward and the long-term reward is dynamically set according to the scene type, and the method comprises the following steps:

[0048] If the scene type is a video live streaming scene, ;

[0049] If the scene type is a data collection scene, ; is a comprehensive reward.

[0050] The scene type is automatically identified in the following manner:

[0051] When it is detected that there are video streams in the network and the bandwidth demand of a single channel is , the video live streaming scene is determined;

[0052] When more than 80% of the nodes have a transmission interval of seconds, the data collection scene is determined.

[0053] Under the above condition limitation, when multiple high-definition video streams are detected, the short-term reward weight is automatically increased by the scheme, so as to drive the system to preferentially select a low-delay link. The dynamic target switching mechanism based on traffic feature recognition breaks through the rigid constraint of the fixed optimization target of the traditional protocol, and can achieve a delay reduction of more than 25% in a time-sensitive scene such as video live streaming, which is much higher than that of a conventional parameter optimization scheme.

[0054] S5, multi-agent collaborative decision-making is performed, comprising the following steps: a local topology graph is constructed based on neighbor state information broadcast by nodes; each agent selects an action based on a state vector through a DQN network; and the DQN network weight is updated by using a comprehensive reward.

[0055] The neighbor state information refers to a HELLO packet containing [available bandwidth, remaining hop count, energy level];

[0056] In this step, the local topology graph is updated based on the HELLO packet when constructing the local topology graph , wherein the vertex V is a neighbor node set, and the edge E is an available link.

[0057] The state vector is input into a DQN network, which includes three fully connected layers (128-64-32 neurons). The DQN network outputs the Q value of each action. The action with the maximum Q value is selected Routing is performed.

[0058] The DQN network is trained by the following method:

[0059] An experience tuple is stored in an experience replay pool. The experience tuple is .

[0060] After receiving 50 new tuples, 32 tuples are randomly sampled to calculate the loss function:

[0061] . .

[0062] The network weights are updated by gradient descent, and the learning rate is set to 0.001.

[0063] Further, in this step, a dynamic rerouting triggering mechanism is also provided. When the link packet loss rate in the last three sampling periods is and , the dynamic rerouting triggering mechanism is triggered. After triggering, the following actions are performed:

[0064] An ALERT packet containing [invalid link ID, current timestamp] is sent by the node;

[0065] After receiving the ALERT packet, the neighbor node removes the invalid link and updates within 0.5 seconds.

[0066] Based on the updated , the state vector is recalculated, and a new action is output.

[0067] Based on the above operation, the method can effectively realize collaborative optimization. Firstly, the local topology broadcast mechanism is adopted, which can compress the communication overhead to less than 30% of the traditional OLSR protocol; secondly, the counterfactual baseline credit distribution based on global reward is designed, so that each node can still evaluate the contribution of its own action to the performance of the whole network in the distributed execution environment, and avoid selfish decision; thirdly, the collaborative rerouting triggering condition of link failure is introduced, which can realize fast path reconstruction through the directional notification of the associated node by the ALERT packet. The closed-loop design of "local perception-global reward-directional collaboration" can guarantee the reliability of the distributed architecture while achieving the optimization efficiency close to centralized control, and effectively improve the collaborative speed.

[0068] S6, performing global strategy synchronization: uploading the local DQN network weight of each node to the central coordinator in a period (in this embodiment, five minutes as a period); generating a lightweight benchmark model by the central coordinator through strategy distillation; distributing the benchmark model to all network nodes to replace the local model, so as to realize strategy synchronization.

[0069] Specifically, when generating the lightweight benchmark model, the KL divergence loss function is used to compress the number of output layer neurons to 50% of the original model, so as to realize lightweight.

[0070] In this embodiment, after receiving the local DQN network weight of each node, the central coordinator first calculates the mean and standard deviation of the local DQN network weight of each node, and eliminates the abnormal weight (may be caused by node failure) deviating from the mean more than 3 ; the remaining weights are weighted and averaged, and the weight coefficient is the node computing resource proportion.

[0071] Integrate all the uploaded weights, and combine the trust coefficient for weighted aggregation to construct an integrated teacher model (the parameter size is consistent with the local model of the node): is the weight matrix of the integrated teacher model; is the trust coefficient of node i, which is dynamically adjusted based on the historical upload quality of the node (positively correlated with the punctuality rate and weight stability), and has a positive correlation, is the weight uploaded by the node.

[0072] ​Based on the teacher model, a student model is constructed as a lightweight benchmark model: an incremental compression strategy is adopted, the number of hidden layers remains unchanged (i.e. the same number of layers as the teacher model), but the number of neurons in each layer is reduced by 50%; and a channel pruning technique is used to retain important feature channels (such as based on weight L1 norm sorting, removing the channel with the smallest L1 norm), compressing the output layer neurons to 50% of the original model, and using a clustering algorithm to merge similar action categories (such as merging "select neighbor A + frequency band 1" and "select neighbor A + frequency band 2" into "select neighbor A").

[0073] During the training process of the student model, the decision distribution of the teacher model is learned through the KL divergence loss function, and the parameters of the student model are adjusted through the back propagation algorithm (such as the Adam optimizer) until the KL divergence loss value is reduced to a preset threshold (such as 0.01).

[0074] Based on the above operation, the present scheme can realize incremental global optimization: through periodic strategy distillation, the distributed experience is compressed to a lightweight benchmark model (half the number of neurons), which can enable the new node to complete fine-tuning and integrate into the routing system within 5 minutes after receiving the benchmark model, and the convergence speed is improved by 80% compared with the traditional federated learning scheme. More importantly, this mechanism innovatively freezes the bottom feature extraction layer of the DQN network (accounting for 80% of the parameter amount), and only fine-tunes the top fully connected layer, which not only retains the ability to recognize general network features, but also avoids overfitting to new devices. This hierarchical parameter update strategy can significantly reduce computational overhead, with peak memory usage on embedded NPU devices not exceeding 50MB, fully meeting the resource constraints of mobile terminals.

[0075] The heterogeneous network dynamic routing cooperative optimization method based on reinforcement learning provided by the embodiment can balance transmission real-time performance and sustainability, reduce transmission delay and transmission cost, and is suitable for film and television production and industrial Internet of Things scenes with variable topology and complex interference.

[0076] To verify the application effect of the present scheme, network optimization tests are carried out in the same application scenario using the traditional scheme and the present scheme respectively.

[0077] The traditional method includes an optimization scheme based on the traditional AODV protocol. The traditional AODV (Ad-hoc On-Demand Distance Vector, Ad-hoc On-Demand Distance Vector Routing Protocol) is a routing protocol suitable for mobile ad hoc networks (MANET). Its core features are "on-demand route generation" and "dynamic path maintenance", and it is designed for distributed networks without fixed infrastructure and nodes that can move freely.

[0078] The application scenario is specifically a large-scale variety show live broadcast scene. The device composition includes: 12 4K wireless cameras (transmission end), 3 unmanned aerial vehicle aerial vehicles (mobile relay), 5 ground receiving base stations, and 1 central relay vehicle (global coordinator). The interference sources include: 32 wireless microphones (2.4 GHz frequency band), 8 stage light control systems (5.8 GHz frequency band). The business requirements include: real-time backhaul of 12 4K video streams (single bandwidth 25 Mbps, end-to-end delay requirement ≤200 ms).

[0079] The obtained test data is shown in Table 1:

[0080] Table 1 Comparison results of tests

[0081]

[0082] As can be seen from the above table, compared with the traditional scheme, the network delay time in the present scheme is significantly reduced, the number of 4K stream interruptions is reduced by several times, the maximum path reconstruction time is reduced by several times, and the endurance of the device in the network node is improved to a certain extent. The real-time state perception ability, anti-interference ability, fault recovery ability and energy optimization ability of the present scheme are all strong.

[0083] The above results fully verify the effectiveness of the design of each step in the present scheme. The fundamental advantage of the present scheme over the traditional routing protocol is that it realizes the closed-loop adaptive optimization of environment perception-decision-execution through the reinforcement learning framework, which is specifically reflected in three aspects:

[0084] Firstly, in the perception aspect, the traditional scheme (such as AODV) only relies on the binary link on-off state and the fixed hop number index, while the present scheme quantifies the continuous changes of network load, link quality and node energy consumption in real time through a multi-dimensional state perception model (such as the load grading formula and the link quality score), constructs a high-precision environment portrait, and makes the system predict link degradation before the critical threshold of packet loss rate or delay is reached.

[0085] Secondly, in the decision aspect, the traditional protocol selects paths based on static rules (such as the minimum hop number), which often falls into local optimization (such as selecting high-load short paths). The present scheme converts the scene requirements into mathematical optimization targets through a hierarchical reward function, and dynamically generates a routing strategy by a DQN model, for example, automatically avoids low-energy nodes (energy level ≤1) and prefers high-bandwidth links in the film and television live broadcast scene, to realize the core effect of reducing transmission delay.

[0086] Thirdly, in the execution aspect, the traditional flooding type route reconstruction needs to broadcast throughout the network, consuming Communication resources, the scheme helps multi-agent collaborative mechanism (such as directional ALERT broadcast and local topology update) to compress the communication overhead of re-routing, and periodically eliminates distributed strategy divergence through incremental global optimization (strategy distillation), ensures that the new node can quickly integrate into the network, and finally achieves the collaborative optimization of path reconstruction within 3 seconds and energy consumption reduction in dynamic scenes such as film shooting, which is difficult for traditional protocols to achieve.

[0087] The above is only an embodiment of the present application, and the common knowledge of the specific structure and characteristics of the scheme is not described in detail here. The ordinary skilled person in the art knows all the ordinary technical knowledge in the field of the present application before the application date or the priority date, can know all the prior art in this field, and has the ability to apply conventional experimental means before that date. The ordinary skilled person in the art can improve and implement the present scheme based on their own ability under the guidance of this application. Some typical known structures or known methods should not be an obstacle for the ordinary skilled person in the art to implement the present application. It should be noted that for those skilled in the art, without departing from the structure of the present application, a number of modifications and improvements can be made, which should also be considered within the scope of protection of the present application. These will not affect the effectiveness and practicality of the present application.

Claims

1. A dynamic routing cooperative optimization method for heterogeneous networks based on reinforcement learning, characterized in that, Includes the following steps: S1, Deploy an agent at each node of the heterogeneous network and initialize the state sampling period and action space; S2, real-time sensing of heterogeneous network status and acquisition of sensing data, including: monitoring the percentage of the data queue length to the cache capacity of this node. and the percentage of actual transmission rate relative to nominal bandwidth ; Measure adjacent link delay values and packet loss rate Read the remaining battery percentage of this node and its neighbors. ; S3, encodes the sensed data into a state vector; S4. Design a hierarchical dynamic reward function, including short-term and long-term rewards; dynamically set the weight ratio of short-term and long-term rewards according to the scenario type, and output the comprehensive reward. The short-term and long-term rewards are calculated using the following formula: ; ; in, For short-term rewards, , Reference delay and reference bandwidth; , ; For long-term rewards, The number of node switches within a period. The standard deviation of the network load; , ; The step of dynamically setting the weight ratio of short-term rewards to long-term rewards based on scenario type includes: If the scene type is a live film and television broadcast scene. ; If the scenario type is a data acquisition scenario ; As a comprehensive reward; S5 performs multi-agent collaborative decision-making, including: constructing a local topology graph based on neighbor state information broadcast by nodes; each agent selecting actions based on state vectors through the DQN network; and updating the DQN network weights using comprehensive rewards.

2. The method for dynamic route cooperative optimization in heterogeneous networks based on reinforcement learning according to claim 1, characterized in that, It also includes S6, which performs global policy synchronization: periodically uploading the local DQN network weights of each node to the central coordinator; the central coordinator generates a lightweight baseline model through policy distillation; and the baseline model is distributed to all nodes in the network to replace the local model in order to achieve policy synchronization.

3. The method for dynamic routing cooperative optimization in heterogeneous networks based on reinforcement learning according to claim 1, characterized in that, The intelligent agent is an embedded software module deployed on each network node, including wireless cameras, drone image transmission devices, and relay routers; each intelligent agent performs the following actions: collecting the status of local and neighboring nodes, calculating the optimal action through the DQN network, and broadcasting HELLO messages and responding to ALERT alarms.

4. The method for dynamic route cooperative optimization in heterogeneous networks based on reinforcement learning according to claim 1, characterized in that, The state vector ; in, For based on and The load level being measured; For based on and The generated link quality vector; According to Energy levels are categorized.

5. The method for dynamic routing cooperative optimization in heterogeneous networks based on reinforcement learning according to claim 4, characterized in that, when and hour, ;when and hour, ;when and hour, ; The link quality vector is generated as follows: For each adjacent link, calculate a quality score. : ； And the quality score is mapped to discrete levels; At that time, the corresponding level value is 0; The corresponding level value is 1; When the time is right, the corresponding level value is 2; The energy levels include low, medium, and high, with corresponding values ​​of 0, 1, and 2.

6. The method for dynamic route cooperative optimization in heterogeneous networks based on reinforcement learning according to claim 5, characterized in that, The action space is defined as a set of next-hop neighbors and a combination of communication frequency bands that a node can choose. When selecting an action, an energy constraint rule is executed: nodes with low energy levels are prohibited from being used as the next hop unless there are no other available paths.

7. The method for dynamic route cooperative optimization in heterogeneous networks based on reinforcement learning according to claim 1, characterized in that, The scene type is automatically identified in the following ways: When ≥3 video streams are detected in the network and the bandwidth requirement of a single stream is >5Mbps, it is determined to be a live video streaming scenario. When more than 80% of nodes have a transmission interval >10 seconds, it is determined to be a data collection scenario; otherwise, it is determined to be a general scenario. In the general scenario, the weight ratio of short-term rewards to long-term rewards is set to 0.5, i.e. .

8. The method for dynamic route cooperative optimization in heterogeneous networks based on reinforcement learning according to claim 1, characterized in that, The neighbor status information refers to HELLO messages that include available bandwidth, remaining hop count, and energy level; In S5, when constructing a local topology graph, the local topology graph is updated based on the HELLO message. Where vertex V is the set of neighboring nodes and edge E is the available link; The state vector is then input into a DQN network, which contains three fully connected layers; the DQN network outputs the Q-values ​​of each action; and the action with the largest Q-value is selected. Execute the route.

9. The method for dynamic route cooperative optimization in heterogeneous networks based on reinforcement learning according to claim 8, characterized in that, S5 also includes a dynamic rerouting trigger mechanism; when the link packet loss rate is within 3 consecutive sampling periods... and When triggered, the dynamic rerouting mechanism is activated; upon activation, the following actions are performed: This node sends an ALERT message containing the failed link ID and the current timestamp; After receiving the ALERT message, the neighboring node removes the failed link and updates the status within 0.5 seconds. ; Based on the updated Recalculate the state vector and output the new action.

Images